Newer kernel can improve random read and write IO by 25% – 50% in some cases, at least if you are utilizing SSD storage, or use this "cloud" thing called KVM. This performance boost comes from the newish blk-mq update to the Linux block device subsystem. To upgrade CentOS from Kernel 3.10.x to 3.17.x do the following: '''note: you may need to change the minor versions as the kernel is updated, so if copypasta doesn't work, please make sure you are replacing the versions in the commands with the correct version''' Get the key and install the repo so we can has the access to latest centos kernels. You should make sure there isn't a newer version of elrepo before you copypasta these commands. That should help to ensure you always get the latest kernel for CentOS 7. Keep in mind that removing the stock kernel will set you adrift from the rest of the boring and slow CentOS world, which is ok with me, but it might not be ok with your boss so I'm just giving you a fair warning.
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
Remove the old CentOS 7 kernel tools, headers and libs, in the case of CentOS 7 we are removing the 3.10 Kernel and replacing with the 3.17 kernel. I don't really suggest doing this on a production server, at least until you have finalized everything and are happy with the results. You already knew that though, right? 🙂
yum remove kernel-headers-3.10.0-123.9.2.el7.x86_64 kernel-tools-3.10.0-123.9.2.el7.x86_64 kernel-tools-libs-3.10.0-123.9.2.el7.x86_64
*http://elrepo.org/tiki/tiki-index.php Edit repo and set "enabled=" in the kernel section to "1" to enable access to the latest kernel
vim /etc/yum.repos.d/elrepo.repo
[elrepo-kernel]
name=ELRepo.org Community Enterprise Linux Kernel Repository - el7
baseurl=http://elrepo.org/linux/kernel/el7/$basearch/
http://mirrors.coreix.net/elrepo/kernel/el7/$basearch/
http://jur-linux.org/download/elrepo/kernel/el7/$basearch/
http://repos.lax-noc.com/elrepo/kernel/el7/$basearch/
http://mirror.ventraip.net.au/elrepo/kernel/el7/$basearch/
mirrorlist=http://mirrors.elrepo.org/mirrors-elrepo-kernel.el7
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-elrepo.org
protect=0
Update Yum so that CentOS 7 pulls the newer 3.17 kernel packages from the repo we just installed.
yum update
Install the new 3.17 kernel and it's packages / libraries / tools, you might see a different kernel version than the one talked about here, so you may need to slightly modify the rest of these steps.
yum install kernel-ml.x86_64 kernel-ml-devel.x86_64 kernel-ml-headers.x86_64 kernel-ml-tools.x86_64 kernel-ml-tools-libs.x86_64 kernel-ml-tools-libs-devel.x86_64
Verify the latest kernel is available by running the following command which will parse grub2.cfg and output the available kernels to boot from
awk -F\' '$1=="menuentry " {print $2}' /etc/grub2.cfg
CentOS Linux, with Linux 3.17.1-1.el7.elrepo.x86_64
CentOS Linux, with Linux 3.10.0-123.el7.x86_64
CentOS Linux, with Linux 3.10.0-123.9.2.el7.x86_64
CentOS Linux, with Linux 3.10.0-123.8.1.el7.x86_64
CentOS Linux, with Linux 0-rescue-8a06d0a454c348f5911cbf876baa0e02
CentOS Linux Memtest memtest86+-4.20
If the CentOS kernel version 3.17.x is listed first, then set grub to use 0 which will set the system to boot off the newer kernel. You might want to leave some of the older 3.10.x kernels on the system in case you decide to revert to the stock CentOS 7 kernel, but you can always modify the default kernel to use later on.
grub2-set-default 0
Update the grub2 config so it's aware of the changes we just made.
grub2-mkconfig -o /boot/grub2/grub.cfg
Reboot the server and cross your fingers! If all goes well the server should come back online and boot CentOS 7 with the 3.17 kernel
reboot
To see if this is the case, run
uname -a
==Upgrade CentOS 7 Kernel to 4.0.1== Why upgrade CentOS from the stock 3.10.x kernel to a newer 4.0.1.x kernel? '''[http://wiki.mikejung.biz/Category:Performance Performance]''', especially if you use '''[http://wiki.mikejung.biz/KVM_/_Xen KVM]'''. I found that the 3.17 kernel can improve random read and write IO by 25% – 50% in some cases. The Linux 4.0 Kernel builds on top of all the previous performance gains found in the 3.17 kernel. To upgrade CentOS from Kernel 3.10.x to 4.0.1 do the following: ' Get the key and install the repo so we can has the access to latest centos kernels. You should make sure there isn't a newer version of elrepo before you copypasta these commands. That should help to ensure you always get the latest kernel for CentOS 7. Keep in mind that removing the stock kernel will set you adrift from the rest of the boring and slow CentOS world, which is ok with me, but it might not be ok with your boss so I'm just giving you a fair warning.
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
Remove the old CentOS 7 kernel tools, headers and libs, in the case of CentOS 7 we are removing the 3.10 Kernel and replacing with the 4.0.1 kernel. I don't really suggest doing this on a production server, at least until you have finalized everything and are happy with the results. You already knew that though, right? 🙂
yum remove kernel-headers-3.10.0-123.9.2.el7.x86_64 kernel-tools-3.10.0-123.9.2.el7.x86_64 kernel-tools-libs-3.10.0-123.9.2.el7.x86_64
*http://elrepo.org/tiki/tiki-index.php Edit repo and set "enabled=" in the kernel section to "1" to enable access to the latest kernel
vim /etc/yum.repos.d/elrepo.repo
[elrepo-kernel]
name=ELRepo.org Community Enterprise Linux Kernel Repository - el7
baseurl=http://elrepo.org/linux/kernel/el7/$basearch/
http://mirrors.coreix.net/elrepo/kernel/el7/$basearch/
http://jur-linux.org/download/elrepo/kernel/el7/$basearch/
http://repos.lax-noc.com/elrepo/kernel/el7/$basearch/
http://mirror.ventraip.net.au/elrepo/kernel/el7/$basearch/
mirrorlist=http://mirrors.elrepo.org/mirrors-elrepo-kernel.el7
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-elrepo.org
protect=0
Update Yum so that CentOS 7 pulls the newer 4.0.1 kernel packages from the repo we just installed.
yum update
Install the new 4.0.1 kernel and it's packages / libraries / tools, you might see a different kernel version than the one talked about here, so you may need to slightly modify the rest of these steps.
yum install kernel-ml kernel-ml-headers kernel-ml-tools kernel-ml-tools-libs kernel-ml-tools-libs-devel
If the CentOS kernel version 4.0.1 is listed first, then set grub to use 0 which will set the system to boot off the newer kernel. You might want to leave some of the older 3.10.x kernels on the system in case you decide to revert to the stock CentOS 7 kernel, but you can always modify the default kernel to use later on.
grub2-set-default 0
Update the grub2 config so it's aware of the changes we just made.
grub2-mkconfig -o /boot/grub2/grub.cfg
Reboot the server and cross your fingers! If all goes well the server should come back online and boot CentOS 7 with the 4.0.1 kernel
reboot
== Activate tuned latency-performance for CentOS 7== Tuned should already be installed for Cent 7, and the default profile is balanced. Depending on what you are doing, you may see better '''[http://wiki.mikejung.biz/Performance_Analysis performance]''' by changing the profile to "latency-performance". Please note that if you use "tuned-adm" you are enabling a static profile which is different than running the "tuned" dynamic daemon which will analyze historical performance data and attempt to tune the server's settings on the fly. Personally I don't used the daemon, but I wanted to mention there is a dynamic "tuned" daemon, as well as a static "tuned-adm" utility which is a one off config changer.
tuned-adm profile latency-performance
To see what the active profile is:
tuned-adm active
For more information on tuned-adm options, check out the man page.
man tuned-adm
*'''latency-performance''' — "Profile for low latency performance tuning. Disables power saving mechanisms. CPU governor is set to performance and locked to the low C states (by PM QoS). CPU energy performance bias to performance." This profile is the Parent profile to "network-latency". For CentOS 7, the latency-performance profile includes the following tweaks
force_latency=1 governor=performance energy_perf_bias=performance min_perf_pct=100 kernel.sched_min_granularity_ns=10000000 vm.dirty_ratio=10 vm.dirty_background_ratio=3 vm.swappiness=10 kernel.sched_migration_cost_ns=5000000
*'''network-latency''' — This is a Child profile of "latency-performance". That this means is that if you were to activate network-latency profile via tuned, it would automatically enable latency-performance, then make some additional tweaks to improve network latency. As you can see, the network-latency profile includes "latency-performance", however it also disables transparent hugepages, and makes some net.core kernel tweaks. '''Please note:''' network-latency disables Cent 7's automatic NUMA balancing, so if you are not manually pinning processes to nodes, then you might actually see some reduced performance by enabling network-latency.
include=latency-performance transparent_hugepages=never net.core.busy_read=50 net.core.busy_poll=50 net.ipv4.tcp_fastopen=3 kernel.numa_balancing=0
*'''throughput-performance''' — This is the Parent profile to virtual-guest, virtual-host and network-throughput. This profile is optimized for large, streaming files or any high throughput workloads. *'''virtual-guest''' — "Profile optimized for virtual guests based on throughput-performance profile. It additionally decreases virtual memory swapiness and increases dirty_ratio settings." *'''virtual-host''' — "Profile optimized for virtual hosts based on throughput-performance profile. It additionally enables more aggressive write-back of dirty pages." tuned-adm profiles can be found in this directory
/usr/lib/tuned/
To learn more about each profile you can navigate to the profile directory and take a look at the profile's tuned.conf which shows what tweaks are done. The 3 most common profiles seem to be the virtual-guest, virtual-host, and latency-performance profiles. There is a "balanced" profile, but if you are running a server you will almost always want to set "latency-performance" unless you are running a storage node that deal with large, streaming files, in that case "throughput-performance" might make more sense. CentOS 7 is pretty smart about picking a profile to use when it starts up. If CentOS 7 is run as a guest using '''[http://wiki.mikejung.biz/KVM_/_Xen KVM]''', then you may see that the virtual-guest profile is active. If you are using CentOS 7 to run KVM on, and to host guests then you might see that the profile for virtual-host is active. Personally I prefer to use '''latency-performance''' profile most of the time because it really does seem to boost random read IO at the expense of slightly lower random writes (100 – 200 iops slower than virtual-guest) , and help improve overall CPU performance, at least that's what I've found when I '''[http://wiki.mikejung.biz/Benchmarking benchmark]''' servers using '''[http://wiki.mikejung.biz/Sysbench Sysbench]'''. In general you will almost always see a 1.5x – 2x performance boost for random reads if you are using SSD backed storage. == CentOS 7 RPM commands and options== ===How to search and view specific rpm packages on CentOS 7=== To search for available packages on CentOS 7 you can use the yum search command
yum search $example
Find all packages installed on CentOS 7
rpm -qa
Find all MySQL RPM packages installed on CentOS 7 you can pipe the output of -qa to grep and use -i to match upper and lower case. If you want to find all mysql related rpms use the command below.
rpm -qa | grep -i mysql
=== List all files in RPM package === Simply run this command against any RPM you want to inspect.
rpm -qlp $package.rpm
Here is an example of the output.
rpm -qlp MySQL-shared-compat-5.1.49-1.rhel5.x86_64.rpm /usr/lib64/libmysqlclient.so /usr/lib64/libmysqlclient.so.12 /usr/lib64/libmysqlclient.so.12.0.0 /usr/lib64/libmysqlclient.so.14 /usr/lib64/libmysqlclient.so.14.0.0 /usr/lib64/libmysqlclient.so.15 /usr/lib64/libmysqlclient.so.15.0.0 /usr/lib64/libmysqlclient.so.16 /usr/lib64/libmysqlclient.so.16.0.0 /usr/lib64/libmysqlclient_r.so /usr/lib64/libmysqlclient_r.so.12 /usr/lib64/libmysqlclient_r.so.12.0.0 /usr/lib64/libmysqlclient_r.so.14 /usr/lib64/libmysqlclient_r.so.14.0.0 /usr/lib64/libmysqlclient_r.so.15 /usr/lib64/libmysqlclient_r.so.15.0.0 /usr/lib64/libmysqlclient_r.so.16 /usr/lib64/libmysqlclient_r.so.16.0.0
=== Extract RPM packages Using rpm2cpio === Create a new directory, then copy over the RPM into that directory. Then run this command to extract the files into your directory.
rpm2cpio MySQL-shared-compat-5.1.49-1.rhel5.x86_64.rpm | cpio -idmv ./usr/lib64/libmysqlclient.so ./usr/lib64/libmysqlclient.so.12 ./usr/lib64/libmysqlclient.so.12.0.0 ./usr/lib64/libmysqlclient.so.14 ./usr/lib64/libmysqlclient.so.14.0.0 ./usr/lib64/libmysqlclient.so.15 ./usr/lib64/libmysqlclient.so.15.0.0 ./usr/lib64/libmysqlclient.so.16 ./usr/lib64/libmysqlclient.so.16.0.0 ./usr/lib64/libmysqlclient_r.so ./usr/lib64/libmysqlclient_r.so.12 ./usr/lib64/libmysqlclient_r.so.12.0.0 ./usr/lib64/libmysqlclient_r.so.14 ./usr/lib64/libmysqlclient_r.so.14.0.0 ./usr/lib64/libmysqlclient_r.so.15 ./usr/lib64/libmysqlclient_r.so.15.0.0 ./usr/lib64/libmysqlclient_r.so.16 ./usr/lib64/libmysqlclient_r.so.16.0.0 22600 blocks
== Install EPEL Repo CentOS 7 == Please make sure you are downloading the latest release for EPEL as it gets updated frequently. As of January 19th 2015, the latest EPEL release is 7-5 but before use the commands below, check out this link to make sure you wget the latest release! http://mirrors.liquidweb.com/fedora-epel/7/x86_64/repoview/epel-release.html All you need to do is grab the rpm and install it, then update yum and you should be all set.
wget http://mirrors.liquidweb.com/fedora-epel/7/x86_64/e/epel-release-7-5.noarch.rpm rpm -iv epel-release-7-5.noarch.rpm yum update
== How to view all installed rpms on centos 7 == You can use rpm -qa to view all of the currently installed rpms on centos 7
rpm -qa
== How to install NOVM on CentOS 7 == *https://github.com/google/novm To make and run novm on CentOS 7 do the following
yum install go fakeroot git clone https://github.com/google/novm.git cd novm/ make
To make an RPM of novm on CentOS 7 do the following
yum install go fakeroot git clone https://github.com/google/novm.git cd novm/ make rpm rpm -iv novm-0.0-164.gb7c9c78.x86_64.rpm
If you get the error "bin/sh: gofmt: command not found" when attempting to run the make command, then you should make sure GO is installed, once that is the case the make command should build novm successfully. After you make novm you can find two binaries under the bin directory where you built novm.
/novm/bin/ noguest novmm
novmm command line options
Usage of ./novmm: -cmdline="": linux command line -controlfd=-1: bound control socket -debug=false: devices start debugging -init=false: real in-guest init? -initrd="": initial ramdisk image -paused=false: start with model and vcpus paused -setup="": linux boot params (vmlinuz) -statefd=0: machine state file -step=false: step instructions -stop=false: wait for a SIGCONT before running -sysmap="": kernel symbol map -trace=false: trace kernel symbols on exit -vmlinux="": linux kernel binary (ELF)
novm noguest command line options
./noguest --help Usage of ./noguest: -control="/dev/vport0p0": control file -serverfd=-1: run RPC server
==CPUFreq and CPU Governors == To view what CPUFreq drivers are available for CentOS 7 run this command
ls -1 /lib/modules/`uname -r`/kernel/arch/x86/kernel/cpu/cpufreq/
To load the acpi-cpufreq module run modprobe, if it's already loaded then you can skip this step. Generally speaking you always want to load the '''acpi-cpufreq''' instead of the other CPU freq options, so unless you have a specific reason to load something else, always use acpi-cpufreq. If you are having issues using CPUFreq you might as well make sure it's loaded before trying to troubleshoot too much further.
modprobe acpi-cpufreq
Once the driver is loaded you can install and run cpupowerutils which will let you know what the current frequency of your CPUs is, and what CPU governors are available to be used on your system.
yum install cpupowerutils cpupower frequency-info --governors
Generally speaking, the performance governor is the best option to use if you are configuring a server that's going to be used all the time. Forcing the CPU governor to run in performance mode can help improve IO and reduce latency in some cases.
cpupower frequency-set --governor performance
For Official RHEL docs please visit this [https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Power_Management_Guide/cpufreq_setup.html link] ==How to install FIO on CentOS 7== The easiest way to install FIO on CentOS 7 is to install the lastest version of the EPEL repo, update YUM and then install fio. This will install FIO-2.1.11-1 which appears to be relatively up to date.
wget http://mirrors.liquidweb.com/fedora-epel/7/x86_64/e/epel-release-7-5.noarch.rpm rpm -iv epel-release-7-5.noarch.rpm yum update yum install fio -y
To run a basic '''FIO random write''' test on CentOS 7, use the command below. This uses the "libaio" engine for IO, uses and IO depth of 32, uses file block size of 4KB and runs 8 Jobs, or processes each writing to a 4GB file for a total of 4 minutes, results are then grouped together which basically combines all 8 job results into one overall report summary. Fio is pretty flexible when it comes to displaying output, I suggest you view the fio man page for a list if the filters for the output.
fio --name=randfile --ioengine=libaio --iodepth=32 --rw=randwrite --bs=4k --direct=0 --size=4G --numjobs=8 --runtime=240 --group_reporting
To run a basic '''FIO random read''' test on CentOS 7, use the command below. This uses the "libaio" engine for IO, uses and IO depth of 32, uses file block size of 4KB and runs 8 Jobs, or processes each reading from a 4GB file for a total of 4 minutes, results are then grouped together which basically combines all 8 job results into one overall report summary.
fio --name=randfile --ioengine=libaio --iodepth=32 --rw=randread --bs=4k --direct=0 --size=4G --numjobs=8 --runtime=240 --group_reporting
==How to install Phoronix Test Suite CentOS 7== To install the latest version of Phoronix Test Suite on CentOS 7, run the commands below. Basically we are just installing some dependencies for Phoronix (PHP stuff), grabbing the latest stable generic package, renaming it to something we can extract, extracting the test suite, then running Phoronix's installer script.
yum install php-cli php-xml php-gd php-json -y wget http://www.phoronix-test-suite.com/download.php?file=phoronix-test-suite-5.4.1 mv download.php\?file\=phoronix-test-suite-5.4.1 phoronix-test-suite-5.4.1.tar.gz tar zxvf phoronix-test-suite-5.4.1.tar.gz cd phoronix-test-suite/ ./install-sh
Once the installer script completes you can start using Phoronix to test out performance. For example, if you want to run the nginx test you would run this command '''note''' for whatever reason the nginx and sometimes apache test fails to run, I've found that disabling IPV6 fixes the issue, obviously if you use IPV6 then don't do this, but if you get an error run the command below to resolve the issue. The command is just disabling ipv6 in '''[[Sysctl tweaks|sysctl.conf]]'''
echo "net.ipv6.conf.all.disable_ipv6 = 1" >> /etc/sysctl.conf sysctl -p /usr/bin/phoronix-test-suite benchmark pts/nginx
For more information on Phoronix and available tests, check out this '''[http://wiki.mikejung.biz/Benchmarking#Phoronix_Test_Suite page]'''. == systemctl apache CentOS 7 == To start '''[http://wiki.mikejung.biz/Apache Apache]''' using systemctl
systemctl start httpd.service
To stop httpd using systemctl
systemctl stop httpd.service
To restart httpd using systemctl
systemctl restart httpd.service
==Using Numastat on CentOS 7 == With CentOS 7, numastat has received some updates that make it easier to view memory usage on a per node basis. The updates make numastat's output look more like the output from /proc/meminfo. You can use numastat to determine how much memory each '''[http://wiki.mikejung.biz/NUMA NUMA]''' node is using. Numastat will also show the amount of total NODEs on the server, this can be useful if you are trying to determine if a server / CPU utilizing multiple banks of RAM (like an E5-2690), or if it just uses a single bank of RAM (like an E3-1220). If you use the option "-c" when you run numastat, it will shrink the output based on the data being displayed. This option helps make the output more readable. numastats "-m" makes the memory information appear to be more /proc/meminfo like. "-z" will remove any rows that have no data, cleaning up the output and only displaying relevant information.
numstat -mzcs
CentOS 7 supports automatic NUMA balancing. The kernel is able to detect what memory pages are being used by processes and their threads, then attempts to balance the pages / schedule CPU time on the closest CPU to those memory pages. The idea is to try and get processes to run on the CPU that is the shortest distance from the memory bank. Ideally, data in memory should be processed on the closest CPU to avoid latency, which can significantly slow down a program that is using large amounts of RAM. ==Using lscpu on CentOS 7 == Similar to the numastat utility, you can use lscpu to view advanced CPU information which provides NUMA NODE counts, and information about what CPU / Core uses which NUMA Node. lscpu is part of the util-linux package, you can install this on CentOS 7 by using yum.
yum install util-linux
To run lscpu you can simply run lscpu!
lscpu