Tune Linux for Performance
Backlog queue /Connection Queue
consider increasing net.core.somaxconn, the maximum number of connections that can be queued awaiting attention from NGINX.
it will be queued in the operating system before they are being handed over to Nginx.
etc/sysctl.conf
# Increase size of file handles and inode cache
fs.file-max = 2097152
fs.file-max = 2097152
net.core.somaxconn = 65536
net.ipv4.tcp_max_tw_buckets = 1440000
//If there are error messages in the kernel log, increase the value until errors stop.
If you set the somaxconn key to a value greater than 512, change the backlog parameter of the NGINX listen directive to match:
server {
listen 80 backlog=65536;
# The rest of server configuration
}
File descriptors/ max of open files
sys.fs.file_max defines the system wide limit for file descriptors.
nofile defines the user file descriptor limit,
set in the /etc/security/limits.conf file.
system wide FD Limits
/etc/sysctl.conf
# Increase size of file handles and inode cache fs.file-max = 2097152
fs.file-max = 100000
sysctl -p
Show all system parameters with their values (default or changed)
sysctl -A
Show values of parameters modified by you
sysctl -p
what is ulimit and how to change it permanently?
to use enough system resources
Setting Soft/Hard
NoFiles 32768/65536
Filesize unlimited/unlimited
Data unlimited/unlimited
Stack unlimited/unlimited
Memory unlimited/unlimited
Coredump unlimited/unlimited2. set permanent ulimit values at /etc/security/limits.conf
Restart the system:
esadmin system stopall
esadmin system startall
ulimit -u unlimited
ulimit -u
31878
root@lemp-s-2vcpu-4gb-blr1-01:~# ulimit
unlimited
root@lemp-s-2vcpu-4gb-blr1-01:~# ulimit -u unlimited
root@lemp-s-2vcpu-4gb-blr1-01:~# ulimit -u
unlimited
cat /proc/sys/fs/file-max
813940
root@lemp-s-2vcpu-4gb-blr1-01:~# ulimit -Hn
1048576
root@lemp-s-2vcpu-4gb-blr1-01:~# ulimit -Sn
1024
root@lemp-s-2vcpu-4gb-blr1-01:~# cat /proc/sys/fs/file-max
813940
root@lemp-s-2vcpu-4gb-blr1-01:~# sysctl fs.file-max
fs.file-max = 813940
root@instance-1:~# sysctl fs.file-max
fs.file-max = 9223372036854775807
root@instance-1:~#
fs.file-max = 9223372036854775807
root@instance-1:~#
root@instance-1:~# ulimit -aS
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31814
max locked memory (kbytes, -l) 65536
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 31814
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31814
max locked memory (kbytes, -l) 65536
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 31814
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
root@instance-1:~# ulimit -aH
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31814
max locked memory (kbytes, -l) 65536
max memory size (kbytes, -m) unlimited
open files (-n) 1048576
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) 31814
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
root@instance-1:~#
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31814
max locked memory (kbytes, -l) 65536
max memory size (kbytes, -m) unlimited
open files (-n) 1048576
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) 31814
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
root@instance-1:~#
what is ulimit in linux?
command that can set or report the resource limit of the current user.
ulimit
unlimited
ulimit -a
max process 31814
Difference between hard & soft ulimits in linux?
Hard limits are maintained by the kernel while the soft limits are enforced by the shell.
A soft limit is like a warning and hard limit is a real max limit
A soft limit is like a warning and hard limit is a real max limit
“hard” and “soft”. Hard resource limit defines the physical limit that the user can reach. The “soft” resource limit is manageable by the user. Its value can go up to the “hard” limit.
To check the value of max “core file size”, use the flag “-c”.
ulimit -c
root@instance-1:~# ulimit -c
0
0
For max “data seg size” (in kilobytes)
root@instance-1:~# ulimit -d
unlimited
unlimited
ulimit -u – maximum number of process
ulimit -u
31814
31814
linux check how many processes are running
Command to count the number of processes running in Linux
root@instance-1:~# ps aux | wc -l
118
118
ps -U user1 | wc -l
ps -U www-data | wc -l
ps -U mysql | wc -l
ps -U root | wc -l
ps -C httpd | wc -l
root@instance-1:~# ps -U www-data | wc -l
12
12
root@instance-1:~# ps -U mysql | wc -l
2
2
root@instance-1:~# ps -U root | wc -l
98
98
To know many file descriptors per process*?
ulimit -n
1024
ulimit -t – time limit to each process run
unlimited
ulimit –help
sudo nano /etc/security/limits.conf
core: Core file size (in KB).
data: Max data size (in KB)
fsize: Max file size (in KB).
memlock: Max locked-in-memory address space (in KB).
nofile: Max number of file descriptors
stack: Max size of the stack (in KB).
cpu: Maximum CPU time (in MIN).
maxlogins: Maximum number of logins for the current user/group
priority: Set the priority of the user’s processes
rtprio: Max priority in real-time.
data: Max data size (in KB)
fsize: Max file size (in KB).
memlock: Max locked-in-memory address space (in KB).
nofile: Max number of file descriptors
stack: Max size of the stack (in KB).
cpu: Maximum CPU time (in MIN).
maxlogins: Maximum number of logins for the current user/group
priority: Set the priority of the user’s processes
rtprio: Max priority in real-time.
do changes in /etc/security/limits.conf require a reboot?
No but you should close all active sessions windows.
fs.file-max vs ulimit
ulimit shows the per-process maximum open files by user
ulimit -n
1024
(system wide limit) file-max is the maximum File Descriptors (FD) enforced on a kernel level, which cannot be surpassed by all processes without increasing.
root@instance-1:~# cat /proc/sys/fs/file-max
9223372036854775807
temporary change ulimits for user until restart
-n processes
root@instance-1:~# ulimit -n
1024
ulimit -n 2048
root@instance-1:~# ulimit -n 2048
root@instance-1:~# ulimit -n
2048
root@instance-1:~#
What;s the difference between file descriptor and inode limits?
Files are represented by inodes
All information needed by the filesystem to handle a file is included in a data structure called an inode.
Max files for user account in a system ex: 1,50,000 files in shared server account. //general “file on a disk”
essentially an index into an array of open files kept by the kernel. files can be open at a time.
file descriptor: it is just an integer number that uniquely represents an opened file in operating system
-
User Level FD Limits
- /etc/security/limits.conf
-
www-data soft nproc 100000www-data soft nofile 10000012soft nofile 4096hard nofile 4096two file descriptors for each connection increase no file to support highload by os.File Descriptors (FD)worker_rlimit_nofile 100000;
nano /etc/sysctl.conf
### IMPROVE SYSTEM MEMORY MANAGEMENT ###
# Increase size of file handles and inode cachefs.file-max = 2097152# Do less swappingvm.swappiness = 10vm.dirty_ratio = 60vm.dirty_background_ratio = 2### GENERAL NETWORK SECURITY OPTIONS ###
# Number of times SYNACKs for passive TCP connection.net.ipv4.tcp_synack_retries = 2# Allowed local port rangenet.ipv4.ip_local_port_range = 2000 65535# Protect Against TCP Time-Waitnet.ipv4.tcp_rfc1337 = 1# Decrease the time default value for tcp_fin_timeout connectionnet.ipv4.tcp_fin_timeout = 15# Decrease the time default value for connections to keep alivenet.ipv4.tcp_keepalive_time = 300net.ipv4.tcp_keepalive_probes = 5net.ipv4.tcp_keepalive_intvl = 15### TUNING NETWORK PERFORMANCE ###
# Default Socket Receive Buffernet.core.rmem_default = 31457280# Maximum Socket Receive Buffernet.core.rmem_max = 12582912# Default Socket Send Buffernet.core.wmem_default = 31457280# Maximum Socket Send Buffernet.core.wmem_max = 12582912# Increase number of incoming connectionsnet.core.somaxconn = 65535# Increase number of incoming connections backlognet.core.netdev_max_backlog = 65535# Increase the maximum amount of option memory buffersnet.core.optmem_max = 25165824# Increase the maximum total buffer-space allocatable# This is measured in units of pages (4096 bytes)net.ipv4.tcp_mem = 65535 131072 262144net.ipv4.udp_mem = 65535 131072 262144# Increase the read-buffer space allocatablenet.ipv4.tcp_rmem = 8192 87380 16777216net.ipv4.udp_rmem_min = 16384# Increase the write-buffer-space allocatablenet.ipv4.tcp_wmem = 8192 65535 16777216net.ipv4.udp_wmem_min = 16384# Increase the tcp-time-wait buckets pool size to prevent simple DOS attacks
net.ipv4.tcp_max_tw_buckets = 1440000net.ipv4.tcp_tw_recycle = 1net.ipv4.tcp_tw_reuse = 1Paste this (at the end of the file) to /etc/security/limits.conf (using nano /etc/security/limits.conf) and save it,nginx soft nofile 2097152nginx hard nofile 2097152www-data soft nofile 2097152www-data hard nofile 2097152Paste this (at the end of the file) to /etc/pam.d/common-session (using nano /etc/pam.d/common-session) and save it,session required pam_limits.soChange listen.backlog in /etc/php5/fpm/pool.d/www.conf (using nano /etc/php5/fpm/pool.d/www.conf) and save it,listen.backlog = 65535 //default to systemChange worker_rlimit_nofile in /etc/nginx/nginx.conf (using nano /etc/nginx/nginx.conf) and save it,worker_rlimit_nofile 99999;And finally reboot 🙂Ephemeral ports – When used as a proxyTune file descriptor limits on Linux
php-fpm, also uses user as nginx server
www-data
nginx uses 2 file descriptor per connection. max connections 512 with default linux configuration.
apache uses one process per 1 connection.
default limit 1024 files for one process.
nginx is single threaded /single process.
ulimit-aS
ulimit-aH (you cannot increase above this limit without increasing kernel parameters)
to view current user limits
su www-data
and run above commands.
Updating Default Kernel Parameter Settings
to increase the number of simultaneously connection to db server
msgmni
kernel.shmmax=268435456 for 32-bit
kernel.shmmax=1073741824 for 64-bit
kernel.msgmni=1024
fs.file-max=8192
kernel.sem=”250 32000 32 1024″
kernel.shmmax=1073741824 for 64-bit
kernel.msgmni=1024
fs.file-max=8192
kernel.sem=”250 32000 32 1024″
useful commands
too, htop, vimsat,iosat, sar
sudo apt-get clean
sudo apt-get autoremove
ubuntu 20.04 server optimization
Tuning Nginx to handle more than 50.000 reqs/s
linux sysctl conf linux security limits conf
-
Disable unnecessary daemons to save memory & cpu
Tuned – Automatic Performance Tuning of CentOS/RHEL Servers
how to apply changes to /etc/security/limits.conf immediately
applies after reboot
without reboot, modify /etc/pam.d/common-session by adding this line at the end of file:
session required pam_limits.so
logout & login
maxfiles limit in linux unix
max files for a process default 1024MB., my mysql server running on 8GB RAM and 100 connections or process 100*2files=200opened files
you may increase when server error says max open files reached. for tuning you can increse max open fiels based on availablity of RAM.
The maximum number of file descriptors available to a process is controlled by user limit parameters. The “nofiles” parameter is the number of file descriptors available to a process. The default maximum number of open files in a Linux system is 1024
Inode limit:
maximum inode count maxfiles
The maximum number of inodes and maxfiles depends on the volume capacity and the type of file system:
he default maxfiles limit is roughly one file per 32 KiB of volume capacity
fs.file-max counts max opened file in the system. ideally it should be 10% RAM in the server.
df -i: This command displays the number of inodes available
root@srv443573:~# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/root 25804800 426222 25378578 2% /
tmpfs 2050188 1 2050187 1% /dev/shm
tmpfs 819200 525 818675 1% /run
tmpfs 2050188 3 2050185 1% /run/lock
/dev/sda15 0 0 0 – /boot/efi
tmpfs 410037 26 410011 1% /run/user/0