linux performance tuning inode limit file descriptors tco, kernel etc

Inode limit:

Tune Linux for Performance

Backlog queue /Connection Queue

consider increasing net.core.somaxconn, the maximum number of connections that can be queued awaiting attention from NGINX.

it will be queued in the operating system before they are being handed over to Nginx.

etc/sysctl.conf

# Increase size of file handles and inode cache
fs.file-max = 2097152

net.core.somaxconn = 65536

net.ipv4.tcp_max_tw_buckets = 1440000

//If there are error messages in the kernel log, increase the value until errors stop.

If you set the somaxconn key to a value greater than 512, change the backlog parameter of the NGINX listen directive to match:

server {

listen 80 backlog=65536;

# The rest of server configuration

}

File descriptors/ max of open files

sys.fs.file_max defines the system wide limit for file descriptors.

nofile defines the user file descriptor limit,

set in the /etc/security/limits.conf file.

system wide FD Limits

/etc/sysctl.conf

# Increase size of file handles and inode cache
fs.file-max = 2097152

fs.file-max = 100000

sysctl -p

Show all system parameters with their values (default or changed)

sysctl -A
Show values of parameters modified by you

sysctl -p

what is ulimit and how to change it permanently?

to use enough system resources

Setting       Soft/Hard
NoFiles       32768/65536
Filesize      unlimited/unlimited
Data          unlimited/unlimited
Stack         unlimited/unlimited
Memory        unlimited/unlimited
Coredump      unlimited/unlimited

2. set permanent ulimit values at /etc/security/limits.conf

Restart the system:

esadmin system stopall
esadmin system startall

ulimit -u unlimited

ulimit -u

31878

root@lemp-s-2vcpu-4gb-blr1-01:~# ulimit

unlimited

root@lemp-s-2vcpu-4gb-blr1-01:~# ulimit -u unlimited

root@lemp-s-2vcpu-4gb-blr1-01:~# ulimit -u

unlimited

cat /proc/sys/fs/file-max

813940

root@lemp-s-2vcpu-4gb-blr1-01:~# ulimit -Hn

1048576

root@lemp-s-2vcpu-4gb-blr1-01:~# ulimit -Sn

1024

root@lemp-s-2vcpu-4gb-blr1-01:~# cat /proc/sys/fs/file-max

813940

root@lemp-s-2vcpu-4gb-blr1-01:~# sysctl fs.file-max

fs.file-max = 813940

root@instance-1:~# sysctl fs.file-max
fs.file-max = 9223372036854775807
root@instance-1:~#

root@instance-1:~# ulimit -aS
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31814
max locked memory (kbytes, -l) 65536
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 31814
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

root@instance-1:~# ulimit -aH
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31814
max locked memory (kbytes, -l) 65536
max memory size (kbytes, -m) unlimited
open files (-n) 1048576
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) 31814
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
root@instance-1:~#

what is ulimit in linux?

command that can set or report the resource limit of the current user.

ulimit

unlimited

ulimit -a

max process 31814

Difference between hard & soft ulimits in linux?

Hard limits are maintained by the kernel while the soft limits are enforced by the shell.
A soft limit is like a warning and hard limit is a real max limit

“hard” and “soft”. Hard resource limit defines the physical limit that the user can reach. The “soft” resource limit is manageable by the user. Its value can go up to the “hard” limit.

To check the value of max “core file size”, use the flag “-c”.

ulimit -c

root@instance-1:~# ulimit -c
0

For max “data seg size” (in kilobytes)

root@instance-1:~# ulimit -d
unlimited

ulimit -u – maximum number of process

ulimit -u
31814

linux check how many processes are running

Command to count the number of processes running in Linux

root@instance-1:~# ps aux | wc -l
118

ps -U user1 | wc -l

ps -U www-data | wc -l

ps -U mysql | wc -l

ps -U root | wc -l

ps -C httpd | wc -l

root@instance-1:~# ps -U www-data | wc -l
12

root@instance-1:~# ps -U mysql | wc -l
2

root@instance-1:~# ps -U root | wc -l
98

To know many file descriptors per process*?

ulimit -n

1024

ulimit -t – time limit to each process run

unlimited

ulimit –help

sudo nano /etc/security/limits.conf

core: Core file size (in KB).
data: Max data size (in KB)
fsize: Max file size (in KB).
memlock: Max locked-in-memory address space (in KB).
nofile: Max number of file descriptors
stack: Max size of the stack (in KB).
cpu: Maximum CPU time (in MIN).
maxlogins: Maximum number of logins for the current user/group
priority: Set the priority of the user’s processes
rtprio: Max priority in real-time.

do changes in /etc/security/limits.conf require a reboot?

No but you should close all active sessions windows.

fs.file-max vs ulimit

ulimit shows the per-process maximum open files by user

ulimit -n
1024

(system wide limit) file-max is the maximum File Descriptors (FD) enforced on a kernel level, which cannot be surpassed by all processes without increasing.

root@instance-1:~# cat /proc/sys/fs/file-max
9223372036854775807

temporary change ulimits for user until restart

-n processes

root@instance-1:~# ulimit -n
1024

ulimit -n 2048

root@instance-1:~# ulimit -n 2048
root@instance-1:~# ulimit -n
2048
root@instance-1:~#

What;s the difference between file descriptor and inode limits?

Files are represented by inodes

All information needed by the filesystem to handle a file is included in a data structure called an inode.

Max files for user account in a system ex: 1,50,000 files in shared server account. //general “file on a disk”

essentially an index into an array of open files kept by the kernel. files can be open at a time.

file descriptor: it is just an integer number that uniquely represents an opened file in operating system

User Level FD Limits

/etc/security/limits.conf

www-data soft nproc 100000

www-data soft nofile 100000

1 2	soft nofile 4096 hard nofile 4096

two file descriptors for each connection increase no file to support highload by os.

File Descriptors (FD)

worker_rlimit_nofile 100000;

nano /etc/sysctl.conf

### IMPROVE SYSTEM MEMORY MANAGEMENT ###

# Increase size of file handles and inode cache

fs.file-max = 2097152

# Do less swapping

vm.swappiness = 10

vm.dirty_ratio = 60

vm.dirty_background_ratio = 2

### GENERAL NETWORK SECURITY OPTIONS ###

# Number of times SYNACKs for passive TCP connection.

net.ipv4.tcp_synack_retries = 2

# Allowed local port range

net.ipv4.ip_local_port_range = 2000 65535

# Protect Against TCP Time-Wait

net.ipv4.tcp_rfc1337 = 1

# Decrease the time default value for tcp_fin_timeout connection

net.ipv4.tcp_fin_timeout = 15

# Decrease the time default value for connections to keep alive

net.ipv4.tcp_keepalive_time = 300

net.ipv4.tcp_keepalive_probes = 5

net.ipv4.tcp_keepalive_intvl = 15

### TUNING NETWORK PERFORMANCE ###

# Default Socket Receive Buffer

net.core.rmem_default = 31457280

# Maximum Socket Receive Buffer

net.core.rmem_max = 12582912

# Default Socket Send Buffer

net.core.wmem_default = 31457280

# Maximum Socket Send Buffer

net.core.wmem_max = 12582912

# Increase number of incoming connections

net.core.somaxconn = 65535

# Increase number of incoming connections backlog

net.core.netdev_max_backlog = 65535

# Increase the maximum amount of option memory buffers

net.core.optmem_max = 25165824

# Increase the maximum total buffer-space allocatable

# This is measured in units of pages (4096 bytes)

net.ipv4.tcp_mem = 65535 131072 262144

net.ipv4.udp_mem = 65535 131072 262144

# Increase the read-buffer space allocatable

net.ipv4.tcp_rmem = 8192 87380 16777216

net.ipv4.udp_rmem_min = 16384

# Increase the write-buffer-space allocatable

net.ipv4.tcp_wmem = 8192 65535 16777216

net.ipv4.udp_wmem_min = 16384

# Increase the tcp-time-wait buckets pool size to prevent simple DOS attacks

net.ipv4.tcp_max_tw_buckets = 1440000

net.ipv4.tcp_tw_recycle = 1

net.ipv4.tcp_tw_reuse = 1

Paste this (at the end of the file) to /etc/security/limits.conf (using nano /etc/security/limits.conf) and save it,

nginx soft nofile 2097152

nginx hard nofile 2097152

www-data soft nofile 2097152

www-data hard nofile 2097152

Paste this (at the end of the file) to /etc/pam.d/common-session (using nano /etc/pam.d/common-session) and save it,

session required pam_limits.so

Change listen.backlog in /etc/php5/fpm/pool.d/www.conf (using nano /etc/php5/fpm/pool.d/www.conf) and save it,

listen.backlog = 65535 //default to system

Change worker_rlimit_nofile in /etc/nginx/nginx.conf (using nano /etc/nginx/nginx.conf) and save it,

worker_rlimit_nofile 99999;

And finally reboot 🙂

Ephemeral ports – When used as a proxy

Tune file descriptor limits on Linux

php-fpm, also uses user as nginx server

www-data

nginx uses 2 file descriptor per connection. max connections 512 with default linux configuration.

apache uses one process per 1 connection.

default limit 1024 files for one process.

nginx is single threaded /single process.

ulimit-aS

ulimit-aH (you cannot increase above this limit without increasing kernel parameters)

to view current user limits

su www-data

and run above commands.

Updating Default Kernel Parameter Settings

to increase the number of simultaneously connection to db server

msgmni

kernel.shmmax=268435456 for 32-bit
kernel.shmmax=1073741824 for 64-bit
kernel.msgmni=1024
fs.file-max=8192
kernel.sem=”250 32000 32 1024″

useful commands

too, htop, vimsat,iosat, sar

sudo apt-get clean

sudo apt-get autoremove

ubuntu 20.04 server optimization

Tuning Nginx to handle more than 50.000 reqs/s

read nginx.conf settings , php -fpm conf pool config , mysql conf config

linux sysctl conf linux security limits conf

Disable unnecessary daemons to save memory & cpu

Tuned – Automatic Performance Tuning of CentOS/RHEL Servers

how to apply changes to /etc/security/limits.conf immediately

applies after reboot

without reboot, modify /etc/pam.d/common-session by adding this line at the end of file:

session required pam_limits.so

logout & login

linux performance tuning inode limit file descriptors tco, kernel etc

Tune Linux for Performance

etc/sysctl.conf

File descriptors/ max of open files

/etc/sysctl.conf

what is ulimit and how to change it permanently?

2. set permanent ulimit values at /etc/security/limits.conf

what is ulimit in linux?

Difference between hard & soft ulimits in linux?

To know many file descriptors per process*?

do changes in /etc/security/limits.conf require a reboot?

fs.file-max vs ulimit

temporary change ulimits for user until restart

What;s the difference between file descriptor and inode limits?

User Level FD Limits

nano /etc/sysctl.conf

### IMPROVE SYSTEM MEMORY MANAGEMENT ###

### GENERAL NETWORK SECURITY OPTIONS ###

### TUNING NETWORK PERFORMANCE ###

# Increase the tcp-time-wait buckets pool size to prevent simple DOS attacks

Tune file descriptor limits on Linux

Updating Default Kernel Parameter Settings

ubuntu 20.04 server optimization

how to apply changes to /etc/security/limits.conf immediately

wordpress

Server Admin Cloud

Tune Linux for Performance

etc/sysctl.conf

File descriptors/ max of open files

/etc/sysctl.conf

what is ulimit and how to change it permanently?

2. set permanent ulimit values at /etc/security/limits.conf

what is ulimit in linux?

Difference between hard & soft ulimits in linux?

To know many file descriptors per process*?

do changes in /etc/security/limits.conf require a reboot?

fs.file-max vs ulimit

temporary change ulimits for user until restart

What;s the difference between file descriptor and inode limits?

User Level FD Limits

nano /etc/sysctl.conf

### IMPROVE SYSTEM MEMORY MANAGEMENT ###

### GENERAL NETWORK SECURITY OPTIONS ###

### TUNING NETWORK PERFORMANCE ###

# Increase the tcp-time-wait buckets pool size to prevent simple DOS attacks

Tune file descriptor limits on Linux

Updating Default Kernel Parameter Settings

ubuntu 20.04 server optimization

how to apply changes to /etc/security/limits.conf immediately

Related topics:

wordpress

Server Admin Cloud