1st set at system level then user level then process level.
nginx open file limit rlimit should twice than connection limit
UNIX offers several ways to set the number of FDs per process:
- The
ulimitcommand if you start NGINX from a shell - The
initscript orsystemdservice manifest variables if you start NGINX as a service - The /etc/security/limits.conf file
There is also a system‑wide limit on the number of FDs, which you can set with the OS’s sysctl fs.file-max command. It is usually large enough, but it is worth verifying that the maximum number of file descriptors all NGINX worker processes might use (worker_rlimit_nofile * worker_processes) is significantly less than fs.file‑max.
worker_processes auto equals number cores, 4 core 4*1024= 4086 files = 2048 connections.
check nginx rmlit nofiles
finding process id, check existing and set in daemon , after adding to etc secuirty limit conf file.
check the current file limit on the master process
cat /proc/$(cat /var/run/nginx.pid)/limits | grep open.files
root@srv443573:~# cat /proc/$(cat /var/run/nginx.pid)/limits | grep open.files
Max open files 30000 30000 files
check the current limit on the nginx worker process
ps --ppid $(cat /var/run/nginx.pid) -o %p|sed '1d'|xargs -I{} cat /proc/{}/limits|grep open.files
root@srv443573:~# ps –ppid $(cat /var/run/nginx.pid) -o %p|sed ‘1d’|xargs -I{} cat /proc/{}/limits|grep open.files
Max open files 30000 30000 files
Max open files 30000 30000 files
Max open files 30000 30000 files
Max open files 30000 30000 files
worker_rlimit_nofile directive in {,/usr/local}/etc/nginx/nginx.conf
Not works limits by system.
set nolimit without Systemd in ect security limits
# /etc/security/limits.conf
# /etc/default/nginx (ULIMIT)
$ nano /etc/security/limits.d/nginx.conf
nginx soft nofile 65536
nginx hard nofile 65536
set nolimit with Systemd ubuntu
nano /lib/systemd/system/nginx.service
LimitNOFILE=30000
check existing open file limit
process level file descriptor limit
cat /proc/$(cat /var/run/nginx.pid)/limits
root@instance-1:~# cat /proc/491113/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 63948 63948 processes
Max open files 1024 524288 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 63948 63948 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
root@instance-1:~#
nano /lib/systemd/system/nginx.service
LimitNOFILE=30000
systemctl daemon-reload
nginx -s reload
cat /lib/systemd/system/nginx.service
[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t -q -g ‘daemon on; master_process on;’
ExecStart=/usr/sbin/nginx -g ‘daemon on; master_process on;’
ExecReload=/usr/sbin/nginx -g ‘daemon on; master_process on;’ -s reload
ExecStop=-/sbin/start-stop-daemon –quiet –stop –retry QUIT/5 –pidfile /run/nginx.pid
TimeoutStopSec=5
KillMode=mixed
LimitNOFILE=30000
[Install]
WantedBy=multi-user.target
root@instance-1:~# cat /proc/493972/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 63948 63948 processes
Max open files 30000 30000 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 63948 63948 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
User Level FD file descriptors Limits
cat /etc/security/limits.conf
#@student – maxlogins 4
nginx soft nofile 100000
nginx hard nofile 300000
www-data soft nofile 200000
www-data hard nofile 500000
mysql soft nofile 30000
mysql hard nofile 50000
* soft nofile 60000 (any user)
* hard nofile 60000
* soft core 0
root limits comments
nano /etc/security/limits.conf
After you have changed the ulimit value, restart the computer, or below.
reload etc security limits without restart
pam_limits.so in /etc/pam.d/common-session
set dynamically user limits with ulimits command
ulimit -n 100000
System Max Open Files Limit
also check sysctl,conf
to know system wide limits
# Controls the maximum number of open files in oeratng system
fs.file-max=65536
root@srv443573:~# cat /proc/sys/fs/file-max
500000
ideal should be equal to 10% of memory.
nginx connection dropped error
error open socket #3442 left in connection 3412
alert open socket ## left in connection # happens when a socket is left open after a nginx process thread is shutdown. There is a debugging note about this on the nginx wiki. This type of problem can exhaust your available TCP sockets, preventing new connections. Make sure your processes running under nginx are closing their sockets properly, if you believe they are already supposed to be doing that then something is interrupting the proper closing of sockets.
solution 1:
timeout setting should be equal in php-fpm and nginx fascgi timeout
Warning: The unit file, source configuration file or drop-ins of nginx.service changed on disk. Run ‘systemctl daemon-reload’ to reload units.
root@instance-1:~# systemctl daemon-reload
root@instance-1:~# tail -f /var/log/nginx/error.log
2023/09/01 07:30:58 [alert] 489857#489857: *161898 open socket #6669 left in connection 9913
2023/09/01 07:30:58 [alert] 489857#489857: *141616 open socket #50 left in connection 9923
2023/09/01 07:30:58 [alert] 489857#489857: *162218 open socket #971 left in connection 9924
2023/09/01 07:30:58 [alert] 489857#489857: *135246 open socket #4006 left in connection 9942
2023/09/01 07:30:58 [alert] 489857#489857: *150263 open socket #3764 left in connection 9950
2023/09/01 07:30:58 [alert] 489857#489857: *148091 open socket #1065 left in connection 9960
2023/09/01 07:30:58 [alert] 489857#489857: *158398 open socket #1459 left in connection 9966
2023/09/01 07:30:58 [alert] 489857#489857: *131779 open socket #6085 left in connection 9977
2023/09/01 07:30:58 [alert] 489857#489857: *146189 open socket #1021 left in connection 9983
2023/09/01 07:30:58 [alert] 489857#489857: aborting
also ngincx worker process high cpu
soltion increaeing rmlimt no files
in nginx.conf not worked
worker_rlimit_nofile 4096;
^[[A^[[A^[[A^[[A^C
a. find the nginx process ID
$ ps aux -P | head -1
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root@instance-1:~# ps aux |grep nginx
root 491113 0.0 0.0 261488 1844 ? Ss 07:30 0:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data 491114 19.7 0.4 325600 70908 ? S 07:30 1:08 nginx: worker process
www-data 491115 38.6 0.8 391384 137060 ? S 07:30 2:14 nginx: worker process
www-data 491116 2.8 0.1 274932 20188 ? S 07:30 0:09 nginx: worker process
www-data 491117 8.5 0.2 289024 34456 ? S 07:30 0:29 nginx: worker process
www-data 491118 0.0 0.0 261636 3252 ? S 07:30 0:00 nginx: cache manager process
root 492266 0.0 0.0 5132 700 pts/0 S+ 07:36 0:00 grep nginx
cat /proc/{PID}/limits
cat /proc/493972/limits
cat /proc/518/limits