tune linux sysctl to increase network connections for nginx, lemp, mysql etc. also increase file descriptors to support connections. identify the difference between user limits & system limits.
net.core.somaxconn nginx
connections per seconds
0-65356max
# # file: '/etc/sysctl.conf' # vm.swappiness = 0 vm.max_map_count = 262144 net.ipv4.tcp_wmem = 4096 65536 33554432 net.ipv4.tcp_syn_retries = 3 net.ipv4.tcp_syncookies = 1 net.ipv4.tcp_synack_retries = 3 net.ipv4.tcp_slow_start_after_idle = 0 net.ipv4.tcp_rmem = 4096 87380 33554432 net.ipv4.tcp_max_tw_buckets = 5880000 net.ipv4.tcp_max_syn_backlog = 3240000 net.ipv4.tcp_max_orphans = 262144 net.ipv4.tcp_keepalive_probes = 5 net.ipv4.tcp_keepalive_intvl = 30 net.ipv4.tcp_fin_timeout = 10 net.ipv4.tcp_congestion_control = cubic net.ipv4.neigh.default.gc_thresh3 = 450560 net.ipv4.neigh.default.gc_thresh2 = 450560 net.ipv4.neigh.default.gc_thresh1 = 225280 net.ipv4.neigh.default.gc_stale_time = 7200 net.ipv4.ip_local_port_range = 1024 65535 net.ipv4.ip_forward = 1 net.ipv4.conf.default.send_redirects = 0 net.ipv4.conf.default.secure_redirects = 0 net.ipv4.conf.default.accept_source_route = 0 net.ipv4.conf.default.accept_source_route = 0 net.ipv4.conf.default.accept_redirects = 0 net.ipv4.conf.all.send_redirects = 0 net.ipv4.conf.all.secure_redirects = 0 net.ipv4.conf.all.rp_filter = 1 net.ipv4.conf.all.log_martians = 0 net.ipv4.conf.all.accept_source_route = 0 net.ipv4.conf.all.accept_redirects = 0 net.core.wmem_max = 67108864 net.core.rmem_max = 67108864 net.core.rmem_default = 67108864 net.core.wmem_default = 67108864 net.ipv4.tcp_sack = 0 net.ipv4.tcp_dsack = 0 net.ipv4.tcp_fack = 0
# Increase number of incoming connections
net.core.somaxconn = 65535
# Increase number of incoming connections backlog
net.core.netdev_max_backlog = 65535
net.core.default_qdisc = fq kernel.randomize_va_space = 1 kernel.pid_max = 65536 kernel.msgmnb = 65536 kernel.msgmax = 65536 fs.nr_open = 4000000 fs.file-max = 4000000
FIle descriptors fs max open files
root@instance-1:~# cat /proc/sys/fs/file-max
9223372036854775807
NO Files
NO proc are user level set at /etc/security/limit.conf
How to check the `net.core.somaxconn` value?
root@instance-1:~# sysctl -a | grep net.core.somaxconn
net.core.somaxconn = 4096
sysctl -a | grep net.core.netdev_max_backlog
root@instance-1:~# sysctl -a | grep net.core.netdev_max_backlog
net.core.netdev_max_backlog = 1000
sysctl –all
or
sysctl –a
How to reload sysctl.conf variables on Linux
sysctl --load
sudo sysctl -p /etc/sysctl.d/nginx.conf
sudo sysctl -p /etc/sysctl.confvariable=value
sysctl -w variable=value
sysctl -w net.core.somaxconn = 65535
sysctl -w net.core.netdev_max_backlog = 65535
sysctl -w net.core.somaxconn = 4096
reload the sysctl
sysctl -p
permanent changes at vi /etc/sysctl.conf
net.core.netdev_max_backlog vs net.ipv4.tcp_max_syn_backlog
net.core.netdev_max_backlog is a per CPU core setting.
The maximum number of connections in the queue is set in the net.ipv4.tcp_max_syn_backlog kernel setting
linux kernels up through v5.3, while SOMAXCONN was raised to 4096 in
net.core.netdev_max_backlog – The rate at which packets are buffered by the network card before being handed off to the CPU.
net.core.somaxconn – The maximum number of connections that can be queued for acceptance by NGINX.
if error message in kernel log indicate that the value is too small.
512 connections per second
raise both the value of somaxconn and tcp_max_syn_backlog to get effect.
cat /proc/sys/net/core/somaxconn
cat /proc/sys/net/ipv4/tcp_max_syn_backlog
sysctl net.core.netdev_max_backlog
root@-s-4vcpu-8gb-blr1-01:~# sysctl net.core.netdev_max_backlog
net.core.netdev_max_backlog = 3240000
number of packets buffer at NIC network card before handles to cpu.
Maximum number of remembered unacknowledged connection requests from connecting client.
maximal size of ESTABLISHED queue
root@-s-4vcpu-8gb-blr1-01:~# sysctl net.ipv4.tcp_max_syn_backlog
net.ipv4.tcp_max_syn_backlog = 256
root@-s-4vcpu-8gb-blr1-01:~# cat /proc/sys/fs/file-nr
2272 0 2097152
the server has 2272 opened files out of 2097152.
net.ipv4.tcp_syncookies
to avoid A TCP SYN flood attack DOS denial of service
root@-s-4vcpu-8gb-blr1-01:~# sysctl net.ipv4.tcp_syncookies
net.ipv4.tcp_syncookies = 1
sysctl command list examples
to view all values
sysctl -a
to read value of variable
sysctl somaxconn
root@-s-4vcpu-8gb-blr1-01:~# cat /proc/sys/net/core/somaxconn
65536
to modify variable value
sysctl -w parameter=value
to save changes in /etc/sysctl.conf
sysctl -p
to reload all system configuration files
sysctl –system
part 2 /etc/security/limits.conf file for user level limits
sys.fs.file-max – The system‑wide limit for file descriptors
nofile – The user file descriptor limit, set in the
net.ipv4.ip_local_port_range if running out (Ephemeral) ports increase 1024 to 65000
root@instance-1:~# systemctl show nginx | grep LimitNOFILE
LimitNOFILE=524288
LimitNOFILESoft=1024
root@instance-1:~# cat /lib/systemd/system/nginx.service
[Unit]
Description=nginx – high performance web server
Documentation=http://nginx.org/en/docs/
After=network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target
[Service]
Type=forking
PIDFile=/var/run/nginx.pid
ExecStart=/usr/sbin/nginx -c /etc/nginx/nginx.conf
ExecReload=/bin/sh -c “/bin/kill -s HUP $(/bin/cat /var/run/nginx.pid)”
ExecStop=/bin/sh -c “/bin/kill -s TERM $(/bin/cat /var/run/nginx.pid)”
[Install]
WantedBy=multi-user.target
root@instance-1:~#
root@instance-1:~# cat /etc/security/limits.d/nginx.conf
cat: /etc/security/limits.d/nginx.conf: No such file or directory
root@instance-1:~# nano /etc/security/limits.d/nginx.conf
root@instance-1:~# cat /etc/security/limits.d/nginx.conf
nginx soft nofile 64000
nginx hard nofile 64000
root@instance-1:~#
root@instance-1:~# systemctl show nginx | grep LimitNOFILE
LimitNOFILE=524288
LimitNOFILESoft=1024
system limit:
set desire
fs.file-max = 3261780
Worker or threads limit of a process
system
root@instance-1:~# cat /proc/sys/kernel/threads-max
63628
kernel setting kernel.threads-max
present running threads
root@instance-1:~# ps -eo nlwp | tail -n +2 | \
> awk ‘{ num_threads += $1 } END { print num_threads }’
194
User Limit processes
root@instance-1:~# ulimit -u // processes
31814
root@instance-1:~# systemctl show nginx | grep LimitNPROC
LimitNPROC=31814
LimitNPROCSoft=31814
file descriptor limit per user
root@instance-1:~# ulimit -n
1024
nofile – max number of open files
nproc – max number of processes
To set ulimit value on a parameter use the below command.
# ulimit -p [new_value]
ulimit -n 2048
root@instance-1:~# ulimit -n
1024
root@instance-1:~# ulimit -n 2048
root@instance-1:~# ulimit -n
2048
you can set as variable but already set to high by default
kernel.threads-max = 3261780
cat /etc/security/limits.d/nginx.conf
nano /etc/security/limits.d/nginx.conf
nginx soft nofile 64000
nginx hard nofile 64000
nginx soft nproc 64000
nginx hard nproc 64000
/etc/sysctl.d/00-network.conf # Receive Queue Size per CPU Core, number of packets # Example server: 8 cores net.core.netdev_max_backlog = 4096# SYN Backlog Queue, number of half-open connections net.ipv4.tcp_max_syn_backlog = 32768# Accept Queue Limit, maximum number of established # connections waiting for accept() per listener. net.core.somaxconn = 65535# Maximum number of SYN and SYN+ACK retries before # packet expires. net.ipv4.tcp_syn_retries = 1 net.ipv4.tcp_synack_retries = 1# Timeout in seconds to close client connections in # TIME_WAIT after receiving FIN packet. net.ipv4.tcp_fin_timeout = 5# Disable SYN cookie flood protection net.ipv4.tcp_syncookies = 0# Maximum number of threads system can have, total. # Commented, may not be needed. See user limits. #kernel.threads-max = 3261780# Maximum number of file descriptors system can have, total. # Commented, may not be needed. See user limits. #fs.file-max = 3261780 mysql ulimit open files / mysql open_files_limit / mysql max_open_files Add the following for all users to the bottom for of the file and save it. * soft nofile 1024000 * hard nofile 1024000 * soft nproc 10240 * hard nproc 10240 root soft nproc unlimited * means all users my.cnf /etc/mysql/my.cnf [mysqld] open_files_limit = 102400 SHOW VARIABLES LIKE 'open_files_limit'; nginx settings net.ipv4.ip_local_port_range = 1024 64999 net.ipv4.tcp_wmem = 4096 65536 16777216 net.ipv4.tcp_rmem = 4096 87380 16777216 net.core.wmem_max = 16777216 net.core.rmem_max = 16777216 net.ipv4.tcp_tw_reuse = 1 net.core.netdev_max_backlog = 30000 net.core.somaxconn = 32768 net.ipv4.tcp_max_orphans = 32768 What is linux somaxconn & how to increase and check? linux somaxconn tcp_max_syn_backlog somaxconn" max number of tcp established connections to the server from all clients. default 4096 ex: nginx default connection backlog_que 511 — truncated to 128 on linux kernels through v5.3 tcp_max_syn_backlog: maximum number of unacknowledged connections from in a 3 way tcp handshake application backlog que ex:pho if connections are full they put in backlog queue. threads limits by process system cat /proc/sys/kernel/threads-max maximum number of threads a user can spin up: $ ulimit -u 4096
sysctl reload without restart
edit /etc/pam.d/common-session and add the following line to the end:
session required pam_limits.so
sysctl -p
sudo systemctl status systemd-sysctl.service
sysctl command
sysctl oid security mac proc_enforce is read only
systemd sysctl service loaded failed failed apply kernel variables
sysctl hardening
# Enable TCP SYN cookie protection
net.ipv4.tcp_syncookies = 1
# Enable IP spoofing protection
net.ipv4.conf.all.rp_filter = 1
# Enable ignoring to ICMP requests and broadcasts request
net.ipv4.icmp_echo_ignore_all = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1
# Enable logging of spoofed packets, source routed packets and redirect packets
net.ipv4.conf.all.log_martians = 1
# Disable IP source routing
net.ipv4.conf.all.accept_source_route = 0
# Disable ICMP redirect acceptance
net.ipv4.conf.all.accept_redirects = 0
spoofing attack against the IP address
syn flood attack prevention
sends massive numbers of SYN requests to a server to overwhelm it with open connections.
net.ipv4.tcp_syncookies = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1
ICMP (ping) broadcasts and multicasts are usually a sign of Smurf attack.
etc more
sysctl net.ipv4.tcp_syncookies
root@-s-4vcpu-8gb-blr1-01:~# sysctl net.ipv4.tcp_syncookies
net.ipv4.tcp_syncookies = 1
How to enable IP Forwarding in Linux
root@-s-4vcpu-8gb-blr1-01:~# sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 0
disabled
to enable
sysctl -w net.ipv4.ip_forward=1
sysctl net ipv4 ip_forward 1
65536 connection limit
65535 connection ports limit for single user to server.
65535 vs 65536 (0 zero is missing here)
65536 IP packet limit max data that tcp protocol send per packet.
IPv4 “Total Length” header field has 16 bits to indicate the size of the packet in bytes.
1500 MTU maximum segment size: by ethernet card in most systems.
references https://medium.com/snapt/haproxy-performance-tweaks-sysctl-and-config-50605b84d32d https://community.mellanox.com/s/article/linux-sysctl-tuning