linux sysctl command tweaks & hardening

tune linux sysctl to increase network connections for nginx, lempo, mysql etc. also increase file descriptors  to support connections. identify the difference between user limits & system limits.

net.core.somaxconn nginx

connections per seconds

0-65356max

 

 

#
# file: '/etc/sysctl.conf'
#

vm.swappiness = 0
vm.max_map_count = 262144

net.ipv4.tcp_wmem = 4096 65536 33554432
net.ipv4.tcp_syn_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_synack_retries = 3
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_rmem = 4096 87380 33554432
net.ipv4.tcp_max_tw_buckets = 5880000
net.ipv4.tcp_max_syn_backlog = 3240000
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_congestion_control = cubic

net.ipv4.neigh.default.gc_thresh3 = 450560
net.ipv4.neigh.default.gc_thresh2 = 450560
net.ipv4.neigh.default.gc_thresh1 = 225280
net.ipv4.neigh.default.gc_stale_time = 7200

net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.ip_forward = 1

net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.default.secure_redirects = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.all.log_martians = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.accept_redirects = 0

net.core.wmem_max = 67108864
net.core.rmem_max = 67108864
net.core.rmem_default = 67108864
net.core.wmem_default = 67108864

net.ipv4.tcp_sack = 0
net.ipv4.tcp_dsack = 0
net.ipv4.tcp_fack = 0

# Increase number of incoming connections
net.core.somaxconn = 65535

# Increase number of incoming connections backlog
net.core.netdev_max_backlog = 65535

net.core.default_qdisc = fq kernel.randomize_va_space = 1 kernel.pid_max = 65536 kernel.msgmnb = 65536 kernel.msgmax = 65536 fs.nr_open = 4000000 fs.file-max = 4000000

FIle descriptors fs max open files

root@instance-1:~# cat /proc/sys/fs/file-max
9223372036854775807

NO Files

NO proc are user level set at /etc/security/limit.conf

How to check the `net.core.somaxconn` value?

root@instance-1:~# sysctl -a | grep net.core.somaxconn
net.core.somaxconn = 4096

sysctl -a | grep net.core.netdev_max_backlog

root@instance-1:~# sysctl -a | grep net.core.netdev_max_backlog
net.core.netdev_max_backlog = 1000

sysctl –all

or
sysctl –a

How to reload sysctl.conf variables on Linux

sysctl --load

sudo sysctl -p /etc/sysctl.d/nginx.conf

sudo sysctl -p /etc/sysctl.conf

variable=value

sysctl -w variable=value

sysctl -w net.core.somaxconn = 65535

sysctl -w net.core.netdev_max_backlog = 65535

sysctl -w  net.core.somaxconn = 4096

reload the sysctl

sysctl -p

permanent changes at vi /etc/sysctl.conf

 

net.core.netdev_max_backlog vs net.ipv4.tcp_max_syn_backlog

 

net.core.netdev_max_backlog is a per CPU core setting.

The maximum number of connections in the queue is set in the net.ipv4.tcp_max_syn_backlog kernel setting

linux kernels up through v5.3, while SOMAXCONN was raised to 4096 in

net.core.netdev_max_backlog – The rate at which packets are buffered by the network card before being handed off to the CPU.

net.core.somaxconn – The maximum number of connections that can be queued for acceptance by NGINX.

if error message in kernel log indicate that the value is too small.

512 connections per second

 

sysctl command list examples

to view all values

sysctl -a

to read value of variable

sysctl somaxconn

root@-s-4vcpu-8gb-blr1-01:~# cat /proc/sys/net/core/somaxconn
65536

 

to modify variable value

sysctl -w parameter=value

to save changes in /etc/sysctl.conf

sysctl -p

to reload all system configuration files

sysctl –system

part 2 /etc/security/limits.conf file for user level limits

sys.fs.file-max – The system‑wide limit for file descriptors
nofile – The user file descriptor limit, set in the

net.ipv4.ip_local_port_range if running out (Ephemeral) ports increase 1024 to 65000

 

root@instance-1:~# systemctl show nginx | grep LimitNOFILE
LimitNOFILE=524288
LimitNOFILESoft=1024

root@instance-1:~# cat /lib/systemd/system/nginx.service
[Unit]
Description=nginx – high performance web server
Documentation=http://nginx.org/en/docs/
After=network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target

[Service]
Type=forking
PIDFile=/var/run/nginx.pid
ExecStart=/usr/sbin/nginx -c /etc/nginx/nginx.conf
ExecReload=/bin/sh -c “/bin/kill -s HUP $(/bin/cat /var/run/nginx.pid)”
ExecStop=/bin/sh -c “/bin/kill -s TERM $(/bin/cat /var/run/nginx.pid)”

[Install]
WantedBy=multi-user.target
root@instance-1:~#

root@instance-1:~# cat /etc/security/limits.d/nginx.conf
cat: /etc/security/limits.d/nginx.conf: No such file or directory
root@instance-1:~# nano /etc/security/limits.d/nginx.conf
root@instance-1:~# cat /etc/security/limits.d/nginx.conf
nginx soft nofile 64000
nginx hard nofile 64000
root@instance-1:~#

root@instance-1:~# systemctl show nginx | grep LimitNOFILE
LimitNOFILE=524288
LimitNOFILESoft=1024

system limit:

set desire

fs.file-max = 3261780

Worker or threads limit of a process

system

root@instance-1:~# cat /proc/sys/kernel/threads-max
63628

kernel setting kernel.threads-max

present running threads

root@instance-1:~# ps -eo nlwp | tail -n +2 | \
> awk ‘{ num_threads += $1 } END { print num_threads }’
194

User Limit processes

root@instance-1:~# ulimit -u // processes
31814

root@instance-1:~# systemctl show nginx | grep LimitNPROC
LimitNPROC=31814
LimitNPROCSoft=31814

file descriptor limit per user

root@instance-1:~# ulimit -n
1024

nofile – max number of open files

nproc – max number of processes

To set ulimit value on a parameter use the below command.

# ulimit -p [new_value]

ulimit -n 2048

root@instance-1:~# ulimit -n
1024
root@instance-1:~# ulimit -n 2048
root@instance-1:~# ulimit -n
2048

 

you can set as variable but already set to high by default

kernel.threads-max = 3261780

cat /etc/security/limits.d/nginx.conf

nano /etc/security/limits.d/nginx.conf

nginx soft nofile 64000
nginx hard nofile 64000
nginx soft nproc 64000
nginx hard nproc 64000

 

/etc/sysctl.d/00-network.conf
# Receive Queue Size per CPU Core, number of packets
# Example server: 8 cores
net.core.netdev_max_backlog = 4096# SYN Backlog Queue, number of half-open connections
net.ipv4.tcp_max_syn_backlog = 32768# Accept Queue Limit, maximum number of established
# connections waiting for accept() per listener.
net.core.somaxconn = 65535# Maximum number of SYN and SYN+ACK retries before
# packet expires.
net.ipv4.tcp_syn_retries = 1
net.ipv4.tcp_synack_retries = 1# Timeout in seconds to close client connections in
# TIME_WAIT after receiving FIN packet.
net.ipv4.tcp_fin_timeout = 5# Disable SYN cookie flood protection
net.ipv4.tcp_syncookies = 0# Maximum number of threads system can have, total.
# Commented, may not be needed. See user limits.
#kernel.threads-max = 3261780# Maximum number of file descriptors system can have, total.
# Commented, may not be needed. See user limits.
#fs.file-max = 3261780

mysql ulimit open files / mysql open_files_limit / mysql max_open_files

Add the following for all users to the bottom for of the file and save it.
* soft nofile 1024000
* hard nofile 1024000
* soft nproc 10240
* hard nproc 10240
root soft nproc unlimited

* means all users


my.cnf  /etc/mysql/my.cnf
[mysqld]
open_files_limit = 102400

SHOW VARIABLES LIKE 'open_files_limit';

nginx settings

net.ipv4.ip_local_port_range = 1024 64999
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.core.wmem_max = 16777216
net.core.rmem_max = 16777216
net.ipv4.tcp_tw_reuse = 1
net.core.netdev_max_backlog = 30000
net.core.somaxconn = 32768
net.ipv4.tcp_max_orphans = 32768


What is linux somaxconn & how to increase and check?


linux somaxconn tcp_max_syn_backlog

somaxconn" max number of tcp established connections to the server from all clients.
default 4096 
ex:
nginx default connection backlog_que  511 —  truncated to 128 on linux kernels through v5.3

tcp_max_syn_backlog: maximum number of unacknowledged connections from in a 3 way tcp handshake
application backlog que ex:pho
if connections are full they put in backlog queue.


threads limits by process
system

cat /proc/sys/kernel/threads-max 

 maximum number of threads a user can spin up:
$ ulimit -u
4096

sysctl reload without restart

edit /etc/pam.d/common-session and add the following line to the end:

session required pam_limits.so

 

sysctl -p

 

sudo systemctl status systemd-sysctl.service

 

sysctl command

sysctl oid security mac proc_enforce is read only

systemd sysctl service loaded failed failed apply kernel variables

 

sysctl hardening

# Enable TCP SYN cookie protection
net.ipv4.tcp_syncookies = 1

# Enable IP spoofing protection
net.ipv4.conf.all.rp_filter = 1

# Enable ignoring to ICMP requests and broadcasts request
net.ipv4.icmp_echo_ignore_all = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1

# Enable logging of spoofed packets, source routed packets and redirect packets
net.ipv4.conf.all.log_martians = 1

# Disable IP source routing
net.ipv4.conf.all.accept_source_route = 0

# Disable ICMP redirect acceptance
net.ipv4.conf.all.accept_redirects = 0

 

spoofing attack against the IP address

syn flood attack prevention

sends massive numbers of SYN requests to a server to overwhelm it with open connections.

net.ipv4.tcp_syncookies = 1

net.ipv4.icmp_echo_ignore_broadcasts = 1

ICMP (ping) broadcasts and multicasts are usually a sign of Smurf attack.

etc more

sysctl net.ipv4.tcp_syncookies

root@-s-4vcpu-8gb-blr1-01:~# sysctl net.ipv4.tcp_syncookies
net.ipv4.tcp_syncookies = 1

How to enable IP Forwarding in Linux

root@-s-4vcpu-8gb-blr1-01:~# sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 0

disabled

to enable

sysctl -w net.ipv4.ip_forward=1

sysctl net ipv4 ip_forward 1

65536 connection limit

65535 connection ports limit for single user to server.

65535 vs 65536  (0 zero is missing here)

65536 IP packet limit  max data that tcp protocol send per packet.

IPv4 “Total Length” header field has 16 bits to indicate the size of the packet in bytes.

1500 MTU maximum segment size:  by ethernet card in most systems.

 

references https://medium.com/snapt/haproxy-performance-tweaks-sysctl-and-config-50605b84d32d https://community.mellanox.com/s/article/linux-sysctl-tuning