Haproxy作为MySQL中间层如何避免TCP端口耗尽

Haproxy作为MySQL中间层是很成熟的方案,特别是解决从库的负载均衡和故障切换,在生产环境中有着广泛的应用。

在实际使用过程中,有两个问题比较容易发生:
1. TCP端口耗尽
2. 网卡带宽跑满
本文重点讲讲如何优化问题1,问题2暂不讨论。

优化一: 使用尽可能多的端口

Linux系统默认提供了65K个端口,每当Haproxy建立了一个到MySQL的连接,就会消耗一个端口;当Haproxy断开和MySQL的连接时,该端口并不会立即释放,而是会处于TIME_WAIT状态(2*MSL),超时后才会释放此端口供新的连接使用。

我的环境中,tcp_fin_timeout为15秒,也就是说如果我环境中的haproxy可以承载的最大并发连接数为64K/(15*2)=2.1K,可实际上达不到这个上限,原因如下:

$ sysctl net.ipv4.ip_local_port_range
net.ipv4.ip_local_port_range = 15000 65000

linux会保留一段端口,实际能参与分配的端口数只有50K,为了获得尽可能多的可分配端口,做如下调整:

# sysctl net.ipv4.ip_local_port_range="1025 65000"

记得修改/etc/sysctl.conf中对应的内容

优化二: 复用处于TIME_WAIT的端口

调整两个参数:

net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1

第一个参数很安全,可以不用过多关注。需要注意的是第二个参数,某些情况下会导致数据包被丢弃。

例如:client通过NAT连接haproxy,并且haproxy端打开了tcp_tw_recycle,同时saw_tstamp也没有关闭,当第一个连接建立并关闭后,此端口(句柄)处于TIME_WAIT状态,在2*MSL时间内又一个client(相同IP,如果打开了xfrm还要相同PORT)发一个syn包,此时linux内核就会认为这个数据包异常,从而丢掉这个包,并发送rst包.

不过通常情况下,client都是通过内网直接连接haproxy,所以可以认为tcp_tw_recycle是安全的,只是需要记住此坑。

优化三: 缩短TIME_WAIT时间

Linux系统默认MSL为60秒,也就是正常情况下,120秒后处于TIME_WAIT的端口(句柄)才会释放,可以将MSL的时间缩小,缩短端口的释放周期。

# cat /proc/sys/net/ipv4/tcp_fin_timeout
60
# echo 15 > /proc/sys/net/ipv4/tcp_fin_timeout

这是一个折中的数值,太小也会导致其它问题

以下内容摘录自 http://www.speedguide.net/articles/linux-tweaking-121


Other TCP Parameters to consider

TCP_FIN_TIMEOUT
This setting determines the time that must elapse before TCP/IP can release a closed connection and reuse its resources. During this TIME_WAIT state, reopening the connection to the client costs less than establishing a new connection. By reducing the value of this entry, TCP/IP can release closed connections faster, making more resources available for new connections. Addjust this in the presense of many connections sitting in the TIME_WAIT state:

# echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout
(default: 60 seconds, recommended 15-30 seconds)

Notes:
You can use any of the earlier described methods to reapply these settings at boot time.
Here is a quick way to view the number of connections and their states:

netstat -tan | grep ':80 ' | awk '{print $6}' | sort | uniq -c

TCP_KEEPALIVE_INTERVAL
This determines the wait time between isAlive interval probes. To set:

echo 30 > /proc/sys/net/ipv4/tcp_keepalive_intvl
(default: 75 seconds, recommended: 15-30 seconds)

TCP_KEEPALIVE_PROBES
This determines the number of probes before timing out. To set:

echo 5 > /proc/sys/net/ipv4/tcp_keepalive_probes
(default: 9, recommended 5)

TCP_TW_RECYCLE
It enables fast recycling of TIME_WAIT sockets. The default value is 0 (disabled). The sysctl documentation incorrectly states the default as enabled. It can be changed to 1 (enabled) in many cases. Known to cause some issues with hoststated (load balancing and fail over) if enabled, should be used with caution.

echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle
(boolean, default: 0)
  

TCP_TW_REUSE
This allows reusing sockets in TIME_WAIT state for new connections when it is safe from protocol viewpoint. Default value is 0 (disabled). It is generally a safer alternative to tcp_tw_recycle

echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse
(boolean, default: 0)

Note: The tcp_tw_reuse setting is particularly useful in environments where numerous short connections are open and left in TIME_WAIT state, such as web servers. Reusing the sockets can be very effective in reducing server load.

Another option for applying the above kernel parameters is to enter them in /etc/sysctl.conf in the format:
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_tw_recycle=1
...

Linux Netfilter Tweaks

Try this for a list netfilter parameters:  sysctl -a | grep netfilter

We can add the following commands to the /etc/sysctl.conf file to tune individual parameters, as follows.
To reduce the number of connections in TIME_WAIT state, we can decrease the number of seconds connections are kept in this state before being dropped:

# reduce TIME_WAIT from the 120s default
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 30

You can commit the sysctl.conf changes without rebooting (and test for possible syntax errors) by executing: sysctl -p
To check sysctl parameters, use: sysctl -a

Misc Notes: You may want to reduce net.netfilter.nf_conntrack_tcp_timeout_established to 900 or some manageable number as well.
To check the actual number of current connections in the TIME_WAIT state, for example, try: netstat -n | grep TIME_WAIT | wc -l

Kernel Recompile Option

There is another method one can use to set TCP/IP parameters, involving kernel recompile... If you're brave enough. Look for the parameters in the following files:
/LINUX-SOURCE-DIR/include/linux/skbuff.h  (Look for SK_WMEM_MAX & SK_RMEM_MAX)
/LINUX-SOURCE-DIR/include/net/tcp.h (Look for MAX_WINDOW & MIN_WINDOW)


优化四: 使用多IP

如优化一中所说,我们已经尽可能多的使用了系统提供的端口范围。但最多依然不超过65K。
Haproxy提供了内建的端口管理方法,可以充分利用以扩大我们的端口范围。

server mysql0 10.0.3.1:3306 check source 10.0.3.100:1025-65000
server mysql1 10.0.3.1:3306 check source 10.0.3.101:1025-65000

如果使用两个ip,我们可用的端口数就接近130K。扩展多个IP,就可以不断增加端口数。

优化五: 使用长连接

服务最好使用长连接,一是避免频繁的申请连接,导致端口耗尽;二是避免创建连接带来的时间消耗。

还没有评论,快来抢沙发!

发表评论

  • 😉
  • 😐
  • 😡
  • 😈
  • 🙂
  • 😯
  • 🙁
  • 🙄
  • 😛
  • 😳
  • 😮
  • emoji-mrgree
  • 😆
  • 💡
  • 😀
  • 👿
  • 😥
  • 😎
  • ➡
  • 😕
  • ❓
  • ❗
  • 66 queries in 0.491 seconds