= 2011-04-13 =
== Apache Performance Tuning ==
* 4/11~4/13 多次重新開機 trac 網站，一直無法找出記憶體需求突然高漲的主因是因為升級到 Squeeze，改用 Apache MPM 版本（[http://packages.debian.org/apache2-mpm-prefork apache2-mpm-prefork]），還是其他原因。
* [[Image(jazz/11-04-13:11-04-14_memory-week.png)]]
* 花了兩天時間，總算找出造成 trac 網站大量記憶體需求的特徵：刻意發起造成 CLOSE_WAIT 的攻擊行為。
{{{
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15652 www-data 20 0 783m 763m 4624 R 97.0 75.4 17:31.07 apache2
20059 www-data 20 0 52836 31m 4732 S 2.7 3.2 0:06.86 apache2
jazz@trac-pool:~$ sudo netstat -nap | grep 15652
tcp6 1 0 140.110.X.X:80 220.181.93.1:55903 CLOSE_WAIT 15652/apache2
}}}
* 手動砍掉 15652 這個 process 之後，同樣這個 IP，後來又產生了一次 CLOSE_WAIT 現象，不禁懷疑這是一種網站攻擊方式。
{{{
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23708 www-data 20 0 414m 396m 4544 R 99.7 39.2 2:44.46 apache2
23025 www-data 20 0 74940 55m 4752 S 0.0 5.5 0:27.58 apache2
tcp6 0 0 140.110.X.X:80 220.181.93.1:64932 TIME_WAIT -
tcp6 0 0 140.110.X.X:80 220.181.93.1:47074 TIME_WAIT -
tcp6 0 0 140.110.X.X:80 220.181.93.1:50047 TIME_WAIT -
tcp6 0 0 140.110.X.X:80 220.181.93.1:43913 TIME_WAIT -
}}}
* <解決方法> 降低 net.ipv4.tcp_keepalive_time 的數值，預設是 7200 秒（兩小時），真的太長了！！
{{{
echo "decrease TCP socket TIME_WAIT time"
sysctl -w net.ipv4.tcp_fin_timeout=10
sysctl -w net.ipv4.tcp_tw_reuse=1
sysctl -w net.ipv4.tcp_tw_recycle=1
echo "decrease TCP socket CLOSE_WAIT time"
sysctl -w net.ipv4.tcp_keepalive_time=30
sysctl -w net.ipv4.tcp_keepalive_probes=2
sysctl -w net.ipv4.tcp_keepalive_intvl=2
}}}
* <參考> [http://tw.myblog.yahoo.com/yaitoo-richzal/article?mid=212&sc=1 大量 CLOSE_WAIT 的影響]
{{{
大量的 CLOSE_WAIT 連接，直接佔滿 TCP 連線佇列，
導致 Apache 失去回應，且 CPU 使用量與記憶體使用量快速提高！！
}}}
* <參考> [http://haka.sharera.com/blog/BlogTopic/32309.htm CLOSE_WAIT 生成的原因]
* 另外我也做了兩個小調校：
* <1> 限制每個 process 的記憶體使用 - 在 /etc/profile 中加入
{{{
ulimit -v 524288
}}}
* 其結果是最高可以使用的記憶體是 524288 = 512 * 1024 kbytes = 512 MB。
{{{
virtual memory (kbytes, -v) 524288
}}}
* <2> 把 /etc/apache2/apache2.conf 改成只使用 prefork 模組，並且降低 !KeepAliveTimeout 的時間(15秒降低到2秒)，還有降低預設開啟的 apache process 數量。
{{{
#!diff
--- /etc/apache2/apache2.conf.org 2011-04-12 10:32:53.000000000 +0800
+++ /etc/apache2/apache2.conf 2011-04-14 00:38:39.000000000 +0800
@@ -64,7 +64,7 @@
#
# Timeout: The number of seconds before receives and sends time out.
#
-Timeout 300
+Timeout 30
#
# KeepAlive: Whether or not to allow persistent connections (more than
@@ -83,7 +83,7 @@
# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
#
-KeepAliveTimeout 15
+KeepAliveTimeout 2
##
## Server-Pool Size Regulation (MPM specific)
@@ -96,11 +96,11 @@
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
- StartServers 5
- MinSpareServers 5
- MaxSpareServers 10
- MaxClients 150
- MaxRequestsPerChild 0
+ StartServers 1
+ MinSpareServers 1
+ MaxSpareServers 1
+ MaxClients 5
+ MaxRequestsPerChild 100
# worker MPM
@@ -113,15 +113,15 @@
# and starting Apache.
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
-
- StartServers 2
- MinSpareThreads 25
- MaxSpareThreads 75
- ThreadLimit 64
- ThreadsPerChild 25
- MaxClients 150
- MaxRequestsPerChild 0
-
+#
+# StartServers 1
+# MinSpareThreads 1
+# MaxSpareThreads 1
+# ThreadLimit 1
+# ThreadsPerChild 1
+# MaxClients 5
+# MaxRequestsPerChild 50
+#
# event MPM
# StartServers: initial number of server processes to start
@@ -130,15 +130,15 @@
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
-
- StartServers 2
- MaxClients 150
- MinSpareThreads 25
- MaxSpareThreads 75
- ThreadLimit 64
- ThreadsPerChild 25
- MaxRequestsPerChild 0
-
+#
+# StartServers 1
+# MinSpareThreads 1
+# MaxSpareThreads 1
+# ThreadLimit 1
+# ThreadsPerChild 1
+# MaxClients 5
+# MaxRequestsPerChild 50
+#
# These need to be set in /etc/apache2/envvars
User ${APACHE_RUN_USER}
}}}
* <參考> [http://bobcares.com/blog/?p=71 Configuring Apache for Maximum Performance]
* <參考> [http://wiki.vpslink.com/Low_memory_MySQL_/_Apache_configurations Low memory MySQL / Apache configurations]
* <參考> [http://www.devside.net/articles/apache-performance-tuning Apache Performance Tuning]
{{{
prefork [default MPM for Apache 2.0 and 1.3]:
* Apache 1.3-based.
* Multiple processes, 1 thread per process, processes handle requests.
* Used for security and stability.
* Has higher memory consumption and lower performance over the newer Apache 2.0-based threaded MPMs.
worker:
* Apache 2.0-based.
* Multiple processes, many threads per process, threads handle requests.
* Used for lower memory consumption and higher performance.
* Does not provide the same level of isolation request-to-request, as a process-based MPM does.
}}}
{{{
MaxClients ≈ (RAM - size_all_other_processes)/(size_apache_process)
}}}