2011-04-13

Apache Performance Tuning

• 4/11~4/13 多次重新開機 trac 網站，一直無法找出記憶體需求突然高漲的主因是因為升級到 Squeeze，改用 Apache MPM 版本（​apache2-mpm-prefork），還是其他原因。
• 
• 花了兩天時間，總算找出造成 trac 網站大量記憶體需求的特徵：刻意發起造成 CLOSE_WAIT 的攻擊行為。

    PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
    15652 www-data  20   0  783m 763m 4624 R 97.0 75.4  17:31.07 apache2
    20059 www-data  20   0 52836  31m 4732 S  2.7  3.2   0:06.86 apache2
  
  jazz@trac-pool:~$ sudo netstat -nap | grep 15652
  tcp6       1      0 140.110.X.X:80      220.181.93.1:55903      CLOSE_WAIT  15652/apache2
  

• 手動砍掉 15652 這個 process 之後，同樣這個 IP，後來又產生了一次 CLOSE_WAIT 現象，不禁懷疑這是一種網站攻擊方式。

    PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
  23708 www-data  20   0  414m 396m 4544 R 99.7 39.2   2:44.46 apache2
  23025 www-data  20   0 74940  55m 4752 S  0.0  5.5   0:27.58 apache2
  
  tcp6       0      0 140.110.X.X:80      220.181.93.1:64932      TIME_WAIT   -               
  tcp6       0      0 140.110.X.X:80      220.181.93.1:47074      TIME_WAIT   -               
  tcp6       0      0 140.110.X.X:80      220.181.93.1:50047      TIME_WAIT   -               
  tcp6       0      0 140.110.X.X:80      220.181.93.1:43913      TIME_WAIT   -               
  

• <解決方法> 降低 net.ipv4.tcp_keepalive_time 的數值，預設是 7200 秒（兩小時），真的太長了！！

  echo "decrease TCP socket TIME_WAIT time"
  sysctl -w net.ipv4.tcp_fin_timeout=10
  sysctl -w net.ipv4.tcp_tw_reuse=1
  sysctl -w net.ipv4.tcp_tw_recycle=1
  echo "decrease TCP socket CLOSE_WAIT time"
  sysctl -w net.ipv4.tcp_keepalive_time=30
  sysctl -w net.ipv4.tcp_keepalive_probes=2
  sysctl -w net.ipv4.tcp_keepalive_intvl=2
  

• <參考> ​大量 CLOSE_WAIT 的影響

  大量的 CLOSE_WAIT 連接，直接佔滿 TCP 連線佇列，
  導致 Apache 失去回應，且 CPU 使用量與記憶體使用量快速提高！！
  

• <參考> ​CLOSE_WAIT 生成的原因

• 另外我也做了兩個小調校：
• <1> 限制每個 process 的記憶體使用 - 在 /etc/profile 中加入

  ulimit -v 524288
  

• 其結果是最高可以使用的記憶體是 524288 = 512 * 1024 kbytes = 512 MB。

  virtual memory          (kbytes, -v) 524288
  

• <2> 把 /etc/apache2/apache2.conf 改成只使用 prefork 模組，並且降低 KeepAliveTimeout 的時間(15秒降低到2秒)，還有降低預設開啟的 apache process 數量。

  /etc/apache2/apache2.conf

  @@ -64 +64 @@
   #
   # Timeout: The number of seconds before receives and sends time out.
   #
  -Timeout 300
  +Timeout 30
   
   #
   # KeepAlive: Whether or not to allow persistent connections (more than
  @@ -83 +83 @@
   # KeepAliveTimeout: Number of seconds to wait for the next request from the
   # same client on the same connection.
   #
  -KeepAliveTimeout 15
  +KeepAliveTimeout 2
   
   ##
   ## Server-Pool Size Regulation (MPM specific)
  @@ -96 +96 @@
   # MaxClients: maximum number of server processes allowed to start
   # MaxRequestsPerChild: maximum number of requests a server process serves
   <IfModule mpm_prefork_module>
  -    StartServers          5
  -    MinSpareServers       5
  -    MaxSpareServers      10
  -    MaxClients          150
  -    MaxRequestsPerChild   0
  +    StartServers          1
  +    MinSpareServers       1
  +    MaxSpareServers       1
  +    MaxClients            5
  +    MaxRequestsPerChild 100
   </IfModule>
   
   # worker MPM
  @@ -113 +113 @@
   #              and starting Apache.
   # ThreadsPerChild: constant number of worker threads in each server process
   # MaxRequestsPerChild: maximum number of requests a server process serves
  -<IfModule mpm_worker_module>
  -    StartServers          2
  -    MinSpareThreads      25
  -    MaxSpareThreads      75 
  -    ThreadLimit          64
  -    ThreadsPerChild      25
  -    MaxClients          150
  -    MaxRequestsPerChild   0
  -</IfModule>
  +#<IfModule mpm_worker_module>
  +#    StartServers          1
  +#    MinSpareThreads       1
  +#    MaxSpareThreads       1 
  +#    ThreadLimit           1
  +#    ThreadsPerChild       1
  +#    MaxClients            5
  +#    MaxRequestsPerChild  50
  +#</IfModule>
   
   # event MPM
   # StartServers: initial number of server processes to start
  @@ -130 +130 @@
   # MaxSpareThreads: maximum number of worker threads which are kept spare
   # ThreadsPerChild: constant number of worker threads in each server process
   # MaxRequestsPerChild: maximum number of requests a server process serves
  -<IfModule mpm_event_module>
  -    StartServers          2
  -    MaxClients          150
  -    MinSpareThreads      25
  -    MaxSpareThreads      75 
  -    ThreadLimit          64
  -    ThreadsPerChild      25
  -    MaxRequestsPerChild   0
  -</IfModule>
  +#<IfModule mpm_event_module>
  +#    StartServers          1
  +#    MinSpareThreads       1
  +#    MaxSpareThreads       1 
  +#    ThreadLimit           1
  +#    ThreadsPerChild       1
  +#    MaxClients            5
  +#    MaxRequestsPerChild  50
  +#</IfModule>
   
   # These need to be set in /etc/apache2/envvars
   User ${APACHE_RUN_USER}

• <參考> ​Configuring Apache for Maximum Performance
• <參考> ​Low memory MySQL / Apache configurations
• <參考> ​Apache Performance Tuning

  prefork [default MPM for Apache 2.0 and 1.3]:
  
      * Apache 1.3-based.
      * Multiple processes, 1 thread per process, processes handle requests.
      * Used for security and stability.
      * Has higher memory consumption and lower performance over the newer Apache 2.0-based threaded MPMs.
  worker:
  
      * Apache 2.0-based.
      * Multiple processes, many threads per process, threads handle requests.
      * Used for lower memory consumption and higher performance.
      * Does not provide the same level of isolation request-to-request, as a process-based MPM does.
  

  MaxClients ≈ (RAM - size_all_other_processes)/(size_apache_process)