Lustre File System Performance Report (Uses bonnie++ command)

Machine Informance

Node	4 nodes (1 server , 3 client provide disks)
CPU	Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz (each node)
Memory	2GB DDR2 667 (each node)
Disk	320G+160G (each node), All nodes: (320G+ 160G) * 3 = 1.4T
NIC	Intel Corporation 82566DM Gigabit Network Connection
Switch	ZyXEL 8 port GE switch

Offical Web

​http://www.coker.com.au/bonnie++/
​http://www.coker.com.au/bonnie++/readme.html

Lustre Performance

• without tunning NFS, kernel io scheduler = default
  

  [root@lustre bonnie++-1.03c]# ./bonnie++ -d /mnt/client_fs01/ -s 16000:1048576  -r 1024 -u root -m first
  Using uid:0, gid:0.
  Writing with putc()...done
  Writing intelligently...done
  Rewriting...done
  Reading with getc()...done
  Reading intelligently...done
  start 'em...done...done...done...
  Create files in sequential order...done.
  Stat files in sequential order...done.
  Delete files in sequential order...done.
  Create files in random order...done.
  Stat files in random order...done.
  Delete files in random order...done.
  Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                      -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
  Machine   Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
  first  16000M:1024k 44277  73 44487   6 20676   7 44695  90 49761  12  60.2  17
                      ------Sequential Create------ --------Random Create--------
                      -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
                files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                   16  6706  35  2971  31  5324  36  6314  37  2995  33  5812  36
  first,16000M:1024k,44277,73,44487,6,20676,7,44695,90,49761,12,60.2,17,16,6706,35,2971,31,5324,36,6314,37,2995,33,5812,36
  
  

• with NFS tunning, kernel io scheduler=default
  

  [root@lustre bonnie++-1.03c]# ./bonnie++ -d /mnt/client_fs01/ -s 16000:1048576  -r 1024 -u root -m first
  Using uid:0, gid:0.
  Writing with putc()...done
  Writing intelligently...done
  Rewriting...done
  Reading with getc()...done
  Reading intelligently...done
  start 'em...done...done...done...
  Create files in sequential order...done.
  Stat files in sequential order...done.
  Delete files in sequential order...done.
  Create files in random order...done.
  Stat files in random order...done.
  Delete files in random order...done.
  Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                      -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
  Machine   Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
  first  16000M:1024k 44359  72 51748   7 23290   8 46002  92 57361  15  64.5  18
                      ------Sequential Create------ --------Random Create--------
                      -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
                files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                   16  6859  38  2978  33  5514  37  6494  36  3013  32  5577  35
  first,16000M:1024k,44359,72,51748,7,23290,8,46002,92,57361,15,64.5,18,16,6859,38,2978,33,5514,37,6494,36,3013,32,5577,35
  
  

• with NFS tunning, kernel io scheduler = deadline
  

  [root@lustre bonnie++-1.03c]# ./bonnie++ -d /mnt/client_fs01/ -s 16000:1048576  -r 1024 -u root
  Using uid:0, gid:0.
  Writing with putc()...done
  Writing intelligently...done
  Rewriting...done
  Reading with getc()...done
  Reading intelligently...done
  start 'em...done...done...done...
  Create files in sequential order...done.
  Stat files in sequential order...done.
  Delete files in sequential order...done.
  Create files in random order...done.
  Stat files in random order...done.
  Delete files in random order...done.
  Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                      -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
  Machine   Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
  lustre 16000M:1024k 51892  90 54387   8 24917   9 47740  95 63936  17  69.9  22
                      ------Sequential Create------ --------Random Create--------
                      -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
                files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                   16  6745  31  3013  32  6486  38  6850  33  3079  30  6327  38
  lustre,16000M:1024k,51892,90,54387,8,24917,9,47740,95,63936,17,69.9,22,16,6745,31,3013,32,6486,38,6850,33,3079,30,6327,38