Lustre File System Performance Report (Uses bonnie++ command)
Machine Informance
| Node | 4 nodes (1 server , 3 client provide disks) |
| CPU | Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz (each node) |
| Memory | 2GB DDR2 667 (each node) |
| Disk | 320G+160G (each node), All nodes: (320G+ 160G) * 3 = 1.4T |
| NIC | Intel Corporation 82566DM Gigabit Network Connection |
| Switch | ZyXEL 8 port GE switch |
Offical Web
http://www.coker.com.au/bonnie++/
http://www.coker.com.au/bonnie++/readme.html
Lustre Performance
- 設定不同的 kernel i/o scheduler，根據 lustre 官方文件 3-8 頁所描述，最適合 Lustre 的 kernel i/o scheduler 為 deadline。
- 在 Create/Delete檔案方面，則是以 as 有較佳的表現。
- 當多個 Client 同時存取時，lustre file system 的效能有顯著的提升。(由於機器不足的關係，Client的測試乃是在單一機器上開起多個測試行程，而非真正獨立之 clients ，故此結果僅供參考。)
- 以下是一些測試的數據：
- without NFS tunning, kernel io scheduler = default
[root@lustre bonnie++-1.03c]# ./bonnie++ -d /mnt/client_fs01/ -s 16000:1048576 -r 1024 -u root -m first Using uid:0, gid:0. Writing with putc()...done Writing intelligently...done Rewriting...done Reading with getc()...done Reading intelligently...done start 'em...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.03c ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP first 16000M:1024k 44277 73 44487 6 20676 7 44695 90 49761 12 60.2 17 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 6706 35 2971 31 5324 36 6314 37 2995 33 5812 36 first,16000M:1024k,44277,73,44487,6,20676,7,44695,90,49761,12,60.2,17,16,6706,35,2971,31,5324,36,6314,37,2995,33,5812,36 - with NFS tunning, kernel io scheduler=default
[root@lustre bonnie++-1.03c]# ./bonnie++ -d /mnt/client_fs01/ -s 16000:1048576 -r 1024 -u root -m first Using uid:0, gid:0. Writing with putc()...done Writing intelligently...done Rewriting...done Reading with getc()...done Reading intelligently...done start 'em...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.03c ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP first 16000M:1024k 41030 75 59535 9 26237 10 47504 96 64738 17 68.0 21 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 6802 36 3005 32 5248 36 6308 38 3029 35 5351 35 first,16000M:1024k,41030,75,59535,9,26237,10,47504,96,64738,17,68.0,21,16,6802,36,3005,32,5248,36,6308,38,3029,35,5351,35 - with NFS tunning, kernel io scheduler = deadline
[root@lustre bonnie++-1.03c]# ./bonnie++ -d /mnt/client_fs01/ -s 16000:1048576 -r 1024 -u root -m first Using uid:0, gid:0. Writing with putc()...done Writing intelligently...done Rewriting...done Reading with getc()...done Reading intelligently...done start 'em...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.03c ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP first 16000M:1024k 48900 79 59534 10 26680 10 47398 96 66517 18 70.0 22 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 6954 38 3050 33 6233 35 7125 34 2981 34 5416 36 first,16000M:1024k,48900,79,59534,10,26680,10,47398,96,66517,18,70.0,22,16,6954,38,3050,33,6233,35,7125,34,2981,34,5416,36 - without nfs tunning, kernel io scheduler = cfq
[root@lustre bonnie++-1.03c]# ./bonnie++ -d /mnt/client_fs01/ -s 16000:1048576 -r 1024 -u root Using uid:0, gid:0. Writing with putc()...done Writing intelligently...done Rewriting...done Reading with getc()...done Reading intelligently...done start 'em...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.03c ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP lustre 16000M:1024k 50986 80 46204 8 21749 8 46982 95 56664 15 65.2 21 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 8117 37 3062 31 6504 38 7041 35 2963 33 5579 34 lustre,16000M:1024k,50986,80,46204,8,21749,8,46982,95,56664,15,65.2,21,16,8117,37,3062,31,6504,38,7041,35,2963,33,5579,34 - with nfs tunning, kernel io scheduler = cfq
[root@lustre bonnie++-1.03c]# ./bonnie++ -d /mnt/client_fs01/ -s 16000:1048576 -r 1024 -u root Using uid:0, gid:0. Writing with putc()...done Writing intelligently...done Rewriting...done Reading with getc()...done Reading intelligently...done start 'em...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.03c ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP lustre 16000M:1024k 51603 85 48624 7 22740 8 46542 97 59096 16 67.7 21 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 6940 33 2944 34 5327 41 6141 35 2999 33 5549 39 lustre,16000M:1024k,51603,85,48624,7,22740,8,46542,97,59096,16,67.7,21,16,6940,33,2944,34,5327,41,6141,35,2999,33,5549,39 - with nfs tunning, kernel io scheduler = as
[root@lustre bonnie++-1.03c]# ./bonnie++ -d /mnt/client_fs01/ -s 16000:1048576 -r 1024 -u root Using uid:0, gid:0. Writing with putc()...done Writing intelligently...done Rewriting...done Reading with getc()...done Reading intelligently...done start 'em...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.03c ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP lustre 16000M:1024k 48623 78 51374 8 23598 9 47865 96 60014 16 68.6 21 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 8914 36 3153 31 7046 38 8020 39 3108 32 6087 39 lustre,16000M:1024k,48623,78,51374,8,23598,9,47865,96,60014,16,68.6,21,16,8914,36,3153,31,7046,38,8020,39,3108,32,6087,39 - with nfs tunning, kernel io scheduler = noop
[root@lustre bonnie++-1.03c]# ./bonnie++ -d /mnt/client_fs01/ -s 16000:1048576 -r 1024 -u root Using uid:0, gid:0. Writing with putc()...done Writing intelligently...done Rewriting...done Reading with getc()...done Reading intelligently...done start 'em...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.03c ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP lustre 16000M:1024k 46297 74 54749 9 24791 9 47547 96 61390 16 68.4 21 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 7982 34 3036 33 5398 36 7063 35 3056 33 5341 39 lustre,16000M:1024k,46297,74,54749,9,24791,9,47547,96,61390,16,68.4,21,16,7982,34,3036,33,5398,36,7063,35,3056,33,5341,39 - with nfs tunning, kernel io scheduler = noop, isolated client.
[root@lustre005 bonnie++-1.03c]# ./bonnie++ -d /mnt/client_fs/ -s 16000:1048576 -r 1024 -u root Using uid:0, gid:0. Writing with putc()...done Writing intelligently...done Rewriting...done Reading with getc()...done Reading intelligently...done start 'em...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.03c ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP lustre 16000M:1024k 35309 98 51334 8 24285 8 35666 98 58716 14 68.2 19 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 2090 9 1593 14 1877 9 2024 9 1638 11 1891 9 lustre005,16000M:1024k,35309,98,51334,8,24285,8,35666,98,58716,14,68.2,19,16,2090,9,1593,14,1877,9,2024,9,1638,11,1891,9 - with nfs tunning, kernel io scheduler = default, just sdb mounted , total 881 GB
[root@lustre bonnie++-1.03c]# ./bonnie++ -d /mnt/client_fs01/ -s 16000:1048576 -r 1024 -u root -m first Using uid:0, gid:0. Writing with putc()...done Writing intelligently...done Rewriting...done Reading with getc()...done Reading intelligently...done start 'em...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.03c ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP first 16000M:1024k 62065 98 64754 10 27672 10 47468 96 64299 17 67.2 20 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 8385 38 3080 31 5924 37 6350 37 3056 33 5901 35 first,16000M:1024k,62065,98,64754,10,27672,10,47468,96,64299,17,67.2,20,16,8385,38,3080,31,5924,37,6350,37,3056,33,5901,35 - without DRBL, kernel io scheduler = default, just sdb mounted , total 881 GB
[root@lustre bonnie++-1.03c]# ./bonnie++ -d /mnt/client_fs01/ -s 16000:1048576 -r 1024 -u root -m first Using uid:0, gid:0. Writing with putc()...done Writing intelligently...done Rewriting...done Reading with getc()...done Reading intelligently...done start 'em...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.03c ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP first 16000M:1024k 50937 85 53757 8 24032 9 44920 88 53996 14 60.1 18 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 6718 36 2996 32 5342 34 6891 36 2925 29 5413 38 first,16000M:1024k,50937,85,53757,8,24032,9,44920,88,53996,14,60.1,18,16,6718,36,2996,32,5342,34,6891,36,2925,29,5413,38
- without NFS tunning, kernel io scheduler = default
Last modified 17 years ago
Last modified on Jun 16, 2008, 2:57:29 PM
Attachments (3)
- Lustre with diff kernel_scheduler.PNG (19.3 KB) - added by chris 17 years ago.
- Lustre create with diff kernel_scheduler.PNG (20.4 KB) - added by chris 17 years ago.
- fake multi-clients.PNG (15.6 KB) - added by chris 17 years ago.
Download all attachments as: .zip
