| Version 19 (modified by chris, 16 years ago) (diff) |
|---|
Lustre File System Performance Report (Uses bonnie++ command)
Machine Informance
| Node | 4 nodes (1 server , 3 client provide disks) |
| CPU | Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz (each node) |
| Memory | 2GB DDR2 667 (each node) |
| Disk | 320G+160G (each node), All nodes: (320G+ 160G) * 3 = 1.4T |
| NIC | Intel Corporation 82566DM Gigabit Network Connection |
| Switch | ZyXEL 8 port GE switch |
Offical Web
http://www.coker.com.au/bonnie++/
http://www.coker.com.au/bonnie++/readme.html
Lustre Performance
- without NFS tunning, kernel io scheduler = default
[root@lustre bonnie++-1.03c]# ./bonnie++ -d /mnt/client_fs01/ -s 16000:1048576 -r 1024 -u root -m first Using uid:0, gid:0. Writing with putc()...done Writing intelligently...done Rewriting...done Reading with getc()...done Reading intelligently...done start 'em...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.03c ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP first 16000M:1024k 44277 73 44487 6 20676 7 44695 90 49761 12 60.2 17 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 6706 35 2971 31 5324 36 6314 37 2995 33 5812 36 first,16000M:1024k,44277,73,44487,6,20676,7,44695,90,49761,12,60.2,17,16,6706,35,2971,31,5324,36,6314,37,2995,33,5812,36 - with NFS tunning, kernel io scheduler=default
[root@lustre bonnie++-1.03c]# ./bonnie++ -d /mnt/client_fs01/ -s 16000:1048576 -r 1024 -u root -m first Using uid:0, gid:0. Writing with putc()...done Writing intelligently...done Rewriting...done Reading with getc()...done Reading intelligently...done start 'em...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.03c ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP first 16000M:1024k 44359 72 51748 7 23290 8 46002 92 57361 15 64.5 18 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 6859 38 2978 33 5514 37 6494 36 3013 32 5577 35 first,16000M:1024k,44359,72,51748,7,23290,8,46002,92,57361,15,64.5,18,16,6859,38,2978,33,5514,37,6494,36,3013,32,5577,35 - with NFS tunning, kernel io scheduler = deadline
[root@lustre bonnie++-1.03c]# ./bonnie++ -d /mnt/client_fs01/ -s 16000:1048576 -r 1024 -u root Using uid:0, gid:0. Writing with putc()...done Writing intelligently...done Rewriting...done Reading with getc()...done Reading intelligently...done start 'em...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.03c ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP lustre 16000M:1024k 51892 90 54387 8 24917 9 47740 95 63936 17 69.9 22 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 6745 31 3013 32 6486 38 6850 33 3079 30 6327 38 lustre,16000M:1024k,51892,90,54387,8,24917,9,47740,95,63936,17,69.9,22,16,6745,31,3013,32,6486,38,6850,33,3079,30,6327,38 - without nfs tunning, kernel io scheduler = cfq
[root@lustre bonnie++-1.03c]# ./bonnie++ -d /mnt/client_fs01/ -s 16000:1048576 -r 1024 -u root Using uid:0, gid:0. Writing with putc()...done Writing intelligently...done Rewriting...done Reading with getc()...done Reading intelligently...done start 'em...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.03c ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP lustre 16000M:1024k 50986 80 46204 8 21749 8 46982 95 56664 15 65.2 21 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 8117 37 3062 31 6504 38 7041 35 2963 33 5579 34 lustre,16000M:1024k,50986,80,46204,8,21749,8,46982,95,56664,15,65.2,21,16,8117,37,3062,31,6504,38,7041,35,2963,33,5579,34 - with nfs tunning, kernel io scheduler = cfq
[root@lustre bonnie++-1.03c]# ./bonnie++ -d /mnt/client_fs01/ -s 16000:1048576 -r 1024 -u root Using uid:0, gid:0. Writing with putc()...done Writing intelligently...done Rewriting...done Reading with getc()...done Reading intelligently...done start 'em...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.03c ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP lustre 16000M:1024k 51603 85 48624 7 22740 8 46542 97 59096 16 67.7 21 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 6940 33 2944 34 5327 41 6141 35 2999 33 5549 39 lustre,16000M:1024k,51603,85,48624,7,22740,8,46542,97,59096,16,67.7,21,16,6940,33,2944,34,5327,41,6141,35,2999,33,5549,39 - with nfs tunning, kernel io scheduler = as
[root@lustre bonnie++-1.03c]# ./bonnie++ -d /mnt/client_fs01/ -s 16000:1048576 -r 1024 -u root Using uid:0, gid:0. Writing with putc()...done Writing intelligently...done Rewriting...done Reading with getc()...done Reading intelligently...done start 'em...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.03c ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP lustre 16000M:1024k 48623 78 51374 8 23598 9 47865 96 60014 16 68.6 21 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 8914 36 3153 31 7046 38 8020 39 3108 32 6087 39 lustre,16000M:1024k,48623,78,51374,8,23598,9,47865,96,60014,16,68.6,21,16,8914,36,3153,31,7046,38,8020,39,3108,32,6087,39 - with nfs tunning, kernel io scheduler = noop
[root@lustre bonnie++-1.03c]# ./bonnie++ -d /mnt/client_fs01/ -s 16000:1048576 -r 1024 -u root Using uid:0, gid:0. Writing with putc()...done Writing intelligently...done Rewriting...done Reading with getc()...done Reading intelligently...done start 'em...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.03c ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP lustre 16000M:1024k 46297 74 54749 9 24791 9 47547 96 61390 16 68.4 21 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 7982 34 3036 33 5398 36 7063 35 3056 33 5341 39 lustre,16000M:1024k,46297,74,54749,9,24791,9,47547,96,61390,16,68.4,21,16,7982,34,3036,33,5398,36,7063,35,3056,33,5341,39 - with nfs tunning, kernel io scheduler = noop, isolated client.
[root@lustre005 bonnie++-1.03c]# ./bonnie++ -d /mnt/client_fs/ -s 16000:1048576 -r 1024 -u root Using uid:0, gid:0. Writing with putc()...done Writing intelligently...done Rewriting...done Reading with getc()...done Reading intelligently...done start 'em...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.03c ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP lustre 16000M:1024k 35309 98 51334 8 24285 8 35666 98 58716 14 68.2 19 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 2090 9 1593 14 1877 9 2024 9 1638 11 1891 9 lustre005,16000M:1024k,35309,98,51334,8,24285,8,35666,98,58716,14,68.2,19,16,2090,9,1593,14,1877,9,2024,9,1638,11,1891,9 - with nfs tunning, kernel io scheduler = default, just sdb mounted , total 881 GB
[root@lustre bonnie++-1.03c]# ./bonnie++ -d /mnt/client_fs01/ -s 16000:1048576 -r 1024 -u root -m first Using uid:0, gid:0. Writing with putc()...done Writing intelligently...done Rewriting...done Reading with getc()...done Reading intelligently...done start 'em...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.03c ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP first 16000M:1024k 62065 98 64754 10 27672 10 47468 96 64299 17 67.2 20 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 8385 38 3080 31 5924 37 6350 37 3056 33 5901 35 first,16000M:1024k,62065,98,64754,10,27672,10,47468,96,64299,17,67.2,20,16,8385,38,3080,31,5924,37,6350,37,3056,33,5901,35
Attachments (3)
- Lustre with diff kernel_scheduler.PNG (19.3 KB) - added by chris 16 years ago.
- Lustre create with diff kernel_scheduler.PNG (20.4 KB) - added by chris 16 years ago.
- fake multi-clients.PNG (15.6 KB) - added by chris 16 years ago.
Download all attachments as: .zip
