= GPFS Performance Report (Uses gpfsperf Command) = rock, rock@nchc.org.tw [[BR]] [[BR]] == Machine Informance == Node 8 nodes (1 server , 7 client provide disks) CPU Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz (each node) Memory 2GB DDR2 667 (each node) Disk 320G+160G (each node) All nodes: WD 320G * 7 + 160G * 7 = 3.36T NIC Intel Corporation 82566DM Gigabit Network Connection Switch D-link 24 port GE switch 1. 8 Nodes, Replicate, Adjust Parameters Context: Create 16G data (sequence) gpfs-server:/usr/lpp/mmfs/samples/perf# ./gpfsperf create seq /home/gpfs_mount/gpfsperf_16G -n 16g -r 1m ./gpfsperf create seq /home/gpfs_mount/gpfsperf_16G recSize 1M nBytes 16G fileSize 16G nProcesses 1 nThreadsPerProcess 1 file cache flushed before test not using data shipping not using direct I/O offsets accessed will cycle through the same file segment not using shared memory buffer not releasing byte-range token after open no fsync at end of test Data rate was 54649.55 Kbytes/sec, thread utilization 1.000 Context: Read 16G data (sequence) gpfs-server:/usr/lpp/mmfs/samples/perf# ./gpfsperf read seq /home/gpfs_mount/gpfsperf_16G -n 16g -r 1m ./gpfsperf read seq /home/gpfs_mount/gpfsperf_16G recSize 1M nBytes 16G fileSize 16G nProcesses 1 nThreadsPerProcess 1 file cache flushed before test not using data shipping not using direct I/O offsets accessed will cycle through the same file segment not using shared memory buffer not releasing byte-range token after open Data rate was 83583.30 Kbytes/sec, thread utilization 1.000 Context: Write 16G data (sequence) gpfs-server:/usr/lpp/mmfs/samples/perf# ./gpfsperf write seq /home/gpfs_mount/gpfsperf_16G -n 16g -r 1m ./gpfsperf write seq /home/gpfs_mount/gpfsperf_16G recSize 1M nBytes 16G fileSize 16G nProcesses 1 nThreadsPerProcess 1 file cache flushed before test not using data shipping not using direct I/O offsets accessed will cycle through the same file segment not using shared memory buffer not releasing byte-range token after open no fsync at end of test Data rate was 50898.76 Kbytes/sec, thread utilization 1.000 2. 8 Nodes, No Replicate, Adjust Parameters Context: Create 16G data (sequence) gpfs-server:/usr/lpp/mmfs/samples/perf# ./gpfsperf create seq /home/gpfs_mount/gpfsperf_16G_2 -n 16g -r 1m ./gpfsperf create seq /home/gpfs_mount/gpfsperf_16G_2 recSize 1M nBytes 16G fileSize 16G nProcesses 1 nThreadsPerProcess 1 file cache flushed before test not using data shipping not using direct I/O offsets accessed will cycle through the same file segment not using shared memory buffer not releasing byte-range token after open no fsync at end of test Data rate was 108330.24 Kbytes/sec, thread utilization 1.000 Context: Read 16G data (sequence) gpfs-server:/usr/lpp/mmfs/samples/perf# ./gpfsperf read seq /home/gpfs_mount/gpfsperf_16G_2 -n 16g -r 1m ./gpfsperf read seq /home/gpfs_mount/gpfsperf_16G_2 recSize 1M nBytes 16G fileSize 16G nProcesses 1 nThreadsPerProcess 1 file cache flushed before test not using data shipping not using direct I/O offsets accessed will cycle through the same file segment not using shared memory buffer not releasing byte-range token after open Data rate was 82420.96 Kbytes/sec, thread utilization 1.000 Context: Write 16G data (sequence) gpfs-server:/usr/lpp/mmfs/samples/perf# ./gpfsperf write seq /home/gpfs_mount/gpfsperf_16G_2 -n 16g -r 1m ./gpfsperf write seq /home/gpfs_mount/gpfsperf_16G_2 recSize 1M nBytes 16G fileSize 16G nProcesses 1 nThreadsPerProcess 1 file cache flushed before test not using data shipping not using direct I/O offsets accessed will cycle through the same file segment not using shared memory buffer not releasing byte-range token after open no fsync at end of test Data rate was 108820.45 Kbytes/sec, thread utilization 1.000 3. Multi-thread 3.1 Create Operation Context: Create 16G data, 1 thread gpfs-server:/usr/lpp/mmfs/samples/perf# ./gpfsperf create seq /home/gpfs_mount/gpfsperf_16G_3 -n 16g -r 1m -th 1 ./gpfsperf create seq /home/gpfs_mount/gpfsperf_16G_3 recSize 1M nBytes 16G fileSize 16G nProcesses 1 nThreadsPerProcess 1 file cache flushed before test not using data shipping not using direct I/O offsets accessed will cycle through the same file segment not using shared memory buffer not releasing byte-range token after open no fsync at end of test Data rate was 50800.95 Kbytes/sec, thread utilization 1.000 Context: Create 16G data, 2 thread gpfs-server:/usr/lpp/mmfs/samples/perf# ./gpfsperf create seq /home/gpfs_mount/gpfsperf_16G_3 -n 16g -r 1m -th 2 ./gpfsperf create seq /home/gpfs_mount/gpfsperf_16G_3 recSize 1M nBytes 16G fileSize 16G nProcesses 1 nThreadsPerProcess 2 file cache flushed before test not using data shipping not using direct I/O offsets accessed will cycle through the same file segment not using shared memory buffer not releasing byte-range token after open no fsync at end of test Data rate was 50297.13 Kbytes/sec, thread utilization 0.999 Context: Create 16G data, 4 thread gpfs-server:/usr/lpp/mmfs/samples/perf# ./gpfsperf create seq /home/gpfs_mount/gpfsperf_16G_3 -n 16g -r 1m -th 4 ./gpfsperf create seq /home/gpfs_mount/gpfsperf_16G_3 recSize 1M nBytes 16G fileSize 16G nProcesses 1 nThreadsPerProcess 4 file cache flushed before test not using data shipping not using direct I/O offsets accessed will cycle through the same file segment not using shared memory buffer not releasing byte-range token after open no fsync at end of test Data rate was 50848.45 Kbytes/sec, thread utilization 0.998 Context: Create 16G data, 8 thread gpfs-server:/usr/lpp/mmfs/samples/perf# ./gpfsperf create seq /home/gpfs_mount/gpfsperf_16G_3 -n 16g -r 1m -th 8 ./gpfsperf create seq /home/gpfs_mount/gpfsperf_16G_3 recSize 1M nBytes 16G fileSize 16G nProcesses 1 nThreadsPerProcess 8 file cache flushed before test not using data shipping not using direct I/O offsets accessed will cycle through the same file segment not using shared memory buffer not releasing byte-range token after open no fsync at end of test Data rate was 50469.88 Kbytes/sec, thread utilization 0.963 Context: Create 16G data, 16 thread gpfs-server:/usr/lpp/mmfs/samples/perf# ./gpfsperf create seq /home/gpfs_mount/gpfsperf_16G_3 -n 16g -r 1m -th 16 ./gpfsperf create seq /home/gpfs_mount/gpfsperf_16G_3 recSize 1M nBytes 16G fileSize 16G nProcesses 1 nThreadsPerProcess 16 file cache flushed before test not using data shipping not using direct I/O offsets accessed will cycle through the same file segment not using shared memory buffer not releasing byte-range token after open no fsync at end of test Data rate was 52578.33 Kbytes/sec, thread utilization 0.919 Context: Create 16G data, 32 thread gpfs-server:/usr/lpp/mmfs/samples/perf# ./gpfsperf create seq /home/gpfs_mount/gpfsperf_16G_3 -n 16g -r 1m -th 32 ./gpfsperf create seq /home/gpfs_mount/gpfsperf_16G_3 recSize 1M nBytes 16G fileSize 16G nProcesses 1 nThreadsPerProcess 32 file cache flushed before test not using data shipping not using direct I/O offsets accessed will cycle through the same file segment not using shared memory buffer not releasing byte-range token after open no fsync at end of test Data rate was 53107.28 Kbytes/sec, thread utilization 0.966 Context: Create 16G data, 64 thread gpfs-server:/usr/lpp/mmfs/samples/perf# ./gpfsperf create seq /home/gpfs_mount/gpfsperf_16G_3 -n 16g -r 1m -th 64 ./gpfsperf create seq /home/gpfs_mount/gpfsperf_16G_3 recSize 1M nBytes 16G fileSize 16G nProcesses 1 nThreadsPerProcess 64 file cache flushed before test not using data shipping not using direct I/O offsets accessed will cycle through the same file segment not using shared memory buffer not releasing byte-range token after open no fsync at end of test Data rate was 53019.53 Kbytes/sec, thread utilization 0.978 3.2 Read Operation Context: Read 16G data, 1 thread gpfs-server:/usr/lpp/mmfs/samples/perf# ./gpfsperf read seq /home/gpfs_mount/gpfsperf_16G_3 -r 1m -n 16g -th 1 ./gpfsperf read seq /home/gpfs_mount/gpfsperf_16G_3 recSize 1M nBytes 16G fileSize 16G nProcesses 1 nThreadsPerProcess 1 file cache flushed before test not using data shipping not using direct I/O offsets accessed will cycle through the same file segment not using shared memory buffer not releasing byte-range token after open Data rate was 81685.18 Kbytes/sec, thread utilization 1.000 Context: Read 16G data, 2 thread gpfs-server:/usr/lpp/mmfs/samples/perf# ./gpfsperf read seq /home/gpfs_mount/gpfsperf_16G_3 -r 1m -n 16g -th 2 ./gpfsperf read seq /home/gpfs_mount/gpfsperf_16G_3 recSize 1M nBytes 16G fileSize 16G nProcesses 1 nThreadsPerProcess 2 file cache flushed before test not using data shipping not using direct I/O offsets accessed will cycle through the same file segment not using shared memory buffer not releasing byte-range token after open Data rate was 90844.61 Kbytes/sec, thread utilization 0.999 Context: Read 16G data, 4 thread gpfs-server:/usr/lpp/mmfs/samples/perf# ./gpfsperf read seq /home/gpfs_mount/gpfsperf_16G_3 -r 1m -n 16g -th 4 ./gpfsperf read seq /home/gpfs_mount/gpfsperf_16G_3 recSize 1M nBytes 16G fileSize 16G nProcesses 1 nThreadsPerProcess 4 file cache flushed before test not using data shipping not using direct I/O offsets accessed will cycle through the same file segment not using shared memory buffer not releasing byte-range token after open Data rate was 89538.89 Kbytes/sec, thread utilization 0.997 Context: Read 16G data, 8 thread gpfs-server:/usr/lpp/mmfs/samples/perf# ./gpfsperf read seq /home/gpfs_mount/gpfsperf_16G_3 -r 1m -n 16g -th 8 ./gpfsperf read seq /home/gpfs_mount/gpfsperf_16G_3 recSize 1M nBytes 16G fileSize 16G nProcesses 1 nThreadsPerProcess 8 file cache flushed before test not using data shipping not using direct I/O offsets accessed will cycle through the same file segment not using shared memory buffer not releasing byte-range token after open Data rate was 87044.97 Kbytes/sec, thread utilization 0.994 Context: Read 16G data, 16 thread gpfs-server:/usr/lpp/mmfs/samples/perf# ./gpfsperf read seq /home/gpfs_mount/gpfsperf_16G_3 -r 1m -n 16g -th 16 ./gpfsperf read seq /home/gpfs_mount/gpfsperf_16G_3 recSize 1M nBytes 16G fileSize 16G nProcesses 1 nThreadsPerProcess 16 file cache flushed before test not using data shipping not using direct I/O offsets accessed will cycle through the same file segment not using shared memory buffer not releasing byte-range token after open Data rate was 94899.75 Kbytes/sec, thread utilization 0.990 Context: Read 16G data, 32 thread gpfs-server:/usr/lpp/mmfs/samples/perf# ./gpfsperf read seq /home/gpfs_mount/gpfsperf_16G_3 -r 1m -n 16g -th 32 ./gpfsperf read seq /home/gpfs_mount/gpfsperf_16G_3 recSize 1M nBytes 16G fileSize 16G nProcesses 1 nThreadsPerProcess 32 file cache flushed before test not using data shipping not using direct I/O offsets accessed will cycle through the same file segment not using shared memory buffer not releasing byte-range token after open Data rate was 90657.18 Kbytes/sec, thread utilization 0.983 Context: Read 16G data, 64 thread gpfs-server:/usr/lpp/mmfs/samples/perf# ./gpfsperf read seq /home/gpfs_mount/gpfsperf_16G_3 -r 1m -n 16g -th 64 ./gpfsperf read seq /home/gpfs_mount/gpfsperf_16G_3 recSize 1M nBytes 16G fileSize 16G nProcesses 1 nThreadsPerProcess 64 file cache flushed before test not using data shipping not using direct I/O offsets accessed will cycle through the same file segment not using shared memory buffer not releasing byte-range token after open Data rate was 89751.67 Kbytes/sec, thread utilization 0.983 3.3 Write Operation Context: Write 16G data, 1 thread gpfs-server:/usr/lpp/mmfs/samples/perf# ./gpfsperf write seq /home/gpfs_mount/gpfsperf_16G_3 -r 1m -n 16g -th 1 ./gpfsperf write seq /home/gpfs_mount/gpfsperf_16G_3 recSize 1M nBytes 16G fileSize 16G nProcesses 1 nThreadsPerProcess 1 file cache flushed before test not using data shipping not using direct I/O offsets accessed will cycle through the same file segment not using shared memory buffer not releasing byte-range token after open no fsync at end of test Data rate was 50819.17 Kbytes/sec, thread utilization 1.000 Context: Write 16G data, 2 thread gpfs-server:/usr/lpp/mmfs/samples/perf# ./gpfsperf write seq /home/gpfs_mount/gpfsperf_16G_3 -r 1m -n 16g -th 2 ./gpfsperf write seq /home/gpfs_mount/gpfsperf_16G_3 recSize 1M nBytes 16G fileSize 16G nProcesses 1 nThreadsPerProcess 2 file cache flushed before test not using data shipping not using direct I/O offsets accessed will cycle through the same file segment not using shared memory buffer not releasing byte-range token after open no fsync at end of test Data rate was 50588.81 Kbytes/sec, thread utilization 1.000 Context: Write 16G data, 4 thread gpfs-server:/usr/lpp/mmfs/samples/perf# ./gpfsperf write seq /home/gpfs_mount/gpfsperf_16G_3 -r 1m -n 16g -th 4 ./gpfsperf write seq /home/gpfs_mount/gpfsperf_16G_3 recSize 1M nBytes 16G fileSize 16G nProcesses 1 nThreadsPerProcess 4 file cache flushed before test not using data shipping not using direct I/O offsets accessed will cycle through the same file segment not using shared memory buffer not releasing byte-range token after open no fsync at end of test Data rate was 50694.87 Kbytes/sec, thread utilization 0.999 Context: Write 16G data, 8 thread gpfs-server:/usr/lpp/mmfs/samples/perf# ./gpfsperf write seq /home/gpfs_mount/gpfsperf_16G_3 -r 1m -n 16g -th 8 ./gpfsperf write seq /home/gpfs_mount/gpfsperf_16G_3 recSize 1M nBytes 16G fileSize 16G nProcesses 1 nThreadsPerProcess 8 file cache flushed before test not using data shipping not using direct I/O offsets accessed will cycle through the same file segment not using shared memory buffer not releasing byte-range token after open no fsync at end of test Data rate was 51648.90 Kbytes/sec, thread utilization 0.985 Context: Write 16G data, 16 thread gpfs-server:/usr/lpp/mmfs/samples/perf# ./gpfsperf write seq /home/gpfs_mount/gpfsperf_16G_3 -r 1m -n 16g -th 16 ./gpfsperf write seq /home/gpfs_mount/gpfsperf_16G_3 recSize 1M nBytes 16G fileSize 16G nProcesses 1 nThreadsPerProcess 16 file cache flushed before test not using data shipping not using direct I/O offsets accessed will cycle through the same file segment not using shared memory buffer not releasing byte-range token after open no fsync at end of test Data rate was 53019.51 Kbytes/sec, thread utilization 0.924 Context: Write 16G data, 32 thread gpfs-server:/usr/lpp/mmfs/samples/perf# ./gpfsperf write seq /home/gpfs_mount/gpfsperf_16G_3 -r 1m -n 16g -th 32 ./gpfsperf write seq /home/gpfs_mount/gpfsperf_16G_3 recSize 1M nBytes 16G fileSize 16G nProcesses 1 nThreadsPerProcess 32 file cache flushed before test not using data shipping not using direct I/O offsets accessed will cycle through the same file segment not using shared memory buffer not releasing byte-range token after open no fsync at end of test Data rate was 53003.69 Kbytes/sec, thread utilization 0.966 Context: Write 16G data, 64 thread gpfs-server:/usr/lpp/mmfs/samples/perf# ./gpfsperf write seq /home/gpfs_mount/gpfsperf_16G_3 -r 1m -n 16g -th 64 ./gpfsperf write seq /home/gpfs_mount/gpfsperf_16G_3 recSize 1M nBytes 16G fileSize 16G nProcesses 1 nThreadsPerProcess 64 file cache flushed before test not using data shipping not using direct I/O offsets accessed will cycle through the same file segment not using shared memory buffer not releasing byte-range token after open no fsync at end of test Data rate was 53590.98 Kbytes/sec, thread utilization 0.971 4. Compare All kind Operation (sequence) Replicate & Adjust Parameters No Replicate & Adjust Parameters Create 54649.55 KB/s 108330.24 KB/s Read 83583.30 KB/s 82420.96 KB/s Write 50898.76 KB/s 108820.45 KB/s Multi-thread (sequence) 1 2 4 8 16 32 64 Create 50800.95 KB/s 50297.13 KB/s 50848.45 KB/s 50469.88 KB/s 52578.33 KB/s 53107.28 KB/s 53019.53 KB/s Read 81685.18 KB/s 90844.61 KB/s 89538.89 KB/s 87044.97 KB/s 94899.75 KB/s 90657.18 KB/s 89751.67 KB/s Write 50819.17 KB/s 50588.81 KB/s 50694.87 KB/s 51648.90 KB/s 53019.51 KB/s 53003.69 KB/s 53590.98 KB/s