NSS Performance with VMFS and ISCSI
ate I have done some testing with NSS and Vmware ESX. And I must say, I’m quite disappointed about the performance! Exspecially with NSS on VMFS with ISCSI. I had the following setup:
One virtual machine with Novell Open Enterprise server 2 on Vmware ESX 3.5 U2. This virtual machine has his VMDK on a local VMFS volume and a second disk witch is a RDM on a ISCSI SAN (Netapp).
I tested the disk performace with a program called Bonnie++ with the following parameters:
bonnie++ -s 2000M -d [path to volume] -v 3
Why a file size of 2000M on three volumes, simple, it’s more than twice the memory so Linux can’t cash the file in his memory.
And here are the results:
| Per Char | Per Block | Rewrite | Per Char | Per Block | 04K | -3 | |||||||
| MB | K/Sec | %CPU | K/sec | %CPU | K/sec | %CPU | K/sec | %CPU | K/sec | %CPU | /sec | %CPU | |
| SLES10 SP1 NSS with VMFS on local storage | 6000M | 23818 | 80,8 | 34224 | 33,1 | 13242 | 1,3 | 8849 | 11,8 | 20194 | 0,5 | 323,4 | 3 |
| SLES10 SP1 NSS with VMFS on ISCSI storage | 6000M | 8952 | 64,6 | 11981 | 36,6 | 10084 | 5,3 | 6540 | 22,4 | 16775 | 4,3 | 240,8 | 2,2 |
| SLES10 SP1 NSS, RDM on ISCSI storage | 6000M | 20384 | 77 | 27691 | 39 | 10468 | 0 | 8780 | 36 | 7327 | 6 | 289,9 | 2 |
So as you see, the combination of NSS, VMFS and ISCSI is killing. I don’t know exactly wath is causing this, but I think it has something to do with block alligment.
Also I did some testing with VMI. I had to install SLES10 SP2 for this, so I can test NSS at this moment. I’ve download the beta of OES2 sp1 but I haven’t had the time to test this.
| Sequential Output (nosync) Sequential | Sequential Input | ||||||||||||
| Per Char | Per Block | Rewrite | Per Char | Per Block | 04K | -3 | |||||||
| MB | K/Sec | %CPU | K/sec | %CPU | K/sec | %CPU | K/sec | %CPU | K/sec | %CPU | /sec | %CPU | |
| SLES10 SP2 Binary ReiserFS (Test 1) | 6000M | 33210 | 98.0 | 47718 | 8,6 | 19611 | 4,9 | 26668 | 62.1 | 61563 | 10,9 | 1784.8 | 1,7 |
| SLES10 SP2 Binary ReiserFS (Test 2) | 6000M | 29833 | 95.2 | 51545 | 9,5 | 21652 | 5,6 | 26205 | 61.2 | 61651 | 10,2 | 1755.2 | 9,3 |
| SLES10 SP2 Binary ReiserFS (Test 3) | 6000M | 32423 | 96.3 | 53426 | 9,7 | 22062 | 1,6 | 25748 | 60.3 | 61361 | 10,2 | 1867.4 | 9,3 |
| SLES10 SP2 VMI ReiserFS (Test 1) | 6000M | 34713 | 97.9 | 55265 | 8,8 | 21152 | 1,1 | 24450 | 47.0 | 52107 | 0,6 | 1744.5 | 0,3 |
| SLES10 SP2 VMI ReiserFS (Test 1) | 6000M | 33526 | 95.9 | 52340 | 8,9 | 21034 | 1,1 | 24695 | 47.1 | 57298 | 1,1 | 1839.1 | 1,1 |
| SLES10 SP2 VMI ReiserFS (Test 1) | 6000M | 32762 | 93.5 | 49264 | 8,2 | 21671 | 1,0 | 25136 | 48.9 | 56911 | 0,9 | 1720.8 | 1,1 |
The disk performance doesn’t increase but the CPU load is lower. So this is interesting. When I have more time I will test the VMI setup with NSS.
I will do some more testing this and the next month, so check this post frequently!
Michael Wilmsen is a experienced VMware Architect with more than 20 years in the IT industry. Main focus is VMware vSphere, Horizon View and Hyper Converged with a deep interest into performance and architecture.
Michael is VCDX 210 certified, has been rewarded with the vExpert title from 2011, Nutanix Tech Champion and a Nutanix Platform Professional.
22 Comments to “NSS Performance with VMFS and ISCSI”
» NSS and VMFS performance OES2 SP1 — December 19, 2008 @ 15:19
» NSS and VMFS performance OES2 SP1 — December 19, 2008 @ 15:19
By Eric Champagne, December 6, 2008 @ 02:54
Hi !
We are at the same place in our test !!! 🙁 🙁 … I’m happy to see some comparaison chart. It will be great to work together to figure out this issue. I’m a Novell Gold Partner in Canada and we work with a huge customers who have around 45k users in their network.
They will go with VMware but OES 2 with NSS seems to have very bad performance even with a SAN FiberChannel 4Gb. When you will do some others tests with NSS on SLES 10 SP2 please let me know as soon as possible.
I will try Bonnie on my server and I can give you the result, but I’d never use Bonnie and I don’t know how to generate the log result. I will probably found …
By Eric Champagne, December 6, 2008 @ 02:54
Hi !
We are at the same place in our test !!! 🙁 🙁 … I’m happy to see some comparaison chart. It will be great to work together to figure out this issue. I’m a Novell Gold Partner in Canada and we work with a huge customers who have around 45k users in their network.
They will go with VMware but OES 2 with NSS seems to have very bad performance even with a SAN FiberChannel 4Gb. When you will do some others tests with NSS on SLES 10 SP2 please let me know as soon as possible.
I will try Bonnie on my server and I can give you the result, but I’d never use Bonnie and I don’t know how to generate the log result. I will probably found …
By Mike, December 8, 2008 @ 16:23
I discovered that block alignment is very important! Even for the OES2 linux partitions (EXT2/3, Reiser).
For EXT2/3 and ReiserFS there are tool to do this, but for NSS is it (as far a I know) not possible to start
a NSS partition/pool at a specific sector.
I will investigate this futher! If you have any updates. Please let me know!
By Mike, December 8, 2008 @ 16:23
I discovered that block alignment is very important! Even for the OES2 linux partitions (EXT2/3, Reiser).
For EXT2/3 and ReiserFS there are tool to do this, but for NSS is it (as far a I know) not possible to start
a NSS partition/pool at a specific sector.
I will investigate this futher! If you have any updates. Please let me know!
By David, December 17, 2008 @ 00:43
Hi,
have you tried OES2 SP1 NSS code ?
By Mike, December 17, 2008 @ 12:35
No not yet, this is a thing on my list.
I am testing with SP1 and VMI. But that you probably saw in the test results.
I think the problem is block alignment. If you create a linux partition (ext2/ext2/reiserfs) than you have to start a block 128 to have
a proper block alignment. With NSS you cannot specify the block on witch NSS has to start. So it never gets aligned!
But when I have tested it I will posted it over here
By David, December 17, 2008 @ 00:43
Hi,
have you tried OES2 SP1 NSS code ?
By Mike, December 17, 2008 @ 12:35
No not yet, this is a thing on my list.
I am testing with SP1 and VMI. But that you probably saw in the test results.
I think the problem is block alignment. If you create a linux partition (ext2/ext2/reiserfs) than you have to start a block 128 to have
a proper block alignment. With NSS you cannot specify the block on witch NSS has to start. So it never gets aligned!
But when I have tested it I will posted it over here
By Mike, December 19, 2008 @ 08:55
I did some testing with a utility that runs on the ESX server and checks a flat.vmdk file is the partitions inside are aligned.Till my surprise it say’s that the NSS partition is aligned. So this cannot be the problem.
By Mike, December 19, 2008 @ 08:55
I did some testing with a utility that runs on the ESX server and checks a flat.vmdk file is the partitions inside are aligned.Till my surprise it say’s that the NSS partition is aligned. So this cannot be the problem.
By Mike, February 24, 2009 @ 12:32
Here a post of aligning NSS on VMFS http://blog.wilmsenit.nl/?p=244.
Check it Out!
By Mike, February 24, 2009 @ 12:32
Here a post of aligning NSS on VMFS http://blog.wilmsenit.nl/?p=244.
Check it Out!
By Charles, May 2, 2009 @ 14:58
I have run these same tests in our environment with OES2 SP1, ESXi Update 4, and an EqualLogic array and the numbers for VMFS based and RDM based volumes are the same.
By Mike, May 5, 2009 @ 15:06
Ok, Nice!
Did you test this with ISCSI or Fiber?
Can you give me the numbers of bonnie? I’m really interested in those numbers.
By Charles, May 2, 2009 @ 14:58
I have run these same tests in our environment with OES2 SP1, ESXi Update 4, and an EqualLogic array and the numbers for VMFS based and RDM based volumes are the same.
By Mike, May 5, 2009 @ 15:06
Ok, Nice!
Did you test this with ISCSI or Fiber?
Can you give me the numbers of bonnie? I’m really interested in those numbers.
By Charles, May 7, 2009 @ 23:35
NSS(VMFS)
—Sequential Output (nosync)— —Sequential Input– –Rnd Seek-
-Per Char- –Block— -Rewrite– -Per Char- –Block— –04k (03)-
Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU
oes2test 6000 33724 82.2 35049 27.3 26426 33.1 22433 83.3 50318 58.3 392.1 9.2
NSS(RDM)
—Sequential Output (nosync)— —Sequential Input– –Rnd Seek-
-Per Char- –Block— -Rewrite– -Per Char- –Block— –04k (03)-
Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU
oes2test 6000 34212 74.1 33871 28.2 23432 31.3 25322 82.3 49716 59.3 395.8 8.6
By Mike, May 8, 2009 @ 07:45
Hi,
Great! Thanks for the results. Next week I’m giving a session at a Congres in the Netherlands called Novellcongres.nl. This is the subject I’m going to talked about.
You confirm my testes. Althoug a Eqlogic is scalled lower than a Netapp. The performance with ISCSI and NSS is better!
Next week I will ask the crowd to test there performance on there SAN with Bonnie and upload the results to here. So keep coming back.
By Charles, May 7, 2009 @ 23:35
NSS(VMFS)
—Sequential Output (nosync)— —Sequential Input– –Rnd Seek-
-Per Char- –Block— -Rewrite– -Per Char- –Block— –04k (03)-
Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU
oes2test 6000 33724 82.2 35049 27.3 26426 33.1 22433 83.3 50318 58.3 392.1 9.2
NSS(RDM)
—Sequential Output (nosync)— —Sequential Input– –Rnd Seek-
-Per Char- –Block— -Rewrite– -Per Char- –Block— –04k (03)-
Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU
oes2test 6000 34212 74.1 33871 28.2 23432 31.3 25322 82.3 49716 59.3 395.8 8.6
By Mike, May 8, 2009 @ 07:45
Hi,
Great! Thanks for the results. Next week I’m giving a session at a Congres in the Netherlands called Novellcongres.nl. This is the subject I’m going to talked about.
You confirm my testes. Althoug a Eqlogic is scalled lower than a Netapp. The performance with ISCSI and NSS is better!
Next week I will ask the crowd to test there performance on there SAN with Bonnie and upload the results to here. So keep coming back.