NSS Performance with VMFS and ISCSI

ate I have done some testing with NSS and Vmware ESX. And I must say, I’m quite disappointed about the performance! Exspecially with NSS on VMFS with ISCSI. I had the following setup:

One virtual machine with Novell Open Enterprise server 2 on Vmware ESX 3.5 U2. This virtual machine has his VMDK on a local VMFS volume and a second disk witch is a RDM on a ISCSI SAN (Netapp).

I tested the disk performace with a program called Bonnie++ with the following parameters:

bonnie++ -s 2000M -d [path to volume] -v 3

Why a file size of 2000M on three volumes, simple, it’s more than twice the memory so Linux can’t cash the file in his memory.

And here are the results:

Per Char Per Block Rewrite Per Char Per Block 04K -3
MB K/Sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU
SLES10 SP1 NSS with VMFS on local storage 6000M 23818 80,8 34224 33,1 13242 1,3 8849 11,8 20194 0,5 323,4 3
SLES10 SP1 NSS with VMFS on ISCSI storage 6000M 8952 64,6 11981 36,6 10084 5,3 6540 22,4 16775 4,3 240,8 2,2
SLES10 SP1 NSS, RDM on ISCSI storage 6000M 20384 77 27691 39 10468 0 8780 36 7327 6 289,9 2

So as you see, the combination of NSS, VMFS and ISCSI is killing. I don’t know exactly wath is causing this, but I think it has something to do with block alligment.

Also I did some testing with VMI. I had to install SLES10 SP2 for this, so I can test NSS at this moment. I’ve download the beta of OES2 sp1 but I haven’t had the time to test this.

Sequential Output (nosync) Sequential Sequential Input
Per Char Per Block Rewrite Per Char Per Block 04K -3
MB K/Sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU
SLES10 SP2 Binary ReiserFS (Test 1) 6000M 33210 98.0 47718 8,6 19611 4,9 26668 62.1 61563 10,9 1784.8 1,7
SLES10 SP2 Binary ReiserFS (Test 2) 6000M 29833 95.2 51545 9,5 21652 5,6 26205 61.2 61651 10,2 1755.2 9,3
SLES10 SP2 Binary ReiserFS (Test 3) 6000M 32423 96.3 53426 9,7 22062 1,6 25748 60.3 61361 10,2 1867.4 9,3
SLES10 SP2 VMI ReiserFS (Test 1) 6000M 34713 97.9 55265 8,8 21152 1,1 24450 47.0 52107 0,6 1744.5 0,3
SLES10 SP2 VMI ReiserFS (Test 1) 6000M 33526 95.9 52340 8,9 21034 1,1 24695 47.1 57298 1,1 1839.1 1,1
SLES10 SP2 VMI ReiserFS (Test 1) 6000M 32762 93.5 49264 8,2 21671 1,0 25136 48.9 56911 0,9 1720.8 1,1

The disk performance doesn’t increase but the CPU load is lower. So this is interesting. When I have more time I will test the VMI setup with NSS.

I will do some more testing this and the next month, so check this post frequently!

About Michael
Michael Wilmsen is a experienced VMware Architect with more than 20 years in the IT industry. Main focus is VMware vSphere, Horizon View and Hyper Converged with a deep interest into performance and architecture. Michael is VCDX 210 certified, has been rewarded with the vExpert title from 2011, Nutanix Tech Champion and a Nutanix Platform Professional.

22 Comments to “NSS Performance with VMFS and ISCSI”

  1. By Eric Champagne, December 6, 2008 @ 02:54

    Hi !
    We are at the same place in our test !!! 🙁 🙁 … I’m happy to see some comparaison chart. It will be great to work together to figure out this issue. I’m a Novell Gold Partner in Canada and we work with a huge customers who have around 45k users in their network.

    They will go with VMware but OES 2 with NSS seems to have very bad performance even with a SAN FiberChannel 4Gb. When you will do some others tests with NSS on SLES 10 SP2 please let me know as soon as possible.

    I will try Bonnie on my server and I can give you the result, but I’d never use Bonnie and I don’t know how to generate the log result. I will probably found …

  2. By Eric Champagne, December 6, 2008 @ 02:54

    Hi !
    We are at the same place in our test !!! 🙁 🙁 … I’m happy to see some comparaison chart. It will be great to work together to figure out this issue. I’m a Novell Gold Partner in Canada and we work with a huge customers who have around 45k users in their network.

    They will go with VMware but OES 2 with NSS seems to have very bad performance even with a SAN FiberChannel 4Gb. When you will do some others tests with NSS on SLES 10 SP2 please let me know as soon as possible.

    I will try Bonnie on my server and I can give you the result, but I’d never use Bonnie and I don’t know how to generate the log result. I will probably found …

  3. By Mike, December 8, 2008 @ 16:23

    I discovered that block alignment is very important! Even for the OES2 linux partitions (EXT2/3, Reiser).
    For EXT2/3 and ReiserFS there are tool to do this, but for NSS is it (as far a I know) not possible to start
    a NSS partition/pool at a specific sector.
    I will investigate this futher! If you have any updates. Please let me know!

  4. By Mike, December 8, 2008 @ 16:23

    I discovered that block alignment is very important! Even for the OES2 linux partitions (EXT2/3, Reiser).
    For EXT2/3 and ReiserFS there are tool to do this, but for NSS is it (as far a I know) not possible to start
    a NSS partition/pool at a specific sector.
    I will investigate this futher! If you have any updates. Please let me know!

  5. By David, December 17, 2008 @ 00:43

    Hi,
    have you tried OES2 SP1 NSS code ?

    • By Mike, December 17, 2008 @ 12:35

      No not yet, this is a thing on my list.
      I am testing with SP1 and VMI. But that you probably saw in the test results.

      I think the problem is block alignment. If you create a linux partition (ext2/ext2/reiserfs) than you have to start a block 128 to have
      a proper block alignment. With NSS you cannot specify the block on witch NSS has to start. So it never gets aligned!

      But when I have tested it I will posted it over here

  6. By David, December 17, 2008 @ 00:43

    Hi,
    have you tried OES2 SP1 NSS code ?

    • By Mike, December 17, 2008 @ 12:35

      No not yet, this is a thing on my list.
      I am testing with SP1 and VMI. But that you probably saw in the test results.

      I think the problem is block alignment. If you create a linux partition (ext2/ext2/reiserfs) than you have to start a block 128 to have
      a proper block alignment. With NSS you cannot specify the block on witch NSS has to start. So it never gets aligned!

      But when I have tested it I will posted it over here

  7. By Mike, December 19, 2008 @ 08:55

    I did some testing with a utility that runs on the ESX server and checks a flat.vmdk file is the partitions inside are aligned.Till my surprise it say’s that the NSS partition is aligned. So this cannot be the problem.

  8. By Mike, December 19, 2008 @ 08:55

    I did some testing with a utility that runs on the ESX server and checks a flat.vmdk file is the partitions inside are aligned.Till my surprise it say’s that the NSS partition is aligned. So this cannot be the problem.

  9. By Mike, February 24, 2009 @ 12:32

    Here a post of aligning NSS on VMFS http://blog.wilmsenit.nl/?p=244.
    Check it Out!

  10. By Mike, February 24, 2009 @ 12:32

    Here a post of aligning NSS on VMFS http://blog.wilmsenit.nl/?p=244.
    Check it Out!

  11. By Charles, May 2, 2009 @ 14:58

    I have run these same tests in our environment with OES2 SP1, ESXi Update 4, and an EqualLogic array and the numbers for VMFS based and RDM based volumes are the same.

    • By Mike, May 5, 2009 @ 15:06

      Ok, Nice!
      Did you test this with ISCSI or Fiber?
      Can you give me the numbers of bonnie? I’m really interested in those numbers.

  12. By Charles, May 2, 2009 @ 14:58

    I have run these same tests in our environment with OES2 SP1, ESXi Update 4, and an EqualLogic array and the numbers for VMFS based and RDM based volumes are the same.

    • By Mike, May 5, 2009 @ 15:06

      Ok, Nice!
      Did you test this with ISCSI or Fiber?
      Can you give me the numbers of bonnie? I’m really interested in those numbers.

  13. By Charles, May 7, 2009 @ 23:35

    NSS(VMFS)

    —Sequential Output (nosync)— —Sequential Input– –Rnd Seek-
    -Per Char- –Block— -Rewrite– -Per Char- –Block— –04k (03)-
    Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU
    oes2test 6000 33724 82.2 35049 27.3 26426 33.1 22433 83.3 50318 58.3 392.1 9.2

    NSS(RDM)

    —Sequential Output (nosync)— —Sequential Input– –Rnd Seek-
    -Per Char- –Block— -Rewrite– -Per Char- –Block— –04k (03)-
    Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU
    oes2test 6000 34212 74.1 33871 28.2 23432 31.3 25322 82.3 49716 59.3 395.8 8.6

    • By Mike, May 8, 2009 @ 07:45

      Hi,

      Great! Thanks for the results. Next week I’m giving a session at a Congres in the Netherlands called Novellcongres.nl. This is the subject I’m going to talked about.
      You confirm my testes. Althoug a Eqlogic is scalled lower than a Netapp. The performance with ISCSI and NSS is better!

      Next week I will ask the crowd to test there performance on there SAN with Bonnie and upload the results to here. So keep coming back.

  14. By Charles, May 7, 2009 @ 23:35

    NSS(VMFS)

    —Sequential Output (nosync)— —Sequential Input– –Rnd Seek-
    -Per Char- –Block— -Rewrite– -Per Char- –Block— –04k (03)-
    Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU
    oes2test 6000 33724 82.2 35049 27.3 26426 33.1 22433 83.3 50318 58.3 392.1 9.2

    NSS(RDM)

    —Sequential Output (nosync)— —Sequential Input– –Rnd Seek-
    -Per Char- –Block— -Rewrite– -Per Char- –Block— –04k (03)-
    Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU
    oes2test 6000 34212 74.1 33871 28.2 23432 31.3 25322 82.3 49716 59.3 395.8 8.6

    • By Mike, May 8, 2009 @ 07:45

      Hi,

      Great! Thanks for the results. Next week I’m giving a session at a Congres in the Netherlands called Novellcongres.nl. This is the subject I’m going to talked about.
      You confirm my testes. Althoug a Eqlogic is scalled lower than a Netapp. The performance with ISCSI and NSS is better!

      Next week I will ask the crowd to test there performance on there SAN with Bonnie and upload the results to here. So keep coming back.

  • » NSS and VMFS performance OES2 SP1 — December 19, 2008 @ 15:19

  • » NSS and VMFS performance OES2 SP1 — December 19, 2008 @ 15:19

  • RSS feed for comments on this post.

    Leave a Reply

    You must be logged in to post a comment.