Updating a single Nutanix node

All the controller VMs of a Nutanix cluster have to have the same version of NOS installed. If you upgrade a cluster. All the CVMs are upgraded one-by-one.If you want to add a new node to the cluster, that node has to have to same version of NOS.

So what if you bought new Nutanix servers and they have a previous version of NOS installed. You cannot add this node to cluster because the NOS version is not the same.

There is a command that you can issue from a CVM that’s part of the cluster to upgrade a single Nutanix node.

  1. Login to a CVM that’s part of the cluster.
  2. Issue: /home/nutanix/cluster/bin/cluster -u [IP of the node that will be upgraded] upgrade_node

Note that this only upgrades the node and doesn’t downgrade it.




VMworld 2014 Europe

This year I’m lucky to have a Bloggers pass for VMworld in Europe. For me this is the 6th time that I will attend VMworld. And as every year I’m really looking forward to go. Not only for the technical sessions, but also for all the good friends I made over the last few years and the interaction with VMware staff/employees and partners.

I just finished building my schedule and wanted to share my highlights with you. The sessesion who I really looking forward to are:

  • STO2496 – vSphere Storage Best Practices: Next-Gen Storage Technologies
  • STO2197 – Storage DRS: Deep Dive and Best Practise
  • NET2745 – vSphere Distributed Switch: Technical Deep Dive
  • EUC2039 – Horizon 6 Storage Architecture Concepts: Designing for success in 2014

I’m planning to write a short blog post about these session. Maybey not in that week but very soon after.

The first few years I booked my agenda full with session. After 2 years I discovered that talking with vendors on the solution exchange can also be mind blowing. Of course there trying to interest you in there products, but after the sales talk you can ask question and talk to technical people how you see there product and how you can use it in your environment.

Some vendor that I en-course you to look at are:

  • PernixData The have a great product for accelerate your storage with local flash or SSD
  • Login VSI The have a nice product for load testing your VDI environment
  • Nutanix Nice solution for a converged infrastructure solution
  • Simplivity Also a nice converged infrastructure solution
  • F5 Nice load-balancer for you Horizon View enviroment.

Go and talk to them, you will be surprised!

Want more information on VMworld are want to attend? Check here for the VMworld page and here for registration


Putty fatal error after upgrading Nutanix NOS

After a upgrading 32 Nutanix servers (8 boxes of 4 servers) I got a error from Putty while trying to connect to a CVM with SSH: “Disconnected: No supported authentication methods available (server sent: publickey)”






Before upgrading I haven’t any problems using Putty to SSH to a CVM.

The error indicates that the server only accepts public key authentication.

After a (better) read of the post actions of the upgrade manual I checked the “Cluster Lockdown ” option in Prism. This option was enabled, thus login using SSH with username and password was disabled.










After placing a mark before “Enable Remote Login with Password” I was able to login again using SSH with a username and password.

Of course using SSH with a username and password isn’t that secure as with public key authentication but sometime it’s needed. If you would like to user Public key authentication with Putty, check this link.


Designing a VMware 5.5 management cluster

In the past we had our vCenter server on his own physical machine. Later we moved vCenter to a virtual machine on the same cluster were all other virtual machine live.

As we get more, bigger and more VMware techniques in our VMware environments, the need for a separate vSphere cluster for VMware services is growing so we can guarantee that the base for our VMware environment isn’t affected by resource consumption of other (production) virtual machines and  of course the same applies the other way around.

A separate VMware management cluster has the following benefits:

  • Separate management resource from production resources.
  • The management services don’t run on the same hardware as your production environment.
  • In case of a complete power-down situation, you first start you management cluster with all VMware services. If all VMware services are up and running you can start the VMware clusters for your production environment making sure that everything you need for a controlled power-up of you production services is in order.

There are several VMware vSphere and vCenter services who can run in a management cluster:

  • Single-Sing-On (SSO)
  • Inventory Services
  • Web client
  • VMware Update Manager (VUM)
  • vShield
  • Cisco Nexus
  • vCenter Operation Manager (VCOPS)
  • Third-party software for exmple anti-virus
  • Active Directory for authentication

Before we start, we have to ask our self some questions:

  1. Do I want a High Available (HA) setup?
  2. Do I want to use self singed SSL certificates or do I want to use a PKI environment (public or private)?
  3. Witch database do I want to use?
  4. Witch type of load balancer do I want to use?

So let us assume that we want to setup a management cluster with the following:

  • 2 SSO servers in HA mode on separate virtual machines
  • 2 vCenter servers load balanced on separate virtual machines (is this scenario we have to separate vSphere clusters)
  • 2 Web clients load balanced on separate virtual machines
  • 2 MS SQL database servers in HA mode on separate virtual machines

Some rules that we have to stick to:

  • Each vCenter server has his own Inventory Service
  • SSO can only be in active/passive mode
  • Each vCenter server has his own VUM server.

When we apply these rules we get the following design.

Virtual Hike














So what do you see here.

Web client
We have two virtual machines running the Web client. These Web clients are behind a load balancer. The load balancer divides the connections equally across the 2 Web client servers. If one Web Client goes down, the other one is still available to manage your VMware environment.

vCenter Server
In this example we have 2 vCenter servers. 1 for a VMware cluster running servers (Exchange, SQL, Etc) and 1 for Horizon View. Nice part is, that both vCenters show up in the same Web Client without linked-mode.

We have 2 Microsoft SQL database servers who are in HA mode with log shipping. This guaranties that if 1 database server goes down, vCenter and other VMware servives (Update Manager) for example still continue to work.

VMware Update Manager
Every vCenter Server needs his own Update Manager. This is a one-on-one connection. To prevent that every VUM server has is own patch repository you can share the repository form one VUM server to the other.

Inventory Services
Same as with VUM, every vCenter server needs his own Inventory server.

So why not place the Inventory Service on the same host a the vCenter server? This way you separate the resources. Inventory is primarily for the Web Client. So to make sure that the Inventory Service doesn’t consume resource from the vCenter server we place them on is own hosts.

So why not place them with the Web Clients? What if you get a third vCenter server. Then you have a problem, because this vCenter server also needs a Inventory Service and you only can run 1 Inventory Service per host.

Single Sign-on
Single Sign-on (SSO) came with vSphere 5 and is your central user authentication mechanism. Single Sign-on can have his own user database or makes a connection to (multiple) other authentications services like Microsoft Active Directory. There for we don’t want only 1 SSO server. If this one fails, nobody can authenticate to vCenter including VUM or other VMware servers as VMware vShield.

SSO can be configured in HA mode, but you have to have a load balancer.

Load Balancer
Most VMware services can be load balanced but they can’t do it by them self. You have to make use of a third-party solution. This can either be a software or a hardware load balancer. Make sure you are aware of the functionality you need. Then pick your load balancer.

Whats missing in this picture?
In this picture we don’t see Microsoft Active Directory services for AD authentication or any other third-party software solutions for example anti-virus. If your going to implement a VMware management cluster it’s very likely that those services also run in the VMware management cluster.

Some resources


New Book – vSphere Virtual Machine Management

A virtual machine (VM) is a software implementation of a machine (a computer) that executes programs like a physical machine. vSphere has clusters of virtual machines that are all interconnected. Virtualization changes everything. Virtual machines have far more functionality than physical servers and are capable of incredible things. Virtual machines should not be treated and administered exactly like physical servers. It is imperative that the administrators know how to correctly create and administer virtual machines.

vSphere Virtual Machine Management depicts how to create a virtual machine step-by-step using multiple creation methods. In this book, you will learn how to develop virtual machine performance, resource allocation, and performance monitoring in detail. You will also be introduced to load balancing and virtual machine availability concepts and configurations.
This book looks at the different ways to create virtual machines, and breaks down the many different configuration options that are available. It will take you through a number of step-by-step instructions that will help you understand virtual machine features and how to configure them.
Not only will you learn how to create and configure virtual machines, but you will also be introduced to many of the advanced configuration options that are available in vSphere. We will also take a look at virtual machine performance, resource allocation options, and how to monitor virtual machines. If your environment uses multi-tiered applications, you will discover why vApp is the perfect container for multi-tiered applications. How to take advantage of virtual machine availability and load balancing will also be discussed.


View Composer error due to failure Windows activation

I’ve been playing around with VMware View in my lab environment testing different scenarios with the View Composer. Today I noticed that a deployment with linked clones failed with the following error:

View Composer agent initialization state error (16): Failed to activate the software license (waited 0 seconds)

When I looked into the log files I noticed that during the linked clone deployment the View Composer tried to activate the Windows 7 software. This lab environment doesn’t have a internet connection so Windows was unable to activate. Normally you would setup a KMS server for Linked Clones but this is a little bit overkill for my lab.

Searching the internet I found the following kb article form VMware saying that they have a workaround for the issue, but you have to contact VMware technical support to get the solution. Searching a little bit further I found the solution. You have to adjust to registry keys telling the View Composer not to activate Windows during a deployment.

You can find the key’s at HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesvmware-viewcomposer-ga

There you find to key’s

  • AllowActivateMAKLicense
  • SkipLicenseActivation

Default both set to 0. Adjust the value to 1 and your done.

Note: This isn’t supported by VMware or Microsoft and only should be used in a lab environment.


vExpert 2014




I’m honored to announce that this year for the 4th time I’m awarded with the vExpert 2014 title.

vExpert is title who you can ear for a year if you have demonstrated significant contribution to the community and a willingness to share their expertise with others.

I’ve got mine for my grey VMware courses that I’ve developed and of course this blog

The complete list of all vExpert 2014 can be found here.



How to add a new node to a Nutanix cluster using CLI

If you want to expand you Nutanix cluster with a new node you have the option to do this from the GUI. But sometime you want to do it from the command line using the CLI.

Before you start check a couple of things:

Is IPV6 working between the Nutanix hosts?

  1. Get the IPV6 address of the node you want to add by entering the command:
    ifconfig etho | grep inet6

    You will get a output something like:
    inet6 addr: fe80::20c::29ff:feb7:a9c7/64 Scope:Link

  2. Try to ping the node you want to add from a cluster node by entering the command:
    ping6 -I eth0 fe80::20c::29ff:feb7:a9c7

    You should get a reply

If IPV6 is working you can add the new node to the cluster.

  1. Enter the NCLI interface by entering ncli
  2. Check if the node you want to add is discovered by entering:
    cluster discover-nodes

    You should get a output something like:
    ncli> cluster discover-nodes
    Cluster Id                           :
    Hypervisor Address         : xx.xx.x.xx
    Ip                                         : fe80::20c:29ff:feb7:a9c7%eth0
    Ipmi Address                    : xx.xx.x.xx
    Node Position                   : A
    Node Serial                        : ZMxxxxxxxxxx
    Node Uuid                         : d1b9d9c3-7f6f-4cb8-bf16-f379fd73e342
    Rackable Unit Model       : NX-3051
    Rackable Unit Serial        : 13SMxxxxxxxx
    Service Vm Address         : xx.xx.xx.xx
    Svm Id                                :

    Notice the Node Uuid

  3. Add the node to the cluster by entering the command:
    cluster add-node node-uuid=[node uuid from step 4]

After this the node will be add to your cluster


How to check your version of Nutanix NOS

On the CVM controller issue the command:

cat /etc/nutanix/svm-version



My first thoughts on Nutanix

Currently I’m working on a project designing a 3000 VMware Horizon View environment for a local government in the west of the Netherlands. When I started this job the hardware was already purchased by the project manager. Not the correct way (according to Prince2) but it’s a fact a have to deal with. Honestly I was very pleased to hear that the vendor chose was Nutanix. On the first day I got a tour in the datacenter and 1 19″ rack was almost fully filled with NX-3050  (8 blocks with 4 servers connect with 10Gb on a Cisco Nexus 5000).

I’m quite experienced with VMware vSphere an d View but not with Nutanix. So the first thing I did was Google for a Nutanix. I didn’t want to search the Nutanix website itself but a independent people who give there thoughts on Nutanix. One of the first website I found was was IT Blood Pressure and the website from Stevan Poitras. Stevan website contains a section called the Nutanix Bible. Really interesting stuff especially the section witch describes in detail how the controller VM handels the storage workloads. You can see that some really smart people developed this system.Looking at the profiles of both authors I discovered that the both work for Nutanix. Oke, that’s by the articles are so in depth and detailed. So maybe not as independent that I was looking for, but nevertheless really interesting stuff.

After reading a lot about Nutanix I posted a Tweet that I was working with Nutanix. The same day I was contacted by Raymon Epping (@repping) that if I have any questions I can contact him. This really enjoyed me. Although Nutanix isn’t a small company the aren’t that spoiled  that talk directly whit engineers. Of course Nutanix has a benefit if this project succeeds (and this is when the user is satisfied) the could think that the sold the hardware and it isn’t there concern anymore. That if I need more information I can go to a training a read the manual. Nutanix even went so far that I had Raymon Epping for a couple of hours for myself talking about how to design a Nutanix cluster. Great!

As far as I now can overlook Nutanix isn’t so difficult to setup. The manuals are really clear and on the Internet you can Google for a of information. But as whit most (large) implementation there are 2 way’s to implement/desing a environment. The way that it runs, and the way that it runs on it’s best and it’s manageable. In my course I often say that 60% till 70% is thinking about how to setup your environment. Think in what if scenarios. The last 40%  till 30% is just clicking your environment together. You cannot design a Nutanix cluster the same way as you would design a traditional cluster. Some stuff you have to think about is:

  • Is it wise to run vCenter on the same hardware because of the dependency of the controller VM?
  • How to size your storage cluster, 1 big one or multiple smaller ones because you have different workloads
  • What is the impact of the controller VM and what if the controller VM isn’t available (backup?)?
  • How do you setup your network (VSS versus VDS)?
  • How to setup time synchronization (because of the database Nutanix uses) and what if time ain’t available.

For sure I’m going to contact Raymon Epping about designing issues.

One thing I learned in a early stage. RTFM! 🙂