Most simple web server ever!

When I install VMware vSphere on a server, I always (oke 9 out of 10 times) use a scripted install. A scripted installation has a couple of advantages over a manual installation like:

  • Every host is installed with the same configuration, limiting the possibilities of a misconfiguration by a administrator (even after a year).
  • The installation is finished in less time
  • You have a good disaster recovery of your hosts available.
  • You can (re)installed multiple hosts at the same time.
  • You have a good documentation how your hosts is configured.
  • You can have a cup of coffee while installing :)

Almost every server has a IPMI, ILO or DRAC interface enabling you to connect a ISO file to a virtual CD-ROM player so you can start your ESXi installation.

In the past I used a USB stick to host the ks.cfg file. So when the ESXi installation kicks-off you just point the kickstart to the USB stick and your ready to go.
But since a couple of years ESXi is more often installed on a local SD card. This SD card is connected to a internal USB interface. So you have to configure your kickstart file to install on the correct USB interface or you will overwrite your USB stick where your kickstart file is located (yes, it happened to me a couple of times).
In large environments you may want to use a PXE boot environment where you can hosts you kickstart file on a NFS share or a web server. But what if you only have to install a couple of hosts in a small environment?

A simple solution to this problem is to host a small web server on your laptop where you host the kickstart file.

For Microsoft Windows there are a couple of simple small web servers available but I’m using a Mac for my daily work. So will searching the Internet for a good, small and free web server I stumbled on a threat where a guy pointed out that Mac OSX has his on built in web server in Python!

Just open a terminal screen an give the following command:

python -m SimpleHTTPServer 8080

This command will start a simple, small and free web server on you Mac in the current directory (beware of your firewall!) on port 8080. Just as simple as that!

I haven’t tested this on a Windows installation where Python is installed, but I can imaging that it will work.


Multiple 1G or 10G in your data center?

The last couple of weeks I had the same discussion with a 3 different customers: Are we going to use multiple 1G (in a LACP or not) connections to the core switch of 2 10G connections?

When designing a VMware vSphere environment including a storage design, this is one of the first questions that pop-up.

As always the truth is in the middle, both solutions have there pro’s and con’s.

But first, let me clear something that you all probably know. 10 x 1G connections to your core switch isn’t the same as 1 x 10G connection to your core switch. Usally I use the following example for a customer to explain the difference.

You can compare 10 x 1G connections to a high way with 10 lanes where every car (packet) can drive at the speed of 1G. So the maximum throughput per session is no more than the speed of that lane. So in this case 1G. Every vSphere host has 1 connections on the high-way where he can place car’s on the road. The advantage is that if you have more vSphere hosts, you can add more cars on the road at the same time.

If you have a 10G connection (or in many cases 2 x 10G because you want to be redundant)  you have on lane on the high-way where all the cares can drive at the speed of 10G. Because the car’s can drive at a higher speed, there is more room on the high-way to place car on. Even when you configure a LACP (which is available with a distributed switch from version 5) you cannot get a higher session speed than 1G.

10G is especially nice when you have a NFS or iSCSI storage solution. Not that most storage solution use the whole 2 x 10G (hence most cannot fully utilize 1 10G connection) but the session between your vSphere hosts and storage solution is more than 1G.

This is in my opinion the most important reason why you  want 10G in you data center. That you can have more than a 1G connection per session.

In the past the price for a 10G solution was a bit of a issue. Now a day’s you can have a redundant 10G L3 core switches from vendors like HP and Cisco under 15K SFP’s and cabling included.

Another pro is the cabling in your rack. You have a cleaner rack who is easier to administer when you have only 2 cables per hosts than 6 or 8 cables per host. Less cables means that your core switch can do with less ports,  resulting in a smaller switch. Of course this switch has to handle a lot of traffic, but rack space is also important nowadays.

So resume. If you have the budget, I would go for 10G. Not that we’re going to use to full 10G but we sure want to use more than 1G. You have to talk to your budget and stack holders what the best solution is for your design.




1 Month after VMworld: What did it brought me?

It has been more than a month ago since I went to VMworld in Barcelona. You see many blog post about new startups, technology and interesting session. I think that VMworld is more than only the (great) technology stuff you can hear about. Eventually you can download almost all the sessions afterwards. I think the whole event is interesting. This includes talking to vendors, other virtualization fanatics and meeting new people.

Here a sum of what it brought me.


Of course you get a lot (and I mean a lot) of gifts. Beside pencils, peppermints, stress balls  and block notes I got the following interesting goodies.

  • SimpliVity was so nice to give all vExpert a Raspberry PI. I’m real excited about this small computer and the stuff that you can do with it. Unfortunately I haven’t had the time to decide what to do with it. But in a few weeks it’s Christmas and that is the perfect time to play around with it.
  • Nutanix gave all vExpert a personal vExpert glass. Even when I didn’t talk to the right person on VMworld. They contacted me after VMworld through Twitter and asked me for my address.
  • PernixData gave me a real nice polo that I can wear during my training classes.

SimpliVity, Nutanix and PernixData thanks! I really appreciate it!


VMworld is also about networking. I had really nice chats with people from Nexenta, PernixData, Nutanix, VMware. Especially talking to guru’s is great. You can drain so much information from them. Especially when it’s on one of the (vendor) party’s. Here you can meet and talk with a lot of VMware guru’s in a informal way. As you all know, the party’s of VMworld a famous. Probably everybody who went to the Veeam party had a hard time to wake up the next morning -:)

Every year I speak to some people that I only see at VMworld. It’s great to catch up and talk about virtualization or oder stuff. 1 month after VMworld one of those person contacted me if I was interested in a job for job designing a 2500 Horizon View environment with Nutanix servers. Although I haven’t had experience working with Nutanix, VMworld gave me enough knowledge to get me this job and I probably will be playing with Nutanix till the summer of 2015.

As you van see. VMworld is more than only technical session and commercial talks from vendors. The whole event is great and I will go for use next year!



Nutanix alert not cleared in Prism

When I visit a Nutanix customer, I always check the cluster health and if there are no alerts.

One customer had a error that wasn’t acknowledged for more than 45 day’s. And that alert repeated itself every hour resulting in more than 500 critical alerts.

After resolving the issue (rebooting the CVM) I wanted to clear the critical alerts in Prism. The procedure for this is quite simple. Just go to Alerts | mark all Alerts| click  Acknowledge and Resolve.

I this case I noticed that the Alerts where not cleared. Although the where Acknowledged and Resolved.  Repeating the action did not solved the problem.

The solution was to resolve the Alert through NCLI with the following script:

for alert in `ncli alert ls | grep ID | awk ‘{print $3}’`;  do echo “Resolving alert $alert”;  ncli alert resolve ids=$alert; sleep 2; done

You may have to run the script multiple times until all Alerts are cleared.


How to check the Nutanix cluster metadata store

Just a quick note.

While upgrading 32 Nutanix nodes for a customer, I wanted to make sure that every node is part of the metadata store.

This can be achieved by entering the command:

nodetool -h localhost ring

Output will look like:

nutanix@NTNX-14SX31290007-C-CVM:$ nodetool -h localhost ring
Address         Status State   Load            Owns    Token
                                                          w7iesrvWOTsU53XPvNTlWCVgub36H9PIJcE3nYDS2rHb4N7XzJEUpGp            lSYYc     Up     Normal     1.99 MB         7.18%   0ZbVFDk5VFoTcX78ofInvdnKw0uwRcxGqQsWrwEqmAFERi9ylGNkW86            9cgAe     Up     Normal     3.32 MB         11.31%  7aP5UmSTUO2pKO0I6AoPOit7jPzFGXlrDPYvimtqqT3qYI5el9ZXHUx            EdZZf     Up     Normal     3.42 MB         6.38%   BXeWgqIKH85IUcIITh1fjgcoYcv1EzVO15vjlxqPKFCUbHEqRhXMLCw            h9xy7     Up     Normal     3.16 MB         7.14%   FxyoAoJjF94Yd548AgmY8sXaoLtw5YRyF6dpDTbnXLzOCThcxzM9JrG            md2wa     Up     Normal     2.51 MB         5.83%   JaB5mJO5Q7DtPoHbPJd7QFRop57CY3dw3V0LwuJk58EnRPDxmRBr6FM            HLN3R     Up     Normal     1.52 MB         6.40%   NYI0KclyR6E8A6Yuok88lWE0yyzBil7vYe6dNrJfaB9iYem7X1D0sHJ            p2oyZ     Up     Normal     1.36 MB         6.28%   RRnlFLPALjpn3sCsA7qaMq7Msf8BEEpds0uSokeYLtlYYrb9gBr3mUw            LVcBS     Up     Normal     1.56 MB         6.81%   VfPTSjlJyU6ZFogXfaHlFKBwScidSEs61CLtW51mMb2SCOcTZauL7lc            xXrzE     Up     Normal     1.32 MB         3.85%   Y3SsPwG3mIRdzZVeED7hKZoCkH32NbDuPqBSH9moN6vtwvD8OGrFR3o            vNXyi     Up     Normal     2.31 MB         7.43%   cewHQfBTGVNdNy1BSABrh3DrI7XmbBLtF1EcvZ248cNygdiyAeYv0rk            SapAL     Up     Normal     2.82 MB         6.36%   gbLIMnJplxOXX4Jbk4jNxXnMo8njhxcrj6RFsMtZ5hQ7ha6hjT2wsi5            7lsrn     Up     Normal     3.16 MB         6.06%   kMR8DmhH2i1YuRgL746IOOXlsv5hitFUHjLO78K1dAlnPBcLdeXKjjX            h2EVF     Up     Normal     2.74 MB         6.24%   oEEJaKrjE1ukEkzB6n6fOVUyNK3P8qLkoFyHqrTzXItNBtNS1fxYBv8            DvmjW     Up     Normal     1.71 MB         6.70%   sNu0haLD0Zmb3JjtUFkk0Iffuqf0EwspDDbfVD4bDtiHCCarjVF18nY            1zBea     Up     Normal     1.8 MB          6.03%   w7iesrvWOTsU53XPvNTlWCVgub36H9PIJcE3nYDS2rHb4N7XzJEUpGp            lSYYc


Updating a single Nutanix node

All the controller VMs of a Nutanix cluster have to have the same version of NOS installed. If you upgrade a cluster. All the CVMs are upgraded one-by-one.If you want to add a new node to the cluster, that node has to have to same version of NOS.

So what if you bought new Nutanix servers and they have a previous version of NOS installed. You cannot add this node to cluster because the NOS version is not the same.

There is a command that you can issue from a CVM that’s part of the cluster to upgrade a single Nutanix node.

  1. Login to a CVM that’s part of the cluster.
  2. Issue: /home/nutanix/cluster/bin/cluster -u [IP of the node that will be upgraded] upgrade_node

Note that this only upgrades the node and doesn’t downgrade it.




VMworld 2014 Europe

This year I’m lucky to have a Bloggers pass for VMworld in Europe. For me this is the 6th time that I will attend VMworld. And as every year I’m really looking forward to go. Not only for the technical sessions, but also for all the good friends I made over the last few years and the interaction with VMware staff/employees and partners.

I just finished building my schedule and wanted to share my highlights with you. The sessesion who I really looking forward to are:

  • STO2496 – vSphere Storage Best Practices: Next-Gen Storage Technologies
  • STO2197 – Storage DRS: Deep Dive and Best Practise
  • NET2745 – vSphere Distributed Switch: Technical Deep Dive
  • EUC2039 – Horizon 6 Storage Architecture Concepts: Designing for success in 2014

I’m planning to write a short blog post about these session. Maybey not in that week but very soon after.

The first few years I booked my agenda full with session. After 2 years I discovered that talking with vendors on the solution exchange can also be mind blowing. Of course there trying to interest you in there products, but after the sales talk you can ask question and talk to technical people how you see there product and how you can use it in your environment.

Some vendor that I en-course you to look at are:

  • PernixData The have a great product for accelerate your storage with local flash or SSD
  • Login VSI The have a nice product for load testing your VDI environment
  • Nutanix Nice solution for a converged infrastructure solution
  • Simplivity Also a nice converged infrastructure solution
  • F5 Nice load-balancer for you Horizon View enviroment.

Go and talk to them, you will be surprised!

Want more information on VMworld are want to attend? Check here for the VMworld page and here for registration


Putty fatal error after upgrading Nutanix NOS

After a upgrading 32 Nutanix servers (8 boxes of 4 servers) I got a error from Putty while trying to connect to a CVM with SSH: “Disconnected: No supported authentication methods available (server sent: publickey)”






Before upgrading I haven’t any problems using Putty to SSH to a CVM.

The error indicates that the server only accepts public key authentication.

After a (better) read of the post actions of the upgrade manual I checked the “Cluster Lockdown ” option in Prism. This option was enabled, thus login using SSH with username and password was disabled.










After placing a mark before “Enable Remote Login with Password” I was able to login again using SSH with a username and password.

Of course using SSH with a username and password isn’t that secure as with public key authentication but sometime it’s needed. If you would like to user Public key authentication with Putty, check this link.


Designing a VMware 5.5 management cluster

In the past we had our vCenter server on his own physical machine. Later we moved vCenter to a virtual machine on the same cluster were all other virtual machine live.

As we get more, bigger and more VMware techniques in our VMware environments, the need for a separate vSphere cluster for VMware services is growing so we can guarantee that the base for our VMware environment isn’t affected by resource consumption of other (production) virtual machines and  of course the same applies the other way around.

A separate VMware management cluster has the following benefits:

  • Separate management resource from production resources.
  • The management services don’t run on the same hardware as your production environment.
  • In case of a complete power-down situation, you first start you management cluster with all VMware services. If all VMware services are up and running you can start the VMware clusters for your production environment making sure that everything you need for a controlled power-up of you production services is in order.

There are several VMware vSphere and vCenter services who can run in a management cluster:

  • Single-Sing-On (SSO)
  • Inventory Services
  • Web client
  • VMware Update Manager (VUM)
  • vShield
  • Cisco Nexus
  • vCenter Operation Manager (VCOPS)
  • Third-party software for exmple anti-virus
  • Active Directory for authentication

Before we start, we have to ask our self some questions:

  1. Do I want a High Available (HA) setup?
  2. Do I want to use self singed SSL certificates or do I want to use a PKI environment (public or private)?
  3. Witch database do I want to use?
  4. Witch type of load balancer do I want to use?

So let us assume that we want to setup a management cluster with the following:

  • 2 SSO servers in HA mode on separate virtual machines
  • 2 vCenter servers load balanced on separate virtual machines (is this scenario we have to separate vSphere clusters)
  • 2 Web clients load balanced on separate virtual machines
  • 2 MS SQL database servers in HA mode on separate virtual machines

Some rules that we have to stick to:

  • Each vCenter server has his own Inventory Service
  • SSO can only be in active/passive mode
  • Each vCenter server has his own VUM server.

When we apply these rules we get the following design.

Virtual Hike














So what do you see here.

Web client
We have two virtual machines running the Web client. These Web clients are behind a load balancer. The load balancer divides the connections equally across the 2 Web client servers. If one Web Client goes down, the other one is still available to manage your VMware environment.

vCenter Server
In this example we have 2 vCenter servers. 1 for a VMware cluster running servers (Exchange, SQL, Etc) and 1 for Horizon View. Nice part is, that both vCenters show up in the same Web Client without linked-mode.

We have 2 Microsoft SQL database servers who are in HA mode with log shipping. This guaranties that if 1 database server goes down, vCenter and other VMware servives (Update Manager) for example still continue to work.

VMware Update Manager
Every vCenter Server needs his own Update Manager. This is a one-on-one connection. To prevent that every VUM server has is own patch repository you can share the repository form one VUM server to the other.

Inventory Services
Same as with VUM, every vCenter server needs his own Inventory server.

So why not place the Inventory Service on the same host a the vCenter server? This way you separate the resources. Inventory is primarily for the Web Client. So to make sure that the Inventory Service doesn’t consume resource from the vCenter server we place them on is own hosts.

So why not place them with the Web Clients? What if you get a third vCenter server. Then you have a problem, because this vCenter server also needs a Inventory Service and you only can run 1 Inventory Service per host.

Single Sign-on
Single Sign-on (SSO) came with vSphere 5 and is your central user authentication mechanism. Single Sign-on can have his own user database or makes a connection to (multiple) other authentications services like Microsoft Active Directory. There for we don’t want only 1 SSO server. If this one fails, nobody can authenticate to vCenter including VUM or other VMware servers as VMware vShield.

SSO can be configured in HA mode, but you have to have a load balancer.

Load Balancer
Most VMware services can be load balanced but they can’t do it by them self. You have to make use of a third-party solution. This can either be a software or a hardware load balancer. Make sure you are aware of the functionality you need. Then pick your load balancer.

Whats missing in this picture?
In this picture we don’t see Microsoft Active Directory services for AD authentication or any other third-party software solutions for example anti-virus. If your going to implement a VMware management cluster it’s very likely that those services also run in the VMware management cluster.

Some resources


New Book – vSphere Virtual Machine Management

A virtual machine (VM) is a software implementation of a machine (a computer) that executes programs like a physical machine. vSphere has clusters of virtual machines that are all interconnected. Virtualization changes everything. Virtual machines have far more functionality than physical servers and are capable of incredible things. Virtual machines should not be treated and administered exactly like physical servers. It is imperative that the administrators know how to correctly create and administer virtual machines.

vSphere Virtual Machine Management depicts how to create a virtual machine step-by-step using multiple creation methods. In this book, you will learn how to develop virtual machine performance, resource allocation, and performance monitoring in detail. You will also be introduced to load balancing and virtual machine availability concepts and configurations.
This book looks at the different ways to create virtual machines, and breaks down the many different configuration options that are available. It will take you through a number of step-by-step instructions that will help you understand virtual machine features and how to configure them.
Not only will you learn how to create and configure virtual machines, but you will also be introduced to many of the advanced configuration options that are available in vSphere. We will also take a look at virtual machine performance, resource allocation options, and how to monitor virtual machines. If your environment uses multi-tiered applications, you will discover why vApp is the perfect container for multi-tiered applications. How to take advantage of virtual machine availability and load balancing will also be discussed.