My VCP5 exam experience

The last and first week of the year I normally reserve for some jobs around the house, some time with wife and kid and some studying.

This time the VCP5 exam was the goal. I combined this with the update of my vSphere4 administration and advanced administration course I give for Ictivity Training. This way I practice the new vSphere 5 features and have the course up-to-date.

Although I have given a vSphere5 administration course (with only the features of that where already available in vSphere4) there were quit some features I didn’t have the time to play with except in the VMware Vsphere: What’s new ESXi 5.0 and vCenter Server 5.0. This includes Storage profiles, Auto deploy and more.

As I follow @esloof on Twitter I read on his blog that the VCP5 exam is different than the VCP4 exam. With the previous VCP exam you had to know the maximum configurations out-of-your-head! There were quite some question: “What is the maximum size of a VMFS volume”. This is not the case any more. The exam is more based on global ESXi 5.0 knowledge with I think is a good thing.

The material I used for my preparation:

• VMware Education (by @esloof) VMware vSphere: What’s new ESXi 5.0 and vCenter Server 5.0
• Scott Lowe (@scott_lowe) Mastering VMware vSphere 5
• Duncan Epping (@DuncanYB) and Frank Denneman (@FrankDenneman) vSphere 5 Clustering Technical Deepdive
• And a lot of testing 🙂 in my home-lab

Gladly all the work payed-off and I passed my exam with a score of 411 where a minimum score of 300 is the passing rate.

As I’m VCAP4-DCD and VCAP4-DCA certified I will go for VCAP5 when it’s available.
After that maybe I will go for VCDX but I’m wondering if this certification brings me more interesting work here in the Netherlands. There aren’t many corporations in the Netherlands that need a VCDX for there environment.
So will we see. For now I’m glad I can call myself VCP5

Some vSphere5 and vCenter5 upgrade considerations

While preparing for my VCP5 exam I was reading a lot of material including the upgrade guide.

As with all upgrade, not all upgrade paths are supported. That’s why I thought to make a list with some upgrade considerations.

1. vSphere5 only comes with the ESXi hypervisor architecture. The main distinction is that ESX comes with a Service Console (also called SC or COS from Console Operating System) and ESXi only has a tech support mode (busybox implementation). If you want to perform command-line administration, you can use vCLI or PowerCLI.
2. You can upgrade from ESX 3.5/4.x and ESXi 4.x installations to ESXi 5.0 preserving your VMFS partitions.
3. New installation and boot devices options are:
   • Unified Extensible Firmware Interface (UEFI) like USB
   • Disk larger than 2TB if the system firmware supports it.
4. A minimum of 2098MB or RAM is required.
5. No graphical installer is available because this requires a Service Console.
6. The (text-base) installer can be used for new installations or upgrades
7. New partitions use the GUID partition table (GPT) instead of the MBR. GUID supports partitions size larger than 2TB.
8. New installations create a 4GB scratch partition. Any remaining disk space is formatted as VMFS datastore.
9. Rolling back to a previous version of ESX/ESXi is not supported.
10. When using a kick-start script (ks.cfg) you can press Shift-O when the ESXi installer screen appears to edit the boot options and provide the kick-start script (Example: ks=nfs//192.168.1.10/vSphere-install/esxi5.cfg nameserver=192.168.1.10 ip=192.168.1.1 netmask=255.255.255.0 gateway=192.168.1.254)
11. The default database for vCenter 5 is Microsoft SQL Server 2008 R2 express. This is bundled with the vCenter DVD. Requirements are:
    • Microsoft Windows Installer version 4.5 (MSI 4.5)
    • 4GB RAM
    • 4GB Disk Storage
    • 64 Bits Operating System
12. A in-place upgrade on Windows XP is not supported
13. When doing a in-place upgrade of your vCenter server, your vCenter server can be down for 40 till 50 minutes. During this downtime, DRS will not function. HA will.
14. A in-place upgrade on a 32Bits Operating System is not supported. You have to perform a migration to a 64Bits Operating System.
15. There is also a vCenter appliance available. This appliance is based on SUSE Linux Enterprise Server 11 (SLES11).
16. Some configuration files are not migrated when performing a upgrade:
    • /etc/sysconfig/mouse
    • /etc/sudoers
    • /etc/yp.conf
    • Custom scripts that are added to /etc/rc.d
17. Configuration files that are migrated are:
    • /etc/vmware/esx.conf
    • /etc/ntp.conf, ntp.drift, ntp.keys
    • /etc/krb.*, /etc/krb5.*
    • /etc/hosts, /etc/resolv.conf
    • /etc/pam.d/*
    • /etc/vmware/vmkiscsid/*
18. Configuration files that are partially migrated are:
    • /etc/passwd, /etc/shadow (only root and vpxusers accounts)
19. When performing a new vSphere5 installation the default partition table that is used will be GPT. When performing a in-place upgrade the MBR partition format will be kept.
20. When migration from ESX the Service Console network interface cards (NICs) are converted to VMkernel NICs and the Service Console port group is removed.
21. Rule set files and customized firewall rules are not preserved.
22. You cannot perform a in-place upgrade from ESX4.x to ESXi 5.0 when the ESX4.x was upgraded from ESX3.x.
23. When performing a in-place upgrade from ESX 4.x the /boot partition has to have more than 350Mb of free space. If the hosts that you are upgrading does not have more than 250MB of free space in the /boot partition, use a scripted or interactive upgrade instead.
24. You can preserve your (local) VMFS datastores when upgrading. Afterward you can upgrade your VMFS3 datastore to VMFS5.

vSphere Performance troubleshooting

“The server has poor performance because the server is running virtual.”
“Virtual server are always slower than physical servers.”
“My vSphere server has 60% CPU utilization but the performance of my Virtual Server is poor.”

When you are a VMware administrator you constantly have to defend your self that it doesn’t matter if a server is virtualized or that it running on physical hardware. But how can you troubleshoot these issues?

In these Performance Troubleshooting series I’m going to explain how you can troubleshoot performance problems in your virtualized environment.
This series will contain the following chapters:

1. CPU
2. Memory
3. Storage
4. Network
5. Virtual Machine

If you think I left something out. Let me know!

Before you dive into the different chapters, first let me explain how esxtop works.

Running esxtop
You can run esxtop in 2 different ways. One directly in the ESXi Console (the so called busybox) or remote through the remote CLI command called resxtop.
In case you want to run esxtop on the ESXi console,  login as root and start esxtop by typing the command esxtop followed by a enter.

esxtop

If you want to use resxtop start resxtop on a OS where the remote CLI commands are installed with the following parameters:

esxtop –server [dns or ip adres of ESXi host] –user [username probably root] –password [password of the user]

The option –user and –password are not required. If you don’t provide them as a parameter, you will be prompted for them.

Alter esxtop view
When esxtop is started (it doesn’t matter if this is remote or local) you can alter you view.

Running esxtop in batch mode
It’s also possible to start esxtop in batch mode. The results are saved in a CSV file. This can be done with the option -b. With the option -d you can specify the delay for the refresh and the option -n specifies the amount of iterations. Example:

esxtop -c ~/myesxtopconf.cfg -b -d 5 -n 10 > ~/esxtop-output.csv

In this example esxtop is started with a custom configruation file called myesxtopconf.cfg, in batch mode, with a delay of 5, and 10 iterations.
With resxtop you have to provide the IP or DNS name of the ESXi host.

The CSV file can be imported in Microsoft Perfmon for example. Note that if you have a large CSV file the import can take a very long time.

vSphere Performance troubleshooting Part1: CPU

Even do vSphere is telling you that the overall CPU utilization is no more than 60%, this doesn’t indicated that your VM isn’t running low on CPU resources.
In this part where going to deepdive in troubleshooting CPU performance related issues on your vSphere environment. Although I’m using vSphere 5.0 for my screenshot, most of the options used also go for vSphere 3 and 4.

Key tool for CPU performance troubleshooting is esxtop or if your running your esxtop remote resxtop.
I my examples I’m using esxtop directly on the vSphere ESXi console but all of these command also work with resxtop.You just have to provide a extra parameter called –server on witch server you want to run the command. Of course you have to provide a username and password to get access to that server.

If we start esxtop on our vSphere ESXi host we will get the following screen:

Click on image to enlarge

First let me explain what we see here.

Click on image to enlarge

The first three lines:

After the first three lines you will see a table with like the following:

Click on image to enlarge

So let me explain what we see there:

This picture (who I have borrowed from a VMworld presentation) explains the overall relationship between the different variables.

Oke, now we now where the different variables stand for and what there relationship is. The next question will be,  which variables do I have to monitor and what are there thresholds?

So that’s what you need to know about monitoring you vSphere ESXi host with esxtop.

vSphere5 and Broadcom iSCSI Adapter IQN name contains localhost

Just a quick post about something I noticed while configuring a new vSphere5 host with a Broadcom iSCSI Adapter. This  is not a real iSCSI HBA but a vmnic with TCP/IP offloading. This offloading feature benefits the performance of iSCSI because the vmkernel doesn’t have to do as must work as with a normal vmnic without the offloading feature.

The Broadcom vmnic will show up as a vmhba in the configuration screen of your vSphere5 host.

no images were found

With the Software iSCSI client provided by VMware the IQN name of this adapter is created when you enable the adapter. This IQN name has a naming convention like: iqn.1998-01.com.vmware:hostname:string where hostname is the name of your vSphere host and string is a random value. Now with the Broadcom card the hostname is where the pain is. As the Broadcom card is enabled by default the hostname is detected as localhost (as seen in the screenshot).
Of course this is not what we want. As when we have a 32 vSphere host all showing up in your iSCSI storage as localhost, it’s now easy to see anymore witch iqn belongs to a vSphere host. Of course the string behind the hostname will be unique and you can document the iqn, but it’s easier and thus better manageable if the iqn contains the real hostname of the server who you have configured during installation.

As of now, the only way I figured out to change the hostname in the iqn name of your vmhba is to go to the properties of you vmhba, on the General tab click configure and change the iqn name in the iSCSI Name field.

no images were found

After doing so you have to to a rescan of the vmhba in order to update the information in your iSCSI target. If you have used the previous iqn name containing the hostname as localhost you have the setup your lun masking in your iSCSI target again.

Note: I have tested this behavior with Broadcom NICs, I don’t know if the behavior is similar on other NICs who have offloading features.