Disk now becoming the limiting factor

Hi

I was conducting a health check today at a site using some very well spected hardware. The site was using dual chase IBM x Servers. These are an amazing bit of kit with a second chase attached which is filled with CPU and RAM. Over all each host had 32 cores and 128GB of RAM. I was onsite conducting a second health check after all the recommendations I had provided from the first health check has been implemented. Everything with the site is looking good until i started conducting a performance analysis of the disk, CPU and MEM. All the hosts are sitting around 40-60% CPU and 10-20% or RAM so we are talking around 23500MHz of CPU and 20 GB of RAM. But when looking at the disk performance of the FC I was seeing some serious delays in storage command processing. The hosts had been built and had been paired with 4 single port 4GB FC cards...... The customer had only connected two and was only pushing traffic down a single HBA. This was causing some serious problems with hi IO VMs such as SQL and exchange.

This is not the first customer I have seen this and it would not appear that the limiting factor in the majority of enterprise size VMware environments is disk IO. I know there are now 8GB FC systems available (i have worked with a Compellent 8GB installation) but It would now appear that storage IO needs to be the first resource investigated when looking into performance problems. This is where I will be looking from now on.

VMware Dropping ESX...... and vSphere 4.1

After the press release of VMware ESX vSPhere 4.1 on Tuesday, i have been reading a few things about it. The biggest change from what I can see is that VMware have said this will be the last version of ESX, after 4.1 ESX will be the only version VMware will make.

This will result in more people using Powershell and the the vMA to manage there virtual machines. I personally have been using both of these management methods for sometime and so the switch isn't going to be a massive headache for me but I can imagine most people having problems switching over.

The other item i noticed is that the vCenter only appears to have been released in a 64Bit version. This from what I can understand is the resolve the very annoying problem with TOMCAT using loads of memory on the majority of 32Bit versions. I have never seen this on the 64bit installs I have done which leads me to believe VMware just want everyone to use the working 64bit version.

I have not had a chance to download the trial and play around but I think I will give the ESXI version a go once I have some time to play with it in the lab.

Thanks

VM Balloning

Hi Again
I have recently been having a large number of calls from clients regarding slow responding VM's. Upon investigating the VMs I have often found that they are ballooning and the MEMCTL driver is showing as active in ESXTOP.

After looking over the ESX/ESXI server there appears to be plenty of free memory, VMware tools is up to date and Virtual hardware is level 7.

I then looked at the VMs settings from the "Edit settings" of the VM in vCenter. I Then went to the resources tab and could see there were a problem.....

For some reason there were limits on the memory for less than the memory granted at the configuration of the VM. I asked why these were set and the customers all say they have not set these settings. Once the limits are moved the VM ballooning stops and the machines start to respond ok.

It was then clear that the majority of all VMs had these limits on them, some causing problems and some just set to the memory of the VM. On a vCenter with 500+ VMs it is very time consuming to remove all of these. I had a quick Google to see if there was a Powershell script to do it, why make one if someone else has already, found a very helpful script that someone had already created. I checked the script over and then ran it in the vSPhere Power CLI

"

Connect-VIServer
$ResourceAllocationInfo = New-Object VMware.Vim.ResourceAllocationInfo
$ResourceAllocationInfo.Limit = -1

# Get the Managed object for each VM
Foreach ($VM in get-vm | get-view) {
# Create a fresh VirtualMachineConfigSpec
$VirtualMachineConfigSpec = New-object VMware.Vim.VirtualMachineConfigSpec
$i = 0
# If the CPU Allocation is not unlimited add CPU to our spec.
IF ($VM.ResourceConfig.CpuAllocation.Limit -ne -1) {
$VirtualMachineConfigSpec.cpuAllocation = $ResourceAllocationIn
$i++
}
# If the Memory Allocation is not unlimited add Memory to our spec.
IF ($VM.ResourceConfig.MemoryAllocation.Limit -ne -1) {
$VirtualMachineConfigSpec.memoryAllocation = $ResourceAllocationInfo
$i++
}
# If $I is gt 0 then we have a VM to fix... Trigger the reconfig task.
IF ($I -gt 0) {
write-verbose "Removing limits on $($VM.Name)"
[void]$VM.ReconfigVM_Task($VirtualMachineConfigSpec)
}
}

"

I take no responsibility for this script so please be careful. You will see all the tasks in vCenter for when the settings are being set to unlimited.


Thanks

W2K8 VM using all free memory but not doing anything.

Hi All
Not updated in a while mainly due to the fact my company has had me doing help desk work to cover sickness. Will never understand why they just don't just get another consultant but that's another story for another time.

My first real work for a while has led me to a massive infrastructure of 45 hosts and 600 VMs, not the biggest environment but not the smallest either, I was onsite doing a check for some upgrade work and a minor health check. One of the internal guys was looking over a VM that was using all the allocated memory but wasn't doing much. I had a look and it was a Windows Server 2008 machine on a 64Bit installation acting as a file server. Now I remember reading an article about this and racked my brains for a few mins before I remembered that I made a note of it.....

http://support.microsoft.com/kb/976618

This is the problem. Windows Server 2008 uses all the free memory to cache files that it thinks are going to be accessed. This saves time and speeds up the file server role. But as my customer found out, it was causing apps that run on the file server to run slow and unresponsive. The above fix was applied to alter the memory behavior, after the reboot I monitored the VM for the rest of the week. Memory was higher then I would expect but there was plenty free for the apps to run. Would appear the above article works.

My customer however did hesitate in applying the fix as it does detail that it is resolved in Windows 7 and Windows sever 2008 R2, but it would appear it has not been addressed in the 64Bit versions of the operating systems.