CBT - Change Block Tracking and its common miss interpretations.

I have been working on a vCloud project for the last few months and it has now come to looking at the backup of the solution.  Commvault is the existing backup vendor and is going to continue to provide backup services in this Private cloud.

Now the customer came to me with an interesting question about how CBT worked, they wanted to understand how the changed blocks in the vmdk were tracked.  This was partly down to there thirst for information and partly because I think they were concerned at contraints that might be seen with IO and CPU.

The perception of most people is that CBT creates snapshots to record all the information in the snapshot and then it is consolidated, much like the usage of a normal snapshot in VMware. This is incorrect and although I am no storage expert I will attempt to explain how it works from my VMware background.

How it actually works is by keeping track of the blocks that have changed in a vmdk based on significant disk events recorded for a specific vmdk.

Simple right?  Well the next question from my customer was "Well how does it do that? And is it going to cause me pain on my storage IO and/or my hosts"  - This is a valid question as recording all this information has to have some overhead.

So how does it do it? Lets use the most common usage of CBT, Backup of a Virtual Machine.

1. We take a full Backup of a VM with CBT enabled on all disks.  This takes a snapshot to record all IO, while the backup is being processed on the original vmdk.  The backup application records the Change Clock timestamp (T1) at the time the snapshot is created to facilitate this backup. After the backup is completed normal snapshot consolidation is conducted and  the changes in the delta are merged into the original vmdk.
2. When the next backup is started 24 Hours later, another snapshot is taken of the VM's vmdk's.  The backup application records the Change Clock timestamp (T2) at the time of creating this snapshot. All changes are written to the delta vmdk while we backup the changed blocks in the origonal vmdk.  We still create a snapshot as we need to have access to the original vmdk.
3. The backup application then backups up only the blocks of the original vmdk that have changed between the two time stamps (C).  

Now the next part of the proccess I wanted to examine was "How does it know what blocks are changed"

When we enable change block tracking, detailed here in KB1020128 and the VM is powered on for the first time. Have a look in the Datastore Browser and the VMs home directory.  You will see a file named "vm_name-ctk.vmdk" this file is used to track the blocks that are changed on a given vmdk.  This auxiliary file has a pointer configured in the vmdk to point to this file to track changes made to blocks in the correcponding vmdk.

So when step three, listed above, is performed the auxiliary file is used to identify blocks based on the two time stamps, and the backup application then performs a backup of the changed blocks.

Now the change clock is not based on a normal clock and utilizes a Unix Epoch based clock,  Now I dont work in engineering and I am under NDA from VMware (being an employee) so I cant share any more information around the nitty gritty details.

But stepping back, my customer wanted to know if this was going to cause them any pain with IO and/or CPU on the hosts.  The simple answer is that the impact of using CBT is minimal.  It will maybe cause the kernal to in use 1 or 2 % more CPU per 20 or 30 VMs. However it is important to remember that the small increase the kernal sees now will help reduce the IO and/or CPU increase if the VM was being fully backed up every night because CBT was not being used.  backups will also be much shorter as well due to smaller amounts of data being backed up.

Hope this has been helpful, many thanks.

Phil

VMware Site Recovery Manager (SRM), EMC RecoverPoint - No Array Pairs

So I have been doing a mixture of many things after joining VMware.  Mainly vCloud Director and vCloud Networking and Security based projects.  However I was tasked with a SRM plan and design for a medium - large (400VM) company.

During this peace of work I hit something that I thought I should highlight as I could only find one article on the problem posted here from our friends over at Xtravirt.  The post was more of less identical to what I had done to resolve the problem but, I was searching to see if there were any additional activities I may have missed that could have impacted my customer.

So what was the problem?

Well I was installing SRM, Configured all the resource mapping for failover and fail-back  Installed the SRA for EMC RecoverPoint, paired the SRA's and run a discovery.  And I found NO paired datastores as shown in the screen below.

So I asked the storage admins to check the policy for the consistency groupes in RecoverPoint.  The customer had a look but was new to EMC and RecoverPoint so I logged on check this for them.

This screenshot is from the Xtravirt blog as I could not use a customer one and dont have any RecoverPoint appliances configured.  Hope Xtravirt dont mind.

When I logged on the Policy was set to "Group is in maintenance mode, It is managed by RecoverPoint , SRM can only Monitor" This in not so many words means that SRM can not action anything to the consistency group and so any Failover and/or test Failover activities can not be performed by the SRA being used with SRM.  As a result SRM classes the consistency group as an object it has no control over and so does not show it in the SRM Array Managers.  Change the Option to "Group is managed by SRM, RecoverPoint can only monitor"   and this changes this behavior   This allows the SRA full control over the LUNs in the consistency group. 

Now that option is configured on the RecoverPoint appliance, go back to SRM and refresh the SRA making sure it is paired with the remote site first.  You should see all the paired LUNs in the display Window now.

vCloud Director - Building Skills

After moving to VMware I have been doing a massive amount of vCloud Director work.  I am working  with a VMware, Cisco and EMC based partner (I am sure you can get the name) working with a VMware colleague Magnus Anderson (http://vcdx56.com/) designing what will be one of the UKs biggest vCloud installations.

I have had good vCloud skills for a while now but working on a project that will be as large as this one and with the level of automation  from vCenter Orchestrator and Cisco Cloud Portal (CCP) has improved my skills massively. 

I have also been working with the guys over on Profesional VMware and have presented on the EMEA Brownbags.  I am due to present on another topic in the near future.  I would jump onto the Brownbags if your studying for the VCAP-CID and the soon to be released VCAP-CIA as they have helped me no end in refreshing and improving my knowledge on many topics.  One of the best BrownBags I have watched is @LawrenceKohan BrownBag in vCenter Change Back Manager.  This is a product I configure in every project but it is something I often need to refresh my knowledge on and Lawrence's knowledge of the product is outstanding. 

Anyway I will be updating the blog a bit more often now.  As I have some interesting stuff coming up that I would love to share. 

VMware vCD with Nexus 100V and Cisco Virtual Secure Gateway

I have just had a request on a pre sales call for vCD to use Cisco VSG.  Now as i posted before i have only recently taken the vCD course so I was very much "ill have to check that out".  Upon looking into it I could see that this was possible with 5.1 utilising VXLAN from the vShield devices.

Cisco Website has some more info on the setup of this.  I am very excited to get stuck in with doing this and see how things go once i have the full requirements.

EMC VNX Round Robin supported without the need for PowerPath

I was recently asked to to a upgrade of a VI3.5 Environment to vSphere 5.1.  After doing the normal day on-sight checking the computability of all the components etc i quickly found that they were going to need full new server estate.

As luck would have it the customer had just purchased a VNX based EMC SAN and decided to use this for the vSphere environment.  So i started my design process and when checking the firmware versons etc on the VMware HCL i found a nice article detailing you can now use Round Robin PSP instead of Fixed.  Previously this was a licensed feature that needed PowerPath to be installed on the hosts.

Have a read here VMware KB2034799 to confirm support and Bios/firmware versions.

vSphere 5.1 Whats New - Platform

User access

VMware have significantly increased the Web based platforms usability and now recommend that the web portal is how the vSphere environment is managed.  This will cause a large number of users upset as moving away from the thick client is not something many users will want to do.

VMware have also introduced the usage of more than one root account for ESXi for the first time.  This will allow for administrators to have there own logins to the console.  This is to assist with tracking configuration changes in much more detail. 

Hardware Accelerated Graphics

This is a feature that will be utilized in the next release of VMware View.  This feature is not likely to be used in in server installations, and is intended to provide better graphics performance for VDI.  The VMware documentation describes the feature below.

With vSphere 5.1, VMware has partnered with NVIDIA to provide hardware-based vGPU support inside the virtual machine. vGPUs improve the graphics capabilities of a virtual machine by off-loading graphic-intensive workloads to a physical GPU installed on the vSphere host. In vSphere 5.1, the new vGPU support targets View environments that run graphic-intensive workloads such as graphic design and medical imaging. Hardware-based vGPU support in vSphere 5.1 is limited to View environments running on vSphere hosts with supported NVIDIA GPU cards (refer to the VMware Compatibility Guide for details on supported GPU adapters). In addition, the initial release of vGPU is supported only with desktop virtual machines running Microsoft Windows 7 or 8. Refer to the View documentation for more information on the vGPU capabilities of vSphere 5.1. NOTE: vGPU support is enabled in vSphere 5.1, but the ability to leverage this feature is dependent on a future release of View. Refer to the View documentation for information on when this feature will be available.

 

Improved CPU virtualization

Improvements have been made to CPU virtualization allowing for more information of the CPU architecture to be exposed to the VM better. This allows for the VMs to have near native access to the CPUs and allow for advanced OS features to run more successful.

Auto Deploy

There have been two new modes added to auto deploy.  However I still anticipate that uptake on this feature will be slow until the new release of vSphere.

Stateless caching mode is a new mode where if the host cannot boot from the PXE network due to a failure of some sort, it will boot from a backup image located on local media.  In normal mode caching mode operates the same as stateless mode deploying an image over the network to the host to run from a memory RAM drive.  However another step is performed in caching mode where that the image deployed to the hosts memory is then written to the internal boot device, HD, SD card or USB stick.  This then makes sure the host can boot in the case of a PXE boot failure.

The second mode that was added to auto deploy was Stateful install mode.  This mode is used to actually install the ESXI image onto the local media of the host.  This is very similar to the Stateless caching however the difference is that the boot order of the BIOS should be changed on the host.  The host only needs to PXE boot once from the Auto Deploy sever to install the ESXi image and then should boot from the local media after.  

vCP5 IaaS Training Course

This week my company have put me on a training course to brush up on some of my vCloud Director Skills.  With the release of vCloud Director 5.1 Suite and the end of support for Lab Manager, we have had a massive uptake in the vCloud Directors 5.1 products.  Many people using the multi tenancy, fast provisioning, advanced network features and the ability to just create test environments and then delete them quickly has resulted in vCloud Director being better in many ways than Lab Manager.

I have the exam booked already to take the VCP5 IaaS exam a week after the end of the course.  Reason for this is the waiting time at my local test center is horrendous and i want to get the exam under my belt by the end of the year. 

As well as using the VMware course material I am using a the trusted Brown Bags.  Some of the podcast videos on here are very good and i would like to thank the guys that do them.  I also like to read books and have found this one very good Cloud Computing with VMware

Anyway I will update here with my thoughts of the design and the course when I have more information.