vCAC 5.2/6.0 Certificate checking - Error "The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel"

During a recent challenging installation I had a situation where I needed to front both the vCAC web servers with a customer generated verified certificate, and keep the rest of the certificates a Self sighed.

This was simple in theory, just replace the certificates on the web servers.

We generated certificate requests from both web servers and submitted them to the CA.  Once I had the certificates they were installed on the vCAC Web servers and I changed the bindings in IIS for 443.  I then placed both the new certs on the MGMT servers and restarted the services.  The MGMT service failed to start and then gave the following error

"The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel"  

I then looked in some of the vCAC documentation and found that I needed the following.

1. The intermediate CA

2. Domain certificate

Placed these on all the machines via group policy and then tried to start the service again.  Still the MGMT service failed.

I then investigated some of the vCAC code as I thought there must be something going on that was not apparent.  I found that all the components try to check sighed certs against the issuing CA authority root.  This is not documented clearly (if at all) so it was a bit of a find.  I then looked at how to over come this.  Some of you who have installed vCAC many times will recall there is an option to "Suppress certificate mismatch" when you install some of the site components.  

This setting stops the full chain check of each service.  But there is no option for this when installing many of the components.  So this is how you add it to the code.

Find the  "ManagerService.exe.config" file located in "/Program Files (x86)/VMware/vCAC/Server" right click and edit the file.  Go to the bottom of the file and just before "</Configuration>" tag insert the following code.

"<system.net>
            <settings>
            <servicePointManager checkCertificateName="false" checkCertificateRevocationList="false"/>
            </settings>

 </system.net>"

This will tell the service not to check the cert chain and will allow it to start.  You will also need to add this to the CONFIG files for all the other services such as the DEM's and the designer.  Same file and insert the same code in the same place. 

Placing all vCAC 5.2 web components and REPO404 error

Hi all

I know vCAC 5.2 is the previous version now with vCAC 6.0 being released.  However there are still a large number of vCAC 5.2 projects running.  Including the one I am working on at the moment.

I had a problem where the initial install was done by some colleagues  and I continued with the installation building out the vCAC web layer to a 2 node load balanced construct.  The install completed by my colleagues  was following my design.  I had placed the vCAC Manager Server role and the vCAC Model Manager Web services role on separate machines.  My reasoning around this was based on the security requirements of the customer.

The installation was functioning well until we came to build the second web server.  We had some major problems with all the portals.  We lost both the vCAC admin portal and the vCAC Self Service Portal and got REPO404 errors when trying to access them.

I looked over the logs of the vCAC web servers and could see loads of SSL errors.  So I check the certificates on all the servers and all of them were correct.

After some discussions internally I decided to place the Model manager web service on the web servers with the other website components.  We installed vCAC from the start and found this removed the error.

I think the problem was a combination of the SSL and communication URL that the model manager was using was being replaced with a new one when the second Admin portal service was installed.  This then conflicted with URLs and certs in the Manager service and maybe in the repository.  We could have over come this with a load balancer, but at the time we didn't have this available. 

Hope this helps if you see the REPO 404 error as there isn't much documented about it. 

Using Network Profiles in vCAC 5.2

Hi All

Another quick thing I thought I would blog during an install I am doing.  The customer is building from my design and came over to me and asked "Where is the Network Profiles option"  After going over and seeing what they were asking it became apparent that they were unaware of the obscure place this is enabled. 

Go to vCAC Administrator > Customization and then tick the box for "Enable Static IP service" Click OK and this will enable the Network Profiles option along the side bar.

Thanks
Phil

Installing ESXi via USB drive

Being mainly an architect or a design lead on many projects I am finding I don't tend  to get too involved with installs as much as I would like.  But when rebuilding my lab I found a good USB boot device tool.

Universal USB installer

Select the Other Linux Distro.  This will then let you select the ESXi install and then your USB. 

Very good tool and works a treat.

vSphere 5.5 - SSO Setup and additional configuration

Hi all,
So when I was rebuilding my lab I took note from internal communications that we had made some changes to SSO, both under the hood and for the install.

So I started the install and selected the "Simple Install" as I was only building a lab.

SSO Admin Account

Over the installation steps you have to configure the SSO admin password.  Now in 5.1 the default admin user was admin@system-domain First thing I noted was this had changed.  The default administrator account for SSO is now Administrator@vsphere.local

 

Attach Identity Source

So after the installation I logged in with the new Administrator@vsphere.local account.  I then logged out to see if the AD that the vCenter was a member of, had been added to the SSO identity sources.  With vSphere 5.1 the AD was added automatically.  I received an error message stating that it could not log in.

So I went to add the identity source to the SSO configuration using the Web Client. 

 

Select the configuration option down the side menu and then select the identity sources tab, and click the green plus.  You will get the below popup window.

 Because this was my Lab I used the domain Administrator account and I used port 389 for LDAP.  In a production environment you would need to configure a Read Only AD account specificity for vSphere to allow you to track AD access.  You would also want to use SSL for LDAP to make the network communication secure.

Test the connection and then click ok.  Give the Web Client a second to perform the action of adding the source.

Once the ID source is added you don't get access straight away, you need to add a group from the ID source to SSO and then to vCenter to allow it access.

Add ID source group to SSO 

Next you need to add a group from the ID source to SSO to allow access from the ID source to vCenter and SSO.

If I just login with an account from the ID source I get presented with an empty vCenter or an access denied as the account has no permissions.

I now need to add the accounts I am going to administer vSphere with, to the SSO group that is assigned Administrator permissions in vCenter.

By default the SSO group called Administrators is added to the vCenter group called Administrators. This is why the account Administrator@vsphere.local has access to the vCenter and all the objects it controls.

The account I want to administer the vCenter with is called vsphere-admin and the domain admin account called Administrator this is more a backup account.

This time load the Web Client select the configuration tab, and under SSO select Users and groups.  This will allow you to see the groups that are configured in SSO.

 

Select the Administrators group and the accounts included in this group are listed below.  If you select the green cross in the window below you can add an account to the group.

Select the correct ID source from the drop down, and then find the account you want to add.  Click add and then select the Ok button.

 

Web Client in my lab took a few mins to add this to the group but after the account has been added you should see a screen like the one below, showing the accounts you have added to the group.

 

 You can add accounts directly to the vCenter  Administrator group using the traditional methods.  So we can add the domain accounts to vCenter and bypass SSO altogether, I would not recommend this as SSO is used in other products and adoption of it now will also make it less painful when it is an essential component in future releases.

 

 

vSphere 5.5 - Whats changed

Hi All

I have tried not to call this a "Whats New" blog post as I plan on doing one of those in the future.  However I did want to post up some info about things I have noticed when building my vSphere 5.5 lab.

Coincidentally I have been asked by one of two customers about things they have noticed and so I thought it was worth publishing something I could reference to future questions. So a few posts will follow about things I have noted in my Lab setup.

My Lab is running the following

1 vCenter - Windows 2K8 R2 SQL Express
1 DC - Windows 2K8 AD, DNS.  AD running 2008 R2 domain level.
2 ESXi hosts - Resource
1 ESXi host - MGMT Runs vCNS
1 vC AC 5.2 Server

I may add more machines to the Lab later on but it is all running on a MacBook with Fusion Pro so I am limited to how many machines I can run.

Keep looking and Ill post updates soon.

vCD without vCNS

One of the most interesting solutions I have been working on recently is a vCD project where the customer has decided to use an alternative to vCNS (vCloud Networking and security).  I can not name the product they want to use as an alternative but for the discussion inside this blog I think the principle applies to most third party products.  

The first thing I asked the customer was why do you not want to use vCNS? The answer was quite simply that there security department doesn't see the vCNS firewall as secure. When asking for a bit more detail we were not provided with anything credible in my view.  One of the main problems they communicated as an issue is that the appliances are deployed on demand and are all applied with the same default password. (this can be changed after the appliance is deployed)  Anyhow this is not the aim of my blog, to detail my customer problems, and so we were asked to use an virtual firewall that they already use. 

Now the first thing I needed to comunicate to the customer we that we HAVE to install vCNS manager, this is a requirement of vCD and all network related instructions are sent to vCNS. This includes port group creations for vCD networks.  I also needed to point out to the customer we will be using VXLAN and vCNS is used in securing multitenancy within the VXLAN.

So once we had cleared all the above and it was agreed that there was no way we could push the customer back to using vCNS we looked at the use cases and the limitations that would be imposed. 

First lets break down what not using vCNS removed from vCD.

1. Limits the use of OrgvDC networks, we can't create a routed or an isolated OrgvDC network with IP addressing ranges specified. - When creating a network with an IP range  a vCNS appliance is applied to the networks corresponding dVportGroup.  This is to assign DHCP. 
2. We can't create any vApp networks with routed connections.
3. Networks and network services configuration is no longer made visible to vCD. This restricts what we can do for network service creation when using vCD. 
4. No vCO plugin for any third party software defined networking service. 
5. No Charge Back monitoring for any Network elements, usage of bandwidth, IP addresses and network services such as firewall rules and NAT rules.

So how did I design around this?

1. The customer was given two options of how to manage IP addressing inside the vCD networks. a) Was to use there third party firewall appliance, b) Was to use a vCO workflow to populate the IP address of the VM's in the vApp from there IPAM system.
2. We had to place an extra VM in the vApp.  This extra VM is the firewall VM and because vCD just thinks this is another VM it creates it in the users OrgvDC and uses the resource provisioned to the user/Organization.  If this was vCNS the appliances are provisioned in the SYSTEM resource pool and do not impact the users resource allocation.
3. The IPAM system and the firewall VM has no integration with vCD and this particular firewall vendors API is not very good. So using vCO to talk to it is a challange.  This gives the customer another system to monitor and manage as we cant pull any information from either of these systems.
4. We can try and develop a vCO plugin, most of the plugin's are developed by VMware but the limitation of this firewall vendors API presents a big challenge and my customer is not going to pay for a plugin development.
5. We can add a fixed cost for the VM in CBM and charge a fixed amount each month.  We can monitor bandwidth but not in the same way we can with a vCNS appliance. 

This is becoming more and more common and the above are some small use cases for my customer.  Other customers may have more risks and may also have more constraints and requirements of vCNS and/or a third party firewall appliance. 

With the release of NSX in the not so distant future I hope people start to see VMware network virtualization as a more credible technology.

So if you get asked this question I hope this post can give you some food for thought on the impact and the design considerations/constraints it will impose.

vCloud Director - Moving vApps Between Clusters In the Same PvDC

 I had a request from a customer of mine to be able to move there vApps and VM's between vSphere clusters in the same vCD Provider Virtual datacenter.  The customers first thoughts were to drop down into the vCenter and then just do a normal vMotion, however they get the standard message detailing that this object is managed by vCD.

This should have made the customer think that maybe doing it this way was not the best idea, But they progressed and migrated the VM anyway.  Once they had done this.  It broke the relationship with vCD and the VM became unmanageable in vCD.

Now if this was a simple migrate a VM/vApp between clusters using the same storage we could go to System>Manage And Monitor> Resource Pools and select the VM from source cluster and the select "Migrate" to move the VM to the destination cluster. 

Now the customer I was working with has been using VBLOCK's in there datacenter to build there vCloud environment.  This is brilliant as the VBLOCK is a very impressive bit of kit and after working with them for a number of months I am very happy with recommending them to customers.  But using a VBLOCK did present us with a challenge.

The customer was using a single provider virtual datacenter in vCD.  This was backed by several clusters each cluster corresponded to a single VBLOCK.  The Clusters were 24-32 hosts.  Each VBLOCK has its own VNX SAN and the storage from this is only presented to the hosts in the same VBLOCK.  There is no storage shred between the VBLOCKs, and thus between the clusters.  

So if we have no shared storage between the clusters backing the Provider Virtual Datacenter. How do we move VMs between the clusters.  The answer is using Storage vMotion! Hang On Storage vMotion is not a selectable option in vCD.

After thinking a little more I decided we could select the VMs storage profile and change this.  This should instruct the VM to conduct a joint "Change Host and Change datastore" migration.

In the example below I have my vApp built on a cluster called "Site1".  This cluster has a corresponding storage profile.  The storage for this profile is only mounted on the hosts in the cluster "Site1"

Now the vApp and the VM must be completely shutdown.  If the vApp is showing a status of "Partially Running" then this operation will fail.  The same applies for if the VM is not shutdown.  You will see an error message of "Invalid Parameter" 

Select a host profile that is linked to another cluster in the Provider Virtual Datacenter.  In my lab this is "Site2" and select OK.

Now the VM will show as being "Busy" while the Storage vMotion Is conducted in the background.  If you drop to the vSphere Client you will see more information. The vSphere client will show the VM as being relocated.

 

Once the Storage vMotion has completed you will see the VM in vCD show as normal and examining the vSphere client will show the VM as being moved to another cluster.

This resolved my customers problem, and we integrated this into a rather cool vCO workflow that could be kicked off from the customers Cisco based cloud portal.  The workflow I may detail in another post but it basically looked at the VMs inside a vApp and then changed there storage profile to relocate them based on the input from the user.

My customer then requested that this activity of moving between the clusters was conducted with no downtime.  The ONLY way this is possible is to use a SWING datastore.  This is a datastore that is presented to all the hosts in the Provider.  This breaks a number of VCE design constraints and would result in having to have a datastore equal to the size of the largest VM.  In my customers case this is 8TB.  So this 8TB is going to be sitting there doing nothing most of the time, as a VCE design constraint is not to share storage between VBLOCK's let alone run workloads on it.

I am working to resolve this at the moment but I am not sure I will find a resolution. We are constrained by the CBT technology used in Storage vMotion and the design constraints of VCE.

vCloud Director - Set Administrator Password and then you cant login?

Hi All

Had a very strange problem today, I was delivering a vCD POC.  I left the database creation to the user and we just connected to it.  I provided them the scripts that located in the Installation Guide. We were provided a SQL user account as normal and ran the ./configure script and connected to the database with no problems.

Once we had completed the configuration we connected the web address for the cell and went over the initial configuration.  We set the password for the administrator and completed the wizard.

After this we then tried to login to the cell

User: administrator
Password: ******** (same as was set in the wizard)

Result = Access denied?

Very strange, as i was 100% certain the password was correct.  So I asked the SQL admin to run the following to rest the cell to a un-configured state. update config set value='false' where name = 'vcloud.system.initialized';   Once this was done we went over the initialization of the cell again and went set the same password, just incase I and typed the password incorrectly.  Still had the same Access Denied message.

So I ran the above SQL query again to reset the cell.  I then asked to have a look at the SQL management studio.  The DBA let me have a look and I could see he had made the account being used by vCD a sysadmin! this was not in the scripts in the install guide so it must have been done after.  Now I have seen strange things when sysadmin accounts have been used in previous VMware products for DB access.  Traditionally VMware products have suggested giving the accounts sysadmin access at install and then lock down after, but most the newer releases give more info on user rights needed so we no longer need to give sysadmin rights at all.

I removed the sysadmin privilege and reconfigured the cell with the same password as the last two times. Amazingly at the login screen I was then able to login.

Now I have no idea why this resolved the login problems and I have asked if I can log a bug internally to see if this is as designed or if it is a small bug.  I was using 5.1.2 vCD binaries on SQL 2008 Express, again I dont know if this had an affect on what was happening.  I just wanted to post this incase anyone has a similar problem, hopefully this will help.

vBrownbag's and Professionalvmware.com

Hi again

I wanted to let everyone know I have been hosting some sessions on the popular community site. professionalvmware.com I have covered 2 objectives from the VCAP-CID blueprint and I am planning on hosting a networking one soon to go a little more in-depth into the networking virtualization we introduce in vCNS and vCD.

Here are the links to my two vBrownbags.  Be sure to add me and the vBrownbag crew on twitter.

Phil Monk  vBrownbags

#vBrownbag Follow-Up Phil Monk Covering VCAP5-CID Objective 4 – vCloud Security

#vBrownbag with Philip Monk covering VCAP5-CID Objective 3

Thanks
Phil

Please Leave Feedback..... I Would love to hear what you think, good or bad

Hi Everyone

I have had some impressive views on some of the stuff I have published, especially the SRM RecoverPoint article I posted.  But next to no one leaves any feedback.  :-(  Good or bad, or even a follow up question.  Please feel free to ask me anything.

Many thanks
Phil

CBT - Change Block Tracking and its common miss interpretations.

I have been working on a vCloud project for the last few months and it has now come to looking at the backup of the solution.  Commvault is the existing backup vendor and is going to continue to provide backup services in this Private cloud.

Now the customer came to me with an interesting question about how CBT worked, they wanted to understand how the changed blocks in the vmdk were tracked.  This was partly down to there thirst for information and partly because I think they were concerned at contraints that might be seen with IO and CPU.

The perception of most people is that CBT creates snapshots to record all the information in the snapshot and then it is consolidated, much like the usage of a normal snapshot in VMware. This is incorrect and although I am no storage expert I will attempt to explain how it works from my VMware background.

How it actually works is by keeping track of the blocks that have changed in a vmdk based on significant disk events recorded for a specific vmdk.

Simple right?  Well the next question from my customer was "Well how does it do that? And is it going to cause me pain on my storage IO and/or my hosts"  - This is a valid question as recording all this information has to have some overhead.

So how does it do it? Lets use the most common usage of CBT, Backup of a Virtual Machine.

1. We take a full Backup of a VM with CBT enabled on all disks.  This takes a snapshot to record all IO, while the backup is being processed on the original vmdk.  The backup application records the Change Clock timestamp (T1) at the time the snapshot is created to facilitate this backup. After the backup is completed normal snapshot consolidation is conducted and  the changes in the delta are merged into the original vmdk.
2. When the next backup is started 24 Hours later, another snapshot is taken of the VM's vmdk's.  The backup application records the Change Clock timestamp (T2) at the time of creating this snapshot. All changes are written to the delta vmdk while we backup the changed blocks in the origonal vmdk.  We still create a snapshot as we need to have access to the original vmdk.
3. The backup application then backups up only the blocks of the original vmdk that have changed between the two time stamps (C).  

Now the next part of the proccess I wanted to examine was "How does it know what blocks are changed"

When we enable change block tracking, detailed here in KB1020128 and the VM is powered on for the first time. Have a look in the Datastore Browser and the VMs home directory.  You will see a file named "vm_name-ctk.vmdk" this file is used to track the blocks that are changed on a given vmdk.  This auxiliary file has a pointer configured in the vmdk to point to this file to track changes made to blocks in the correcponding vmdk.

So when step three, listed above, is performed the auxiliary file is used to identify blocks based on the two time stamps, and the backup application then performs a backup of the changed blocks.

Now the change clock is not based on a normal clock and utilizes a Unix Epoch based clock,  Now I dont work in engineering and I am under NDA from VMware (being an employee) so I cant share any more information around the nitty gritty details.

But stepping back, my customer wanted to know if this was going to cause them any pain with IO and/or CPU on the hosts.  The simple answer is that the impact of using CBT is minimal.  It will maybe cause the kernal to in use 1 or 2 % more CPU per 20 or 30 VMs. However it is important to remember that the small increase the kernal sees now will help reduce the IO and/or CPU increase if the VM was being fully backed up every night because CBT was not being used.  backups will also be much shorter as well due to smaller amounts of data being backed up.

Hope this has been helpful, many thanks.

Phil

VMware Site Recovery Manager (SRM), EMC RecoverPoint - No Array Pairs

So I have been doing a mixture of many things after joining VMware.  Mainly vCloud Director and vCloud Networking and Security based projects.  However I was tasked with a SRM plan and design for a medium - large (400VM) company.

During this peace of work I hit something that I thought I should highlight as I could only find one article on the problem posted here from our friends over at Xtravirt.  The post was more of less identical to what I had done to resolve the problem but, I was searching to see if there were any additional activities I may have missed that could have impacted my customer.

So what was the problem?

Well I was installing SRM, Configured all the resource mapping for failover and fail-back  Installed the SRA for EMC RecoverPoint, paired the SRA's and run a discovery.  And I found NO paired datastores as shown in the screen below.

So I asked the storage admins to check the policy for the consistency groupes in RecoverPoint.  The customer had a look but was new to EMC and RecoverPoint so I logged on check this for them.

This screenshot is from the Xtravirt blog as I could not use a customer one and dont have any RecoverPoint appliances configured.  Hope Xtravirt dont mind.

When I logged on the Policy was set to "Group is in maintenance mode, It is managed by RecoverPoint , SRM can only Monitor" This in not so many words means that SRM can not action anything to the consistency group and so any Failover and/or test Failover activities can not be performed by the SRA being used with SRM.  As a result SRM classes the consistency group as an object it has no control over and so does not show it in the SRM Array Managers.  Change the Option to "Group is managed by SRM, RecoverPoint can only monitor"   and this changes this behavior   This allows the SRA full control over the LUNs in the consistency group. 

Now that option is configured on the RecoverPoint appliance, go back to SRM and refresh the SRA making sure it is paired with the remote site first.  You should see all the paired LUNs in the display Window now.

vCloud Director - Building Skills

After moving to VMware I have been doing a massive amount of vCloud Director work.  I am working  with a VMware, Cisco and EMC based partner (I am sure you can get the name) working with a VMware colleague Magnus Anderson (http://vcdx56.com/) designing what will be one of the UKs biggest vCloud installations.

I have had good vCloud skills for a while now but working on a project that will be as large as this one and with the level of automation  from vCenter Orchestrator and Cisco Cloud Portal (CCP) has improved my skills massively. 

I have also been working with the guys over on Profesional VMware and have presented on the EMEA Brownbags.  I am due to present on another topic in the near future.  I would jump onto the Brownbags if your studying for the VCAP-CID and the soon to be released VCAP-CIA as they have helped me no end in refreshing and improving my knowledge on many topics.  One of the best BrownBags I have watched is @LawrenceKohan BrownBag in vCenter Change Back Manager.  This is a product I configure in every project but it is something I often need to refresh my knowledge on and Lawrence's knowledge of the product is outstanding. 

Anyway I will be updating the blog a bit more often now.  As I have some interesting stuff coming up that I would love to share. 

VMware vCD with Nexus 100V and Cisco Virtual Secure Gateway

I have just had a request on a pre sales call for vCD to use Cisco VSG.  Now as i posted before i have only recently taken the vCD course so I was very much "ill have to check that out".  Upon looking into it I could see that this was possible with 5.1 utilising VXLAN from the vShield devices.

Cisco Website has some more info on the setup of this.  I am very excited to get stuck in with doing this and see how things go once i have the full requirements.

EMC VNX Round Robin supported without the need for PowerPath

I was recently asked to to a upgrade of a VI3.5 Environment to vSphere 5.1.  After doing the normal day on-sight checking the computability of all the components etc i quickly found that they were going to need full new server estate.

As luck would have it the customer had just purchased a VNX based EMC SAN and decided to use this for the vSphere environment.  So i started my design process and when checking the firmware versons etc on the VMware HCL i found a nice article detailing you can now use Round Robin PSP instead of Fixed.  Previously this was a licensed feature that needed PowerPath to be installed on the hosts.

Have a read here VMware KB2034799 to confirm support and Bios/firmware versions.

vSphere 5.1 Whats New - Platform

User access

VMware have significantly increased the Web based platforms usability and now recommend that the web portal is how the vSphere environment is managed.  This will cause a large number of users upset as moving away from the thick client is not something many users will want to do.

VMware have also introduced the usage of more than one root account for ESXi for the first time.  This will allow for administrators to have there own logins to the console.  This is to assist with tracking configuration changes in much more detail. 

Hardware Accelerated Graphics

This is a feature that will be utilized in the next release of VMware View.  This feature is not likely to be used in in server installations, and is intended to provide better graphics performance for VDI.  The VMware documentation describes the feature below.

With vSphere 5.1, VMware has partnered with NVIDIA to provide hardware-based vGPU support inside the virtual machine. vGPUs improve the graphics capabilities of a virtual machine by off-loading graphic-intensive workloads to a physical GPU installed on the vSphere host. In vSphere 5.1, the new vGPU support targets View environments that run graphic-intensive workloads such as graphic design and medical imaging. Hardware-based vGPU support in vSphere 5.1 is limited to View environments running on vSphere hosts with supported NVIDIA GPU cards (refer to the VMware Compatibility Guide for details on supported GPU adapters). In addition, the initial release of vGPU is supported only with desktop virtual machines running Microsoft Windows 7 or 8. Refer to the View documentation for more information on the vGPU capabilities of vSphere 5.1. NOTE: vGPU support is enabled in vSphere 5.1, but the ability to leverage this feature is dependent on a future release of View. Refer to the View documentation for information on when this feature will be available.

 

Improved CPU virtualization

Improvements have been made to CPU virtualization allowing for more information of the CPU architecture to be exposed to the VM better. This allows for the VMs to have near native access to the CPUs and allow for advanced OS features to run more successful.

Auto Deploy

There have been two new modes added to auto deploy.  However I still anticipate that uptake on this feature will be slow until the new release of vSphere.

Stateless caching mode is a new mode where if the host cannot boot from the PXE network due to a failure of some sort, it will boot from a backup image located on local media.  In normal mode caching mode operates the same as stateless mode deploying an image over the network to the host to run from a memory RAM drive.  However another step is performed in caching mode where that the image deployed to the hosts memory is then written to the internal boot device, HD, SD card or USB stick.  This then makes sure the host can boot in the case of a PXE boot failure.

The second mode that was added to auto deploy was Stateful install mode.  This mode is used to actually install the ESXI image onto the local media of the host.  This is very similar to the Stateless caching however the difference is that the boot order of the BIOS should be changed on the host.  The host only needs to PXE boot once from the Auto Deploy sever to install the ESXi image and then should boot from the local media after.  

vCP5 IaaS Training Course

This week my company have put me on a training course to brush up on some of my vCloud Director Skills.  With the release of vCloud Director 5.1 Suite and the end of support for Lab Manager, we have had a massive uptake in the vCloud Directors 5.1 products.  Many people using the multi tenancy, fast provisioning, advanced network features and the ability to just create test environments and then delete them quickly has resulted in vCloud Director being better in many ways than Lab Manager.

I have the exam booked already to take the VCP5 IaaS exam a week after the end of the course.  Reason for this is the waiting time at my local test center is horrendous and i want to get the exam under my belt by the end of the year. 

As well as using the VMware course material I am using a the trusted Brown Bags.  Some of the podcast videos on here are very good and i would like to thank the guys that do them.  I also like to read books and have found this one very good Cloud Computing with VMware

Anyway I will update here with my thoughts of the design and the course when I have more information.

vSphere 5.1 whats new Networking

Networking

 

Networking health check

This feature is aimed to assist with the divide that is often seen between vSphere Administrators and Network Administrators.  Often configuration errors can occur when there are a large number of uplinks to be configured for the vSphere infrastructure. 

The process checks the following items are configured correctly on the VCD switch.

1. VLAN
2. MTU
3. Network Teaming Adapter

VMware document states that this feature works by sending probing packets over the layer 2 network every minutes, to the network equipment connected directly to the DVS uplinks.  REG and ACK packets are sent to probe the network.  If these packets are dropped or rejected a configuration problem is highlighted on the DVS.

VDS management rollback and recovery

One of the major problems with VDS in the past was if there was a complete DC failure and the vCenter was virtualized (VMware recommendation) then when hosts were recovered the networking would not be restored until the vCenter was online to provide the VDS configuration to the hosts.  However when the host containing the vCenter came online, it had no networking because the vCenter was not available to lay down the VDS configuration. This often resulted in the management being placed on a separate VSS.

vSphere 5.1 avoids this buy introducing management rollback. If when the hosts are up and running they cant communicate with each other.  An automatic rollback to a last working configuration is performed (VSS) this will allow for the hosts to communicate with the vCenter and then when the vCenter is fully operational the VDS is recreated.  vSphere 5.1 also allows for interaction of the VDS configuration at the DUCI now as well to provide better troubleshooting.

Link Aggregation control protocol

This has always been a massive point of confusion for administrators.  vSphere documentation has often miss used LACP terms in stating what is and what isn’t supported.  However this has now been clarified in vSphere 5.1

Previously static Link Aggregation was supported however now full Dynamic LACP is supported.  But only on VDS.

Bridge Protocol Data Unit Filter

This is a new feature ad builds on top of the recommendations to disabled STP and enable Port Fast on uplink switches for both VSS and VDS.  It is now recommended to enable bridge protocol unit filtering to stop loop behavior being detected.  The VMware documentation details the behavior below.

VMware virtual switches do not generate BPDU packets. But if a virtual machine sends them, they will be forwarded to the physical switch port over the uplink. When the physical switch port, configured with the BPDU guard setting, detects the packet, that port will be put in err-disabled state. In this err-disabled state, the switch port is completely shut down, which prevents effecting STP. However, the vSphere environment will detect this port failure and will move the virtual machine traffic over another uplink that is connected to another physical switch port. The BPDU packets will be seen on this new physical switch port, and the switch will block that port as well. This ultimately will cause a denial-of-service (DoS) attack situation across the virtual infrastructure cluster.

This configuration is recommendation by VMware and is enabled on the VDS, not the physical uplinked switch. This will be tested and added to the standard building blocks documentation.

Increases in scalability

The below table details the increases in scalability

vSphere 5.1 Whats new - Storage

Storage

Increase of number of hosts able to share a read only file has increased from 8 to 32

A read only file in VMware is often a source VMDK used for linked clones in either VMware View or VMware vCloud Director.  Linked clones are used for quick deployment in both View and VCD.

The current limit is 8 hosts.  This results in View and VCD clusters being limited to 8 hosts as if one of the linked clones was placed on the 9^th host in the cluster, this host would be denied access to the source VMDK file.

This limitation has been increased to allow up to 32 hosts to access the read only file.  This removes the 8 host limitation for View and VCD and the limitation now is defined by the cluster limitation of 32.

Introduction of new VMDK type, SE virtual disk (Space –efficient virtual disk)

A well know problem with thin provisioning VMDK files was that if space was freed in the OS of the VMDK file by deleting files etc., the VMDK file did not shrink.

With SE virtual disks the space can be reclaimed. 

1. VMware tools will scan the OS hard disks for unused allocated blocks.  The blocks are marked as free.
2. Run the SCSI UNMAP command in the guest to instruct the virtual SCSI layer in the VMKernal to mark the blocks as free in the SE vmdk.
3. The once the VMKernal knows what blocks are free it reorganizes the SE vmdk so that the construction of the vmdk is of a continuous data lump with all the free blocks at the end of the vmdk.
4. The VMKernal then sends either a SCSI UNMAP command to the SCSI array or a RPC TRUNCATE command to NFS based storage.

This then frees the unused blocks in the SE vmdk and frees space on the datastore for other VMs.

Please see the below image taken from the VMware white paper for storage in vSphere 5.1

Improvements in detecting APD (all paths down) and PDL (permanent device loss)

It is common when an APD condition is seen by an ESXI host, the host will become unresponsive and will eventually become disconnected from vCenter.  This is due to the hostd process not knowing if the removal of the storage devices is a permanent or a transient state for the lost paths, because of this hostd does not timeout the recan operation for rediscovering the paths or any threads being processed by hostd.  Everything just waits for I/O to be received from the storage array, and because hostd has an infinite number of threads the hostd process often becomes unresponsive and crashes.

There have been improvements over the last few releases of vSphere to improve the detection of APD by introducing PDL detection (Permanent device loss) by detecting specific SCSI sense codes from the target array.  Improvements have been made to this function as well as alterations to how APD conditions are handled.

The following information is an extract from the VMware documentation listed in the appendix of this document.

In vSphere 5.1, a new time-out value for APD is being introduced. There is a new global setting for this

feature called Misc.APDHandlingEnable. If this value is set to 0, the current (vSphere 5.0) condition is used,

i.e., permanently retrying failing I/Os. If Misc.APDHandlingEnable is set to 1, APD handling is enabled to follow the new model, using the time-out value Misc.APDTimeout. This is set to a 140-second time-out by default, but it is tunable. These settings are exposed in the UI. When APD is detected, the timer starts. After 140 seconds, the device is marked as APD Timeout. Any further I/Os are fast-failed with a status of No_Connect, preventing hostd and others from getting hung. If any of the paths to the device recover, subsequent I/Os to the device are issued normally, and special APD treatment concludes.

The above text indicates that the advanced setting Misc.APDHandillingEnable should be set to 1 to allow for APD timeouts and to prevent the hostd process crashing when APD occurs.

Another setting that should be configured is the disk.terminateVMOnPDLDefault this allows HA to restart VMs that were impacted by the APD on another host that is unaffected by the APD issues. There is a know problem with this setting restarting machines that were gracefully shutdown during APD.  Specifying the following advanced setting removes this problem das.maskCleanShutdownEnabled.  Both advanced settings should be used together for best results from an APD condition.

Storage DRS V2.0

Improvements to storage DRS include additional detection for latency and datastore placement on the storage device.  Storage DRS introduces Storage Correlation which is a new feature used to detect if datastores reside on the same SAN spindles. There would be little benefit in moving a VM from one datastore to another if they reside on the same set of physical spindles.  Previously SDRS would analysis constraint on the datastores and move a VM to a less populated datastore with the assumption the VM would receive a performance benefit due to the datastore being less populated.  Now SDRS will investigate if the datastore is sitting on the same spindles and if so it will conclude that there will be little to no benefit and depending the aggressiveness of the SDRS settings will not move the VM’s VMDK files.

VmObservedLatency is another new feature of SDRS and is used to analysis the latency from the time the VMKernal receives the storage command to the time the VMKernal receives a response from the storage array.  This is an improvement over the previous level of monitoring which was monitoring the latency only after the storage request had left the ESXi host.  The new feature allows for latency inside the host to be monitored as well.  This is useful because the latency between the array and the host may be 1 or 2 milliseconds but the latency in the host from the VMKernal could be 20-30milliseconds due to the number of commands being issued/queued on the HBA that is being used for a specific datastore.