Posts Tagged VMWare
This weird behavior was observed on two data stores which we were planning to delete/unmounted from all our ESX/ESXi hosts.
The first error flagged “Can’t remove datastore ‘datastore_name‘ because Storage I/O Control is enabled on it. Correct it and re-try the operation”. So we tried disabling SIOC through vSphere client, the process never completed and got stuck at 4 – 14 %. This process didn’t flag any errors, but I had to increase the SIOC log to the maximum in order to find where the issue is.
To enable Storage I/O log, on each host > configuration > Advanced Settings, change Misc.SIOControlLogLevel to 7 and restart the SIOC on the host by using the following command through a SSH session
[For ESXi] /etc/init.d/storageRM restart
[For ESX] /etc/init.d/vmware-late restart
Error and warning messages can be traced in /var/log/messages
It became apparent that the SIOC wasn’t running on one of the hosts which caused the whole process of deleting a data store to fail.
After upgrading our vCentre server to the latest patch (vCentre 4.1.0-345042), our Update Manager failed to update or even attempt to scan any of our hosts with the error “Host cannot download files from VMware vCenter Update Manager patch store. Check the network connectivity and firewall setup, and check esxupdate logs for details.”
I have searched on the web for the solution to this error and people were mentioning double checking DNS entries on each of the hosts, but that never changed in my case.
I went back to our vCentre server and checked each vCentre server component version installed and it turned out that VUM wasn’t running the latest version. After upgrading and rebooting the machine, ESX host and VM scan and updates were working back again.
I hope you find this training useful.
Coming from a WSUS era, VUM has provided flexibility and good patch/remediation management for virtual machines until you start seeing this error message “Patch scan or remediation is not supported on ServerName because of unsupported or unknown OS”, this has hit me hard especially moving forward to x64 versions of Windows 2008 R2 and Windows 7.
Vmware has released vSphere compatibility matrix found here. I have taken a snippet of this document shown below:
“Update Manager 4.1 and its subsequent update releases are the last releases of the product to support scanning and remediation of patches for Windows and Linux guest operating systems and applications running inside a virtual machine.” Supported document is found here. Make sure you rethink your patching management and deployment strategies for VM environment, I guess WSUS is still the way to go within Windows environment or purchasing MS System Centre Configuration Manager which makes software updates (not just windows updates) much easier with better control with other capabilities on top.
I am sure you have read or come across VMware best practices regarding snapshots. We use vDR in some of our backups in addition to Backup Exec to backup specific applications such as SQL and Exchange.
As a note, VMware snapshots are crash consistent and not application consistent. For that reason VMware do not recommend using snapshots on DC servers. To have an application consistent backup, Backup Exec and other third party backup application can do that for you.
Now back to our issue, running on multiple snapshots is bad. It is even worse when having multiple snapshots on one machine older than 3 days. Your machine would struggle reading from these multiple delta vmdk disks to recreate your data. These snapshots can get very large and can degrade the performance of the server immensely! We had this issue with an Exchange server that had 7 snapshots over 3 days old at various sizes, the server became unusable. The time it took to delete those snapshots were close to 10 hours and that wasn’t fun!
To avoid this crisis, setup alarms on your VCentre to cascade to all nodes in your cluster to alerts you for any VM running on snapshots.
There is a great KB article by VMware.
You will find that mapped RDM datastores in virtual compatibility mode cannot be extended simply by a re-scan of the HBA and extending the drive. In this case you will need to shut down the server, make note of the scsi x:y postion, un-map that volume from the virtual machine and re-map it again. This would recreate the descriptor file with the added space.
A simple rescan within your VM OS would show the added space which you can extended using DISKPART or any other tool out there.
RDM’s in physcial compatibility mode can be extended easily without the above steps.