This weird behavior was observed on two data stores which we were planning to delete/unmounted from all our ESX/ESXi hosts.
The first error flagged “Can’t remove datastore ‘datastore_name‘ because Storage I/O Control is enabled on it. Correct it and re-try the operation”. So we tried disabling SIOC through vSphere client, the process never completed and got stuck at 4 – 14 %. This process didn’t flag any errors, but I had to increase the SIOC log to the maximum in order to find where the issue is.
To enable Storage I/O log, on each host > configuration > Advanced Settings, change Misc.SIOControlLogLevel to 7 and restart the SIOC on the host by using the following command through a SSH session
[For ESXi] /etc/init.d/storageRM restart
[For ESX] /etc/init.d/vmware-late restart
Error and warning messages can be traced in /var/log/messages
It became apparent that the SIOC wasn’t running on one of the hosts which caused the whole process of deleting a data store to fail.