Using NetBackup 7.0.1 and vSphere 4. Tried to backup a new v7 VM and got a 156 failure from NetBackup. Within vSphere client the following error was displayed when NetBackup tried to take a snapshot of the VM:
Cannot create a quiesced snapshot because the create snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine.
Within NetBackup the following was shown in the job Detailed Status tab:
29/11/2010 13:59:10 - Critical bpbrm(pid=3360) from client XXXXXXX: FTL - snapshot creation failed, status 156
29/11/2010 13:59:10 - Warning bpbrm(pid=3360) from client XXXXXXX: WRN - ALL_LOCAL_DRIVES is not frozen
29/11/2010 13:59:18 - end Flash Backup Windows, Delete Snapshot On Exit; elapsed time: 00:00:05 snapshot error encountered(156)
The VM itself is running Windows Server 2008 R2, has one CPU, a 40GB C drive and 2GB RAM. It has VMware tools installed. Plenty of space on the VMFS volume that the VM lives on.
This error was generated from a vSphere host running 4.0.0 164009. Used vMotion to move the VM to a different host running 4.0.0 208167 and now the VM backs up fine. Further to this, I moved the VM to another host running 4.0.0 164009 and the backup again failed with the same error. Seems as though NetBackup 7.0.1 doesn’t like vSphere 4.0.0 164009.
I’ve now also seen a different cause of status 156 failures, related to the snapshot process taking too long to complete. A quick fix is to remove the VMWare VSS writer, though there’s more diagnostics you can do to work out what’s not working. The suggested method is to uninstall the VMware tools, then re-install choosing a custom install, and de-select the VSS writer. Note that the uninstall will remove all the VMware drivers too, so for example you’ll loose your network card if you’re using VMXNET3 etc. Unfortunately the tools that come with vSphere 4.1 don’t have the modify option, so removing and re-installing is the only option.
The VMware tools service provides the ability to disable VSS writers, for diagnostic purposes.