Tidy/Remove VMware Snapshots – possibly created by NetBackup

Diagnosis

I’ve found that sometimes Symatec NetBackup v7.0/7.1 doesn’t remove the snapshots it takes of VMs whilst it’s backing them up direct from the SAN. These snapshots don’t show in the vSphere Client GUI:

However, viewing the VM disk configuration shows that the vmdk files are running on snapshots (note the 00000 in the filename):

From an SSH session (e.g. using Putty) you can also list all disks with snapshots:

find /vmfs/volumes/ -name *delta.vmdk

The tidyup process

From a root SSH session on a host that can see the datastore, CLAR 148 ATA in this example, change to the VM folder: Either:

cd "/vmfs/volumes/CLAR 148 ATA/MY-VMNAME"

Or:

cd /vmfs/volumes/CLAR\ 148\ ATA/MY-VMNAME/

From the VM folder, use:

ls -latr *.vmdk

You’ll see the delta vmdk files:

[root@vfr3-uwe12 MY-VMNAME]# ls -latr *.vmdk
-rw------- 1 root root         560 Nov 18 15:29 MY-VMNAME_3.vmdk
-rw------- 1 root root 10737418240 Nov 18 15:29 MY-VMNAME_3-flat.vmdk
-rw------- 1 root root         586 Nov 18 15:29 MY-VMNAME_2.vmdk
-rw------- 1 root root 10737418240 Nov 18 15:29 MY-VMNAME_2-flat.vmdk
-rw------- 1 root root         560 Nov 18 15:29 MY-VMNAME_1.vmdk
-rw------- 1 root root    16801792 Nov 19 06:43 MY-VMNAME_3-000002-delta.vmdk
-rw------- 1 root root         327 Nov 19 06:45 MY-VMNAME_3-000002.vmdk
-rw------- 1 root root         608 Nov 26 15:11 MY-VMNAME.vmdk
-rw------- 1 root root    16801792 Dec  1 19:30 MY-VMNAME_2-000002-delta.vmdk
-rw------- 1 root root    16801792 Dec  1 19:30 MY-VMNAME_3-000003-delta.vmdk
-rw------- 1 root root         327 Dec  1 19:32 MY-VMNAME_3-000003.vmdk
-rw------- 1 root root         327 Dec  1 19:32 MY-VMNAME_2-000002.vmdk
-rw------- 1 root root    16801792 Dec  4 19:34 MY-VMNAME_2-000003-delta.vmdk
-rw------- 1 root root    16801792 Dec  4 19:35 MY-VMNAME_3-000004-delta.vmdk
-rw------- 1 root root         327 Dec  4 19:36 MY-VMNAME_3-000004.vmdk
-rw------- 1 root root         327 Dec  4 19:36 MY-VMNAME_2-000003.vmdk
-rw------- 1 root root     2621952 Dec  6 20:22 MY-VMNAME-ctk.vmdk
-rw------- 1 root root     2621952 Dec  6 20:22 MY-VMNAME_1-ctk.vmdk
-rw------- 1 root root      655872 Dec  6 20:22 MY-VMNAME_2-ctk.vmdk
-rw------- 1 root root      655872 Dec  6 20:22 MY-VMNAME_3-ctk.vmdk
-rw------- 1 root root 42949672960 Dec  7 00:01 MY-VMNAME_1-flat.vmdk
-rw------- 1 root root 42949672960 Dec  7 19:29 MY-VMNAME-flat.vmdk
-rw------- 1 root root     2621952 Dec  7 19:29 MY-VMNAME-000001-ctk.vmdk
-rw------- 1 root root         398 Dec  7 19:29 MY-VMNAME_1-000001.vmdk
-rw------- 1 root root     2621952 Dec  7 19:29 MY-VMNAME_1-000001-ctk.vmdk
-rw------- 1 root root         398 Dec  7 19:29 MY-VMNAME_3-000001.vmdk
-rw------- 1 root root         398 Dec  7 19:29 MY-VMNAME_2-000001.vmdk
-rw------- 1 root root      655872 Dec  7 19:29 MY-VMNAME_2-000001-ctk.vmdk
-rw------- 1 root root      655872 Dec  7 19:29 MY-VMNAME_3-000001-ctk.vmdk
-rw------- 1 root root    16801792 Dec  7 19:29 MY-VMNAME_3-000005-delta.vmdk
-rw------- 1 root root         327 Dec  7 19:31 MY-VMNAME_3-000005.vmdk
-rw------- 1 root root      655872 Dec  7 19:31 MY-VMNAME_2-000004-ctk.vmdk
-rw------- 1 root root         392 Dec  7 19:32 MY-VMNAME-000001.vmdk
-rw------- 1 root root    50356224 Jan 28 03:36 MY-VMNAME_3-000001-delta.vmdk
-rw------- 1 root root    33579008 Jan 29 13:12 MY-VMNAME_2-000001-delta.vmdk
-rw------- 1 root root  1442926592 Jan 30 11:20 MY-VMNAME_1-000001-delta.vmdk
-rw------- 1 root root  2315341824 Jan 30 11:21 MY-VMNAME-000001-delta.vmdk

Note that there are two types of vmdk file, the metadata and the flat, both have .vmdk as the extension. The easy way to tell the difference is the size, the metadata file will be a few hundred bytes, the flat file will be the size of the virtual disk, as shown above. You can view the contents of the metadata file using a command like cat. There will also be a load of .vmsn files, we’ll move those to a new folder called “old” to get them out of the way.

Create a folder called old, then move all the .vmsn files into it:

mkdir old
mv *.vmsn old

To determine which is the most recent actively in use snapshot vmdk use:

cat MY-VMNAME.vmx | grep .vmdk

Should show the same file as in the VM disk settings via the GUI:

[root@vfr3-uwe12 MY-VMNAME]# cat MY-VMNAME.vmx | grep .vmdk
scsi0:0.fileName = "MY-VMNAME-000001.vmdk"
scsi0:1.fileName = "MY-VMNAME_1-000001.vmdk"
scsi0:2.fileName = "MY-VMNAME_2-000001.vmdk"
scsi0:3.fileName = "MY-VMNAME_3-000001.vmdk"

Also, the date/time stamp will be most recent on the above listed files, other 0000xx vmdk files will have older date/time stamps. Note that vmdk files with underscores (_) in are for separate VM disks, in the example above all four VM disks are running on snapshots.

The primary disk is the one we’re going to resolve. Find the size of the disk, either look in the GUI or from an appropriate ls command output (above) look for the size of the appropriate “flat” vmdk file:

-rw------- 1 root root 42949672960 Dec  7 19:29 MY-VMNAME-flat.vmdk

You need to find a VMFS datastore that has at least that amount of space free to create a new vmdk file with the snapshots rolled up into it, so roughly 40GB in this case. A datastore on a different RAID Group or Pool is a good idea as that way you’re not reading and writing to the same disks.

Create a new folder on your chosen datastore:

mkdir "/vmfs/volumes/CLAR 159 FC/MY-VMNAME"

Issue the following command to start the consolidation process:

vmkfstools -i MY-VMNAME-000001.vmdk "/vmfs/volumes/CLAR 159 FC/MY-VMNAME/MY-VMNAME.vmdk"

You’ll see progress measured in percent:

Destination disk format: VMFS zeroedthick
Cloning disk 'MY-VMNAME-000001.vmdk'...
Clone: 100% done.

Note that you can specify the destination vmdk format by using the -d option if you don’t want the default (which is Eager Zeroed Thick), e.g. -d thin

How long this takes obviously depends on the size of the disk, the number of snapshots and the speed of your storage.

Move the vmdk files relating to the disk you’re working on into the old folder:

mv MY-VMNAME-*.vmdk old mv MY-VMNAME.vmdk old

Copy back the new consolidated vmdk files (you’ll get a <name>.vmdk, a <name>-flat.vmdk and a <name>-ctk.vmdk) from the other location:

cp "/vmfs/volumes/CLAR 159 FC/MY-VMNAME/*.*"

Note that this will not give you any progress indicator, but you can check progress by doing an appropriate ls command on the folder from a second SSH session and looking at the file sizes, which will increase as the file(s) copy.

Finally you need to update the vmx file to tell the VM to use the new consolidated disk files, the easiest way to do this is via the vSphere client GUI. Edit the settings of the VM, pick the disk you’re working on and note down the SCSI settings, e.g. 0:0. Then remove the disk, just choose Remove from virtual machine (the actual vmdk files are now in the old folder which we’ll delete from the command prompt later):

Then add the new consolidated vmdk back in again as a new disk, choosing to Use an existing virtual disk:

Browse to the consolidated vmdk file in the original VM folder. Fire up the VM and make sure it sees the disk properly, this is fairly easy if it’s the OS drive as it won’t boot if something’s wrong.

Once you’ve verified that the VM is functioning normally, delete the files in the “old” folder:

rm old/*.* -f

Then remove the old folder itself:

rmdir old

Job done.

This entry was posted in NetBackup, vSphere and tagged , , , , , . Bookmark the permalink.

3 Responses to Tidy/Remove VMware Snapshots – possibly created by NetBackup

  1. 5rudz says:

    Question: What causes this at the first place? How to prevent this occuring again. I’m experiencing the same problem with snapshot in my environment.

  2. Håkan says:

    Easier to just press “Consolidate” under the snapshot-menu (if you run V5), or just take a new snapshot, and then select Delete All in the snapshot manager.

    • rcmtech says:

      That sometimes doesn’t work though, and also you need to have enough space on the VMFS volume to hold both the original .vmdk, all the snapshots, and the new consolidated .vmdk, whereas the commands above allow you to write the new .vmdk to a new VMFS volume (which if it’s on different spindles will also speed the process somewhat as one VMFS will only be reading and the other only writing).

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s