Nuts and bolts in NetBackup for VMware: Understanding V-Ray vision through backup process flow

Now that we know how VM discovery job works for a VMware Intelligent Policy (VIP), let us move on to the actual backup job. The process flow described here is the same for both VIP and browse-and-select type policies unless specified otherwise.

As we did in the previous blog, let us assume that the NetBackup master server, media server and VMware backup host are three different systems.

1. NetBackup Policy Execution Manager (nbpem) triggers a backup job once the policy, schedule, virtual machine combination is due and backup window is open. It works in conjunction with NetBackup Job Manager (nbjm), NetBackup Resource Broker (nbrb) and NetBackup Enterprise Media Manager (nbemm) to identify the resources (media server, storage unit etc.) for backup. This resource identification process is well detailed in NetBackup Documentation, hence let is not spend time on this. I will focus on things specific to vStorage backups.
2. If this is VIP, the VMware resource limits specified by the backup administrator is also taken into account at this time. NetBackup for VMware (NetBackup 7.1 and NetBackup 5200 series appliances with software version 2.0) provides the flexibility to throttle VMware resources used in vSphere environment. For example, you can specify that you don’t want more than 4 concurrent backup jobs running from a vSphere datastore. This level of control let the administrators to tune the number of backups such that user and application experience for vSphere platform is minimally affected on account of running backup jobs.
3. Once the backup resources are identified and VMware resource limits are set, nbjm contacts the selected media server and starts NetBackup Backup and Restore Manager (bpbrm*) on it. You will see a snapshot job (commonly referred to as parent job) going Active in the Activity monitor at this time.
4. The bpbrm process on media server contacts the VMware backup host and starts NetBackup Frozen Image Service (bpfis) process.
5. The bpfis process is in charge of all things snapshot in NetBackup. This process is capable of creating snapshots by loading respective shared libraries (UNIX/Linux) or dynamic link libraries (Windows) coded to handle various kinds of snapshots. In our example, it loads the code related to creating vSphere snapshots. This dynamic link library implements vStorage APIs for Data Protection (vADP) calls.
6. bpfis armed with vADP contacts the vSphere host (vCenter or ESX/ESXi host for which credentials are stored in NetBackup database) and initiates the snapshot for the VM in question. If there are multiple VMs, a bpfis is started for each VM so that the snapshot operations occur in parallel. As I mentioned in step 2, you can control the number of concurrent snapshots by setting VMware Resource Limits in NetBackup when using VIP. The way bpfis contacts vSphere host is via standard SSL port (default is 443).
7. Once the snapshot is created, bpfis who initiated the snapshot reports back to bpbrm on media server. The snapshot job (parent job) is moved to Done state in Activity Monitor.
8. bpbrm now initiates two things in parallel. It contacts VMware backup host again and starts NetBackup Backup and Archive (bpbkar) process. At the same time it also starts NetBackup Backup Tape Manager (bptm)** to prepare the storage (tape or disk) in order to receive backup stream from VMware Backup Host. This time the backup job (commonly known as the child job) shows up in Activity Monitor.
9. Let us spend some time with bpbkar for now. It is imperative to understand what happens here because this is an area where none of the competitors can match NetBackup’s patent pending granular recovery technology. This is part of Symantec’s V-Ray vision to provide unique visibility into virtual machines. Many vendors can claim that they can restore full virtual machine or individual files from the same backup. If you ask them how it is implemented, you are likely to see their weakness quickly.

The bpbkar process loads VERITAS Mapping Services (VxMS) which among many other things, also implement VMware Disk Development Kit (VDDK) APIs. The VDDK APIs are used for reading from vSphere datastore. The real beauty of V-Ray comes from VxMS which has patent pending technology to map the stream during runtime and identify what is actually inside the VMDK file being read.

Here are some of the salient aspects of Symantec V-Ray within VxMS:

9.a) You are generating the catalog of all the files inside the VMDK (supported for VMDK files from both Windows and Linux VMs). Remember that this is not a two-pass process, the operation occurs while backup data is being streamed. This catalog information is send to master server via bpbrm process on media server.

9.b) As you are looking at file system inode level, you also get to identify unused and deleted blocks. This is quite powerful vision into the VM data. If the application on VM had allocated 1TB of space for a file, of which just 100GB is currently used, the backup stream would include just that 100GB. Similarly if you deleted a 1TB file that was fully allocated in the past, VxMS knows to skip the deleted blocks (unless the blocks are now allocated for a new file) from the backup stream. This optimization not only speeds up the backup stream, but it reduces needed storage even when deduplication is not enabled.

9.c) If source side deduplication feature is enabled, the VMware backup host does the deduplication. The NetBackup deduplication plug-in is not seeing just a giant VMDK file, it understands the mapping info generated by VxMS and hence it sees (with the V-Ray vision!) the actual files in the file system within the VMDK. This V-Ray vision is established by NetBackup deduplication plug-in loading a dedicated stream handler that understands the mapping info generated by VxMS.

9.d) Remember that all these operations occur at the VMware backup host. Your ESX resources (and hence the resources from VMs) are not used in these cases. This is true offhost backup with no burden on your production vSphere. Compare this with other vendors running index engines on VMs or helper VMs and including unallocated blocks and deleted files in all backups. Even the source side deduplication is happening in an offhost system in case of NetBackup for VMware.

1. Now let us go back to media server, we mentioned that bpbrm had started bptm. (Technically speaking, bptm forks another bptm as well. One is in charge of preparing the storage and writing backup streams to it and another one receives backup stream from VMware backup host.  As this part is covered well in NetBackup Documentation, I am not distinguishing various bptm processes here.) The bptm process prepares the backup storage to receive data. If the storage unit allocated is tape based (known as Media Manager Storage Unit), it works with Media Manager daemons to get appropriate tape mounted into a drive. If the storage is BasicDisk, AdvancedDisk or third party OpenStorage; appropriate plug-ins are loaded by bptm. The backup stream sent by bpbkar (see step 9) is directed to the storage by bptm. In order to understand another aspect of V-Ray, we need to talk about a special case here when the storage chosen is a NetBackup Deduplication Pool.

For those new to NetBackup Deduplication, I would recommend reading this blog series. If you chose not to do deduplication at VMware Backup host (see 9.c), you are not losing V-Ray vision. The deduplication plug-in for media server deduplication pool, NetBackup 5200 series appliances and NetBackup 5000 series appliances also have V-Ray vision by making use of the dedupe stream handler which sees the actual files within a VMDK in backup streams. In fact, this V-Ray vision is one of the key differentiators for NetBackup deduplication when compared with third party target-only deduplication devices.

1. Once the backup stream is complete, bpbkar on VMware Backup host notifies bpbrm on media server and exits.
2. bpbrm on media server contacts VMware backup host again and starts a bpfis process to release the snapshot.
3. bpfis on VMware backup hosts loads the dynamic link library implementing vADP. It contacts the vSphere host and initiates snapshot deletion.
4. Once the snapshot deletion is over, bpfis on VMware backup host notifies bpbrm on media server and exits.
5. bpbrm waits for bptm to finish the final housekeeping tasks. Once bptm processes exit, bpbrm notifies nbjm on master server.
6. nbjm works in conjunction with nbrb and nbemm to release the resources used for backups. In parallel, it also notifies nbpem about the status of the job.
7. nbpem calculates when the backup for the VM will become due again and sets it timer accordingly. The backup job now moves to Done state in Activity Monitor.

*For those new to NetBackup, the ‘bp’ in bpbrm, bptm, bpbkar etc. stands for ‘backupplus’, the old name of NetBackup.

**Although ‘t’ in bptm stands for tape, bptm has undergone significant overhaul during NetBackup 6.x. This process now handles tape as well as disk destinations.

Back to Nuts and Bolts in NetBackup for VMware series