Veritas NetBackup™ for MongoDB Administrator's Guide

In a sharded MongoDB cluster with high availability that contains multiple mongos processes, before starting a restore and recover operation, only the mongos process on the restore destination for the Config Server Replica Set (CSRS) image should be running.

Manually stop any other mongos processes in the cluster before starting a restore and recover operation.

After recovery, reconfigure the mongos services to point to the recovered cluster.

If mongos process is not shut down on all nodes except one, the additional mongos processes might conflict with the restore and recover operation, causing the data that is restored to be inaccessible via a connection to mongos.

In case the mongos processes are not shutdown before starting the restore and recovery, then after recovery you must manually shutdown the stale mongos processes and then restart all the recovered mongod and mongos processes under cluster.

You must start the MongoDB processes with an absolute path to the configuration files. You must use the absolute paths for the certificate files and the CA file as well. You must specify the absolute paths for the CA file, PEM file and Key Files as well.

N/A

If the authentication type that was present during backup changes and you run a recovery job that requires a different authentication, the recovery process might fail.

Ensure that the authentication type during recovery remains the same as the type used during the backup.

After running a backup if you rename the volume group or the logical volume, the subsequent backup might fail.

N/A

During recovery, ensure that you select only one full backup image and its relevant subsequent incremental images. If you select more than one image, the recovery may fail as the restored data could be corrupted.

N/A

After your recover the MongoDB cluster, the cluster information for only the restored node is available.

After the recovery process is complete, manually add the secondary nodes to the cluster.

For more information, refer to the following article: add-members-to-the-replica-set

During the backup process, if the MongoDB import operation is running, it can become unresponsive. Avoid the MongoDB import operation during the backup or restore process.

N/A

During the restore process, "The restore was successfully initiated" popup is displayed, but the restore job does not start.

This issue occurs when you enter the Application Server in both the Source Client and Destination Client in the BAR UI.

Ensure that Source Client and Destination Client are entered correctly. The Source Client must be the Application Server and the Destination Client must be the backup host.

If your environment has DNAT, ensure that the backup host or the restore host and all the MongoDB nodes are in the same private network.

N/A

The NetBackup for MongoDB plug-in does not support the command line bprestore options -w and -print_jobid.

N/A

MongoDB restores are not supported from the backup hosts. All the restore operations for MongoDB must be initiated from the NetBackup master.

N/A

If your restore job submission is not displaying the restore job, check if your destination node has a MongoDB plug-in installed on it.

N/A

If you restore the MongoDB database to a non-LVM location and then try to take a backup from this non-LVM location, the backup fails.

Restore the data to an LVM location and then try to take a backup of the restored data.

The NetBackup for MongoDB plug-in does not support hard or soft links in the data path folders. Do not add any hard or soft links that point to locations in a different logical volume or a non-logical volume.

NetBackup cannot ensure that the data is consistent at the time of backups if you have hard or soft links in the data path folder. During the restore process, the hard or soft links are created as folders and not links.

N/A

When you cancel a child restore job during the MongoDB restore and recovery process, the thin client (mdbserver) is not removed immediately. The thin client is removed after the next restore operation.

N/A

MongoDB restore fails and displays error 2850.

Ensure that the destination host and port is valid and has the credentials configured using the tpconfig command and the credentials file. For more information, refer to the tar logs.

After recovery, the MongoDB shard node fails to restart manually and the following error is seen in the MongoDB logs:

NoSuchKey: Missing expected field "configsvrConnectionString"

On the MongoDB shard where the problem occurs, start MongoDB in the maintenance mode and run the following method on the system.version collection in the admin database:

use admin
db.system.version.deleteOne
( { _id: "minOpTimeRecovery" } )

In a restore and recover operation containing one or more replica sets, replica set members are restored to the replica set using the default "cfg.members[#].host" value provided by rs.config().

If this value was previously updated from the default value, after the restore and recover completes, this value may need to be updated (for example, from shortname to FQDN), to match the original configuration.

Workaround:

Backup jobs fail and the following error codes are displayed:

Ensure that the backup windows for incremental backups are different for the same MongoDB cluster. The backup windows must not overlap each other for incremental backups for the same MongoDB cluster.

Ensure that permissions are in place for the mdbserver location, oplog location, and snapshot mount location. For more information, See Using a non-root user as a host user.

In a sharded MongoDB cluster environment, a 112 error can indicate that the mongos process is not running on the client defined in the backup policy.

An error 112 can also indicate that same hosts names for multiple backup hosts are added to the BigData policy. Use unique host names for multiple backup hosts that are running the backup operations.

After a restore and recovery operation, if you try to stop and restart the mongod or mongos services (service mongod stop or service mongod restart), the commands fail.

This error occurs when the mongod or mongos processes are launched as service using the service or systemctl commands and not using a direct command.

Workaround:

Stop the mongod or mongos services using alternative methods. For example, mongod -f /etc/mongod.conf --shutdown or kill <PID>. After stopping the services, you can use the service or systemctl commands again.

Backup operation fails if a command that generates output in .bashrc is added.

Backup fails with error 6646 and displays the following error:

Error: Unable to communicate with the server.

Ensure that no output is generated by .bashrc (echo or any other output generating command). The output should not return STDERR or STDOUT when the shell is non-interactive.

When you select two full backup images and try to restore to a point-in-time image that is between the two full backup images, the latest full backup image is restored.

Workaround:

During the restore and recovery operation, do not select more than one full backup image.

For an effective point-in-time recovery, ensure that you run differential incremental backups of shorter duration.

Unable to see the restore job progress in the Activity Monitor.

Workaround:

For compound restore jobs that use a non-master server as the restore host, you must use the Update Task List button to display the restore job progress in the Activity Monitor.

Backup fails with the following error:

(6625) The backup host is either unauthorized to complete the operation or it is unable to establish a connection with the application server.

Workaround:

On the server where MongoDB is intalled, ensure that PasswordAuthentication is not disabled in /etc/ssh/sshd_config file.

Run the sudo service sshd restart command.

Backup fails with the following error:

(6646) Unable to communicate with the server.

Workaround:

Ensure that the backup host can access the defined port in mongodb.conf file or the default mdbserver_port (11000).

The following error is displayed in the mdbserver logs:

error-sudo: sorry, you must have a tty to run sudo

Workaround:

The nbaapireq_handler log folder is not created on a Flex Container, even after running the mklogdir command.

Workaround:

When a Flex Appliance is upgraded from version 8.1.2 to 8.2 and the Flex media server is used as backup host, then for logging the MongoDB plug-in restore logs create the nbaapireq_handler folder in the /usr/openv/netbackup/logs/ directory.

MongoDB Restore fails with the error 2850

The target database path does not exist and there are insufficient permissions for the non-root user.

Workaround:

Ensure that the target database path exists and there are sufficient permissions for the non-root user.

The snapshot size as described by the free_space_percentage_snapshot parameter must be set according the MongoDB cluster size and must be large enough. If these criteria are not met, the backup fails and displays the following error:

invalid command parameter (20)

Validate the free_space_percentage_snapshot value with the MongoDB cluster.

Backup fails with the following error:

(13) file read failed for Media

Ensure that the:

The mdb_progress_loglevel parameter is missing from the MongoDB configuration tool.

To modify the mdb_progress_loglevel parameter, update the mongodb.conf file after it is created by the MongoDB configuration tool.

For more information, refer to the MongoDB Administrator's Guide.

Snapshots are not deleted and stale mdbserver instances are seen. This scenario might cause Cannot lstat errors during backup and partially successful backups.

Change the configuration settings for the following parameters in the mongodb.conf file:

Set the values such that the stale snapshots and stale instances of mdbserver are cleared before the next full or incremental backup schedule.

If the backup host has NetBackup version earlier than 8.3 and master and media server have the latest version of NetBackup, the following invalid error codes can be seen for various scenarios:

13302, 13303, 13304, 13305, 13306, 13307, 13308, 13309, 13310, 13311, 13312, 13313, 13314, 13315

Workaround:

Refer to the following list of corresponding actual error codes if you see the invalid error codes for the actual scenarios and recommended actions:

For detailed information and recommended actions, refer to the NetBackup Status Codes Reference Guide.

Restore button in the NetBackup BAR UI can get disabled for the imported MongoDB backup images.

Workaround:

If you import the images to the same NetBackup master server that was originally used to back them up, use either of the following methods:

If you import the images to a different NetBackup master server than the one that was originally used to back them up, use the bprestore command to run the restore operation.

Recovery operation fails on an alternate, sharded MongoDB cluster. The following error is displayed:

Unable to find the configuration parameter. (6661)

This issue occurs during an alternate cluster recovery because the pre-recovery check is unable to find the mongos port for the alternate cluster in the mongodb.conf file. This is because of the way the MongoDB configuration tool creates the mongodb.conf file when you add the alternate MongDB cluster details using the Update option from the tool.

Workaround:

Before you start the recovery process, update the mongodb.conf file to separate the alternate cluster from the original cluster.

For example:

Existing mongodb.conf file:

 "application_servers":
   {
    "original.mongodb.cluster.com:26050":
		 {
     "alternate_config_server":
			  [
       {
         "hostname:port": "alt.mongodb.cluster.com:26000",
         "mongos_port": "26001"
       }
      ],
    "mongos_port": "26051"
    }
   }

Suggested updated to the mongodb.conf file:

"application_servers": 
   {
    "original.mongodb.cluster.com:26050":
   {
      "mongos_port": "26051"
   },
      "alt.mongodb.cluster.com:26000":
   {
   "mongos_port": "26001"
   }
   }

The MUI tool displays the following error:

Unable to delete configuration.

Recommended Action:

Work Around:

Delete the <hostname-port>.conf file manually from /usr/openv/var/global.