Veritas NetBackup™ Troubleshooting Guide

Last Published:
Product(s): NetBackup (8.2)
  1. Introduction
    1.  
      NetBackup logging and status code information
    2.  
      Troubleshooting a problem
    3.  
      Problem report for Technical Support
    4.  
      About gathering information for NetBackup-Java applications
  2. Troubleshooting procedures
    1.  
      About troubleshooting procedures
    2. Troubleshooting NetBackup problems
      1.  
        Verifying that all processes are running on UNIX servers
      2.  
        Verifying that all processes are running on Windows servers
    3.  
      Troubleshooting installation problems
    4.  
      Troubleshooting configuration problems
    5.  
      Device configuration problem resolution
    6.  
      Testing the master server and clients
    7.  
      Testing the media server and clients
    8.  
      Resolving network communication problems with UNIX clients
    9.  
      Resolving network communication problems with Windows clients
    10. Troubleshooting vnetd proxy connections
      1.  
        vnetd proxy connection requirements
      2.  
        Where to begin to troubleshoot vnetd proxy connections
      3.  
        Verify that the vnetd process and proxies are active
      4.  
        Verify that the host connections are proxied
      5.  
        Test the vnetd proxy connections
      6.  
        Examine the log files of the connecting and accepting processes
      7.  
        Viewing the vnetd proxy log files
    11. Troubleshooting security certificate revocation
      1.  
        Troubleshooting cloud provider's revoked SSL certificate issues
      2.  
        Troubleshooting cloud provider's CRL download issues
      3.  
        How a host's CRL affects certificate revocation troubleshooting
      4.  
        NetBackup job fails because of revoked certificate or unavailability of CRLs
      5.  
        NetBackup job fails because of apparent network error
      6.  
        NetBackup job fails because of unavailable resource
      7.  
        Master server security certificate is revoked
      8.  
        Determining a NetBackup host's certificate state
      9.  
        Troubleshooting issues with external CA-signed certificate revocation
    12.  
      About troubleshooting networks and host names
    13. Verifying host name and service entries in NetBackup
      1.  
        Example of host name and service entries on UNIX master server and client
      2.  
        Example of host name and service entries on UNIX master server and media server
      3.  
        Example of host name and service entries on UNIX PC clients
      4.  
        Example of host name and service entries on UNIX server that connects to multiple networks
    14.  
      About the bpclntcmd utility
    15.  
      Using the Host Properties window to access configuration settings
    16.  
      Resolving full disk problems
    17. Frozen media troubleshooting considerations
      1.  
        Logs for troubleshooting frozen media
      2.  
        About the conditions that cause media to freeze
    18. Troubleshooting problems with the NetBackup web services
      1.  
        Viewing NetBackup web services logs
      2.  
        Troubleshooting web service issues after external CA configuration
    19.  
      Troubleshooting problems with the NetBackup web server certificate
    20. Resolving PBX problems
      1.  
        Checking PBX installation
      2.  
        Checking that PBX is running
      3.  
        Checking that PBX is set correctly
      4.  
        Accessing the PBX logs
      5.  
        Troubleshooting PBX security
      6.  
        Determining if the PBX daemon or service is available
    21. Troubleshooting problems with validation of the remote host
      1.  
        Viewing logs pertaining to host validation
      2.  
        Enabling insecure communication with NetBackup 8.0 and earlier hosts
      3.  
        Approving pending host ID-to-host name mappings
      4.  
        Clearing host cache
    22. About troubleshooting Auto Image Replication
      1. Troubleshooting Auto Image Replication
        1.  
          Targeted AIR trusted master server operation failed in case of external certificate configuration
      2.  
        About troubleshooting automatic import jobs
    23.  
      Troubleshooting network interface card performance
    24.  
      About SERVER entries in the bp.conf file
    25.  
      About unavailable storage unit problems
    26.  
      Resolving a NetBackup Administration operations failure on Windows
    27.  
      Resolving garbled text displayed in NetBackup Administration Console on a UNIX computer
    28.  
      Unable to logon to the NetBackup Administration Console after external CA configuration
    29.  
      Troubleshooting file-based external certificate issues
    30.  
      Troubleshooting Windows certificate store issues
    31.  
      Troubleshooting backup failures
    32.  
      Troubleshooting backup failure issues with NAT clients
    33.  
      Troubleshooting issues with the NetBackup Messaging Broker (or nbmqbroker) service
  3. Using NetBackup utilities
    1.  
      About NetBackup troubleshooting utilities
    2.  
      About the analysis utilities for NetBackup debug logs
    3.  
      About the Logging Assistant
    4.  
      About network troubleshooting utilities
    5. About the NetBackup support utility (nbsu)
      1.  
        Output from the NetBackup support utility (nbsu)
      2.  
        Example of a progress display for the NetBackup support utility (nbsu)
    6. About the NetBackup consistency check utility (NBCC)
      1.  
        Output from the NetBackup consistency check utility (NBCC)
      2.  
        Example of an NBCC progress display
    7.  
      About the NetBackup consistency check repair (NBCCR) utility
    8.  
      About the nbcplogs utility
    9. About the robotic test utilities
      1.  
        Robotic tests on UNIX
      2.  
        Robotic tests on Windows
  4. Disaster recovery
    1.  
      About disaster recovery
    2.  
      About disaster recovery requirements
    3.  
      Disaster recovery packages
    4.  
      About disaster recovery settings
    5.  
      Recommended backup practices
    6. About disk recovery procedures for UNIX and Linux
      1. About recovering the master server disk for UNIX and Linux
        1.  
          Recovering the master server when root is intact
        2.  
          Recovering the master server when the root partition is lost
      2.  
        About recovering the NetBackup media server disk for UNIX
      3.  
        Recovering the system disk on a UNIX client workstation
    7. About clustered NetBackup server recovery for UNIX and Linux
      1.  
        Replacing a failed node on a UNIX or Linux cluster
      2.  
        Recovering the entire UNIX or Linux cluster
    8. About disk recovery procedures for Windows
      1. About recovering the master server disk for Windows
        1.  
          Recovering the master server with Windows intact
        2.  
          Recovering the master server and Windows
      2.  
        About recovering the NetBackup media server disk for Windows
      3.  
        Recovering a Windows client disk
    9. About clustered NetBackup server recovery for Windows
      1.  
        Replacing a failed node on a Windows VCS cluster
      2.  
        Recovering the shared disk on a Windows VCS cluster
      3.  
        Recovering the entire Windows VCS cluster
    10.  
      Generating a certificate on a clustered master server after disaster recovery installation
    11.  
      About restoring disaster recovery package
    12.  
      About the DR_PKG_MARKER_FILE environment variable
    13.  
      Restoring disaster recovery package on Windows
    14.  
      Restoring disaster recovery package on UNIX
    15. About recovering the NetBackup catalog
      1.  
        About NetBackup catalog recovery on Windows computers
      2.  
        About NetBackup catalog recovery from disk devices
      3.  
        About NetBackup catalog recovery and symbolic links
      4. About NetBackup catalog recovery and OpsCenter
        1.  
          Specifying the NetBackup job ID number after a catalog recovery
      5.  
        NetBackup disaster recovery email example
      6. About recovering the entire NetBackup catalog
        1.  
          Recovering the entire NetBackup catalog using the Catalog Recovery Wizard
        2.  
          Recovering the entire NetBackup catalog using bprecover -wizard
      7. About recovering the NetBackup catalog image files
        1.  
          Recovering the NetBackup catalog image files using the Catalog Recovery Wizard
        2.  
          Recovering the NetBackup catalog image files using bprecover -wizard
      8. About recovering the NetBackup relational database
        1.  
          Recovering NetBackup relational database files from a backup
        2.  
          Recovering the NetBackup relational database files from staging
        3.  
          About processing the relational database in staging
      9.  
        Recovering the NetBackup catalog when NetBackup Access Control is configured
      10.  
        Recovering the NetBackup catalog from a nonprimary copy of a catalog backup
      11.  
        Recovering the NetBackup catalog without the disaster recovery file
      12.  
        Recovering a NetBackup user-directed online catalog backup from the command line
      13.  
        Restoring files from a NetBackup online catalog backup
      14.  
        Unfreezing the NetBackup online catalog recovery media
      15.  
        Steps to carry out when you see exit status 5988 during catalog recovery

Recovering the NetBackup catalog image files using bprecover -wizard

You must have root (administrative) privileges to perform this procedure.

You must be logged on to the master server on which you want to recover the catalog. The Catalog Recovery Wizard does not work after you perform a change server operation.

Note:

This wizard relies on the disaster recovery file that was generated during the catalog backup. The path to the disaster recovery file is specified in the catalog backup policy.

Note:

During the catalog recovery process, services may be shut down and restarted. If NetBackup is configured as a highly available application (cluster or global cluster), freeze the cluster before starting the recovery process to prevent a failover. Then unfreeze the cluster after the recovery process is complete.

Warning:

Do not run any client backups before you recover the NetBackup catalog.

To recover the catalog image files using bprecover -wizard

  1. If recovering the catalog to a new NetBackup installation, such as at a disaster recovery site, do the following:

  2. If the EMM server is on a different host than the master server, start the NetBackup services on that host by entering the following command:

    • On Windows:

      install_path\NetBackup\bin\bpup

    • On UNIX and Linux:

      /usr/openv/netbackup/bin/bp.start_all

  3. Start the NetBackup services on the master server by entering the following command:

    • On Windows:

      install_path\NetBackup\bin\bpup

    • On UNIX and Linux:

      /usr/openv/netbackup/bin/bp.start_all

  4. Start the bprecover wizard by entering the following command:

    bprecover -wizard

    The following is displayed:

    Welcome to the NetBackup Catalog Recovery Wizard!
    Please make sure the devices and media that contain catalog
    disaster recovery data are available
    Are you ready to continue?(Y/N)
  5. Enter Y to continue. You are prompted to enter the full path name of the disaster recovery file, as follows:

    Please specify the full pathname to the catalog disaster recovery file:

  6. Enter the fully qualified path name to the disaster recovery file for the backup that you want to restore. For example:

    /mnt/hdd2/netbackup/dr-file/Backup-Catalog_1318222845_FULL

    If the most recent catalog backup was an incremental backup, use the disaster recovery file from the incremental backup. (There is no need to first restore the full backup and then follow with the incremental backup.) Alternately, you can recover from earlier version of the catalog.

    If you specified a DR file for a full backup, a message similar to the following appears:

    vm2.example.com_1318222845
    All media resources were located
    
    Do you want to recover the entire NetBackup catalog? (Y/N)

    If you specified a DR file for an incremental backup, a message similar to the following is displayed:

    vm2.example.com_1318309224
    All media resources were located
    
    The last catalog backup in the catalog disaster recovery file is 
    an incremental.
    If no catalog backup images exist in the catalog,
    a PARTIAL catalog recovery will only restore the NetBackup catalog
    files backed up in that incremental backup.
    
    However, all of the catalog backup images up to the last full catalog
    backup are restored.  Then you can restore the remaining NetBackup 
    catalog files from the Backup, Archive, and Restore user interface.
    If catalog backup images already exist, all files that were included
    in the related set of catalog backups are restored.
    
    Do you want to recover the entire NetBackup catalog? (Y/N)
  7. Enter N to continue. The following is displayed:
    A PARTIAL catalog recovery includes the images directory
    containing the dotf files and staging of the NetBackup relational 
    database (NBDB) for further processing.
    
    Do you also want to include policy data?(Y/N)
  8. Enter Y or N to continue. The following is displayed:

    Do you also want to include licensing data?(Y/N)

  9. Enter Y or N to continue. The following is displayed:
    Catalog recovery is in progress. Please wait...
    
    Completed successful recovery of NBDB in staging directory on 
    vm2.example.com
    
    This portion of the catalog recovery has completed.
    Because this was a PARTIAL recovery of the NetBackup catalog,
    any remaining files included in the catalog backup can be restored
    using the Backup, Archive, and Restore user interface.
    
    The image metadata that is stored in NBDB in the staging directory
    can be exported using "cat_export -staging", and, imported using 
    "cat_import".
    
    The "nbdb_unload -staging" command can be used to unload one or more
    database tables from NBDB in the staging directory.
    
    The "nbdb_restore -recover -staging" command can be used to replace
    NBDB in the data directory with the contents from the staging 
    directory.
    
    WRN - NetBackup will not run scheduled backup jobs until NetBackup 
    is restarted.
    
    For more information, please review the log file:
    /usr/openv/netbackup/logs/user_ops/root/logs/Recover1318357550.log
    
  10. When the recovery job is finished, each image file is restored to the proper image directory and the configuration files are restored. If you chose to recover the policy data and licensing data, it is restored also.
  11. Export the image metadata from the relational database in the staging directory, as follows:

    cat_export -all -staging -source_master source-master-server-name

    The export is required so that the image metadata can be imported into the relational database. A catalog image file recovery does not recover the relational database.

  12. Import the image metadata into the relational database, as follows:

    cat_import -all -replace_destination

  13. If you recovered the catalog from a disk device, you may have to fix the disk media ID references in the image headers. The image headers were recovered from the catalog backup.

    See About NetBackup catalog recovery from disk devices.

    To fix the disk media IDs in the image headers, run the following command:

    nbcatsync -backupid image_id -prune_catalog

    Replace image_id with the ID of the catalog backup. The bprecover output contains the image ID of the catalog backup being restored. Alternatively, you can find the image ID of the catalog backup by examining the DR file.

  14. Before you continue, be aware of the following points:

    • If you recovered the catalog from removable media, NetBackup freezes the catalog media.

    • Before you restart NetBackup, Veritas recommends that you freeze the media that contains the backups more recent than the date of the catalog from which you recovered.

    • NetBackup does not run scheduled backup jobs until you stop and then restart NetBackup.

      You can submit backup jobs manually before you stop and restart NetBackup. However, if you do not freeze the media that contains the backups more recent than the date of the catalog from which you recovered, NetBackup may overwrite that media.

    • Because this operation is a partial recovery, you must recover the relational database portion of the catalog.

  15. Clean up whitelist cache for all hosts.
  16. Stop and restart NetBackup services on the master server and other hosts, as follows:

    • On Windows:

      install_path\NetBackup\bin\bpdown
      install_path\NetBackup\bin\bpup
    • On UNIX:

      /usr/openv/netbackup/bin/bp.kill_all
      /usr/openv/netbackup/bin/bp.start_all
  17. After the services are restarted, run the following command:
    • If NetBackup (or host ID-based) certificates are used in your NetBackup domain, do the following:

      On a non-clustered setup:

      Windows:

      install_path\netbackup\bin\nbcertcmd -renewcertificate

      UNIX:

      /usr/openv/netbackup/bin/nbcertcmd -renewcertificate

      On a clustered setup:

      Windows:

      install_path\netbackup\bin\nbcertcmd -renewcertificate -cluster

      UNIX:

      /usr/openv/netbackup/bin/nbcertcmd -renewcertificate -cluster
    • If external CA-signed certificates are used in your NetBackup domain, do the following:

      If external CA-signed certificates are used in your NetBackup domain, do the following On a non-clustered setup

      UNIX:

      /usr/openv/netbackup/bin/nbcertcmd -enrollCertificate

      Windows:

      install_path\netbackup\bin\nbcertcmd -enrollCertificate

      On a clustered setup:

      UNIX:

      /usr/openv/netbackup/bin/nbcertcmd -enrollCertificate -cluster

      Windows:

      install_path\netbackup\bin\nbcertcmd -enrollCertificate -cluster
  18. If the catalog recovery is part of a server recovery procedure, complete the remaining steps in the appropriate recovery procedure.

    This procedure can include the following tasks:

    • Importing the backups from the backup media into the catalog

    • Write protecting the media

    • Ejecting the media and setting it aside

    • Freezing the media