Veritas NetBackup™ Troubleshooting Guide

Last Published:
Product(s): NetBackup (8.2)
  1. Introduction
    1.  
      NetBackup logging and status code information
    2.  
      Troubleshooting a problem
    3.  
      Problem report for Technical Support
    4.  
      About gathering information for NetBackup-Java applications
  2. Troubleshooting procedures
    1.  
      About troubleshooting procedures
    2. Troubleshooting NetBackup problems
      1.  
        Verifying that all processes are running on UNIX servers
      2.  
        Verifying that all processes are running on Windows servers
    3.  
      Troubleshooting installation problems
    4.  
      Troubleshooting configuration problems
    5.  
      Device configuration problem resolution
    6.  
      Testing the master server and clients
    7.  
      Testing the media server and clients
    8.  
      Resolving network communication problems with UNIX clients
    9.  
      Resolving network communication problems with Windows clients
    10. Troubleshooting vnetd proxy connections
      1.  
        vnetd proxy connection requirements
      2.  
        Where to begin to troubleshoot vnetd proxy connections
      3.  
        Verify that the vnetd process and proxies are active
      4.  
        Verify that the host connections are proxied
      5.  
        Test the vnetd proxy connections
      6.  
        Examine the log files of the connecting and accepting processes
      7.  
        Viewing the vnetd proxy log files
    11. Troubleshooting security certificate revocation
      1.  
        Troubleshooting cloud provider's revoked SSL certificate issues
      2.  
        Troubleshooting cloud provider's CRL download issues
      3.  
        How a host's CRL affects certificate revocation troubleshooting
      4.  
        NetBackup job fails because of revoked certificate or unavailability of CRLs
      5.  
        NetBackup job fails because of apparent network error
      6.  
        NetBackup job fails because of unavailable resource
      7.  
        Master server security certificate is revoked
      8.  
        Determining a NetBackup host's certificate state
      9.  
        Troubleshooting issues with external CA-signed certificate revocation
    12.  
      About troubleshooting networks and host names
    13. Verifying host name and service entries in NetBackup
      1.  
        Example of host name and service entries on UNIX master server and client
      2.  
        Example of host name and service entries on UNIX master server and media server
      3.  
        Example of host name and service entries on UNIX PC clients
      4.  
        Example of host name and service entries on UNIX server that connects to multiple networks
    14.  
      About the bpclntcmd utility
    15.  
      Using the Host Properties window to access configuration settings
    16.  
      Resolving full disk problems
    17. Frozen media troubleshooting considerations
      1.  
        Logs for troubleshooting frozen media
      2.  
        About the conditions that cause media to freeze
    18. Troubleshooting problems with the NetBackup web services
      1.  
        Viewing NetBackup web services logs
      2.  
        Troubleshooting web service issues after external CA configuration
    19.  
      Troubleshooting problems with the NetBackup web server certificate
    20. Resolving PBX problems
      1.  
        Checking PBX installation
      2.  
        Checking that PBX is running
      3.  
        Checking that PBX is set correctly
      4.  
        Accessing the PBX logs
      5.  
        Troubleshooting PBX security
      6.  
        Determining if the PBX daemon or service is available
    21. Troubleshooting problems with validation of the remote host
      1.  
        Viewing logs pertaining to host validation
      2.  
        Enabling insecure communication with NetBackup 8.0 and earlier hosts
      3.  
        Approving pending host ID-to-host name mappings
      4.  
        Clearing host cache
    22. About troubleshooting Auto Image Replication
      1. Troubleshooting Auto Image Replication
        1.  
          Targeted AIR trusted master server operation failed in case of external certificate configuration
      2.  
        About troubleshooting automatic import jobs
    23.  
      Troubleshooting network interface card performance
    24.  
      About SERVER entries in the bp.conf file
    25.  
      About unavailable storage unit problems
    26.  
      Resolving a NetBackup Administration operations failure on Windows
    27.  
      Resolving garbled text displayed in NetBackup Administration Console on a UNIX computer
    28.  
      Unable to logon to the NetBackup Administration Console after external CA configuration
    29.  
      Troubleshooting file-based external certificate issues
    30.  
      Troubleshooting Windows certificate store issues
    31.  
      Troubleshooting backup failures
    32.  
      Troubleshooting backup failure issues with NAT clients
    33.  
      Troubleshooting issues with the NetBackup Messaging Broker (or nbmqbroker) service
  3. Using NetBackup utilities
    1.  
      About NetBackup troubleshooting utilities
    2.  
      About the analysis utilities for NetBackup debug logs
    3.  
      About the Logging Assistant
    4.  
      About network troubleshooting utilities
    5. About the NetBackup support utility (nbsu)
      1.  
        Output from the NetBackup support utility (nbsu)
      2.  
        Example of a progress display for the NetBackup support utility (nbsu)
    6. About the NetBackup consistency check utility (NBCC)
      1.  
        Output from the NetBackup consistency check utility (NBCC)
      2.  
        Example of an NBCC progress display
    7.  
      About the NetBackup consistency check repair (NBCCR) utility
    8.  
      About the nbcplogs utility
    9. About the robotic test utilities
      1.  
        Robotic tests on UNIX
      2.  
        Robotic tests on Windows
  4. Disaster recovery
    1.  
      About disaster recovery
    2.  
      About disaster recovery requirements
    3.  
      Disaster recovery packages
    4.  
      About disaster recovery settings
    5.  
      Recommended backup practices
    6. About disk recovery procedures for UNIX and Linux
      1. About recovering the master server disk for UNIX and Linux
        1.  
          Recovering the master server when root is intact
        2.  
          Recovering the master server when the root partition is lost
      2.  
        About recovering the NetBackup media server disk for UNIX
      3.  
        Recovering the system disk on a UNIX client workstation
    7. About clustered NetBackup server recovery for UNIX and Linux
      1.  
        Replacing a failed node on a UNIX or Linux cluster
      2.  
        Recovering the entire UNIX or Linux cluster
    8. About disk recovery procedures for Windows
      1. About recovering the master server disk for Windows
        1.  
          Recovering the master server with Windows intact
        2.  
          Recovering the master server and Windows
      2.  
        About recovering the NetBackup media server disk for Windows
      3.  
        Recovering a Windows client disk
    9. About clustered NetBackup server recovery for Windows
      1.  
        Replacing a failed node on a Windows VCS cluster
      2.  
        Recovering the shared disk on a Windows VCS cluster
      3.  
        Recovering the entire Windows VCS cluster
    10.  
      Generating a certificate on a clustered master server after disaster recovery installation
    11.  
      About restoring disaster recovery package
    12.  
      About the DR_PKG_MARKER_FILE environment variable
    13.  
      Restoring disaster recovery package on Windows
    14.  
      Restoring disaster recovery package on UNIX
    15. About recovering the NetBackup catalog
      1.  
        About NetBackup catalog recovery on Windows computers
      2.  
        About NetBackup catalog recovery from disk devices
      3.  
        About NetBackup catalog recovery and symbolic links
      4. About NetBackup catalog recovery and OpsCenter
        1.  
          Specifying the NetBackup job ID number after a catalog recovery
      5.  
        NetBackup disaster recovery email example
      6. About recovering the entire NetBackup catalog
        1.  
          Recovering the entire NetBackup catalog using the Catalog Recovery Wizard
        2.  
          Recovering the entire NetBackup catalog using bprecover -wizard
      7. About recovering the NetBackup catalog image files
        1.  
          Recovering the NetBackup catalog image files using the Catalog Recovery Wizard
        2.  
          Recovering the NetBackup catalog image files using bprecover -wizard
      8. About recovering the NetBackup relational database
        1.  
          Recovering NetBackup relational database files from a backup
        2.  
          Recovering the NetBackup relational database files from staging
        3.  
          About processing the relational database in staging
      9.  
        Recovering the NetBackup catalog when NetBackup Access Control is configured
      10.  
        Recovering the NetBackup catalog from a nonprimary copy of a catalog backup
      11.  
        Recovering the NetBackup catalog without the disaster recovery file
      12.  
        Recovering a NetBackup user-directed online catalog backup from the command line
      13.  
        Restoring files from a NetBackup online catalog backup
      14.  
        Unfreezing the NetBackup online catalog recovery media
      15.  
        Steps to carry out when you see exit status 5988 during catalog recovery

Troubleshooting backup failure issues with NAT clients

Backup fails with the following error: bpbrm (pid=31553) cannot send mail because BPCD on host exited with status 21: socket open failed

This issue may occur because of one of the following reasons:

  • Media server cannot connect to the NetBackup Messaging Broker (or nbmqbroker) service.

  • The nbmqbroker service may not be up and running on the master server.

  • The NAT client is not configured to accept the reverse connection.

  • The client is not a NAT client.

  • The client is 8.1.2 or earlier.

  • Port configuration for the nbmqbroker service is updated.

  • The master server services are restarted.

Cause 1

Media server cannot connect to the nbmqbroker service.

Cause 2

The nbmqbroker service may not be up and running on the master server.

Cause 1 and Cause 2 have the same solution as follows:

  • Check the bpbrm logs on the media server at Install_Path/logs/bpbrm.

  • Check the nbmqbroker log file at:

    UNIX: /usr/openv/mqbroker/logs

    Windows: Install_Path/mqbroker/logs

  • Ensure that the nbmqbroker service is running on the master server. Use the following commands:

    • Run the bpps command.

    • Run the bptestbpcd -host hostname command from the master or media server and check the admin logs at Install_Path/logs/admin.

Cause 3: The NAT client is not configured to accept the reverse connection

Do the following:

  • Check the subscriber logs at:

    UNIX: usr/openv/logs/nbsubscriber

    Windows: Install_Path/logs/nbsubscriber

  • Check the vnetd logs at Install_Path/logs/vnetd.

  • Run the bptestbpcd -host hostname command on the master or media server and check the admin logs at Install_Path/logs/admin.

  • Run the nbmqutil -publish -master hostname -message message_text -remoteHost hostname command.

  • Ensure that the ACCEPT_REVERSE_CONNECTION configuration option is set to TRUE using the nbgetconfig command.

  • Check the subscriber service is running on the NAT client by running the bpps command.

Cause 4: The client is not a NAT client

Do the following:

Ensure that the ENABLE_DIRECT_CONNECTION configuration option is set to TRUE on the master or media server using the nbgetconfig command.

Cause 5: The client is 8.1.2 or earlier

Do the following:

Ensure that the ENABLE_DIRECT_CONNECTION configuration option is set to TRUE on the master or media server using the nbgetconfig command.

Cause 6: Port configuration for the nbmqbroker service is updated

Do the following:

  • Wait until the cache is cleared.

  • Clear host cache on the media server using the bpclntcmd -clear_host_cache command.

Cause 7: The master server services are restarted

Do the following:

  • Check the subscriber service logs at:

    UNIX: usr/openv/logs/nbsubscriber

    Windows: Install_Path/logs/nbsubscriber

  • Wait until the subscriber service starts on the client.

  • Restart the subscriber service.

Backup fails with the following error: bpbrm (pid=9880) bpcd on host exited with status 48: client hostname could not be found

This issue may occur because of one of the following reasons:

  • The NAT client's host name is not mapped to it's host ID.

  • Host ID that is associated with the client is null or is not valid.

Do the following:

  • Check the bpbrm logs at Install_Path/logs/bpbrm

  • Check the existing host ID-to-host name mapping of the client by running the Install_Path/bin/admincmd/nbhostmgmt -li -json command on the master or media server.

  • If the client name is not mapped to the host ID, add a new name for the client and map it to existing host ID using the Install_Path/bin/admincmd/nbhostmgmt -add -hostid hostid -mappingname hostname command.

  • Clear host cache on the client using Install_Path/bin/bpclntcmd -clear_host_cache.

Backup takes too long to complete

This issue may occur because of one of the following reasons:

  • Client's configuration file (bp.conf file on UNIX or Windows registry) contains wrong media server entry.

  • The ENABLE_DATA_CHANNEL_ENCRYPTION option is set to TRUE on the client.

Cause 1: Client's configuration file contains wrong media server entry

Do the following:

  • Run the the Install_Path/bin/admincmd/bptestbpcd -host hostname from the master or media server and check the admin logs at Install_Path/logs/admin.

  • Add the media server name in the /etc/hosts file on the client.

  • Add the media server name in the configuration file on the client using the nbsetconfig command.

Cause 2: The ENABLE_DATA_CHANNEL_ENCRYPTION option is enabled

Do the following:

  • Set the ENABLE_DATA_CHANNEL_ENCRYPTION to FALSE using the nbsetconfig command.

Backup fails as the job is hung and no new job is triggered for the policy

This issue may occur because of the following reason:

  • The NAT client awaits an incoming message, but the nbmqbroker service has closed the client connection, and client cannot detect the closed connection.

Do the following:

  • Check the client logs to see if it contains the following message:

    Trying to get Message from MQ Broker:[master server name]     
  • Check the current heartbeat value that is set for the SUBSCRIBER_HEARTBEAT_TIMEOUT configuration option on the server. Use the nbgetconfig command.

  • Set the SUBSCRIBER_HEARTBEAT_TIMEOUT option value to minimum so that the client can detect a closed connection.

  • Restart the subscriber service on the client.

Backup or restore jobs fail after CLIENT_CONNECT_TIMEOUT

This issue may occur because of the following reason:

  • Subscriber was not able to reverse connect to media server.

  • Message is delivered by publisher but subscriber did not receive the message.

Do the following:

  • Check the subscriber service logs to ensure that the subscriber service is able to connect to the PBX Transient ID.

  • Check the subscriber service logs to ensure that the publisher message is delivered to the subscriber.

    Log message:

    Got Message from MQ Broker:[<message>] with return:<status code> total timeout,reset:<timeout reset>