Veritas NetBackup™ Troubleshooting Guide

Last Published:
Product(s): NetBackup (8.2)
  1. Introduction
    1.  
      NetBackup logging and status code information
    2.  
      Troubleshooting a problem
    3.  
      Problem report for Technical Support
    4.  
      About gathering information for NetBackup-Java applications
  2. Troubleshooting procedures
    1.  
      About troubleshooting procedures
    2. Troubleshooting NetBackup problems
      1.  
        Verifying that all processes are running on UNIX servers
      2.  
        Verifying that all processes are running on Windows servers
    3.  
      Troubleshooting installation problems
    4.  
      Troubleshooting configuration problems
    5.  
      Device configuration problem resolution
    6.  
      Testing the master server and clients
    7.  
      Testing the media server and clients
    8.  
      Resolving network communication problems with UNIX clients
    9.  
      Resolving network communication problems with Windows clients
    10. Troubleshooting vnetd proxy connections
      1.  
        vnetd proxy connection requirements
      2.  
        Where to begin to troubleshoot vnetd proxy connections
      3.  
        Verify that the vnetd process and proxies are active
      4.  
        Verify that the host connections are proxied
      5.  
        Test the vnetd proxy connections
      6.  
        Examine the log files of the connecting and accepting processes
      7.  
        Viewing the vnetd proxy log files
    11. Troubleshooting security certificate revocation
      1.  
        Troubleshooting cloud provider's revoked SSL certificate issues
      2.  
        Troubleshooting cloud provider's CRL download issues
      3.  
        How a host's CRL affects certificate revocation troubleshooting
      4.  
        NetBackup job fails because of revoked certificate or unavailability of CRLs
      5.  
        NetBackup job fails because of apparent network error
      6.  
        NetBackup job fails because of unavailable resource
      7.  
        Master server security certificate is revoked
      8.  
        Determining a NetBackup host's certificate state
      9.  
        Troubleshooting issues with external CA-signed certificate revocation
    12.  
      About troubleshooting networks and host names
    13. Verifying host name and service entries in NetBackup
      1.  
        Example of host name and service entries on UNIX master server and client
      2.  
        Example of host name and service entries on UNIX master server and media server
      3.  
        Example of host name and service entries on UNIX PC clients
      4.  
        Example of host name and service entries on UNIX server that connects to multiple networks
    14.  
      About the bpclntcmd utility
    15.  
      Using the Host Properties window to access configuration settings
    16.  
      Resolving full disk problems
    17. Frozen media troubleshooting considerations
      1.  
        Logs for troubleshooting frozen media
      2.  
        About the conditions that cause media to freeze
    18. Troubleshooting problems with the NetBackup web services
      1.  
        Viewing NetBackup web services logs
      2.  
        Troubleshooting web service issues after external CA configuration
    19.  
      Troubleshooting problems with the NetBackup web server certificate
    20. Resolving PBX problems
      1.  
        Checking PBX installation
      2.  
        Checking that PBX is running
      3.  
        Checking that PBX is set correctly
      4.  
        Accessing the PBX logs
      5.  
        Troubleshooting PBX security
      6.  
        Determining if the PBX daemon or service is available
    21. Troubleshooting problems with validation of the remote host
      1.  
        Viewing logs pertaining to host validation
      2.  
        Enabling insecure communication with NetBackup 8.0 and earlier hosts
      3.  
        Approving pending host ID-to-host name mappings
      4.  
        Clearing host cache
    22. About troubleshooting Auto Image Replication
      1. Troubleshooting Auto Image Replication
        1.  
          Targeted AIR trusted master server operation failed in case of external certificate configuration
      2.  
        About troubleshooting automatic import jobs
    23.  
      Troubleshooting network interface card performance
    24.  
      About SERVER entries in the bp.conf file
    25.  
      About unavailable storage unit problems
    26.  
      Resolving a NetBackup Administration operations failure on Windows
    27.  
      Resolving garbled text displayed in NetBackup Administration Console on a UNIX computer
    28.  
      Unable to logon to the NetBackup Administration Console after external CA configuration
    29.  
      Troubleshooting file-based external certificate issues
    30.  
      Troubleshooting Windows certificate store issues
    31.  
      Troubleshooting backup failures
    32.  
      Troubleshooting backup failure issues with NAT clients
    33.  
      Troubleshooting issues with the NetBackup Messaging Broker (or nbmqbroker) service
  3. Using NetBackup utilities
    1.  
      About NetBackup troubleshooting utilities
    2.  
      About the analysis utilities for NetBackup debug logs
    3.  
      About the Logging Assistant
    4.  
      About network troubleshooting utilities
    5. About the NetBackup support utility (nbsu)
      1.  
        Output from the NetBackup support utility (nbsu)
      2.  
        Example of a progress display for the NetBackup support utility (nbsu)
    6. About the NetBackup consistency check utility (NBCC)
      1.  
        Output from the NetBackup consistency check utility (NBCC)
      2.  
        Example of an NBCC progress display
    7.  
      About the NetBackup consistency check repair (NBCCR) utility
    8.  
      About the nbcplogs utility
    9. About the robotic test utilities
      1.  
        Robotic tests on UNIX
      2.  
        Robotic tests on Windows
  4. Disaster recovery
    1.  
      About disaster recovery
    2.  
      About disaster recovery requirements
    3.  
      Disaster recovery packages
    4.  
      About disaster recovery settings
    5.  
      Recommended backup practices
    6. About disk recovery procedures for UNIX and Linux
      1. About recovering the master server disk for UNIX and Linux
        1.  
          Recovering the master server when root is intact
        2.  
          Recovering the master server when the root partition is lost
      2.  
        About recovering the NetBackup media server disk for UNIX
      3.  
        Recovering the system disk on a UNIX client workstation
    7. About clustered NetBackup server recovery for UNIX and Linux
      1.  
        Replacing a failed node on a UNIX or Linux cluster
      2.  
        Recovering the entire UNIX or Linux cluster
    8. About disk recovery procedures for Windows
      1. About recovering the master server disk for Windows
        1.  
          Recovering the master server with Windows intact
        2.  
          Recovering the master server and Windows
      2.  
        About recovering the NetBackup media server disk for Windows
      3.  
        Recovering a Windows client disk
    9. About clustered NetBackup server recovery for Windows
      1.  
        Replacing a failed node on a Windows VCS cluster
      2.  
        Recovering the shared disk on a Windows VCS cluster
      3.  
        Recovering the entire Windows VCS cluster
    10.  
      Generating a certificate on a clustered master server after disaster recovery installation
    11.  
      About restoring disaster recovery package
    12.  
      About the DR_PKG_MARKER_FILE environment variable
    13.  
      Restoring disaster recovery package on Windows
    14.  
      Restoring disaster recovery package on UNIX
    15. About recovering the NetBackup catalog
      1.  
        About NetBackup catalog recovery on Windows computers
      2.  
        About NetBackup catalog recovery from disk devices
      3.  
        About NetBackup catalog recovery and symbolic links
      4. About NetBackup catalog recovery and OpsCenter
        1.  
          Specifying the NetBackup job ID number after a catalog recovery
      5.  
        NetBackup disaster recovery email example
      6. About recovering the entire NetBackup catalog
        1.  
          Recovering the entire NetBackup catalog using the Catalog Recovery Wizard
        2.  
          Recovering the entire NetBackup catalog using bprecover -wizard
      7. About recovering the NetBackup catalog image files
        1.  
          Recovering the NetBackup catalog image files using the Catalog Recovery Wizard
        2.  
          Recovering the NetBackup catalog image files using bprecover -wizard
      8. About recovering the NetBackup relational database
        1.  
          Recovering NetBackup relational database files from a backup
        2.  
          Recovering the NetBackup relational database files from staging
        3.  
          About processing the relational database in staging
      9.  
        Recovering the NetBackup catalog when NetBackup Access Control is configured
      10.  
        Recovering the NetBackup catalog from a nonprimary copy of a catalog backup
      11.  
        Recovering the NetBackup catalog without the disaster recovery file
      12.  
        Recovering a NetBackup user-directed online catalog backup from the command line
      13.  
        Restoring files from a NetBackup online catalog backup
      14.  
        Unfreezing the NetBackup online catalog recovery media
      15.  
        Steps to carry out when you see exit status 5988 during catalog recovery

Troubleshooting file-based external certificate issues

This issue may occur because of one of the following reasons:

  • The web service certificate that is used for communication is not configured properly.

  • Some of the NetBackup core services have not started.

  • The required prerequisites for external certificate are not met.

  • External certificate configuration path (ECA_CERT_PATH) is not configured properly.

  • Certificate revocation check failed.

To resolve the issue, review the following causes and run the following command to determine the current state of the problem.

Install_Path/bin/nbcertcmd -enrollCertificate -preCheck -server server_name

Install_Path refers to the following:

On Windows: VERITAS\NetBackup\bin

On Unix: /usr/openv/netbackup/bin

Cause 1: The web server certificate that is used for communication is not configured properly.
  • The NetBackup web server is not configured to use external certificates.

    The following error is displayed:

    EXIT STATUS 26: client/server handshaking failed.

    • Run the following command on the master server to check if external CA is configured (ON) or not (OFF).

      Install_Path/nbcertcmd -getSecConfig -caUsage

      On Windows: C:\Program Files\ VERITAS\NetBackup\bin\nbcertcmd -getSecConfig -caUsage

      On Unix: /usr/openv/netbackup/bin/netbackup/bin/nbcertcmd -getSecConfig -caUsage

      For example: C:\Program Files\Veritas\NetBackup\bin>nbcertcmd -getSecConfig -caUsage

      Output:

      NBCA:OFF ECA:ON

      If an external CA is not configured, run the configureWebServerCerts command on the web server.

      In certain cases, you may also get the following error when an external CA is not configured on the web server.

      EXIT STATUS 5982: The certificate revocation list is unavailable.

      In this case, first check the value of the ECA parameter. If it is OFF, run the configureWebServerCerts command.

  • The web service certificate that is used for communication is not trusted by a certificate authority.

    • Check the certificate path (the configureWebServerCert -certPath option) must have a leaf certificate with the entire chain of CA certificates except the trust anchor (root CA).

    • Run the following command to list the certificates that are configured for the web server.

      nbcertcmd -listallcertificates -jks

      On Windows: C:\Program Files\ VERITAS\NetBackup\bin\nbcertcmd -listallcertificates -jks

      On Unix: /usr/openv/netbackup/bin/netbackup/bin/nbcertcmd -listallcertificates -jks

    • Run the following command to list the host certificate details of the NetBackup master server.

      Install_Path/goodies/vxsslcmd x509 -in certificate_path -noout -text -purpose

      On Windows: C:\Program Files\ VERITAS\NetBackup\bin\goodies\vxsslcmd x509 -in certificate_path -noout -text -purpose

      On Unix: /usr/openv/netbackup/bin/netbackup/bin/goodies/vxsslcmd x509 -in certificate_path -noout -text -purpose

      Validate whether the host certificate of the master server is issued by the same root CA as of the web server certificate.

      If host certificate is not issued by the same root CA as of web server certificate then issue new certificate with that CA for NetBackup Master server and enroll certificate again.

  • The specified server name was not found in the web service certificate.

    The server name does not match any of the host names listed in the server's certificate.

    Names listed in the server's certificate are:

    DNS: nb-master _ext

    DNS: nb-master .some.domain.com

    DNS: nb-master _web_svr EXIT STATUS 8509:

    Either update the configuration on the NetBackup host so that it uses one of the names that are present in the web server certificate to refer to the master server or Include all names of the master server that are known to the NetBackup domain in the certificate.

For more information, refer to the following article:

https://www.veritas.com/support/en_US/article.000126751

Cause 2

Some of the NetBackup core services have not started.

Carry out the following procedure to resolve the issue:

  • Check the status of the following services by running the bpps command from the NetBackup/bin directory:

    • nbsl

    • vnetd -standalone

    • NB_dbsrv (on UNIX) or the dbsrv16 (on Windows)

    For more details on the NetBackup commands, refer to the NetBackup Commands Reference Guide.

  • Start the nbsl and the vnetd services, if they are not running.

  • Start the NB_dbsrv (on Unix) service or the dbsrv16 (on Windows) service, if it is not running.

Restart nbsl, vnetd, and NB_dbsrv (or dbsrv16) services as follows:

On Windows:

Install_Path\bin\bpdown -e "NetBackup Service Layer" -f -v

Install_Path\bin\bpup -e "NetBackup Service Layer" -f -v

Install_Path\bin\bpdown -e "NetBackup Legacy Network Service" -f -v

Install_Path\bin\bpup -e "NetBackup Legacy Network Service" -f -v

Install_Path\bin\bpdown -e "SQLANYs_VERITAS_NB" -f -v

Install_Path\bin\bpup -e "SQLANYs_VERITAS_NB" -f -v

Alternatively, you may use the Service Control Manager to restart the NetBackup Service Layer (NBSL), NetBackup Legacy Network Service (vnetd) , and SQLANYs_VERITAS_NB services.

For example:

C:\Program Files\Veritas\NetBackup\bin\bpdown -e "NetBackup Service Layer" -f -v

C:\Program Files\Veritas\NetBackup\bin\bpup -e "NetBackup Service Layer" -f -v

C:\Program Files\Veritas\NetBackup\bin\bpdown -e "NetBackup Legacy Network Service" -f -v

C:\Program Files\Veritas\NetBackup\bin\bpup -e "NetBackup Legacy Network Service" -f -v

C:\Program Files\Veritas\NetBackup\bin\bpdown -e "SQLANYs_VERITAS_NB" -f -v

C:\Program Files\Veritas\NetBackup\bin\bpup -e "SQLANYs_VERITAS_NB" -f -v

On Unix:

Install_Path/netbackup/bin/nbsl -terminate

Install_Path/netbackup/bin/nbsl

To stop vnetd and NB_dbsrv, refer to the following example:

To start vnetd and NB_dbsrv, run the following commands:

install_path/netbackup/bin/vnetd -standalone install_path/db/bin/NB_dbsrv

For example:

/usr/openv/netbackup/bin/nbsl -terminate

/usr/openv/netbackup/bin/nbsl

# ps -fed | grep vnetd | grep standalone

root 16018 1 4 08:47:35 ? 0:01 ./vnetd -standalone

# kill 16018

# ps -fed |grep NB_dbsrv

root 11959 1 4 08:47:35 ? 0:01 ./NB_dbsrv

root 16174 16011 0 08:47:39 pts/2 0:00 grep ./NB_dbsrv

# kill 11959

/usr/openv/netbackup/bin/vnetd -standalone

/usr/openv/db/bin/NB_dbsrv

If the problem persists, contact the Veritas Technical Support team.

Cause 3

The required prerequisites for external certificate are not met.

Review the following prerequisites:

  • Subject DN should be unique and stable for each host. It should have less than 255 characters and should not be empty.

  • Only ASCII 7 characters are supported in the certificate subject DN and X509v3 Subject Alternative Name.

  • Server and client authentication attributes (SSL server and SSL client) should be set (or should be true) in the certificate.

  • Certificate is in PEM format.

  • CRL distribution points (CDPs) are supported only for HTTP/HTTPS.

Run the following command to verify if the prerequisites are met.

Install_Path/goodies/vxsslcmd x509 -in certificate_path -noout -text -purpose

Note:

The certificate paths that are provided for the configureWebServerCert -certPath option and the ECA_CERT_PATH option must have a leaf certificate with the entire chain of the CA certificates except the trust anchor (root CA).

Desirable conditions:

  • Host name (CLIENT_NAME) that is used for certificate enrollment should be part of X509v3 Subject Alternative Name under DNS type.

  • Common name (CN) of the subject name should not be empty.

Note:

The following warning is generated when the vxsslcmd command is run and can be safely ignored:

WARNING: can't open config file: /usr/local/ssl/openssl.cnf

Cause 4

External certificate configuration path is not configured properly.

Ensure the following external certificate configuration options are configured properly:

  • ECA_CERT_PATH

  • ECA_TRUST_STORE_PATH

  • ECA_PRIVATE_KEY_PATH

  • ECA_CRL_PATH

  • ECA_CRL_CHECK

Ensure the following:

  • The peer host certificate has the CRL distribution point (CDP).

    If you have not specified ECA_CRL_PATH, NetBackup uses the CRLs on the URLs that are specified in the peer host certificate's CDP.

  • ECA_CRL_PATH is not a volumeID path on Windows.

Run the following command and validate the external certificate configuration parameters.

On UNIX: Install_Path/bin/nbgetconfig | grep ECA

Windows: Install_Path/bin/nbgetconfig | findstr ECA

.

For more information about the configuration options, refer to the NetBackup Security and Encryption Guide.

Cause 5

The requirements that are mentioned in Cause 3 are not met.

  • Host name (CLIENT_NAME) used for the certificate enrollment is not part of X509v3 Subject Alternative Name under the DNS type.

    If enrollment fails with this error, do one of the following:

    • Generate new certificate having host name in subject alternative name of the certificate.

    • Add or update (first delete and then add) the subject name of the certificate (RFC 2253 compliant) in the external certificate database on the master server.

      Run the following command to add an entry for the host and the associated subject name in the NetBackup certificate database (only administrator can perform this operation):

      Install_Path/bin/nbcertcmd -createECACertEntry -host host_name | -hostId host_id -subject subject name of external cert [-server master_server_name]

      Alternatively, run the following command to delete an entry for the host and the associated subject name from the NetBackup certificate database and then add an entry using the -createECACertEntry command (only administrator can perform this operation):

      Install_Path/bin/nbcertcmd -deleteECACertEntry -subject subject name of external cert [-server master_server_name]

  • Common name (CN) of the subject name is not present in the certificate.

    If certificate enrollment fails with this error, do one of the following:

    • Generate a new certificate with the common name in the certificate.

    • Generate a new certificate with the host name in the subject alternative name of the certificate.

    • Add host in the NetBackup host database and add an entry for the host and the associated subject name in the NetBackup certificate database.

      Run the following command to add a host in the NetBackup host database (only administrator can perform this operation):

      Install_Path/bin/admincmd/nbhostmgmt -addhost -host host_name | -hostId host_id [-server master_server_name]

      Run the following command to add an entry for the host and the associated subject name in the NetBackup certificate database.

      Install_Path/bin/nbcertcmd -createECACertEntry -host host_name | -hostId host_id -subject subject name of external cert [-server master_server_name]

      Subject name of the external certificate should be RFC 2253 compliant.

Cause 6

Certificate revocation check failed.

External certificate enrollment can fail with the certificate revocation error for the following reasons:

  • The external certificate is revoked.

  • The web server certificate is revoked.

  • CRL is unavailable on either the host or the master server.

See Troubleshooting issues with external CA-signed certificate revocation.

For more details on enrollment of external certificates in NetBackup, refer to the NetBackup Security and Encryption Guide.