Veritas NetBackup™ Troubleshooting Guide

Last Published:
Product(s): NetBackup (8.3.0.1)
  1. Introduction
    1.  
      NetBackup logging and status code information
    2.  
      Troubleshooting a problem
    3.  
      Problem report for Technical Support
    4.  
      About gathering information for NetBackup-Java applications
  2. Troubleshooting procedures
    1.  
      About troubleshooting procedures
    2. Troubleshooting NetBackup problems
      1.  
        Verifying that all processes are running on UNIX servers
      2.  
        Verifying that all processes are running on Windows servers
    3.  
      Troubleshooting installation problems
    4.  
      Troubleshooting configuration problems
    5.  
      Device configuration problem resolution
    6.  
      Testing the master server and clients
    7.  
      Testing the media server and clients
    8.  
      Resolving network communication problems with UNIX clients
    9.  
      Resolving network communication problems with Windows clients
    10. Troubleshooting vnetd proxy connections
      1.  
        vnetd proxy connection requirements
      2.  
        Where to begin to troubleshoot vnetd proxy connections
      3.  
        Verify that the vnetd process and proxies are active
      4.  
        Verify that the host connections are proxied
      5.  
        Test the vnetd proxy connections
      6.  
        Examine the log files of the connecting and accepting processes
      7.  
        Viewing the vnetd proxy log files
    11. Troubleshooting security certificate revocation
      1.  
        Troubleshooting cloud provider's revoked SSL certificate issues
      2.  
        Troubleshooting cloud provider's CRL download issues
      3.  
        How a host's CRL affects certificate revocation troubleshooting
      4.  
        NetBackup job fails because of revoked certificate or unavailability of CRLs
      5.  
        NetBackup job fails because of apparent network error
      6.  
        NetBackup job fails because of unavailable resource
      7.  
        Master server security certificate is revoked
      8.  
        Determining a NetBackup host's certificate state
      9.  
        Troubleshooting issues with external CA-signed certificate revocation
    12.  
      About troubleshooting networks and host names
    13. Verifying host name and service entries in NetBackup
      1.  
        Example of host name and service entries on UNIX master server and client
      2.  
        Example of host name and service entries on UNIX master server and media server
      3.  
        Example of host name and service entries on UNIX PC clients
      4.  
        Example of host name and service entries on UNIX server that connects to multiple networks
    14.  
      About the bpclntcmd utility
    15.  
      Using the Host Properties window to access configuration settings
    16.  
      Resolving full disk problems
    17. Frozen media troubleshooting considerations
      1.  
        Logs for troubleshooting frozen media
      2.  
        About the conditions that cause media to freeze
    18. Troubleshooting problems with the NetBackup web services
      1.  
        Viewing NetBackup web services logs
      2.  
        Troubleshooting web service issues after external CA configuration
    19.  
      Troubleshooting problems with the NetBackup web server certificate
    20. Resolving PBX problems
      1.  
        Checking PBX installation
      2.  
        Checking that PBX is running
      3.  
        Checking that PBX is set correctly
      4.  
        Accessing the PBX logs
      5.  
        Troubleshooting PBX security
      6.  
        Determining if the PBX daemon or service is available
    21. Troubleshooting problems with validation of the remote host
      1.  
        Viewing logs pertaining to host validation
      2.  
        Enabling insecure communication with NetBackup 8.0 and earlier hosts
      3.  
        Approving pending host ID-to-host name mappings
      4.  
        Clearing host cache
    22. Troubleshooting Auto Image Replication
      1.  
        Rules for master servers used with Auto Image Replication and SLPs
      2. Targeted AIR trusted master server operation failed in case of external certificate configuration
        1.  
          Add or update trust
        2.  
          Remove trust
      3.  
        About troubleshooting automatic import jobs that SLP components manage
    23.  
      Troubleshooting network interface card performance
    24.  
      About SERVER entries in the bp.conf file
    25.  
      About unavailable storage unit problems
    26.  
      Resolving a NetBackup Administration operations failure on Windows
    27.  
      Resolving garbled text displayed in NetBackup Administration Console on a UNIX computer
    28.  
      Troubleshooting error messages in the NetBackup Administration Console
    29.  
      Extra disk space required for logs and temporary files for the NetBackup Administration Console
    30.  
      Unable to logon to the NetBackup Administration Console after external CA configuration
    31.  
      Troubleshooting file-based external certificate issues
    32.  
      Troubleshooting Windows certificate store issues
    33.  
      Troubleshooting backup failures
    34.  
      Troubleshooting backup failure issues with NAT clients or NAT servers
    35.  
      Troubleshooting issues with the NetBackup Messaging Broker (or nbmqbroker) service
    36.  
      Issues with email notifications for Windows systems
  3. Using NetBackup utilities
    1.  
      About NetBackup troubleshooting utilities
    2.  
      About the analysis utilities for NetBackup debug logs
    3.  
      About the Logging Assistant
    4.  
      About network troubleshooting utilities
    5. About the NetBackup support utility (nbsu)
      1.  
        Output from the NetBackup support utility (nbsu)
      2.  
        Example of a progress display for the NetBackup support utility (nbsu)
    6. About the NetBackup consistency check utility (NBCC)
      1.  
        Output from the NetBackup consistency check utility (NBCC)
      2.  
        Example of an NBCC progress display
    7.  
      About the NetBackup consistency check repair (NBCCR) utility
    8.  
      About the nbcplogs utility
    9. About the robotic test utilities
      1.  
        Robotic tests on UNIX
      2.  
        Robotic tests on Windows
  4. Disaster recovery
    1.  
      About disaster recovery
    2.  
      About disaster recovery requirements
    3.  
      Disaster recovery packages
    4.  
      About disaster recovery settings
    5.  
      Recommended backup practices
    6. About disk recovery procedures for UNIX and Linux
      1. About recovering the master server disk for UNIX and Linux
        1.  
          Recovering the master server when root is intact
        2.  
          Recovering the master server when the root partition is lost
      2.  
        About recovering the NetBackup media server disk for UNIX
      3.  
        Recovering the system disk on a UNIX client workstation
    7. About clustered NetBackup server recovery for UNIX and Linux
      1.  
        Replacing a failed node on a UNIX or Linux cluster
      2.  
        Recovering the entire UNIX or Linux cluster
    8. About disk recovery procedures for Windows
      1. About recovering the master server disk for Windows
        1.  
          Recovering the master server with Windows intact
        2.  
          Recovering the master server and Windows
      2.  
        About recovering the NetBackup media server disk for Windows
      3.  
        Recovering a Windows client disk
    9. About clustered NetBackup server recovery for Windows
      1.  
        Replacing a failed node on a Windows VCS cluster
      2.  
        Recovering the shared disk on a Windows VCS cluster
      3.  
        Recovering the entire Windows VCS cluster
    10.  
      Generating a certificate on a clustered master server after disaster recovery installation
    11.  
      About restoring disaster recovery package
    12.  
      About the DR_PKG_MARKER_FILE environment variable
    13.  
      Restoring disaster recovery package on Windows
    14.  
      Restoring disaster recovery package on UNIX
    15. About recovering the NetBackup catalog
      1.  
        About NetBackup catalog recovery on Windows computers
      2.  
        About NetBackup catalog recovery from disk devices
      3.  
        About NetBackup catalog recovery and symbolic links
      4. About NetBackup catalog recovery and
        1.  
          Specifying the NetBackup job ID number after a catalog recovery
      5.  
        NetBackup disaster recovery email example
      6. About recovering the entire NetBackup catalog
        1.  
          Recovering the entire NetBackup catalog using the Catalog Recovery Wizard
        2.  
          Recovering the entire NetBackup catalog using bprecover -wizard
      7. About recovering the NetBackup catalog image files
        1.  
          Recovering the NetBackup catalog image files using the Catalog Recovery Wizard
        2.  
          Recovering the NetBackup catalog image files using bprecover -wizard
      8. About recovering the NetBackup relational database
        1.  
          Recovering NetBackup relational database files from a backup
        2.  
          Recovering the NetBackup relational database files from staging
        3.  
          About processing the relational database in staging
      9.  
        Recovering the NetBackup catalog when NetBackup Access Control is configured
      10.  
        Recovering the NetBackup catalog from a nonprimary copy of a catalog backup
      11.  
        Recovering the NetBackup catalog without the disaster recovery file
      12.  
        Recovering a NetBackup user-directed online catalog backup from the command line
      13.  
        Restoring files from a NetBackup online catalog backup
      14.  
        Unfreezing the NetBackup online catalog recovery media
      15.  
        Steps to carry out when you see exit status 5988 during catalog recovery
  5.  
    Index

Troubleshooting web service issues after external CA configuration

Problem

The web service does not start or respond after external certificate (ECA) configuration.

Cause

Check the web server logs at the following location:

install_path/wmc/webserver/logs/catalina.log

Check if the logs contain any of the following strings:

SEVERE [main] org.apache.tomcat.util.net.SSLUtilBase.getStore Failed to load keystore type [JKS] with path [C:\Program Files\Veritas\NetBackup\var\global\wsl\credentials\tpcredentials\nbwebservice.jks] due to [Illegal character in opaque part at index 2: C:\Program Files\Veritas\NetBackup\var\global\wsl\credentials\tpcredentials\nbwebservice.jks]

Caused by: java.lang.IllegalArgumentException: Keystore was tampered with, or password was incorrect

The root cause can be: The keystore of the external CA used by the NetBackup web service is tampered or deleted.

Solution
  • Verify that NetBackup Web Management Console service is running.

    Run the following command:

    On UNIX: /usr/openv/netbackup/bin/bpps -x

    On Windows: Use the NetBackup Activity Monitor or the services application of the Windows Control Panel.

  • If the status is FAIL, reconfigure the external certificate by executing the following command:

    On Windows:Install path\netbackup\wmc\bin\configureWebServerCerts -addExternalCert -nbHost -certPath file_path -privateKeyPath file_path -trustStorePath file_path

    On Unix:/usr/openv/netbackup/bin/configureWebServerCerts -addExternalCert -nbHost -certPath file_path -privateKeyPath file_path -trustStorePath file_path

  • Try to start the NetBackup web service.

    For windows:Install path\netbackup\wmc\bin\nbwmc.exe -start -srvname "NetBackup Web Management Console"

    For Unix:/usr/openv/netbackup/bin/nbwmc start

Problem

External certificate is not configured.

Cause

The issue can occur because of the following:

  • Invalid certificate, private key, or trust store.

    Error message : The certificate could not be added. Please check the configureWebServerCerts logs.

  • Certificate does not contain server name in the subject alternative name (SAN) of the certificate.

Solution for cause: Invalid Certificate, private key or trust store
  • Open web server configuration logs

    Location: <install dir>/NetBackup/wmc/webserver/logs/configureWebServerCerts.log

  • Review the log messages:

    • If the logs have the following message:

      unable to load private key 22308:error:0906D06C:PEM routines:PEM_read_bio:no start line:.\crypto\pem\pem_lib.c:697:Expecting: ANY PRIVATE KEY Could not export certificates in PKCS#12 format, 1.

      The private key does not t match the private key of the certificate that is provided.

      Provide the appropriate private key.

    • If the logs have following message:

      Error occurred while adding certificate to keystore. Exception: java.security.cert.CertificateParsingException: signed overrun, bytes = 918 Exiting.. Could not import CA certificates in JAVA keystore, -1.

      The file path that is provided for the -trustStorePath option is not a valid file path or a valid trust store CA certificate is not present at the given file path.

      Provide the trust store bundle path for the -trustStorePath option.

Solution for cause: Certificate does not contain server name in the subject alternative name (SAN)

The following error message is displayed:

The server name server_name was not found in the web service certificate.

The certificate could not be added. Please check configureWebServerCerts logs.

For successful configuration, ensure the following:

  • Common name of the subject name and the SAN names should not be empty at the same time.

  • If the SAN is not empty, host name must be present in the SAN entry.

  • If SAN is empty, common name of the subject name must be host name.

    Only PEM formatted certificates are allowed.

    Note:

    The host name is the name provided for the master server at the time of installation. Host name can be found in the setenv file with the NB_HOSTNAME property.

    Location of the file:

    On UNIX : /usr/openv/wmc/bin/setenv

    On Windows: install_path\Veritas\NetBackup\wmc\bin\setenv

    Communication can be successful in the following scenarios:

    • The certificate contains all host names that the master server is known by (host names that are listed in the SERVER entries of other hosts in the domain) in the SAN field of the certificate.

    • Server authentication attributes are set in the certificate.

  • Check the logs for the missing entry.

    Add the missing host name in the SAN of the certificate.