Veritas NetBackup™ Troubleshooting Guide

Last Published:
Product(s): NetBackup (8.2)
  1. Introduction
    1.  
      NetBackup logging and status code information
    2.  
      Troubleshooting a problem
    3.  
      Problem report for Technical Support
    4.  
      About gathering information for NetBackup-Java applications
  2. Troubleshooting procedures
    1.  
      About troubleshooting procedures
    2. Troubleshooting NetBackup problems
      1.  
        Verifying that all processes are running on UNIX servers
      2.  
        Verifying that all processes are running on Windows servers
    3.  
      Troubleshooting installation problems
    4.  
      Troubleshooting configuration problems
    5.  
      Device configuration problem resolution
    6.  
      Testing the master server and clients
    7.  
      Testing the media server and clients
    8.  
      Resolving network communication problems with UNIX clients
    9.  
      Resolving network communication problems with Windows clients
    10. Troubleshooting vnetd proxy connections
      1.  
        vnetd proxy connection requirements
      2.  
        Where to begin to troubleshoot vnetd proxy connections
      3.  
        Verify that the vnetd process and proxies are active
      4.  
        Verify that the host connections are proxied
      5.  
        Test the vnetd proxy connections
      6.  
        Examine the log files of the connecting and accepting processes
      7.  
        Viewing the vnetd proxy log files
    11. Troubleshooting security certificate revocation
      1.  
        Troubleshooting cloud provider's revoked SSL certificate issues
      2.  
        Troubleshooting cloud provider's CRL download issues
      3.  
        How a host's CRL affects certificate revocation troubleshooting
      4.  
        NetBackup job fails because of revoked certificate or unavailability of CRLs
      5.  
        NetBackup job fails because of apparent network error
      6.  
        NetBackup job fails because of unavailable resource
      7.  
        Master server security certificate is revoked
      8.  
        Determining a NetBackup host's certificate state
      9.  
        Troubleshooting issues with external CA-signed certificate revocation
    12.  
      About troubleshooting networks and host names
    13. Verifying host name and service entries in NetBackup
      1.  
        Example of host name and service entries on UNIX master server and client
      2.  
        Example of host name and service entries on UNIX master server and media server
      3.  
        Example of host name and service entries on UNIX PC clients
      4.  
        Example of host name and service entries on UNIX server that connects to multiple networks
    14.  
      About the bpclntcmd utility
    15.  
      Using the Host Properties window to access configuration settings
    16.  
      Resolving full disk problems
    17. Frozen media troubleshooting considerations
      1.  
        Logs for troubleshooting frozen media
      2.  
        About the conditions that cause media to freeze
    18. Troubleshooting problems with the NetBackup web services
      1.  
        Viewing NetBackup web services logs
      2.  
        Troubleshooting web service issues after external CA configuration
    19.  
      Troubleshooting problems with the NetBackup web server certificate
    20. Resolving PBX problems
      1.  
        Checking PBX installation
      2.  
        Checking that PBX is running
      3.  
        Checking that PBX is set correctly
      4.  
        Accessing the PBX logs
      5.  
        Troubleshooting PBX security
      6.  
        Determining if the PBX daemon or service is available
    21. Troubleshooting problems with validation of the remote host
      1.  
        Viewing logs pertaining to host validation
      2.  
        Enabling insecure communication with NetBackup 8.0 and earlier hosts
      3.  
        Approving pending host ID-to-host name mappings
      4.  
        Clearing host cache
    22. About troubleshooting Auto Image Replication
      1. Troubleshooting Auto Image Replication
        1.  
          Targeted AIR trusted master server operation failed in case of external certificate configuration
      2.  
        About troubleshooting automatic import jobs
    23.  
      Troubleshooting network interface card performance
    24.  
      About SERVER entries in the bp.conf file
    25.  
      About unavailable storage unit problems
    26.  
      Resolving a NetBackup Administration operations failure on Windows
    27.  
      Resolving garbled text displayed in NetBackup Administration Console on a UNIX computer
    28.  
      Unable to logon to the NetBackup Administration Console after external CA configuration
    29.  
      Troubleshooting file-based external certificate issues
    30.  
      Troubleshooting Windows certificate store issues
    31.  
      Troubleshooting backup failures
    32.  
      Troubleshooting backup failure issues with NAT clients
    33.  
      Troubleshooting issues with the NetBackup Messaging Broker (or nbmqbroker) service
  3. Using NetBackup utilities
    1.  
      About NetBackup troubleshooting utilities
    2.  
      About the analysis utilities for NetBackup debug logs
    3.  
      About the Logging Assistant
    4.  
      About network troubleshooting utilities
    5. About the NetBackup support utility (nbsu)
      1.  
        Output from the NetBackup support utility (nbsu)
      2.  
        Example of a progress display for the NetBackup support utility (nbsu)
    6. About the NetBackup consistency check utility (NBCC)
      1.  
        Output from the NetBackup consistency check utility (NBCC)
      2.  
        Example of an NBCC progress display
    7.  
      About the NetBackup consistency check repair (NBCCR) utility
    8.  
      About the nbcplogs utility
    9. About the robotic test utilities
      1.  
        Robotic tests on UNIX
      2.  
        Robotic tests on Windows
  4. Disaster recovery
    1.  
      About disaster recovery
    2.  
      About disaster recovery requirements
    3.  
      Disaster recovery packages
    4.  
      About disaster recovery settings
    5.  
      Recommended backup practices
    6. About disk recovery procedures for UNIX and Linux
      1. About recovering the master server disk for UNIX and Linux
        1.  
          Recovering the master server when root is intact
        2.  
          Recovering the master server when the root partition is lost
      2.  
        About recovering the NetBackup media server disk for UNIX
      3.  
        Recovering the system disk on a UNIX client workstation
    7. About clustered NetBackup server recovery for UNIX and Linux
      1.  
        Replacing a failed node on a UNIX or Linux cluster
      2.  
        Recovering the entire UNIX or Linux cluster
    8. About disk recovery procedures for Windows
      1. About recovering the master server disk for Windows
        1.  
          Recovering the master server with Windows intact
        2.  
          Recovering the master server and Windows
      2.  
        About recovering the NetBackup media server disk for Windows
      3.  
        Recovering a Windows client disk
    9. About clustered NetBackup server recovery for Windows
      1.  
        Replacing a failed node on a Windows VCS cluster
      2.  
        Recovering the shared disk on a Windows VCS cluster
      3.  
        Recovering the entire Windows VCS cluster
    10.  
      Generating a certificate on a clustered master server after disaster recovery installation
    11.  
      About restoring disaster recovery package
    12.  
      About the DR_PKG_MARKER_FILE environment variable
    13.  
      Restoring disaster recovery package on Windows
    14.  
      Restoring disaster recovery package on UNIX
    15. About recovering the NetBackup catalog
      1.  
        About NetBackup catalog recovery on Windows computers
      2.  
        About NetBackup catalog recovery from disk devices
      3.  
        About NetBackup catalog recovery and symbolic links
      4. About NetBackup catalog recovery and OpsCenter
        1.  
          Specifying the NetBackup job ID number after a catalog recovery
      5.  
        NetBackup disaster recovery email example
      6. About recovering the entire NetBackup catalog
        1.  
          Recovering the entire NetBackup catalog using the Catalog Recovery Wizard
        2.  
          Recovering the entire NetBackup catalog using bprecover -wizard
      7. About recovering the NetBackup catalog image files
        1.  
          Recovering the NetBackup catalog image files using the Catalog Recovery Wizard
        2.  
          Recovering the NetBackup catalog image files using bprecover -wizard
      8. About recovering the NetBackup relational database
        1.  
          Recovering NetBackup relational database files from a backup
        2.  
          Recovering the NetBackup relational database files from staging
        3.  
          About processing the relational database in staging
      9.  
        Recovering the NetBackup catalog when NetBackup Access Control is configured
      10.  
        Recovering the NetBackup catalog from a nonprimary copy of a catalog backup
      11.  
        Recovering the NetBackup catalog without the disaster recovery file
      12.  
        Recovering a NetBackup user-directed online catalog backup from the command line
      13.  
        Restoring files from a NetBackup online catalog backup
      14.  
        Unfreezing the NetBackup online catalog recovery media
      15.  
        Steps to carry out when you see exit status 5988 during catalog recovery

About troubleshooting networks and host names

In a configuration with multiple networks and clients with more than one host name, NetBackup administrators must configure the policy entries carefully. They must consider the network configuration (physical, host names and aliases, NIS/DNS, routing tables, and so on). If administrators want to direct backup and restore data across specific network paths, they especially need to consider these things.

For a backup, NetBackup connects to the host name as configured in the policy. The operating system's network code resolves this name and sends the connection across the network path that the system routing tables define. The bp.conf file is not a factor making this decision.

For restores from the client, the client connects to the master server. For example, on a UNIX computer, the master server is the first one named in the /usr/openv/netbackup/bp.conf file. On a Windows computer, the master server is specified on the Server to use for backups and restores drop-down of the Specify NetBackup Machines and Policy Type dialog box. To open this dialog, start the NetBackup Backup, Archive, and Restore interface and click Specify NetBackup Machines and Policy Type on the File menu. The client's network code that maps the server name to an IP address determines the network path to the server.

Upon receipt of the connection, the server determines the client's configured name from the peer name of its connection to the server.

The peer name is derived from the IP address of the connection. This means that the address must translate into a host name (using the gethostbyaddr() network routine). This name is visible in the bprd debug log when a connection is made as in the line:

Connection from host peername ipaddress ...

The client's configured name is then derived from the peer name by querying the bpdbm process on UNIX computers. On Windows computers, you must query the NetBackup Database Manager service.

The bpdbm process compares the peer name to a list of client names that are generated from the following:

  • All clients for which a backup has been attempted

  • All clients in all policies

The comparison is first a string comparison. The comparison is verified by comparing host names and the aliases that are retrieved by using the network function gethostbyname().

If none of the comparisons succeed, a more brute force method is used, which compares all names and aliases using gethostbyname().

The configured name is the first comparison that succeeds. Note that other comparisons might also have succeeded if aliases or other "network names" are configured.

If the comparison fails, the client's host name as returned by the gethostname() function on the client is used as the configured name. An example of a failed comparison: the client changes its host name but its new host name is not yet reflected in any policies.

These comparisons are recorded in the bpdbm debug log if VERBOSE is set. You can determine a client's configured name by using the bpclntcmd command on the client. For example:

# /usr/openv/netbackup/bin/bpclntcmd -pn (UNIX)

# install_path\NetBackup\bin\bpclntcmd -pn (Windows)

expecting response from server wind.abc.me.com 
danr.abc.me.com danr 194.133.172.3 4823

Where the first output line identifies the server to which the request is directed. The second output line is the server's response in the following order:

  • Peer name of the connection to the server

  • Configured name of the client

  • IP address of the connection to the server

  • Port number that is used in the connection

When the client connects to the server, it sends the following three names to the server:

  • Browse client

  • Requesting client

  • Destination client

The browse client name is used to identify the client files to list or restore from. The user on the client can modify this name to restore files from another client. For example, on a Windows client, the user can change the client name by using the Backup, Archive, and Restore interface. (See the NetBackup online Help for instructions). For this change to work, however, the administrator must also have made a corresponding change on the server.

See the NetBackup Administrator's Guide, Volume I.

The requesting client is the value from the gethostname() function on the client.

The destination client name is a factor only if an administrator pushes a restore to a client from a server. For a user restore, the destination client and the requesting client are the same. For an administrator restore, the administrator can specify a different name for the destination client.

By the time these names appear in the bprd debug log, the requesting client name has been translated into the client's configured name.

The name that used to connect back to the client to complete the restore is either the client's peer name or its configured name. The type of restore request (for example, from root on a server, from a client, to a different client, and so on) influences this action.

When you modify client names in NetBackup policies to accommodate specific network paths, the administrator needs to consider:

  • The client name as configured on the client. For example, on UNIX the client name is CLIENT_NAME in the client's bp.conf file. On a Windows client, it is on the General tab of the NetBackup Client Properties dialog box. To open this dialog box, select NetBackup Client Properties from the File menu in the Backup, Archive, and Restore interface.

  • The client as currently named in the policy configuration.

  • The client backup and archive images that already exist as recorded in the images directory on the master server. On a UNIX server, the images directory is /usr/openv/netbackup/db/images. On a Windows NetBackup server, the images directory is install_path\NetBackup\db\images.

Any of these client names can require manual modification by the administrator if the following: a client has multiple network connections to the server and restores from the client fail due to a connection-related problem.

On UNIX, the public domain program traceroute (not included with NetBackup) often can provide valuable information about a network's configuration. Some system vendors include this program with their systems. For Windows, use the tracert command.

The master server may be unable to reply to client requests, if the Domain Name Services (DNS) are used and the following is true: The name that the client obtains through its gethostname() library (UNIX) or gethostbyname() network (Windows) function is unknown to the DNS on the master server. The client and the server configurations can determine if this situation exists. gethostname() or gethostbyname()on the client may return an unqualified host name that the DNS on the master server cannot resolve.

Although you can reconfigure the client or the master server DNS hosts file, this solution is not always desirable. For this reason, NetBackup provides a special file on the master server. This file is as follows:

/usr/openv/netbackup/db/altnames/host.xlate (UNIX)

install_path\NetBackup\db\altnames\host.xlate (Windows)

You can create and edit this file to force the desired translation of NetBackup client host names.

Each line in the host.xlate file has three elements: a numeric key and two host names. Each line is left justified, and a space character separates each element of the line.

key hostname_from_client client_as_known_by_server

The following describes the preceding variables:

  • key is a numeric value used by NetBackup to specify the cases where the translation is to be done. Currently this value must always be 0, which indicates a configured name translation.

  • hostname_from_client is the value to translate. This value must correspond to the name that the client's gethostname() function obtains and sends to the server in the request.

  • client_as_known_by_server is the name to substitute for hostname_from_client when the client responds to requests. This name must be the name that is configured in the NetBackup configuration on the master server. It must also be known to the master server's network services.

This following is an example:

0 danr danr.eng.aaa.com

When the master server receives a request for a configured client name (numeric key 0), the name danr always replaces the name danr.eng.aaa.com. The problem is resolved, assuming the following:

  • The client's gethostname() function returns danr.

  • The master server's network services gethostbyname() function did not recognize the name danr.

  • The client was configured and named in the NetBackup configuration as danr.eng.aaa.com and this name is also known to network services on the master server.