Veritas NetBackup™ Troubleshooting Guide

Last Published:
Product(s): NetBackup (8.2)
  1. Introduction
    1.  
      NetBackup logging and status code information
    2.  
      Troubleshooting a problem
    3.  
      Problem report for Technical Support
    4.  
      About gathering information for NetBackup-Java applications
  2. Troubleshooting procedures
    1.  
      About troubleshooting procedures
    2. Troubleshooting NetBackup problems
      1.  
        Verifying that all processes are running on UNIX servers
      2.  
        Verifying that all processes are running on Windows servers
    3.  
      Troubleshooting installation problems
    4.  
      Troubleshooting configuration problems
    5.  
      Device configuration problem resolution
    6.  
      Testing the master server and clients
    7.  
      Testing the media server and clients
    8.  
      Resolving network communication problems with UNIX clients
    9.  
      Resolving network communication problems with Windows clients
    10. Troubleshooting vnetd proxy connections
      1.  
        vnetd proxy connection requirements
      2.  
        Where to begin to troubleshoot vnetd proxy connections
      3.  
        Verify that the vnetd process and proxies are active
      4.  
        Verify that the host connections are proxied
      5.  
        Test the vnetd proxy connections
      6.  
        Examine the log files of the connecting and accepting processes
      7.  
        Viewing the vnetd proxy log files
    11. Troubleshooting security certificate revocation
      1.  
        Troubleshooting cloud provider's revoked SSL certificate issues
      2.  
        Troubleshooting cloud provider's CRL download issues
      3.  
        How a host's CRL affects certificate revocation troubleshooting
      4.  
        NetBackup job fails because of revoked certificate or unavailability of CRLs
      5.  
        NetBackup job fails because of apparent network error
      6.  
        NetBackup job fails because of unavailable resource
      7.  
        Master server security certificate is revoked
      8.  
        Determining a NetBackup host's certificate state
      9.  
        Troubleshooting issues with external CA-signed certificate revocation
    12.  
      About troubleshooting networks and host names
    13. Verifying host name and service entries in NetBackup
      1.  
        Example of host name and service entries on UNIX master server and client
      2.  
        Example of host name and service entries on UNIX master server and media server
      3.  
        Example of host name and service entries on UNIX PC clients
      4.  
        Example of host name and service entries on UNIX server that connects to multiple networks
    14.  
      About the bpclntcmd utility
    15.  
      Using the Host Properties window to access configuration settings
    16.  
      Resolving full disk problems
    17. Frozen media troubleshooting considerations
      1.  
        Logs for troubleshooting frozen media
      2.  
        About the conditions that cause media to freeze
    18. Troubleshooting problems with the NetBackup web services
      1.  
        Viewing NetBackup web services logs
      2.  
        Troubleshooting web service issues after external CA configuration
    19.  
      Troubleshooting problems with the NetBackup web server certificate
    20. Resolving PBX problems
      1.  
        Checking PBX installation
      2.  
        Checking that PBX is running
      3.  
        Checking that PBX is set correctly
      4.  
        Accessing the PBX logs
      5.  
        Troubleshooting PBX security
      6.  
        Determining if the PBX daemon or service is available
    21. Troubleshooting problems with validation of the remote host
      1.  
        Viewing logs pertaining to host validation
      2.  
        Enabling insecure communication with NetBackup 8.0 and earlier hosts
      3.  
        Approving pending host ID-to-host name mappings
      4.  
        Clearing host cache
    22. About troubleshooting Auto Image Replication
      1. Troubleshooting Auto Image Replication
        1.  
          Targeted AIR trusted master server operation failed in case of external certificate configuration
      2.  
        About troubleshooting automatic import jobs
    23.  
      Troubleshooting network interface card performance
    24.  
      About SERVER entries in the bp.conf file
    25.  
      About unavailable storage unit problems
    26.  
      Resolving a NetBackup Administration operations failure on Windows
    27.  
      Resolving garbled text displayed in NetBackup Administration Console on a UNIX computer
    28.  
      Unable to logon to the NetBackup Administration Console after external CA configuration
    29.  
      Troubleshooting file-based external certificate issues
    30.  
      Troubleshooting Windows certificate store issues
    31.  
      Troubleshooting backup failures
    32.  
      Troubleshooting backup failure issues with NAT clients
    33.  
      Troubleshooting issues with the NetBackup Messaging Broker (or nbmqbroker) service
  3. Using NetBackup utilities
    1.  
      About NetBackup troubleshooting utilities
    2.  
      About the analysis utilities for NetBackup debug logs
    3.  
      About the Logging Assistant
    4.  
      About network troubleshooting utilities
    5. About the NetBackup support utility (nbsu)
      1.  
        Output from the NetBackup support utility (nbsu)
      2.  
        Example of a progress display for the NetBackup support utility (nbsu)
    6. About the NetBackup consistency check utility (NBCC)
      1.  
        Output from the NetBackup consistency check utility (NBCC)
      2.  
        Example of an NBCC progress display
    7.  
      About the NetBackup consistency check repair (NBCCR) utility
    8.  
      About the nbcplogs utility
    9. About the robotic test utilities
      1.  
        Robotic tests on UNIX
      2.  
        Robotic tests on Windows
  4. Disaster recovery
    1.  
      About disaster recovery
    2.  
      About disaster recovery requirements
    3.  
      Disaster recovery packages
    4.  
      About disaster recovery settings
    5.  
      Recommended backup practices
    6. About disk recovery procedures for UNIX and Linux
      1. About recovering the master server disk for UNIX and Linux
        1.  
          Recovering the master server when root is intact
        2.  
          Recovering the master server when the root partition is lost
      2.  
        About recovering the NetBackup media server disk for UNIX
      3.  
        Recovering the system disk on a UNIX client workstation
    7. About clustered NetBackup server recovery for UNIX and Linux
      1.  
        Replacing a failed node on a UNIX or Linux cluster
      2.  
        Recovering the entire UNIX or Linux cluster
    8. About disk recovery procedures for Windows
      1. About recovering the master server disk for Windows
        1.  
          Recovering the master server with Windows intact
        2.  
          Recovering the master server and Windows
      2.  
        About recovering the NetBackup media server disk for Windows
      3.  
        Recovering a Windows client disk
    9. About clustered NetBackup server recovery for Windows
      1.  
        Replacing a failed node on a Windows VCS cluster
      2.  
        Recovering the shared disk on a Windows VCS cluster
      3.  
        Recovering the entire Windows VCS cluster
    10.  
      Generating a certificate on a clustered master server after disaster recovery installation
    11.  
      About restoring disaster recovery package
    12.  
      About the DR_PKG_MARKER_FILE environment variable
    13.  
      Restoring disaster recovery package on Windows
    14.  
      Restoring disaster recovery package on UNIX
    15. About recovering the NetBackup catalog
      1.  
        About NetBackup catalog recovery on Windows computers
      2.  
        About NetBackup catalog recovery from disk devices
      3.  
        About NetBackup catalog recovery and symbolic links
      4. About NetBackup catalog recovery and OpsCenter
        1.  
          Specifying the NetBackup job ID number after a catalog recovery
      5.  
        NetBackup disaster recovery email example
      6. About recovering the entire NetBackup catalog
        1.  
          Recovering the entire NetBackup catalog using the Catalog Recovery Wizard
        2.  
          Recovering the entire NetBackup catalog using bprecover -wizard
      7. About recovering the NetBackup catalog image files
        1.  
          Recovering the NetBackup catalog image files using the Catalog Recovery Wizard
        2.  
          Recovering the NetBackup catalog image files using bprecover -wizard
      8. About recovering the NetBackup relational database
        1.  
          Recovering NetBackup relational database files from a backup
        2.  
          Recovering the NetBackup relational database files from staging
        3.  
          About processing the relational database in staging
      9.  
        Recovering the NetBackup catalog when NetBackup Access Control is configured
      10.  
        Recovering the NetBackup catalog from a nonprimary copy of a catalog backup
      11.  
        Recovering the NetBackup catalog without the disaster recovery file
      12.  
        Recovering a NetBackup user-directed online catalog backup from the command line
      13.  
        Restoring files from a NetBackup online catalog backup
      14.  
        Unfreezing the NetBackup online catalog recovery media
      15.  
        Steps to carry out when you see exit status 5988 during catalog recovery

About the analysis utilities for NetBackup debug logs

The debug log analysis utilities enhance NetBackup's existing debug capabilities by providing a consolidated view of a job debug log.

NetBackup jobs span multiple processes that are distributed across servers.

To trace a NetBackup job you must view and correlate messages in multiple log files on multiple hosts. The log analysis utilities provide a consolidated view of the job debug logs. The utilities scan the logs for all processes that are traversed or run for the job. The utilities can consolidate job information by client, job ID, start time for the job, and policy that is associated with the job.

Table: Analysis utilities for NetBackup debug logs describes the log analysis utilities. To see the parameters, limitations, and examples of use for each utility, use the command with the -help option. All the commands require administrative privileges. The log analysis utilities are available for all platforms that are supported for NetBackup servers.

Note:

The utilities must be initiated on supported platforms. However, the utilities can analyze debug log files from most NetBackup client and server platforms for UNIX and Windows.

Table: Analysis utilities for NetBackup debug logs

Utility

Description

backupdbtrace

Consolidates the debug log messages for specified NetBackup database backup jobs and writes them to standard output. It sorts the messages by time. backupdbtrace attempts to compensate for time zone changes and clock drift between remote servers and clients.

At a minimum, you must enable debug logging for admin on the master server, and for bptm and bpbkar on the media server. For best results, set the verbose logging level to 5 and enable debug logging for the following: bpdbm on the master server and bpcd on all servers in addition to the processes already identified.

A complete description of backupdbtrace is in the NetBackup Commands Reference Guide.

backuptrace

Copies to standard output the debug log lines relevant to the specified backup jobs, including online (hot) catalog backups.

The backuptrace utility can be used for regular file system, database extension, and alternate backup method backup jobs. It consolidates the debug logs for specified NetBackup jobs. The utility writes the relevant debug log messages to standard output and sorts the messages by time. backuptrace attempts to compensate for time zone changes and clock drift between remote servers and clients. The format of the output makes it relatively easy to sort or grep by timestamp, program name, and server or client name.

The backuptrace utility works with the nbpem, nbjm, and nbrb logs on the master server. You should enable debug logging for bpbrm and bptm or bpdm on the media server and for bpbkar on the client. For best results, set the verbose logging level to 5. Enable debug logging for the following: bpdbm and bprd on the master server and for bpcd on all servers and clients in addition to the processes already identified.

A complete description of backuptrace is in the NetBackup Commands Reference Guide.

bpgetdebuglog

A helper program for backuptrace and restoretrace. It can also be useful as a standalone program and is available for all NetBackup server platforms.

bpgetdebuglog prints to standard output the contents of a specified debug log file. If only the remote machine parameter is specified, bpgetdebuglog prints the following to standard output: the number of seconds of clock drift between the local computer and the remote computer.

A complete description of bpgetdebuglog is in the NetBackup Commands Reference Guide.

duplicatetrace

Consolidates the debug logs for the specified NetBackup duplicate jobs and writes them to standard output. It sorts the messages by time. duplicatetrace attempts to compensate for time zone changes and clock drift between remote servers and clients.

At a minimum, you must enable debug logging for admin on the master server and for bptm or bpdm on the media server. For best results, set the verbose logging level to 5 and enable debug logging for the following: bpdbm on the master server and bpcd on all servers and clients in addition to the processes already identified.

A complete description of duplicatetrace is in the NetBackup Commands Reference Guide.

importtrace

Consolidates the debug log messages for the specified NetBackup import jobs and writes them to standard output. It sorts the messages by time. importtrace attempts to compensate for time zone changes and clock drift between remote servers and clients.

At a minimum, you must enable debug logging for admin on the master server. And for bpbrm, you must enable debug logging for bptm and tar on the media server. For best results, set the verbose logging level to 5 and enable debug logging for the following: bpdbm on the master server and bpcd on all servers and clients in addition to the processes already identified.

A complete description of importtrace is in the NetBackup Commands Reference Guide.

restoretrace

Copies to standard output the debug log lines relevant to the specified restore jobs.

The restoretrace utility consolidates the debug logs for specified NetBackup restore jobs. The utility writes debug log messages relevant to the specified jobs to standard output and sorts the messages by time. restoretrace attempts to compensate for time zone changes and clock drift between remote servers and clients. The format of the output makes it relatively easy to sort or grep by timestamp, program name, and server or client name.

At a minimum, you must enable debug logging for bprd on the master server. Enable debug logging for bpbrm and bptm or bpdm on the media server and tar on the client. For best results, set the verbose logging level to 5. Enable debug logging for bpdbm on the master server and for bpcd on all servers and clients.

A complete description of restoretrace is in the NetBackup Commands Reference Guide.

verifytrace

Consolidates the debug log messages for the specified verify jobs and writes them to standard output. It sorts the messages by time. The verifytrace command attempts to compensate for time zone changes and clock drift between remote servers and clients.

At a minimum, you must enable debug logging as follows: for admin on the master server and for bpbrm, bptm (or bpdm) and tar on the media server. For best results, set the verbose logging level to 5 and enable debug logging for the following: bpdbm on the master server and bpcd on all servers and clients in addition to the processes already identified.

A complete description of verifytrace is in the NetBackup Commands Reference Guide.

The analysis utilities have the following limitations:

  • Media and device management logs are not analyzed.

  • The legacy debug log files must be in standard locations on the servers and clients.

    UNIX

    /usr/openv/netbackup/logs/<PROGRAM_NAME>/log.mmddyy

    Windows

    install_path\NetBackup\Logs\<PROGRAM_NAME>\mmddyy.log

    An option may be added later that allows the analyzed log files to reside on alternate paths.

    Note:

    For the processes that use unified logging, log directories are automatically created.

  • The consolidated debug log may contain messages from unrelated processes. You can ignore messages with timestamps outside the duration of the job from the following: bprd, nbpem, nbjm, nbrb, bpdbm, bpbrm, bptm, bpdm, and bpcd.

An output line from the log analysis utilities uses the following format:

daystamp.millisecs.program.sequence machine log_line

daystamp

The date of the log that is in the format yyyymmdd.

millisecs

The number of milliseconds since midnight on the local computer.

program

The name of program (BPCD, BPRD, etc.) being logged.

sequence

Line number within the debug log file.

machine

The name of the NetBackup server or client.

log_line

The line that appears in the debug log file.

For more information, see the NetBackup Commands Reference Guide.