Veritas NetBackup™ Troubleshooting Guide

Last Published:
Product(s): NetBackup (8.2)
  1. Introduction
    1.  
      NetBackup logging and status code information
    2.  
      Troubleshooting a problem
    3.  
      Problem report for Technical Support
    4.  
      About gathering information for NetBackup-Java applications
  2. Troubleshooting procedures
    1.  
      About troubleshooting procedures
    2. Troubleshooting NetBackup problems
      1.  
        Verifying that all processes are running on UNIX servers
      2.  
        Verifying that all processes are running on Windows servers
    3.  
      Troubleshooting installation problems
    4.  
      Troubleshooting configuration problems
    5.  
      Device configuration problem resolution
    6.  
      Testing the master server and clients
    7.  
      Testing the media server and clients
    8.  
      Resolving network communication problems with UNIX clients
    9.  
      Resolving network communication problems with Windows clients
    10. Troubleshooting vnetd proxy connections
      1.  
        vnetd proxy connection requirements
      2.  
        Where to begin to troubleshoot vnetd proxy connections
      3.  
        Verify that the vnetd process and proxies are active
      4.  
        Verify that the host connections are proxied
      5.  
        Test the vnetd proxy connections
      6.  
        Examine the log files of the connecting and accepting processes
      7.  
        Viewing the vnetd proxy log files
    11. Troubleshooting security certificate revocation
      1.  
        Troubleshooting cloud provider's revoked SSL certificate issues
      2.  
        Troubleshooting cloud provider's CRL download issues
      3.  
        How a host's CRL affects certificate revocation troubleshooting
      4.  
        NetBackup job fails because of revoked certificate or unavailability of CRLs
      5.  
        NetBackup job fails because of apparent network error
      6.  
        NetBackup job fails because of unavailable resource
      7.  
        Master server security certificate is revoked
      8.  
        Determining a NetBackup host's certificate state
      9.  
        Troubleshooting issues with external CA-signed certificate revocation
    12.  
      About troubleshooting networks and host names
    13. Verifying host name and service entries in NetBackup
      1.  
        Example of host name and service entries on UNIX master server and client
      2.  
        Example of host name and service entries on UNIX master server and media server
      3.  
        Example of host name and service entries on UNIX PC clients
      4.  
        Example of host name and service entries on UNIX server that connects to multiple networks
    14.  
      About the bpclntcmd utility
    15.  
      Using the Host Properties window to access configuration settings
    16.  
      Resolving full disk problems
    17. Frozen media troubleshooting considerations
      1.  
        Logs for troubleshooting frozen media
      2.  
        About the conditions that cause media to freeze
    18. Troubleshooting problems with the NetBackup web services
      1.  
        Viewing NetBackup web services logs
      2.  
        Troubleshooting web service issues after external CA configuration
    19.  
      Troubleshooting problems with the NetBackup web server certificate
    20. Resolving PBX problems
      1.  
        Checking PBX installation
      2.  
        Checking that PBX is running
      3.  
        Checking that PBX is set correctly
      4.  
        Accessing the PBX logs
      5.  
        Troubleshooting PBX security
      6.  
        Determining if the PBX daemon or service is available
    21. Troubleshooting problems with validation of the remote host
      1.  
        Viewing logs pertaining to host validation
      2.  
        Enabling insecure communication with NetBackup 8.0 and earlier hosts
      3.  
        Approving pending host ID-to-host name mappings
      4.  
        Clearing host cache
    22. About troubleshooting Auto Image Replication
      1. Troubleshooting Auto Image Replication
        1.  
          Targeted AIR trusted master server operation failed in case of external certificate configuration
      2.  
        About troubleshooting automatic import jobs
    23.  
      Troubleshooting network interface card performance
    24.  
      About SERVER entries in the bp.conf file
    25.  
      About unavailable storage unit problems
    26.  
      Resolving a NetBackup Administration operations failure on Windows
    27.  
      Resolving garbled text displayed in NetBackup Administration Console on a UNIX computer
    28.  
      Unable to logon to the NetBackup Administration Console after external CA configuration
    29.  
      Troubleshooting file-based external certificate issues
    30.  
      Troubleshooting Windows certificate store issues
    31.  
      Troubleshooting backup failures
    32.  
      Troubleshooting backup failure issues with NAT clients
    33.  
      Troubleshooting issues with the NetBackup Messaging Broker (or nbmqbroker) service
  3. Using NetBackup utilities
    1.  
      About NetBackup troubleshooting utilities
    2.  
      About the analysis utilities for NetBackup debug logs
    3.  
      About the Logging Assistant
    4.  
      About network troubleshooting utilities
    5. About the NetBackup support utility (nbsu)
      1.  
        Output from the NetBackup support utility (nbsu)
      2.  
        Example of a progress display for the NetBackup support utility (nbsu)
    6. About the NetBackup consistency check utility (NBCC)
      1.  
        Output from the NetBackup consistency check utility (NBCC)
      2.  
        Example of an NBCC progress display
    7.  
      About the NetBackup consistency check repair (NBCCR) utility
    8.  
      About the nbcplogs utility
    9. About the robotic test utilities
      1.  
        Robotic tests on UNIX
      2.  
        Robotic tests on Windows
  4. Disaster recovery
    1.  
      About disaster recovery
    2.  
      About disaster recovery requirements
    3.  
      Disaster recovery packages
    4.  
      About disaster recovery settings
    5.  
      Recommended backup practices
    6. About disk recovery procedures for UNIX and Linux
      1. About recovering the master server disk for UNIX and Linux
        1.  
          Recovering the master server when root is intact
        2.  
          Recovering the master server when the root partition is lost
      2.  
        About recovering the NetBackup media server disk for UNIX
      3.  
        Recovering the system disk on a UNIX client workstation
    7. About clustered NetBackup server recovery for UNIX and Linux
      1.  
        Replacing a failed node on a UNIX or Linux cluster
      2.  
        Recovering the entire UNIX or Linux cluster
    8. About disk recovery procedures for Windows
      1. About recovering the master server disk for Windows
        1.  
          Recovering the master server with Windows intact
        2.  
          Recovering the master server and Windows
      2.  
        About recovering the NetBackup media server disk for Windows
      3.  
        Recovering a Windows client disk
    9. About clustered NetBackup server recovery for Windows
      1.  
        Replacing a failed node on a Windows VCS cluster
      2.  
        Recovering the shared disk on a Windows VCS cluster
      3.  
        Recovering the entire Windows VCS cluster
    10.  
      Generating a certificate on a clustered master server after disaster recovery installation
    11.  
      About restoring disaster recovery package
    12.  
      About the DR_PKG_MARKER_FILE environment variable
    13.  
      Restoring disaster recovery package on Windows
    14.  
      Restoring disaster recovery package on UNIX
    15. About recovering the NetBackup catalog
      1.  
        About NetBackup catalog recovery on Windows computers
      2.  
        About NetBackup catalog recovery from disk devices
      3.  
        About NetBackup catalog recovery and symbolic links
      4. About NetBackup catalog recovery and OpsCenter
        1.  
          Specifying the NetBackup job ID number after a catalog recovery
      5.  
        NetBackup disaster recovery email example
      6. About recovering the entire NetBackup catalog
        1.  
          Recovering the entire NetBackup catalog using the Catalog Recovery Wizard
        2.  
          Recovering the entire NetBackup catalog using bprecover -wizard
      7. About recovering the NetBackup catalog image files
        1.  
          Recovering the NetBackup catalog image files using the Catalog Recovery Wizard
        2.  
          Recovering the NetBackup catalog image files using bprecover -wizard
      8. About recovering the NetBackup relational database
        1.  
          Recovering NetBackup relational database files from a backup
        2.  
          Recovering the NetBackup relational database files from staging
        3.  
          About processing the relational database in staging
      9.  
        Recovering the NetBackup catalog when NetBackup Access Control is configured
      10.  
        Recovering the NetBackup catalog from a nonprimary copy of a catalog backup
      11.  
        Recovering the NetBackup catalog without the disaster recovery file
      12.  
        Recovering a NetBackup user-directed online catalog backup from the command line
      13.  
        Restoring files from a NetBackup online catalog backup
      14.  
        Unfreezing the NetBackup online catalog recovery media
      15.  
        Steps to carry out when you see exit status 5988 during catalog recovery

About troubleshooting automatic import jobs

The automatic import jobs that the storage lifecycle policy (SLP) components manage are different from legacy import jobs. Automatic import jobs asynchronously notify NetBackup that an image needs to be imported. Also, Auto Image Replication jobs provide catalog entries to the storage device so that the job does not have to read the entire image. An automatic import job reads the catalog record off the storage device and adds it into its own catalog. This process is so fast that NetBackup batches images for import for efficiency. A pending import is the state where NetBackup has been notified, but the import has not yet occurred.

More information is available about the import operation in an SLP and how to tune the batch interval of the import manager process.

See the NetBackup Administrator's Guide, Volume I.

The notify event from the storage server provides the following: the image name, the storage server location to read the catalog for this image, and the name of the SLP that processes the image. Images for automatic import jobs are batched by storage lifecycle policy name and disk volume. The import job consumes an input-output stream on the disk volume.

To view the images that are pending import, run the following command:

# nbstlutil pendimplist -U
Image:
 Master Server            : bayside.example.com
 Backup ID                : gdwinlin04_1280299412
 Client                   : gdwinlin04
 Backup Time              : 1280299412 (Wed Jul 28 01:43:32 2010)
 Policy                   : (none specified)
 Client Type              : 0
 Schedule Type            : 0
 Storage Lifecycle Policy : (none specified)
 Storage Lifecycle State  : 1 (NOT_STARTED) 
 Time In Process          : 0 (Wed Dec 31 18:00:00 1969)
 Data Classification ID   : (none specified)
 Version Number           : 0
 OriginMasterServer       : master_tlk
 OriginMasterServerID     : 00000000-0000-0000-0000-000000000000
 Import From Replica Time : 0 (Wed Dec 31 18:00:00 1969)
 Required Expiration Date : 0 (Wed Dec 31 18:00:00 1969)
 Created Date Time        : 1287678771 (Thu Oct 21 11:32:51 2010)

 Copy:
   Master Server       : bayside.example.com
   Backup ID           : gdwinlin04_1280299412
   Copy Number         : 1
   Copy Type           : 4
   Expire Time         : 0 (Wed Dec 31 18:00:00 1969)
   Expire LC Time      : 0 (Wed Dec 31 18:00:00 1969)
   Try To Keep Time    : 0 (Wed Dec 31 18:00:00 1969)
   Residence           : (none specified)
   Copy State          : 1 (NOT_STARTED) 
   Job ID              : 0
   Retention Type      : 0 (FIXED) 
   MPX State           : 0 (FALSE)
   Source              : 0
   Destination ID      : 
   Last Retry Time     : 0

   Fragment:
     Master Server     : bayside.example.com
     Backup ID         : gdwinlin04_1280299412
     Copy Number       : 1
     Fragment Number   : -2147482648
     Resume Count      : 0
     Media ID          : @aaaab
     Media Server      : bayside.example.com
     Storage Server    : bayside.example.com
     Media Type        : 0 (DISK) 
     Media Sub-Type    : 0 (DEFAULT) 
     Fragment State    : 1 (ACTIVE) 
     Fragment Size     : 0
     Delete Header     : 1
     Fragment ID       : gdwinlin04_1280299412_C1_IM

The action to take on the automatic import job and the automatic import event depends on several conditions as shown in the following table.

Action

Condition

Automatic import jobs queue

No media server or I/O stream is available for this disk volume.

Automatic import jobs never start (copy stays at storage lifecycle state 1)

  • The storage lifecycle policy is inactive.

  • The storage lifecycle policy import destination is inactive.

  • The storage lifecycle policy is between sessions.

  • The image has exceeded the extended retry count and the extended retry time has not passed.

Automatic import event is discarded and the image is ignored

  • The event specifies a backup ID that already exists in this master server catalog.

  • The event specifies a disk volume that is not configured in NetBackup for this storage server.

Automatic import job is started but the image is expired and deleted to clean up disk space in some cases. The event logs an error in the Problems Report or bperror output. An import job runs, but the import for this image fails showing a status code in the range 1532 - 1535.

  • The storage lifecycle policy that is specified in the event does not contain an import destination.

  • The storage lifecycle policy that is specified in the event has an import destination with a residence that does not include the disk volume that is specified by the event.

  • The storage lifecycle policy that is specified does not exist. By default, the Storage Lifecycle Policies utility automatically creates a storage lifecycle policy with the correct name. Ensure that a storage lifecycle policy with the same case-sensitive name exists in the target master server.

    More information is available for the storage lifecycle policy configuration options.

    See the NetBackup Administrator's Guide, Volume I.

Look at the Problems Report or the bperror list for these cases.

To troubleshoot the job flow for automatic import jobs, use the same commands as you would for other storage lifecycle policy managed jobs. To list images for which NetBackup has received notification from storage but not yet initiated import (either pending or failed): use the commands that were previously noted or run the following command:

# nbstlutil list -copy_type import -U -copy_incomplete

To list the images that have been automatically imported, run the following command:

# nbstlutil list -copy_type import -U -copy_state 3 -U
Master Server             : bayside.example.com
 Backup ID                : woodridge_1287610477
 Client                   : woodridge
 Backup Time              : 1287610477 (Wed Oct 20 16:34:37 2010)
 Policy                   : two-hop-with-dup
 Client Type              : 0
 Schedule Type            : 0
 Storage Lifecycle Policy : woodridge2bayside2pearl_withdup
 Storage Lifecycle State  : 3 (COMPLETE) 
 Time In Process          : 1287610714 (Wed Oct 20 16:38:34 2010)
 Data Classification ID   : (none specified)
 Version Number           : 0
 OriginMasterServer       : woodridge.example.com
 OriginMasterServerID     : f5cec09a-da74-11df-8000-f5b3612d8988
 Import From Replica Time : 1287610672 (Wed Oct 20 16:37:52 2010)
 Required Expiration Date : 1290288877 (Sat Nov 20 15:34:37 2010)
 Created Date Time        : 1287610652 (Wed Oct 20 16:37:32 2010)

The OriginMasterServer, OriginMasterServerID, Import From Replica Time, and Required Expiration Date are not known until after the image is imported so a pending record may look like the following:

Image:
 Master Server            : bayside.example.com
 Backup ID                : gdwinlin04_1280299412
 Client                   : gdwinlin04
 Backup Time              : 1280299412 (Wed Jul 28 01:43:32 2010)
 Policy                   : (none specified)
 Client Type              : 0
 Schedule Type            : 0
 Storage Lifecycle Policy : (none specified)
 Storage Lifecycle State  : 1 (NOT_STARTED) 
 Time In Process          : 0 (Wed Dec 31 18:00:00 1969)
 Data Classification ID   : (none specified)
 Version Number           : 0
 OriginMasterServer       : master_tlk
 OriginMasterServerID     : 00000000-0000-0000-0000-000000000000
 Import From Replica Time : 0 (Wed Dec 31 18:00:00 1969)
 Required Expiration Date : 0 (Wed Dec 31 18:00:00 1969)
 Created Date Time        : 1287680533 (Thu Oct 21 12:02:13 2010)

The OriginMasterServer here is not empty, although it may be in some cases. In cascading Auto Image Replication, the master server sends the notification.