Tomcat certificate renewal on Windows or VCS cluster does not update the Java keystores

Problem

When the Tomcat certificate is within 180 days of expiration, it should be renewed automatically by nbwmc. This is done in two steps:

1. The Tomcat certificate is renewed and updated in the certstore folder:

   /usr/openv/var/global/vxss/tomcatcreds/nbwebsvc/.VRTSat/profile/certstore

2. This renewed certificate is imported into the Java Keystores used by Tomcat

   /usr/openv/var/global/vxss/tomcatcreds/nbwebservice.jks
/usr/openv/var/global/wsl/credentials/nbwebservice.jks

However, on a cluster, the second step fails.  While the certificate is renewed in the certstore, the Java keystore (JKS) files still contain the old certificate, which will eventually expire.

Error Message

Once the certificate in the keystore expires, communication between NetBackup hosts will fail, and various nbcertcmd commands (e.g. -getcrl) will return an error of 8506: The certificate has expired

If the certificate is not yet expired, but still contains the previous version , it will not generate errors, but NBCertkeycheck will flag the discrepancy, as will nbcheck during a NetBackup 8.2 upgrade.

not ok java_key_store: [11:22:33:44:55:66:11:22:33:33:33:33:33:11:1:1:1:1:1] fingerprint does not match any CA/tomcat certificate files.

At least one Java Keystore nbwmc fingerprint does not match the Tomcat certificate fingerprint.

Cause

The process used in step 2 above checks if the machine is clustered, and if so, if the node is active. However, the cluster check uses commands that will only work if run by a privileged user, i.e. root or Administrator. Because the certificate renewal is being run by web services, as the non-privileged WEBSVC_USER, those commands will hang or fail.  Failure will cause the following line to be logged in the configureCerts.log file:

/usr/openv/netbackup/bin/cluster/cluster_active must be run while logged in as root.

Applies to

NetBackup 8.0 - 8.2, 8.3 and 9.0 running on a VCS or Windows cluster.

Fixed in 8.3.0.1.

Please contact support if you think you have this problem in fixed versions.

Solution

This issue is currently under investigation by Veritas Technologies LLC in E-Track 3982719 (UNIX/Linux) and E-Track 3993744 (Windows).

Emergency Engineering Binaries (EEB):

etrack 4004215 applies to NetBackup version 8.2 windows

etrack 3988675 applies to NetBackup version 8.2 unix

etrack 3993744 applies to NetBackup version 8.1.2. windows

etrack 3982719 applies to NetBackup version 8.1.2. unix

Pending the outcome of the investigation, this issue may be resolved by way of a patch or hotfix in current or future revisions of the software. However, this particular issue is not currently scheduled for any release.  If you feel this issue has a direct business impact for you and your continued use of the product, please contact your Veritas Sales representative or the Veritas Sales group to discuss these concerns.

Work-around #1:
Reset/generate a new Tomcat certificate as of today, that is valid for 12 months, and use it to update the Java Key Store (JKS) files.   This is the same work-around as when the Tomcat certificate is not renewing for other issues, see the related article 100044601.

Work-around #2:
If the Tomcat certificate file has recently renewed but was not imported into the Java Key Store (JKS) files, use this shorter set of steps to import the still valid certificate.  It will be valid until 12 months from the date when it was originally generated.

A)    Ensure there is only one Tomcat certificate file on the master server.  
Unix
ls -ld /usr/openv/var/global/vxss/tomcatcreds/nbwebsvc/.VRTSat/profile/certstore/*1556?nbatd*

Windows
dir "<install_path>\NetBackup\var\global\vxss\tomcatcreds\nbwebsvc\certstore\*1556?nbatd*"

If there is more than one file displayed, use Work-around #1.

B)    Check the "Not After" date of the Tomcat certificate file to make sure it is still valid.
Unix
/usr/openv/netbackup/bin/goodies/vxsslcmd x509 -dates -noout -in /usr/openv/var/global/vxss/tomcatcreds/nbwebsvc/.VRTSat/profile/certstore/*1556?nbatd* 2>/dev/null

Unix Example
$ /usr/openv/netbackup/bin/goodies/vxsslcmd x509 -dates -noout -in /usr/openv/var/global/vxss/tomcatcreds/nbwebsvc/.VRTSat/profile/certstore/*1556?nbatd* 2>/dev/null
            Not Before: Dec 20 16:35:13 2019 GMT
            Not After : Dec 19 17:50:13 2020 GMT

Windows
"<install_Path>\NetBackup\bin\goodies\vxsslcmd" x509 -dates -noout -in "<install_path>\NetBackup\var\global\vxss\tomcatcreds\nbwebsvc\certstore\*1556?nbatd*" 2>NUL

Window Example
"C:\Program Files\Veritas\NetBackup\bin\goodies\vxsslcmd" x509 -dates -noout -in "C:\Program Files\Veritas\NetBackup\var\global\vxss\tomcatcreds\nbwebsvc\certstore\*1556?nbatd*" 2>NUL
notBefore=Nov 19 19:07:19 2019 GMT
notAfter=Nov 18 20:22:19 2020 GMT

It should be at least 6 months in the future.  If not, use Work-around #1.

C)    Check the "until" date of the cached Tomcat certificate within the JKS file.  The JKS will contain multiple certificates.  The date of interest is the one immediately after the ‘OU=TOMCAT’ certificate.
Unix
/usr/openv/java/jre/bin/keytool -list -v -keystore /usr/openv/var/global/vxss/tomcatcreds/nbwebservice.jks < /usr/openv/var/global/jkskey 2>/dev/null | egrep 'until|OU=TOMCAT'

Unix Example
$ /usr/openv/java/jre/bin/keytool -list -v -keystore /usr/openv/var/global/vxss/tomcatcreds/nbwebservice.jks < /usr/openv/var/global/jkskey 2>/dev/null | egrep 'until|OU=TOMCAT'
Valid from: Fri Dec 20 11:18:52 EST 2019 until: Thu Dec 15 12:33:52 EST 2039
Owner: O=vx, OU=TOMCAT@BigServer, CN=BigServer
Valid from: Fri Dec 20 11:35:13 EST 2019 until: Sat Dec 19 12:50:13 EST 2020
Valid from: Fri Dec 20 11:18:52 EST 2019 until: Thu Dec 15 12:33:52 EST 2039

Windows
"<install_path>\NetBackup\jre\bin\keytool.exe" -list -v -keystore"<install_path>\NetBackup\var\global\vxss\tomcatcreds\nbwebservice.jks" < "<install_path>\NetBackup\var\global\jkskey" 2>NUL | findstr "until TOMCAT"

Windows Example
"C:\Program Files\Veritas\NetBackup\jre\bin\keytool.exe" -list -v -keystore "C:\Program Files\Veritas\NetBackup\var\global\vxss\tomcatcreds\nbwebservice.jks" < "C:\Program Files\Veritas\NetBackup\var\global\jkskey" 2>NUL | findstr "until TOMCAT"
Valid from: Tue Nov 19 13:06:42 CST 2019 until: Mon Nov 14 14:21:42 CST 2039
Owner: O=vx, OU=TOMCAT@BigServer.com, CN=BigServer.com
Valid from: Tue Nov 19 13:07:19 CST 2019 until: Wed Nov 18 14:22:19 CST 2020
Valid from: Tue Nov 19 13:06:42 CST 2019 until: Mon Nov 14 14:21:42 CST 2039

If the date in the JKS file matches the date in the certificate file, STOP, there is no inconsistency to resolve.

If the dates do not match, the condition described in this article is present; continue with this procedure.

D)    Manually update the JKS file, then on UNIX ensure permissions are set correctly on the new files.
Unix
This step must be run as the root user.
/usr/openv/wmc/bin/install/configureCerts
/usr/openv/wmc/bin/install/setupWmc

Windows
This step must be run with Administrator permissions.
"<install_path>\NetBackup\wmc\bin\install\configureCerts.bat"

The setupWmc step is not needed on Windows systems

E)    Repeat steps B and C to confirm the Tomcat certificate in the JKS file has the same expiration date as the one in the certificate file.

If the dates are not the same, contact Veritas NetBackup Technical Services for assistance.