Java Console displays "Status Code: 130 System Error Occurred" and nbwmc and nbmqbroker may not start

Problem

Java Console displays "Status Code: 130 System Error Occurred" in "Recent Activity" section and in "Security Management". This is applicable to UNIX/Linux master servers.

In addition, it is observed that the NetBackup Web Management Console (nbwmc) and NetBackup Message Queue Broker (nbmqbroker) services are not responding, or possibly not even running.

Error Message

Error observed in the Java Console:



Error observed while trying to do a WEB login:
 
[root@mymaster]# /usr/openv/netbackup/bin/bpnbat -login -logintype WEB
Authentication Broker [master.example.com is default]:
Authentication port [0 is default]:
Authentication type (NIS, NISPLUS, WINDOWS, vx, unixpwd, ldap) [unixpwd is default]:
Domain [mymaster.example.com is default]:  mymaster.example.com
Login Name [root is default]:
Password:
Web authentication failed.
    
Errors observed while starting the NetBackup services (when all services are already up):

[root@mymaster]# ./bp.start_all
Starting vnetd...
Starting nbatd...
Starting nbazd...
Starting nbaudit...
Starting nbwmc...
Starting NetBackup Web Management Console could take a couple of minutes .su: cannot set user id: Resource temporarily unavailable. started.
Starting nbmqbroker...
Starting NetBackup Messaging Queue Broker could take a couple of minutes .su: cannot set user id: Resource temporarily unavailable
.su: cannot set user id: Resource temporarily unavailable
.su: cannot set user id: Resource temporarily unavailable
.su: cannot set user id: Resource temporarily unavailable
 
These errors may not be present when all NetBackup services are stopped and restarted.  A service restart may also resolve the "Status 130 - System error occurred".

Error observed in bpnbat log:

02:26:09.863 [5795.5795] <16> LoginWithUserTokenManager::getToken: get new token failed, retval = 130 for user root
02:26:09.864 [5795.5795] <16> LoginWithUserTokenManager::getToken: Exception encountered, status code: 130
02:26:09.864 [5795.5795] <16> bpnbat@Login: Unable to logon to web app for user root. Error code 130 and VSS error code 93.
02:26:09.864 [5795.5795] <16> bpnbat: Web authentication failed.
02:26:09.864 [5795.5795] <2> bpnbat@main: EXIT status = 93

Error observed in the mqbroker debug log during process startup attempt.

/opt/openv/mqbroker/rabbitmq-server/sbin/rabbitmq-env: fork: retry: Resource temporarily unavailable

Error observed while trying to switch user to the WEBSVC_USER:
 
[root@mymaster]# su --shell /bin/sh nbwebsvc
su: cannot set user id: Resource temporarily unavailable
 
The su works if NetBackup services are down.

Cause

These problems are happening because number of processes and threads running as the NetBackup WEBSVC_USER is constrained.  This may either prevent the forking of new processes while trying to start the nbwmc and/or nbmqbroker services, or prevent the services from creating new threads to handle the necessary work load.  Most likely, the setting for number of user's processes (nproc) is configured to a relatively low value, such as 1024.

The configuration setting location may differ based on operating system type, but on a Linux master server, check the following files.

[root@mymaster]# grep 'nproc' /etc/security/limits.conf /etc/security/limits.d/*-nproc.conf

In this case, the /etc/security/limits.d/90-nproc.conf file contained the following settings which allowed unlimited processes and threads for the root user, but only 1024 for other users including the WEBSVC_USER.

*          soft    nproc     1024
root       soft    nproc     unlimited

On other UNIX platforms, after switching user to the WEBSVC_USER, check the user limits.

[nbwebsvc@mymaster]$ ulimit -a
...snip...
max user processes              (-u) 1024
 

Solution

Increase the number of concurrent user processes and threads that are allowed.

For master servers running inside NetBackup Flex containers, see the related article. 

For standalone master servers, adjust either the ulimits or the /etc/security files. A value of at least 4096 is recommended, on each host, for the WEBSVC_USER on that host.

On Linux, if the /etc/security/limits.d/*-nproc.conf file is in use, make the change there because settings therein override the /etc/security/limits.conf.  But remember that soft limits will be ignored if they exceed the hard limits.  You may have to increase the hard limit if it too is set too low.  For example, when the WEBSVC_USER is 'nbwebsvc':

*         soft       nproc      1024
nbwebsvc  soft       nproc      4096
nbwebsvc  hard       nproc      65536
root      soft       nproc      unlimited

Note: For additional details, see the Related Article: Minimum O/S ulimit settings on primary and media server Linux/UNIX platforms.