How to Enable Core File Generation for RALUS on a Linux Operating System

Setting the OS up to allow core dump generation

# ulimit -c
This command shows if your system can create a core dump. If it returns a 0 you need to run the next command. If not continue reading.

# ulimit -c unlimited
This command allows the core files to be generated with an unlimited size. If disk space is a concern you can replace unlimited with the maximum size that is desired for core files. The value is specified in 1 kb increments.

Example:

# ulimit -c 1024 creates a core file that is up to 1 MB in size
# ulimit -c 1024000 creates a core file that is up to 1 GB in size.

For more information on enabling core files on Red Hat Linux servers see the following Web site:     https://www.unixmen.com/how-to-enable-core-dumps-in-rhel6/

Note: Closing the console session will cause the ulimit to revert to default.

#uname -a > /opt/VRTSralus/bin/uname-a.txt
This command collects information about the Linux OS and writes it to a file.


Setting up RALUS to perform core dump generation

After verifying that core dump generation has been enabled in the OS, the RALUS Agent needs to be set to create a core file using the following steps:

1. cd /opt/VRTSralus/bin
2. Make sure that 'beremote' is not running - ps -ef | grep beremote
3. If it is running, run this command /etc/init.d/VRTSralus.init stop
4. export _VRTS_ALLOW_SIGSEGV_DFL_ACTION=1
5. export _VRTS_ALLOW_SIGINT_DFL_ACTION=1
6. ./beremote --log-console --log-file agent.log &

The purpose of sending debug output to the console and to a file is that when the core dump occurs, the console will have the most up to date information.  The log file will not be able to capture output right up to the point of the core dump.  However the resulting agent.log file will have valuable historical information in it that not be present in the console log.

When the core dump occurs copy as much of the console output as possible to a file and save it so it can be sent in along with the core dump and the agent.log file to Technical Support. Also note, on some systems you will have to press <Enter> to have it return to a console prompt and complete the process.

The core file often is created following this naming convention:

core.####

The file should be created in the /opt/VRTSralus/bin directory by default.


Post processing the core file

There is some information in the core file that is most easily extracted right there on the system where the crash occurred.  This ensures that all the correct libraries are in place during this post processing step.

Copy the core file to /opt/VRTSralus/bin if it isn't already there.
Change directory to the RALUS directory - cd /opt/VRTSralus/bin
Run the command uname -a > uname-a.txt
Run the command file core.#### > filecoreoutput.txt

This shows the process that caused the core file creation.  For example:

 

core.3975: setgid sticky ELF 64-bit LSB core file AMD x86-64, version 1 (SYSV), SVR4-style, from 'beremote'
 

If the output of the file core.#### command does not end in from 'beremote',  that core file was not caused by the Backup Exec RALUS Agent but by the process specified.
 

Then run this command (which creates the gdb.txt file)
 

gdb ./beremote core.#### 2>&1 | tee /opt/VRTSralus/bin/gdb.txt
 

(gdb) bt
 

(gdb) quit
 

What if the gdb command is not available?

The gdb command is the Gnu Debugger that is available for all of the versions of Linux supported by BEWS.  If it is not available on this system, it should be installed.  It is typically available as a package from the Linux distributor as part of the development package for that Linux distribution.

What if the core file doesn't get created?

If the core dump file is not getting created in the / opt/VRTSralus/bin directory then follow the steps below. These steps do not produce a core file, but they do produce enhanced debugging:

1. Make sure that 'beremote' is not running - ps -ef | grep beremote
2. As 'root' cd /opt/VRTSralus/bin
3. Type: gdb ./beremote 2>&1 | tee /opt/VRTSralus/bin/gdb.txt
4. This brings up the debugger
5. Type run
6. This starts up the agent
7. Perform the operation that causes the crash
8. The crash should return the debugger prompt
9. Type bt full to get the stack trace.
10. Copy the stack trace to a the gdb.txt file.
11. Ctrl-d exits the debugger


Sending in files necessary for diagnosing RALUS core

At this point all the files necessary to diagnose a RALUS core file should be available.  Note that if these files are larger than 10MB, they will need to be placed on the Veritas Technical Support FTP site as e-mailing them will not work.

The following files should be gathered and sent to your Technical Support representative:

 

 

/opt/VRTSralus/bin
 

gdb.txt         (Be SURE to include this file.  It is critical for a core dump diagnosis)
 

core.####
 

filecoreoutput.txt
 

uname-a.txt
 

 

/var/VRTSralus/ralus.ver
 

 

/etc/VRTSralus/ralus.cfg