IBM Support

QRadar: Using the journalctl command to view log entries for application framework services

Troubleshooting


Problem

The journalctl command can be used to display messages from services, useful for troubleshooting errors and failures.

Symptom

An application framework or an application container service is not starting. The list of services responsible for the applications and framework functionality is published under technote Services responsible for the applications and framework functionality.

Diagnosing The Problem

Logs collected by systemd can be viewed by using journalctl. The journal is implemented with the journald daemon and it retrieves messages from the kernel, systemd services, and other sources.
 
These logs are gathered in a central location, which makes it easy to review. The log records in the journal are structured and indexed, and as a result, journalctl is able to present your log information in various useful formats.

Resolving The Problem

The journalctl utility can be used to access and manipulate the data held within the journal.
 
You can view logs by service within a date range: 
  
​​​​​​​journalctl -xu <service> --since "2023-07-01 12:10:10" --until "2023-08-01 12:10:10"
To view a journalctl service log, use the command: 
  
journalctl -xu <service name>
Examples of journalctl commands.
 
For a specific service (docker in this case) you would run:
  
[root@apphost system]# journalctl -xu docker
-- Logs begin at Mon 2023-08-11 07:52:04 GMT, end at Mon 2023-08-11 08:29:48 GMT. --
Aug 11 08:25:22 75apphost.isslab.usga.ibm.com systemd[1]: Starting Docker Application Container Engine...
Aug 11 08:25:22 75apphost.isslab.usga.ibm.com configure-docker-network.sh[7752]: Configuring docker networking...
Aug 11 08:25:22 75apphost.isslab.usga.ibm.com configure-docker-network.sh[7752]: Requested docker interfaces: docker0 dockerApps
Aug 11 08:25:22 75apphost.isslab.usga.ibm.com configure-docker-network.sh[7752]: Configuring docker interface docker0...
Aug 11 08:25:22 75apphost.isslab.usga.ibm.com configure-docker-network.sh[7752]: Docker interface docker0 configured.
Aug 11 08:25:22 75apphost.isslab.usga.ibm.com configure-docker-network.sh[7752]: Configuring docker interface dockerApps...
Aug 11 08:25:22 75apphost.isslab.usga.ibm.com configure-docker-network.sh[7752]: Docker interface dockerApps configured.
Aug 11 08:25:22 75apphost.isslab.usga.ibm.com configure-docker-network.sh[7752]: Docker networking configured.
 
For a specific container service you would run:
  
[root@apphost system]# journalctl -xu container@13719764573200128895.service
-- Logs begin at Mon 2023-08-11 07:52:04 GMT, end at Mon 2023-08-11 08:44:34 GMT. --
Aug 11 08:26:29 75apphost.isslab.usga.ibm.com systemd[1]: Starting Container created and managed by the conman service...
Aug 11 08:26:29 75apphost.isslab.usga.ibm.com systemd[1]: Started Container created and managed by the conman service.
Aug 11 08:26:29 75apphost.isslab.usga.ibm.com conwrap[13618]: time="2023-08-11T08:26:29Z" level=info msg="conwrap starting" container=qapp-1055-9yXWC1G4 tid=135563
Aug 11 08:26:29 75apphost.isslab.usga.ibm.com conwrap[13618]: time="2023-08-11T08:26:29Z" level=error msg="Unable to retrieve data for URL" container=qapp-1055-9yX
Aug 11 08:26:29 75apphost.isslab.usga.ibm.com conwrap[13618]: time="2023-08-11T08:26:29Z" level=error msg="Failed to generate arguments for docker run." container=
Aug 11 08:26:29 75apphost.isslab.usga.ibm.com conwrap[13618]: time="2023-08-11T08:26:29Z" level=error msg="Failed to generate docker run arguments from the environ
Aug 11 08:26:29 75apphost.isslab.usga.ibm.com systemd[1]: container@13719764573200128895.service: main process exited, code=exited, status=1/FAILURE
Aug 11 08:26:29 75apphost.isslab.usga.ibm.com systemd[1]: Unit container@13719764573200128895.service entered failed state.
Aug 11 08:26:29 75apphost.isslab.usga.ibm.com systemd[1]: container@13719764573200128895.service failed.
Aug 11 08:26:32 75apphost.isslab.usga.ibm.com systemd[1]: container@13719764573200128895.service holdoff time over, scheduling restart.
Aug 11 08:26:32 75apphost.isslab.usga.ibm.com systemd[1]: Stopped Container created and managed by the conman service.
Aug 11 08:26:32 75apphost.isslab.usga.ibm.com systemd[1]: Starting Container created and managed by the conman service...
Aug 11 08:26:32 75apphost.isslab.usga.ibm.com systemd[1]: Started Container created and managed by the conman service.
 
Procedure to generate the journalctl output as files:
 
  1. Login as root with SSH.
  2. Make a directory to your preferred location: 
    # mkdir /tmp/appjournallogs
  3. Change to that directory:  
    # cd /tmp/appjournallogs
  4. Copy and paste the following line, and hit enter: 
    # for service in {docker-distribution,docker,qradarca-monitor,conman,traefik}; do journalctl -u $service --no-pager > "$service""_journal.txt"; done
  5. List the log files output generated by the journalctl: 
    [root@con-primary]# ls -lhtr /tmp/appjournallogs
-rw-r--r--     1 root     root          17 Aug 11 08:38 conman_journal.txt
-rw-r--r--     1 root     root          17 Aug 11 08:38 docker_journal.txt
-rw-r--r--     1 root     root        3851 Aug 11 08:38 qradarca-monitor_journal.txt
-rw-r--r--     1 root     root          17 Aug 11 08:38 docker-distribution_journal.txt
-rw-r--r--     1 root     root       23642 Aug 11 08:38 traefik_journal.txt

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwt3AAA","label":"QRadar Apps"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"7.4.3;7.5.0"}]

Document Information

Modified date:
31 October 2023

UID

ibm17027784

Page Feedback