IBM Support

QRadar EDR (formerly ReaQta): Troubleshooting Windows agent issues

Question & Answer


Question

What are some common troubleshooting steps for Windows agent endpoints?

Answer

BSOD

In the presence of a blue screen, collect the following information:

  1. Dashboard or client name if it is an MSSP.
  2. Affected endpoint or endpoints.
  3. If the blue screen is reproducible, then collect the steps to reproduce it, and eventually the versions of the software involved.
  4. Time and date of when it occurred.
On the endpoint:
 
Check the presence of crash dumps in the following directories:
 
  • C:\Windows\Minidump
  • C:\Windows\MEMORY.DMP
Handling BSOD
 
In the sight of a blue screen of death, it is necessary to verify whether the root cause is ReaQta Hive or not. Once verified, do the following:
 
  1. Collect the default information (see BSOD section).
  2. Boot in SafeMode.
  3. Rename: C:\windows\system32\drivers\rqtsentry.sys to rqtsentry-off.sys
  4. Rename: C:\windows\system32\drivers\rqtnetsentry.sys to rqtnetsentry-off.sys
  5. Rename: C:\windows\system32\drivers\i00.sys to i00-off.sys
  6. Reboot.
  7. Follow the BSOD information steps and contact support.  

Forensics Kit failures

In the presence of a failure in generating the Forensics:

  1. Hover with the mouse on the Failed icon available in the Status column.
  2. Collect the error string.
  3. If you see the following message, it means that the PowerShell execution is blocked at endpoint level. Contact the IT administrator to allow it and try again. 
    Package generation failed: Launch script failed: 669
  4. If the error is different, contact support.

Agent services

* Make sure you follow and respect the stated start and stop order.

For stopping the ReaQta services, do as follows:

  1. Keeper
  2. Rqtsentry
  3. i00
Note: rqtnetsentry can be switched off at any time as it is not impacting the order.

For starting the ReaQta services, do as follows: 
 
  1. i00
  2. Rqtsentry
  3. Keeper
Note: rqtnetsentry can be turned on at any time as it is not impacting the order.

General note: All services run as system, and it is not possible to change that

Keeper crash 

If you catch sight of 'keeper' crashes from the event viewer and or the system crash location, proceed as follows:

  1. From services.msc or cmd.exe as admin, run sc query keeper. Verify whether keeper can stay up and running.
  2. In the event keeper cannot stay stable, collect the dump files beginning with “keeper” located in the following folder:

    C:\Windows\System32\config\systemprofile\AppData\Local\CrashDump
  3. Contact support and provide the crash dump collection. 
Important note: Keeper can auto-restart and recover from internal crashes by restarting automatically. This process ensures that the endpoint continues to be protected.

Performance

 

To classify the performance issue, assess which category it falls under:

  • Application-related issue: A specific application or group of applications is slow in performance
  • System wide: Performance degradation
     
When the performance issue is associated to a specific application, you can follow next steps:
  1. Identify the actions performed by the application and the corresponding executable file full path.
  2. If applicable, disable the Anti-Ransomware from the agent “Live Response” and issuing the command: 'antiransomware off'
    image-20220329160821-3
    image-20220329160905-4
  3. Verify whether the issue is solved or not.
  4. If the issue is no longer seen, create a behaviour-based allowlist either by using App Directory or Binary Hash and go with “ransomware behavior” as the trigger type for the specific application.
    image-20220329160459-2
  5. If issue is solved, enable the Anti-Ransomware again by issuing the command 'antiransomware on'
    image-20220329161444-5
  6. If the issue is not solved, verify from the threat hunt the presence of extra processes, potentially involved at the same timeframe, and create more ransomware behavior-based allowlist.
  7. If the issue persists, disable keeper from cmd.exe as admin by issuing the command sc stop keeper.
  8. Try again and note whether the performances change.
  9. Disable rqtsentry from cmd.exe as admin by issuing the command sc stop keeper.
  10. Try again and verify whether the performance changes.
  11. Enable the keeper again from cmd.exe by following the commands: sc start rqtsentry and sc start keeper
  12. If any of the points mentioned previously do not solve the issue grab date, time, endpoint name, full application path, and contact support.

General slowness

  1. Collect information about the performance impact, such as memory, CPU, and the specific operations impacted. Windows Task Manager or Process Hacker (Process Hacker) can be used to collect the memory consumption of the process 'keeper.exe'.

    image-20220330085234-2
  2. Install Microsoft Windows Performance Analyzer
  3. Using xperf, issue the following command from an admin cmd.exe:
      
    xperf -on PROC_THREAD+LOADER+PROFILE+FLT_IO_INIT+FLT_IO+FLT_FASTIO+FLT_IO_FAILURE+FILENAME+DISK_IO+DISK_IO_INIT -stackwalk Profile+MiniFilterPreOpInit+MiniFilterPostOpInit+DiskReadInit+DiskWriteInit+DiskFlushInit -BufferSize 1024 -MinBuffers 256 -MaxBuffers 256 -MaxFile 256 -FileMode Circular
  4. Wait for 5 minutes and issue the following command to stop the capture. If a specific application is noticeably slow, open and use it during the xperf capture time. 
    xperf -stop -d minifilter_and_diskio.etl
    ​​​​​​image-20220329154407-3
This command generates an .etl file that is stored in the current working directory of where xperf tool is executed.
image-20220329155309-1

What ReaQta services run on the endpoint

The services running on a Windows endpoint are:

  • Keeper
  • Rqtsentry
  • Rqtnetsentry
  • i00
Brief Windows agent services description:
 
keeper
 
Keeper is the core service for the agent, the user mod service running with the system privileges. It also is a vital component. Installations without this service are expected to fail or not work properly.
 
Responsible for:
 
  • Gathering all of the data from NanoOs, kernel, and drivers.
  • Assembling collected data for creating alert data.
  • Communicating with Hive.
  • Propagating policies to rqtsentry.
  • Performing the agent update when new build deployed as enabled
Note: There is a child keeper service for each user logged in to the system. It means one main keeper and n keeper services where n = logged in users
 
rqtsentry
 
It is the main service, the filter driver. Installations without this component are expected to fail as this component is a vital one.
 
Responsible for:
 
  • Collecting data from the kernel.
  • Applying policies.
  • Killing processes.
  • Communicating with rqtnetsentry and i00.
rqtnetsentry
 
It is the network filter service. It is also a nonvital component, which means, installation can succeed and work properly without this service

Responsible for:
 
  • Collecting Network connection data.
Note: ReaQta solution can work without this service, but ReaQta lacks visibility into the network connection activity

i00
 
It is the NanoOS service.

Responsible for:
  • Communicates with rqtsentry.
Important: If i00 is ACTIVE (fully functioning) on the system, manually stopping it can cause system instability. In contrast, if it is not active (nonfunctional), it is safe to stop the service.

Note: This service still runs, even though, it is not being used. 

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSVOEH","label":"IBM Security ReaQta"},"ARM Category":[{"code":"a8m3p000000hBSLAA2","label":"Agent-\u003EPerformance"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Product Synonym

ReaQta

Document Information

Modified date:
17 May 2023

UID

ibm16565441

Page Feedback