QRadar EDR: EDR Linux agent 0.80.1 fails to start on some endpoints due to an eBPF probe loading issue.

Troubleshooting

Problem

EDR linux agent 0.80.1 fails to start on some endpoints due to eBPF probe loading issue.

Symptom

The log traces will show a long eBPF probe dump similar to following:

Apr 10 15:33:36 debian10.localdomain keeperx-loader.sh[1736]: from 1992 to 1993: R0=inv(id=0,umin_value=18446744071562067968,var_off=(0xffffffff80000000; 0x7fffffff)) R6=ctx(id=0,off=0,imm=0) R7=map_value(id=0,off=0,ks=4,vs=197,imm=0) R8=inv(id=0,umin_value=77826,umax_value=4295098342,var_off=(0x0; 0x1ffffffff)) R9=inv(id=0,umin_value=18446744071562067968,var_off=(0xffffffff80000000; 0x7fffffff)) R10=fp0,call_-1 fp-88=map_value fp-96=map_value fp-104=map_value fp-112=ctx
Apr 10 15:33:36 debian10.localdomain keeperx-loader.sh[1736]: 1993: (bf) r9 = r0
Apr 10 15:33:36 debian10.localdomain keeperx-loader.sh[1736]: 1994: (67) r9 <<= 32
Apr 10 15:33:36 debian10.localdomain keeperx-loader.sh[1736]: 1995: (c7) r9 s>>= 32
Apr 10 15:33:36 debian10.localdomain keeperx-loader.sh[1736]: 1996: (b7) r1 = 2
Apr 10 15:33:36 debian10.localdomain keeperx-loader.sh[1736]: 1997: (6d) if r1 s> r9 goto pc-1526
Apr 10 15:33:36 debian10.localdomain keeperx-loader.sh[1736]:  R0=inv(id=0,umin_value=18446744071562067968,var_off=(0xffffffff80000000; 0x7fffffff)) R1=inv2 R6=ctx(id=0,off=0,imm=0) R7=map_value(id=0,off=0,ks=4,vs=197,imm=0) R8=inv(id=0,umin_value=77826,umax_value=4295098342,var_off=(0x0; 0x1ffffffff)) R9=inv(id=0,umin_value=18446744071562067968,var_off=(0xffffffff80000000; 0x7fffffff)) R10=fp0,call_-1 fp-88=map_value fp-96=map_value fp-104=map_value fp-112=ctx
Apr 10 15:33:36 debian10.localdomain keeperx-loader.sh[1736]: 1998: (bf) r2 = r8
Apr 10 15:33:36 debian10.localdomain keeperx-loader.sh[1736]: 1999: (07) r2 += -1
Apr 10 15:33:36 debian10.localdomain keeperx-loader.sh[1736]: 2000: (57) r2 &= 131071
Apr 10 15:33:36 debian10.localdomain keeperx-loader.sh[1736]: 2001: (79) r1 = *(u64 *)(r10 -88)
Apr 10 15:33:36 debian10.localdomain keeperx-loader.sh[1736]: 2002: (0f) r1 += r2

Cause

This issue is caused by the Falco eBPF probe used by QRadar EDR Linux Agent v0.80.1, which has a known issue with Debian 10 kernels.

Environment

QRadar EDR Linux Agent 0.80.1
Issue is currently reported in Debian kernel 4.19.0-26 and Ubuntu 5.4.0-1106 kernel.

Diagnosing The Problem

Verify Kernel version using following command:

uname -a

Resolving The Problem

This issue is planned to be fixed in future releases. Temporary workaround is provided below. There are 2 possible scenarios for this workaround:

Scenario 1: New Installation

Install prerequisite packages as described in our installation document.

sudo apt-get install --no-install-recommends curl dkms gcc linux-headers-$(uname -r) make

Use the following command to force usage of Falco kernel module (skipping eBPF probe):
1. ```
sudo sh -c "echo FORCE_KMOD=1 >> /etc/reaqtahive.d/keeperx.env"
```
Use the following command to load unsigned module ignoring kernel taint state:
1. ```
sudo sh -c "echo KMOD_IGNORE_TAINT=1 >> /etc/reaqtahive.d/keeperx.env"
```

Restart the agent service:

```
sudo systemctl reset-failed keeperx
```
```
sudo systemctl restart keeperx
```

Scenario 2: Upgrade from previous version of QRadar EDR agent.
1. Install dkms as additional pre-requisite package:
  1. ```
  sudo apt-get install --no-install-recommends dkms
```
2. Use the following command to force usage of Falco kernel module (skipping eBPF probe):
  1. ```
  sudo sh -c "echo FORCE_KMOD=1 >> /etc/reaqtahive.d/keeperx.env"
```
3. Assuming KMOD_IGNORE_TAINT flag is already set for the installed agent, proceed with upgrade command:
  1. ```
  sudo dpkg -i hive-installer-0.80.1-x86_64.deb
```
2. For more information on agent upgrade refer this.

Related Information

Agent Upgrade

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSOO77","label":"IBM Security QRadar EDR"},"ARM Category":[{"code":"a8m3p000000PCPsAAO","label":"Support"},{"code":"a8m3p0000000rbnAAA","label":"Support-\u003EAdministration Task"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Tips