QRadar: Best practice guidance for application developers

Development teams who write apps need them to be functional when installed and work correctly when installed. This section includes best practice information on communication, infrastructure, and external library guidance for apps to operate in a QRadar environment.

Required

• Do not make assumptions about application containers

  Applications that run on QRadar are containerized and the infrastructure on which the application runs ought to be considered black boxes. Communication to the QRadar Console is defined in the application manifest file. Developers are expected to keep their apps self-contained with communications to QRadar Console on ports 443 for HTTP API traffic and 514 for Syslog messages. Calls to external services are not subject to these restrictions.

  Assumptions that can cause application issues:

  • Assume the app container runs on a local-link 169.254.x.x IP address. In reality, the application could be running on any IP at any time between the QRadar and App Host.
  • Using /proc/meminfo as a gauge for available host memory. The use of meminfo can be dangerous as the app container is liable to be moved to another host at any time where host resources can differ.
  • Providing SSH capabilities to the host QRadar platform. Application should never access QRadar using SSH as raises security issues by providing a means for applications to effectively break out of their contained environments. We do not support this behavior on App Hosts.

• Syslog message restrictions

  RFC 3164 and RFC 5424 Syslog messages are currently supported over port 514 on the Console with TCP or UDP. All QRadar appliances listen by default on port 514 Syslog data. QRadar does not support multiline-syslog on TCP or UDP port 514 currently.

• Avoid making changes to qpylib

  The qpylib library exists to provide helper functions when you interact with QRadar. We are required to update this library from time to time. Currently, the version within in the app takes precedence over the qpylib version on the system; however, this behavior is liable to change in the future. Therefore, it is important that developers do not make changes to qpylib within their apps as it can be overridden. The library itself is hosted open source here: https://github.com/IBM/qpylib. Developers are encouraged to raise issues to be addressed. Developers can move their changes into a separate Python module; however you are responsible for maintaining any forked code.

• Installing extensions that are not present on the X-Force App Exchange

  Only extensions that are published to the X-Force App Exchange run without security warnings. API endpoints such as /api/gui_app_framework/application_creation_task and /config/extension_management/extensions must not be invoked from within the app to install more app content as it might not be available unless it is released.
  
  Note: This restriction does not apply to extensions, which can contain multiple apps. We locate and deploy every app present within an extension on the X-Force App Exchange, so the app is available regardless of whether or not it has a top-level card provided and instead is packaged as a part of your extension.

• Do not log sensitive data

  Any sensitive data such as API tokens and passwords must not be logged within the app, including debug logs. Applications in breach of this security policy are liable to be delisted from the X-Force App Exchange immediately due to the serious security implications of clear text data. Developers can consider a placeholder value, such as '********' when you log sensitive data. Developers must also be careful with event payloads or event messages that contain sensitive data in plain text.
  
  Example of how administrators can obfuscate token values:

  from qpylib import qpylib
  
  endpoint = 'https://example-host/example/endpoint'
  token = 'example-password'
  
  # Exposes the token when debug is enabled
  qpylib.log('Calling endpoint {} (token={}).'.format(endpoint, token), level='debug')
  
  # Logs an obfuscated value if it is set, otherwise None when debug is enabled
  qpylib.log('Calling endpoint {} (token-{}).'.format(endpoint, '****' if token is not None else None), level=debug)

• Ensure that API calls verify certificates

  All requests made to the QRadar must have certificate verification turned on for security. The necessary certificate is provided to apps and the path is /etc/pki/tls/certs/ca-bundle.crt. Developers can use the qpylib REST functions where possible as qpylib performs certificate handling for you. Certificate validation is checked during app validation. Here is an example that displays incorrect validation handling and a better alternative.

  from qpylib import qpylib
  import os.path
  import requests
  
  # No certificate validation performed during the request. Causes errors in startup.log
  response = requests.get('https://qradar-host/api/endpoint', verify=False)
   
  # Use the default cert, then verify the path
  cert_path = '/etc/pki/tls/certs/ca-bundle.crt'
  response = requests.get('https://qradar-host/api/endpoint', verify=cert_path)
   
  # Handles certificate verification for you
  response = qpylib.REST('GET', '/api/endpoint')

• Avoid UDP when you communicate with on-premise resources

  There are a number of app use-cases that require access to resources running on customer's private networks, such as querying an LDAP server for user data. These types of queries are supported, but only when you use TCP for the communication. This requirement might change in the future, but developers cannot use UDP in these scenarios for now.

Recommended

• Incorporate graceful shutdowns into your apps

  This is not currently easy to achieve, therefore we are only currently recommending it. Moving forward developers ought to be mindful of the effects of shutting down apps can have on the integrity of their data.
  
  Currently, applications support /src_deps/cleanup.sh as a means of performing shutdown operations. Anything within this script is executed before the app is shut down. For example, take the following Python code that can be included in a flask app:

  import atexit
  from qpylib import qpylib
  
  def shutdown():
      """
      Function registered with atexit to perform
      shutdown operations
      """
      qpylib.log("Performing shutdown operations.")
  
  atexit.register(shutdown)

  This registers an example function that is invoked when a signal is sent to the app. This signal can currently be sent with the cleanup.sh script:

  supervisorctl stop startflask

• Avoid UDP when you communicate with on-premise resources

  There are a number of app use-cases that require access to resources running on customer's private networks such as querying an LDAP server for user data. These types of queries are supported, but only when you use TCP for the communication. This restriction might change in the future; however, for now developers must avoid UDP in these scenarios for now.

• Be sensible when you make API calls

  Developers are expected to not export excessive amounts of data to external services, instead minimize or only export required data. This does not currently apply to API calls to the QRadar platform; however, is it possible that your app fails validation if it is found to make excessive API calls. For example,

  import requests
  
  success = False
   
  while not success:
    response = requests.get('https://some.system/some/endpoint')
    success = (response.status_code == 200) # if some.system is down this will never return True
• Ensure you have appropriate logging and log levels

  It is important the logs produced by the app are meaningful for use in diagnosing issues and debugging. This often a balancing act between providing a meaningful context of what is going on within the app and too much unnecessary noise. Storing and retaining logs incurs costs, so we ask developers to be sensible with logging. Important information should be logged in a clear and concise manner, with more verbose messages reserved for the debug log level.

  All errors must be logged with the error log level. We see scenarios like the following, which should be avoided at all costs as it leads to errors being swallowed:

  from qpylib import qpylib
  
  try:
    some_function()
  except Exception as e:
    qpylib.log("An error occurred: {}".format(e), level='debug') # Incorrect level for errors

  try:
    some_function()
  except Exception as e:
    return # No logging to indicate an error
• Handle connectivity problems

  It is inevitable that networking issues can occur during normal operations of your application. When you develop your app, you need to handle these errors gracefully. A common pitfall seen is that developers presume successful responses to data requests:

  import requests
  from flask import render_template
   
  response = requests.get('https://some.system/some/endpoint')
  data = response.json()
  field = data['field'] # If the request fails this will result in an unhandled error - crashing the page
  return render_template(some-page.html, field=field)

  Instead, developers should do their best to handle these cases, and ideally return an HTML page, which can render when an error occurs:

  import requests
  from flask import render_template
  from qpylib import qpylib
   
  response = requests.get('https://some.system/some/endpoint')
  if response.status_code == 200:
    data = response.json()
    field = data['field'] # If the request fails this will result in an unhandled error - crashing the page
    return render_template('some-page.html', field=field)
  else:
    qpylib.log('An error occurred while retrieving data. Request to https://some.system/some/endpoint returned {} with text {}'.format(response.status_code, response.text), level='error')
    return render_template('error-page.html', error_response=response)

  Rendering a connection error page can go a long way in improving usability and support for your app.

Required

• Ensure that stored sensitive information is encrypted

  It is a common requirement for apps to require a token for accessing the QRadar APIs or an external service. This is normally persisted under the /store directory within the application. Under no circumstances should this be stored in plain text.
  
  The qpylib library comes with support for encrypting and decrypting values. The following is a simple example of how encryption for tokens can be achieved:
   

  from qpylib.encdec import Encryption
   
  # Encryption object used to perform encryption. Takes a name key used to identify the encrypted value and a user parameter
  token_encryptor = Encryption({
    'name': 'api_token',
    'user': 'myappuser'
  })
   
  # Encrypts the value of the API token and stores it for the user at the provided name key
  token_encryptor.encrypt('some-api-token')
   
  # Reference the same name key and user when decrypting
  token_decryptor = Encryption({
    'name': 'api_token',
    'user': 'myappuser'
  })
   
  # API token is read from disk, decrypted and set for use
  api_token = token_decryptor.decrypt()
  print(api_token) # 'some-api-token'
• Do not store functional data in the store log directory

  All app logs should be written to /store/log. The /store/log directory is regularly walked with its contents written to stdout for log collection and truncated. Therefore developers must avoid writing any data, including logs, to this directory and its sub-folders if they later expect to read from it within their app. Functional data can be written to any other sub-directory under /store and is persisted for later use within your app.

• Ensure file ownership is set correctly during app initialization

  If data in /store is owned by a user other than root then developers must ensure that file/folder permissions are correctly set during the initialization of the app. Ownership is configured in /src_deps/init/ordering.txt, for example:

  chown -R someuser:somegroup /store/data

  If the location is a directory, then ensure you include the recursive option for sub-directory ownership as defined in the example. If the correct ownership permissions are not set, then the app might fail to function properly for users.

Recommended

• Only persist to /store when necessary

  The /store directory is there for persistent data and should only be used as such. For anything that does not absolutely need persisted use a directory such as /tmp instead. Conversely always insure that data that should be persisted, such as configuration files, are written to /store. If this is not done then you will lose the data on app restarts.

This section relates to the resource usage of an app (memory and CPU).

Required

• Understand your memory requirements

  Developers should have an awareness of how much memory their apps will require while under load. The required memory should be gauged on what you expect when your app is running at its highest expected capacity. Failing to set a suitable memory value will result in crashes within the app. A part of this should be supplying the validation team with the necessary inputs to stress test your app to ensure its requested memory is sufficient.
  
  You can monitor the resource usage of an app container on your development environment. To perform this first run /opt/qradar/support/recon ps to obtain the ID of your app while it is running:

  ~# /opt/qradar/support/recon ps
  App-ID  Name                Managed Host ID Workload ID     Service Name    AB  Container Name  CDEGH   Port    IJKL
  1001    Example App         53              apps            qapp-1001       ++  qapp-1001       +++++   5000    ++++

  Using this ID you can then identify which container corresponds to your app with the docker ps command:

  ~# docker ps
  CONTAINER ID        IMAGE                                                                   COMMAND                  CREATED             STATUS              PORTS                     NAMES
  29fb678921d4        console.localdeployment:5000/qapp/1001:1.0.0-20200213115517             "sh /start_container…"   5 days ago          Up 5 days           0.0.0.0:328

  You can then use the container name or container ID to check the resource usage of the running app container with docker stats <NAME>:

  ~# docker stats qapp-1001-hb9aw75x
  CONTAINER ID        NAME                 CPU %               MEM USAGE / LIMIT   MEM %               NET I/O             BLOCK I/O           PIDS
  29fb678921d4        qapp-1001-hb9aw75x   0.57%               115.5MiB / 600MiB   19.24%              37.8MB / 463MB      1.3GB / 610MB       21

• Ensure memory is set in the application manifest

  Understanding the memory requirements of your app should enable you to accurately set the memory field within the resources section of the application manifest. By default this is assigned 200Mb but it is highly unlikely that this value is optimal. Increase or reduce it to a point where you feel the app will not have issues. Setting an unnecessarily high memory value will increase the resources required to run your app. The full memory allocation is set aside for the application to use, which can reduce resources from over-utilized appliances.
  
  Memory is a field within resources, which itself is a top-level field of the application manifest. The value should be an integer representing the required memory in megabytes. For example:

  {
    "resources": {
      "memory": 400
    }
  }
• Understanding CPU usage for your app

  Docker stats can be used to identify the CPU% used by your application. For example,

  ~# docker stats qapp-1001-hb9aw75x
  CONTAINER ID        NAME                 CPU %               MEM USAGE / LIMIT   MEM %               NET I/O             BLOCK I/O           PIDS
  29fb678921d4        qapp-1001-hb9aw75x   0.57%               115.5MiB / 600MiB   19.24%              37.8MB / 463MB      1.3GB / 610MB       21

  The 0.57% in this scenario shows the app is consuming 0.57% of a single core, well within the 0.1 (10%) of the single core we default to. A value of 100% would correspond to 1 core, 200% for 2 cores and so forth.