IBM Support

QRadar: Troubleshooting email issues

How To


Summary

QRadar sends notifications for rules and reporting through email, which can fail to send as expected.

Objective

Validate current email configuration and troubleshoot delivery issues

Steps

Validate mail configuration in QRadar

The first step is to verify that the mail server is set properly. Verify mail server configuration for the Console and each Event or Flow Processor. Mail responses are sent from the processor where each rule is matched. Reports are emailed from the Console.
To verify mail server configuration:
  1. Click the Admin tab > System and License Management
  2. Double-click the Host > Email server
  3. In QRadar 7.4 and later you select from your saved mail servers and can manage them from the link on this page:
    image 11397
    If you need to add the mail server, click Manage Email Servers and add the wanted server information:
    image 11398
  4. Verify that localhost is not selected as it might result in email failing to send. If there are no saved mail servers, then enter the mail server information to be set here.
  5. On QRadar 7.4 and later, you can test the connection in the UI by clicking "Test Connection". If the test fails, there is a connection issue between QRadar and the destination mail server. The connection problem might be a network issue or a permissions issue on the destination mail server.  For example, the QRadar server might not be recognized as a valid source IP on the destination mail server.
  6. Next, go to the CLI and test the connection manually. Detailed steps for testing the connection can be seen here:
    QRadar: How can you test email services from QRadar

Review postfix and mail log logging

The errors provided from postfix are verbose so finding issues can be simple. The mail log file to watch is /var/log/maillog in here you can see all the logs from postfix (mail sending, failing to send, postfix restarts). A normal instance of mail sending looks similar to this example:
Mar 11 12:14:29 qradar74-console-primary postfix/pickup[19453]: 811511FB: uid=0 from=<root>
Mar 11 12:14:29 qradar74-console-primary postfix/cleanup[5884]: 811511FB: message-id=<20210311171429.811511FB@qradar74-console-primary.lab.local>
Mar 11 12:14:29 qradar74-console-primary postfix/qmgr[27341]: 811511FB: from=<root@qradar74-console-primary.lab.local>, size=1029, nrcpt=1 (queue active)
Mar 11 12:14:29 qradar74-console-primary postfix/local[5889]: 811511FB: to=<root@qradar74-console-primary.lab.local>, orig_to=<root>, relay=local, delay=440, 
    delays=440/0.02/0/0.01, dsn=2.0.0, status=sent (delivered to mailbox)
Mar 11 12:14:29 qradar74-console-primary postfix/qmgr[27341]: 811511FB: removed

In this example, the ID of the message is 811511FB.  You can grep for a specific message ID to isolate the relevant lines in the mail log file. The "removed" message indicates that the message is removed from the local queue because it was successfully sent to the destination mail server.  If the message is still not delivered to the destination mailbox, then the problem is likely on the mail server, and you need to follow up with the mail server admin to proceed.

The following would indicate that no mail server is set in the UI or is set to localhost (note relay=none):
Jan 26 17:18:13 xxxxx postfix/smtp[63019]: D2579807C90: to=<xxxx@yyyy.com>, relay=none, delay=0.14, 
delays=0.13/0.01/0/0, dsn=4.3.5, status=deferred (mail for localhost loops back to myself)

Common problems

  • Postfix attempting to use MX records:
    You might see mail failing to send in /var/log/maillog with a message like this:
    Apr  7 22:32:39 qradar_hostname postfix/smtp[2504]: 062D9401438: to=<xxxx@yyyy.com>, relay=none, 
    delay=155825, delays=155825/0.03/0.02/0, dsn=4.4.3, status=deferred (Host or domain name not found. 
    Name service error for name=<mailserver> type=MX: Host not found, try again)
     
    The part to look for here is type=MX. This setting indicates that postfix is attempting to use mail exchange records. A DNS 'mail exchange' (MX) record directs email to a mail server. If it is pointing to an incorrect server or there are no records at all, sending an email fails as shown in the error message. To force postfix to use the relay host specified in the UI, look at the file /etc/postfix/main.cf and view the following section:
    # The relayhost parameter specifies the default host to send mail to
    # when no entry is matched in the optional transport(5) table. When
    # no relayhost is given, mail is routed directly to the destination.
    #
    # On an intranet, specify the organizational domain name. If your
    # internal DNS uses no MX records, specify the name of the intranet
    # gateway host instead.
    #
    # In the case of SMTP, specify a domain, host, host:port, [host]:port,
    # [address] or [address]:port; the form [host] turns off MX lookups.
    #
    # If you're connected via UUCP, see also the default_transport parameter.
    #
    relayhost=testmailserver.com

    Make sure that you surround the mail server with [] brackets, which disable MX lookups, in this case it would be "relayhost=[testmailserver.com]". You can modify this file in the CLI and the restart postfix with the command:
    systemctl restart postfix
    
  • Root user inbox full:
    1. Review /var/log/mail for similar issues:
      postfix/local[39997]: 8266FE00211: to=<root@example.postfix.com<root@example.postfix.com>
      <root@example.postfix.com>>, orig_to=<root>, relay=local, delay=5.5, delays=5.4/0.03/0/0.04, 
      dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing
      message: File too large)
    2. Check the file for the number of emails, reports, alerts by using the command:
      cat /var/mail/root
    3. Either increase the queue or truncate the conflicting file to reset the queue by typing the command:
      truncate -s0 /var/mail/root
    4. Change the mail server on the Event Processor to a specific email server instead of localhost.
      image 11398

      For more information about adding an email server, see the document on Adding an email server.
  • Incorrect smtp_host_lookup setting:
    If the mail server IP address association is set correctly in the /etc/hosts file but the smtp_host setting is incorrect, errors like the following might be visible in the maillog file:
    Name service error for name=example.com type=AAAA: Host not found, try again
    In this case, you need to update the main.cf file to ensure that postfix uses the /etc/hosts file to resolve to the mail server instead of DNS.
    1. In the file /etc/postfix/main.cf, look for the setting smtp_host_lookup. The setting is usually near the end of the file.
      smtp_host_lookup = dns, native
    2. If the setting looks like smtp_host_lookup = dns, native, the setting is required to be updated to look like: smtp_host_lookup = native.
    3. To update the smtp_host_lookup setting, edit the file /opt/ibm/si/si-postfix/bin/configure-postfix.sh and change the smtp_host_lookup from "dns, native" to "native" only.
      Before updating the configure-postfix.sh file, the file looks like.
      # Use the native service for hosts not found in the DNS
      configure_relay_hostlookup() {
          postconf -e "smtp_host_lookup = dns, native"
      }
      After modifying the configure-postfix.sh file with a text editor, the file will look like the following.
      # Use the native service for hosts not found in the DNS
      configure_relay_hostlookup() {
          postconf -e "smtp_host_lookup = native"
      }
      Save the configure-postfix.sh file.
    4. Run the /opt/ibm/si/si-postfix/bin/configure-postfix.sh script to update the postfix configuration.
      /opt/ibm/si/si-postfix/bin/configure-postfix.sh
      
      INFO: Backing up the current Postfix config
      INFO: mailbox_size_limit not set. Using default value...
      WARN: The current Postfix config has customizations to mail connection settings.
      WARN: TLS, SMTP, and SASL connection settings will be reset to distribution defaults before continuing.
      INFO: Configure postfix to use native naming service when DNS cannot resolve hostnames
      INFO: TLS support is disabled
      INFO: TLS will not be established for outgoing mail connections
      INFO: OK configured SMTP mail relay
    5. Review the /etc/postfix/main.cf file to confirm the setting is updated.
      smtp_host_lookup = native
    6. Restart the postfix service.
      systemctl restart postfix

 

Checking the mail queue

If QRadar is unable to send email for any amount of time, it is likely that the backlog of mail in the queue is waiting to be sent. To check the current size of the mail queue use the mailq command. There might be thousands of emails waiting to be sent when the issue is resolved. A poorly tuned rule can also generate thousands of email responses and fill the mail queue in a short time.
 
You can delete all mail in the queue with the command:
postsuper -d ALL

If the issue is resolved but there is still mail in the queue, you can run this command from the CLI to force all mail in the queue to be sent:
postfix flush

Results
These steps can help resolve issues whether they are coming from a mail server issue. If incorrect rule tuning is generating excessive mail responses, you can also purge the mail queue. If incorrect rule tuning is generating excessive mail responses, you can also purge the mail queue.

Additional Information

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwsyAAA","label":"Admin Tasks"},{"code":"a8m0z000000cwthAAA","label":"Offenses"},{"code":"a8m0z000000cwtmAAA","label":"Reports"},{"code":"a8m0z000000cwtrAAA","label":"Rules"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]

Document Information

Modified date:
31 August 2023

UID

ibm16484205