How To
Summary
QRadar sends notifications for rules and reporting through email, which can fail to send as expected.
Objective
Validate current email configuration and troubleshoot delivery issues
Steps
Validate mail configuration in QRadar
The first step is to verify that the mail server is set properly. Verify mail server configuration for the Console and each Event or Flow Processor. Mail responses are sent from the processor where each rule is matched. Reports are emailed from the Console.
To verify mail server configuration:
- Click the Admin tab > System and License Management
- Double-click the Host > Email server
- In QRadar 7.4 and later you select from your saved mail servers and can manage them from the link on this page:
If you need to add the mail server, clickManage Email Servers
and add the wanted server information: - Verify that
localhost
is not selected as it might result in email failing to send. If there are no saved mail servers, then enter the mail server information to be set here. - On QRadar 7.4 and later, you can test the connection in the UI by clicking "Test Connection". If the test fails, there is a connection issue between QRadar and the destination mail server. The connection problem might be a network issue or a permissions issue on the destination mail server. For example, the QRadar server might not be recognized as a valid source IP on the destination mail server.
- Next, go to the CLI and test the connection manually. Detailed steps for testing the connection can be seen here:
QRadar: How can you test email services from QRadar
Review postfix and mail log logging
The errors provided from postfix are verbose so finding issues can be simple. The mail log file to watch is /var/log/maillog in here you can see all the logs from postfix (mail sending, failing to send, postfix restarts). A normal instance of mail sending looks similar to this example:
Mar 11 12:14:29 qradar74-console-primary postfix/pickup[19453]: 811511FB: uid=0 from=<root>
Mar 11 12:14:29 qradar74-console-primary postfix/cleanup[5884]: 811511FB: message-id=<20210311171429.811511FB@qradar74-console-primary.lab.local>
Mar 11 12:14:29 qradar74-console-primary postfix/qmgr[27341]: 811511FB: from=<root@qradar74-console-primary.lab.local>, size=1029, nrcpt=1 (queue active)
Mar 11 12:14:29 qradar74-console-primary postfix/local[5889]: 811511FB: to=<root@qradar74-console-primary.lab.local>, orig_to=<root>, relay=local, delay=440,
delays=440/0.02/0/0.01, dsn=2.0.0, status=sent (delivered to mailbox)
Mar 11 12:14:29 qradar74-console-primary postfix/qmgr[27341]: 811511FB: removed
In this example, the ID of the message is 811511FB. You can grep for a specific message ID to isolate the relevant lines in the mail log file. The "removed" message indicates that the message is removed from the local queue because it was successfully sent to the destination mail server. If the message is still not delivered to the destination mailbox, then the problem is likely on the mail server, and you need to follow up with the mail server admin to proceed.
The following would indicate that no mail server is set in the UI or is set to localhost (note relay=none):
Jan 26 17:18:13 xxxxx postfix/smtp[63019]: D2579807C90: to=<xxxx@yyyy.com>, relay=none, delay=0.14,
delays=0.13/0.01/0/0, dsn=4.3.5, status=deferred (mail for localhost loops back to myself)
Common problems
- Postfix attempting to use MX records:
You might see mail failing to send in /var/log/maillog with a message like this:
Apr 7 22:32:39 qradar_hostname postfix/smtp[2504]: 062D9401438: to=<xxxx@yyyy.com>, relay=none, delay=155825, delays=155825/0.03/0.02/0, dsn=4.4.3, status=deferred (Host or domain name not found. Name service error for name=<mailserver> type=MX: Host not found, try again)
The part to look for here istype=MX
. This setting indicates that postfix is attempting to use mail exchange records. A DNS 'mail exchange' (MX) record directs email to a mail server. If it is pointing to an incorrect server or there are no records at all, sending an email fails as shown in the error message. To force postfix to use the relay host specified in the UI, look at the file /etc/postfix/main.cf and view the following section:# The relayhost parameter specifies the default host to send mail to # when no entry is matched in the optional transport(5) table. When # no relayhost is given, mail is routed directly to the destination. # # On an intranet, specify the organizational domain name. If your # internal DNS uses no MX records, specify the name of the intranet # gateway host instead. # # In the case of SMTP, specify a domain, host, host:port, [host]:port, # [address] or [address]:port; the form [host] turns off MX lookups. # # If you're connected via UUCP, see also the default_transport parameter. # relayhost=testmailserver.com
Make sure that you surround the mail server with [] brackets, which disable MX lookups, in this case it would be"relayhost=[testmailserver.com]"
. You can modify this file in the CLI and the restart postfix with the command:systemctl restart postfix
- Root user inbox full:
- Review /var/log/mail for similar issues:
postfix/local[39997]: 8266FE00211: to=<root@example.postfix.com<root@example.postfix.com> <root@example.postfix.com>>, orig_to=<root>, relay=local, delay=5.5, delays=5.4/0.03/0/0.04, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
- Check the file for the number of emails, reports, alerts by using the command:
cat /var/mail/root
- Either increase the queue or truncate the conflicting file to reset the queue by typing the command:
truncate -s0 /var/mail/root
- Change the mail server on the Event Processor to a specific email server instead of localhost.
For more information about adding an email server, see the document on Adding an email server.
- Review /var/log/mail for similar issues:
- Incorrect smtp_host_lookup setting:
If the mail server IP address association is set correctly in the /etc/hosts file but the smtp_host setting is incorrect, errors like the following might be visible in the
maillog
file:Name service error for name=example.com type=AAAA: Host not found, try again
main.cf
file to ensure that postfix uses the/etc/hosts
file to resolve to the mail server instead of DNS.-
In the file
/etc/postfix/main.cf
, look for the settingsmtp_host_lookup
. The setting is usually near the end of the file.smtp_host_lookup = dns, native
- If the setting looks like
smtp_host_lookup = dns, native
, the setting is required to be updated to look like:smtp_host_lookup = native.
-
To update the smtp_host_lookup setting, edit the file /opt/ibm/si/si-postfix/bin/configure-postfix.sh and change the smtp_host_lookup from "dns, native" to "native" only.Before updating the configure-postfix.sh file, the file looks like.
# Use the native service for hosts not found in the DNS configure_relay_hostlookup() { postconf -e "smtp_host_lookup = dns, native" }
After modifying the configure-postfix.sh file with a text editor, the file will look like the following.# Use the native service for hosts not found in the DNS configure_relay_hostlookup() { postconf -e "smtp_host_lookup = native" }
Save the configure-postfix.sh file. -
/opt/ibm/si/si-postfix/bin/configure-postfix.sh script to update the postfix configuration.Run the
/opt/ibm/si/si-postfix/bin/configure-postfix.sh INFO: Backing up the current Postfix config INFO: mailbox_size_limit not set. Using default value... WARN: The current Postfix config has customizations to mail connection settings. WARN: TLS, SMTP, and SASL connection settings will be reset to distribution defaults before continuing. INFO: Configure postfix to use native naming service when DNS cannot resolve hostnames INFO: TLS support is disabled INFO: TLS will not be established for outgoing mail connections INFO: OK configured SMTP mail relay
-
Review the /etc/postfix/main.cf file to confirm the setting is updated.
smtp_host_lookup = native
- Restart the postfix service.
systemctl restart postfix
-
Checking the mail queue
If QRadar is unable to send email for any amount of time, it is likely that the backlog of mail in the queue is waiting to be sent. To check the current size of the mail queue use the
mailq
command. There might be thousands of emails waiting to be sent when the issue is resolved. A poorly tuned rule can also generate thousands of email responses and fill the mail queue in a short time.You can delete all mail in the queue with the command:
postsuper -d ALL
If the issue is resolved but there is still mail in the queue, you can run this command from the CLI to force all mail in the queue to be sent:
postfix flush
Results
These steps can help resolve issues whether they are coming from a mail server issue. If incorrect rule tuning is generating excessive mail responses, you can also purge the mail queue. If incorrect rule tuning is generating excessive mail responses, you can also purge the mail queue.
Additional Information
For more information, see QRadar: Email queue fills up from rule response.
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwsyAAA","label":"Admin Tasks"},{"code":"a8m0z000000cwthAAA","label":"Offenses"},{"code":"a8m0z000000cwtmAAA","label":"Reports"},{"code":"a8m0z000000cwtrAAA","label":"Rules"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]
Was this topic helpful?
Document Information
Modified date:
31 August 2023
UID
ibm16484205