IBM Support

IBM AIX: Using IPSec Rules to Filter Network Traffic

How To


Summary

This document explains how to use basic IPSec configuration to filter network traffic originating from or destined to the underlying AIX host.

Steps

Configuring AIX host to deny incoming ICMP packets (ping requests):
AIX host can be configured to drop ping requests as follows:
1. Run IPv4 IPSec SMIT menu, as follows:
# smitty ipsec4
2. Choose "Advanced IP Security Configuration":
Move cursor to desired item and press Enter.

  Start/Stop IP Security
  Basic IP Security Configuration  
  Advanced IP Security Configuration  <----
3.Pick "Configure IP Security Filter Rules":
  Configure IP Security Filter Rules  <----
  List Active IP Security Filter Rules
  Activate/Update/Deactivate IP Security Filter Rule
...
  View IKE XML DTD
4. Then, choose "Add an IP Security Filter Rule":
  List IP Security Filter Rules
  Add an IP Security Filter Rule  <----
  Change IP Security Filter Rules
  Move IP Security Filter Rules
  Export IP Security Filter Rules
  Import IP Security Filter Rules
  Delete IP Security Filter Rules
5. Afterwards, the IPSec filtering criteria are displayed, as follows:
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
  
                                                        [Entry Fields]
* Rule Action                                        [permit]                                                                                                                                            +
* IP Source Address                                  []
* IP Source Mask                                     []
  IP Destination Address                             []
  IP Destination Mask                                []
* Apply to Source Routing? (PERMIT/inbound only)     [yes]                                                                                                                                               +
* Protocol                                           [all]                                                                                                                                               +
* Source Port / ICMP Type Operation                  [any]                                                                                                                                               +
* Source Port Number / ICMP Type                     [0]                                                                                                                                                  #
* Destination Port / ICMP Code Operation             [any]                                                                                                                                               +
* Destination Port Number / ICMP Type                [0]                                                                                                                                                  #
* Routing                                            [both]                                                                                                                                              +
* Direction                                          [both]                                                                                                                                              +
* Log Control                                        [no]                                                                                                                                                +
* Fragmentation Control                              [0]                                                                                                                                                 +
* Interface                                          []                                                                                                                                                  +
  Expiration Time  (sec)                             []                                                                                                                                                   #
  Pattern Type                                       [none]                                                                                                                                              +
  Pattern / Pattern File                             []
  Description                                        []
Now let's explain the filtering criteria we're going to tune which defines the IPSec rule behavior:
  • Rule Action: Sets the action taken by the current IPsec filter rule, whether to permit or deny the packet that meets the following criteria. Default is "permit".
  • IP Source Address: The IP source address of the TCP or UDP packet undergoing the filtering process.
  • IP Source Mask: The netmask for the IP source address.
  • IP Destination Address: The IP source address of the TCP or UDP packet undergoing the filtering process.
  • IP Destination Mask: The netmask for the IP destination address.
  • Apply to Source Routing? (PERMIT/inbound only): Applies the current rule to the inbound traffic that meets the rule criteria. Default value is "yes".
  • Protocol: Sets the filter to a specific protocol within the TCP/IP protocol stack. Some routing protocol like OSPF can be specified as well. Default is "all".
  • Source Port / ICMP Type Operation: The logical operation that defines the port (or range of ports) targeted by the current rule. Options being: "eq", "neq", "lt", "gt", "le" and "ge", which mean "equal", "not equal", less than", "greater than", "less than or equal" and "greater than or equal" respectively. Default option is "any".
  • Source Port Number / ICMP Type: Filter based on the TCP or UDP source port of the packet undergoing the IPSec filtering process.
  • Destination Port / ICMP Code Operation: Same as source port.
  • Destination Port Number / ICMP Type: Filter based on the TCP or UDP destination port of the packet undergoing the IPSec filtering process.
  • Direction: Filter based on the packet direction. Options being: "inbound", "outbound" or "both", with "both" being the default.
  • Interface: Specifies which network interface the current rule applies to. It can be specific to a single network interface or general for all interfaces.
6. Next step is to set filtering criteria for the IPSec rule based on the desired action. We are going to configure our IPSec rule to deny ping requests that come the underlying AIX host.
The IPSec rule looks like the following:
  Rule Action                                        [deny]                                                                                                                                              +
  IP Source Address                                  [0.0.0.0]
  IP Source Mask                                     [255.255.255.255]
  IP Destination Address                             [9.40.205.53]
  IP Destination Mask                                [255.255.255.0]
  Apply to Source Routing? (PERMIT/inbound only)     [yes]                                                                                                                                               +
  Protocol                                           [icmp]                                                                                                                                              +
  Source Port / ICMP Type Operation                  [any]                                                                                                                                               +
  Source Port Number / ICMP Type                     [0]                                                                                                                                                  #
  Destination Port / ICMP Code Operation             [any]                                                                                                                                               +
  Destination Port Number / ICMP Type                [0]                                                                                                                                                  #
  Routing                                            [both]                                                                                                                                             +
  Direction                                          [inbound]                                                                                                                                           +
  Log Control                                        [no]                                                                                                                                                +
  Fragmentation Control                              [all packets]                                                                                                                                       +
  Tunnel ID                                          [0]                                                                                                                                                 +#
  Interface                                          [en0]                                                                                                                                               +
  Expiration Time  (sec)                             [0]                                                                                                                                                  #
  Pattern Type                                       [none]                                                                                                                                              +
  Pattern / Pattern File                             []
  Description                                        []
7. Press "Enter" to configure the changes. Once done, you receive a message like the following:
Command: OK            stdout: yes           stderr: no

Before command completion, additional instructions may appear below.

Filter rule 3 for IPv4 has been added successfully.

8.Now that we successfully configured an IPSec filter rule, we need to activate IPSec filtering, as follows:
# smitty ipsec4
<<Enter>>

Move cursor to desired item and press Enter.

  Start/Stop IP Security   <---- Choose this one
  Basic IP Security Configuration
  Advanced IP Security Configuration
<<Enter>>


Move cursor to desired item and press Enter.

  Start IP Security  <---- Choose this one
  Stop IP Security
<<Enter>>
9. Upon starting IPSec, you can choose to start it just in the mean time or on boot time as well. We are going to make it survive the reboot by leaving it at the default value of "Now and After Reboot":
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
  
                                                        [Entry Fields]
  Start IP Security                                  [Now and After Reboot]     <----                                                                                                                         +
  Deny All Non_Secure IP Packets                     [no]         
<<Enter>>
10. Once started successfully, the following "OK" message is displayed:
Command: OK            stdout: yes           stderr: no

Before command completion, additional instructions may appear below.

ipsec_v4 Available
Default rule for IPv4 in ODM has been changed.
Successfully set default action to PERMIT
11. Verify IPSec is started by making sure ipsec_v4 be in the Available state in the ODM customized database, as follows:
# lsdev -Cc ipsec
ipsec_v4 Available  IP Version 4 Security Extension
ipsec_v6 Available  IP Version 6 Security Extension
12. List the IPSec table to make sure the recently configured rule is saved there, as follows:
# lsfilt -v4 -a
Beginning of IPv4 filter rules.
Rule 1:
*** Dynamic filter placement rule for IKE tunnels ***
Logging control     : no

Rule 2:  <---- This is the rule of our interest. 
Rule action         : deny
Source Address      : 0.0.0.0
Source Mask         : 255.255.255.255
Destination Address : 9.40.205.53
Destination Mask    : 255.255.255.0
Source Routing      : yes
Protocol            : icmp
ICMP type           : any 0
ICMP code           : any 0
Scope               : both
Direction           : inbound
Logging control     : no
Fragment control    : all packets
Tunnel ID number    : 0
Interface           : en0
Auto-Generated      : no
Expiration Time     : 0
Description         : 

Rule 3:
Rule action         : permit
Source Address      : 0.0.0.0
Source Mask         : 0.0.0.0
Destination Address : 0.0.0.0
Destination Mask    : 0.0.0.0
Source Routing      : yes
Protocol            : all
Source Port         : any 0
Destination Port    : any 0
Scope               : both
Direction           : both
Logging control     : no
Fragment control    : all packets
Tunnel ID number    : 0
Interface           : all
Auto-Generated      : no
Expiration Time     : 0
Description         : 

End of IPv4 filter rules.
13. Test pinging from a foreign host. This behavior takes place once we activate IPSec:
# ping 9.40.205.53
PING 9.40.205.53 (9.40.205.53): 56 data bytes
64 bytes from 9.40.205.53: icmp_seq=0 ttl=244 time=258.847 ms
64 bytes from 9.40.205.53: icmp_seq=317 ttl=244 time=253.621 ms
64 bytes from 9.40.205.53: icmp_seq=318 ttl=244 time=256.467 ms
64 bytes from 9.40.205.53: icmp_seq=319 ttl=244 time=251.794 ms
64 bytes from 9.40.205.53: icmp_seq=320 ttl=244 time=243.249 ms
64 bytes from 9.40.205.53: icmp_seq=321 ttl=244 time=253.040 ms
64 bytes from 9.40.205.53: icmp_seq=322 ttl=244 time=247.723 ms
64 bytes from 9.40.205.53: icmp_seq=323 ttl=244 time=398.772 ms
64 bytes from 9.40.205.53: icmp_seq=324 ttl=244 time=253.756 ms
64 bytes from 9.40.205.53: icmp_seq=325 ttl=244 time=245.174 ms
Request timeout for icmp_seq 326
Request timeout for icmp_seq 327
Request timeout for icmp_seq 328
Request timeout for icmp_seq 329
Request timeout for icmp_seq 330
Request timeout for icmp_seq 331
Request timeout for icmp_seq 332
Request timeout for icmp_seq 333
Request timeout for icmp_seq 334
Request timeout for icmp_seq 335
Request timeout for icmp_seq 336
Request timeout for icmp_seq 337
Request timeout for icmp_seq 338
14.Testing on the packet level, the IP trace shows only ECHO requests that are not answered by the underlying AIX host with IPSec deny rule in play:
Packet Number 33
ETH: ====( 98 bytes received on interface en0 )==== 13:49:03.366102470
ETH:    [ 3c:8a:b0:00:8f:f0 -> 02:db:1d:33:d7:02 ]  type 800  (IP)
IP:     < SRC =      9.211.34.6 >  
IP:     < DST =     9.40.205.53 >  (tcp53)
IP:     ip_v=4, ip_hl=20, ip_tos=0, ip_len=84, ip_id=24869, ip_off=0
IP:     ip_ttl=53, ip_sum=224e, ip_p = 1 (ICMP)
ICMP:   icmp_type=8 (ECHO_REQUEST)  icmp_id=61398  icmp_seq=13

Packet Number 34
ETH: ====( 98 bytes received on interface en0 )==== 13:49:04.439390156
ETH:    [ 3c:8a:b0:00:8f:f0 -> 02:db:1d:33:d7:02 ]  type 800  (IP)
IP:     < SRC =      9.211.34.6 >  
IP:     < DST =     9.40.205.53 >  (tcp53)
IP:     ip_v=4, ip_hl=20, ip_tos=0, ip_len=84, ip_id=4743, ip_off=0
IP:     ip_ttl=53, ip_sum=70ec, ip_p = 1 (ICMP)
ICMP:   icmp_type=8 (ECHO_REQUEST)  icmp_id=61398  icmp_seq=14

Packet Number 35
ETH: ====( 98 bytes received on interface en0 )==== 13:49:05.386043960
ETH:    [ 3c:8a:b0:00:8f:f0 -> 02:db:1d:33:d7:02 ]  type 800  (IP)
IP:     < SRC =      9.211.34.6 >  
IP:     < DST =     9.40.205.53 >  (tcp53)
IP:     ip_v=4, ip_hl=20, ip_tos=0, ip_len=84, ip_id=14240, ip_off=0
IP:     ip_ttl=53, ip_sum=4bd3, ip_p = 1 (ICMP)
ICMP:   icmp_type=8 (ECHO_REQUEST)  icmp_id=61398  icmp_seq=15

Packet Number 36
ETH: ====( 98 bytes received on interface en0 )==== 13:49:06.368631062
ETH:    [ 3c:8a:b0:00:8f:f0 -> 02:db:1d:33:d7:02 ]  type 800  (IP)
IP:     < SRC =      9.211.34.6 >  
IP:     < DST =     9.40.205.53 >  (tcp53)
IP:     ip_v=4, ip_hl=20, ip_tos=0, ip_len=84, ip_id=20709, ip_off=0
IP:     ip_ttl=53, ip_sum=328e, ip_p = 1 (ICMP)
ICMP:   icmp_type=8 (ECHO_REQUEST)  icmp_id=61398  icmp_seq=16
Which means the ping packets (ECHO requests) reached its destination (tcp53 in our situation) without being answered with any ECHO responses. That means ping requests are being successfully denied by using that IPSec deny rule.
Note:
  • Likewise, we can tailor these IPSec filtering criteria to match our need whether it's IP-based, port-based or protocol-based filtering.
  • Referring to the IPSec filter table shown in step 12, we can see a number or rules displayed despite of configuring just one rule. That introduces the "default IPSec filter rule", which lies at the end of the table.
  • Filter rules in the table are checked from the beginning to the end until a match is found. If no match, the default rule applies.
  • It's good for the default rule to be a permit one unless instructed otherwise.
SUPPORT

If you require more assistance, use the following step-by-step instructions to contact IBM to open a case for software with an active and valid support contract.  

1.  Document (or collect screen captures of) all symptoms, errors, and messages related to your issue.

2.  Capture any logs or data relevant to the situation.

3.  Contact IBM to open a case:

   -For electronic support, see the IBM Support Community:
     https://www.ibm.com/mysupport
   -If you require telephone support, see the web page:
      https://www.ibm.com/planetwide/

4.  Provide a clear, concise description of the issue.

5.  If the system is accessible, collect a system snap, and upload all of the details and data for your case.

 - For guidance, see: Working with IBM AIX Support: Collecting snap data

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB08","label":"Cognitive Systems"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SWG10","label":"AIX"},"ARM Category":[{"code":"a8m0z000000cvxVAAQ","label":"Communication Applications-\u003EIPFILTERS"}],"ARM Case Number":"TS008852516","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Document Information

Modified date:
02 June 2022

UID

ibm16590907