What is an intrusion detection system (IDS)?

An IDS can help accelerate and automate network threat detection by alerting security administrators to known or potential threats, or by sending alerts to a centralized security tool. A centralized security tool such as a security information and event management (SIEM) system can combine data from other sources to help security teams identify and respond to cyberthreats that might slip by other security measures.

IDSs can also support compliance efforts. Certain regulations, such as the Payment Card Industry Data Security Standard (PCI-DSS), require organizations to implement intrusion detection measures.

An IDS cannot stop security threats on its own. Today IDS capabilities are typically integrated with—or incorporated into—intrusion prevention systems (IPSs), which can detect security threats and automatically act to prevent them.

IDSs can be software applications that are installed on endpoints or dedicated hardware devices that are connected to the network. Some IDS solutions are available as cloud services. Whatever form it takes, an IDS uses one or both of two primary threat detection methods: signature-based or anomaly-based detection.

Signature-based detection analyzes network packets for attack signatures—unique characteristics or behaviors that are associated with a specific threat. A sequence of code that appears in a particular malware variant is an example of an attack signature.

A signature-based IDS maintains a database of attack signatures against which it compares network packets. If a packet triggers a match to one of the signatures, the IDS flags it. To be effective, signature databases must be regularly updated with new threat intelligence as new cyberattacks emerge and existing attacks evolve. Brand new attacks that are not yet analyzed for signatures can evade signature-based IDS.

Anomaly-based detection methods use machine learning to create—and continually refine—a baseline model of normal network activity. Then it compares network activity to the model and flags deviations—such as a process that uses more bandwidth than normal, or a device opening a port.

Because it reports any abnormal behavior, anomaly-based IDS can often catch new cyberattacks that might evade signature-based detection. For example, anomaly-based IDSs can catch zero-day exploits—attacks that take advantage of software vulnerabilities before the software developer knows about them or has time to patch them.

But anomaly-based IDSs may also be more prone to false positives. Even benign activity, such as an authorized user accessing a sensitive network resource for the first time, can trigger an anomaly-based IDS.

Reputation-based detection blocks traffic from IP addresses and domains associated with malicious or suspicious activity. Stateful protocol analysis focuses on protocol behavior—for example, it might identify a denial-of-service (DoS) attack by detecting a single IP address, making many simultaneous TCP connection requests in a short period.

Whatever method(s) it uses, when an IDS detects a potential threat or policy violation, it alerts the incident response team to investigate. IDSs also keep records of security incidents, either in their own logs or by logging them with a security information and event management (SIEM) tool (see 'IDS and other security solutions' below). These incident logs can be used to refine the IDS’s criteria, such as by adding new attack signatures or updating the network behavior model.

IDSs are categorized based on where they’re placed in a system and what kind of activity they monitor. 

Network intrusion detection systems (NIDSs) monitor inbound and outbound traffic to devices across the network. NIDS are placed at strategic points in the network, often immediately behind firewalls at the network perimeter so that they can flag any malicious traffic breaking through.

NIDS may also be placed inside the network to catch insider threats or hackers who hijacked user accounts. For example, NIDS might be placed behind each internal firewall in a segmented network to monitor traffic flowing between subnets.

To avoid impeding the flow of legitimate traffic, a NIDS is often placed “out-of-band,” meaning that traffic doesn’t pass directly through it. A NIDS analyzes copies of network packets rather than the packets themselves. That way, legitimate traffic doesn’t have to wait for analysis, but the NIDS can still catch and flag malicious traffic.

Host intrusion detection systems (HIDSs) are installed on a specific endpoint, like a laptop, router, or server. The HIDS only monitors activity on that device, including traffic to and from it. A HIDS typically works by taking periodic snapshots of critical operating system files and comparing these snapshots over time. If the HIDS notices a change, such as log files being edited or configurations being altered, it alerts the security team.

Security teams often combine network-based intrusion detection systems and host-based intrusion detection systems. The NIDS looks at traffic overall, while the HIDS can add extra protection around high-value assets. A HIDS can also help catch malicious activity from a compromised network node, like ransomware spreading from an infected device.

While NIDS and HIDS are the most common, security teams can use other IDSs for specialized purposes. A protocol-based IDS (PIDS) monitors connection protocols between servers and devices. PIDS are often placed on web servers to monitor HTTP or HTTPS connections.

An application protocol-based IDS (APIDS) works at the application layer, monitoring application-specific protocols. An APIDS is often deployed between a web server and an SQL database to detect SQL injections.

While IDS solutions can detect many threats, hackers can get around them. IDS vendors respond by updating their solutions to account for these tactics. However, these solution updates create something of an arm’s race, with hackers and IDSs trying to stay one step ahead of one another. 

Some common IDS evasion tactics include:

• Distributed denial-of-service (DDoS) attacks—taking IDSs offline by flooding them with obviously malicious traffic from multiple sources. When the IDS’s resources are overwhelmed by the decoy threats, the hackers sneak in.

• Spoofing—faking IP addresses and DNS records to make it look like their traffic is coming from a trustworthy source.

• Fragmentation—splitting malware or other malicious payloads into small packets, obscuring the signature and avoiding detection. By strategically delaying packets or sending them out of order, hackers can prevent the IDS from reassembling them and noticing the attack.

• Encryption—using encrypted protocols to bypass an IDS if the IDS doesn’t have the corresponding decryption key.

• Operator fatigue—generating large numbers of IDS alerts on purpose to distract the incident response team from their real activity.

IDSs aren’t standalone tools. They’re designed to be part of a holistic cybersecurity system, and are often tightly integrated with one or more of the following security solutions.

IDSs alerts are often funneled to an organization’s SIEM, where they can be combined with alerts and information from other security tools into a single, centralized dashboard. Integrating IDS with SIEMs enables security teams to enrich IDS alerts with threat intelligence and data from other tools, filter out false alarms‌, and prioritize incidents for remediation.

As noted above, an IPS monitors network traffic for suspicious activity, like an IDS, and intercepts threats in real time by automatically terminating connections or triggering other security tools. Because IPSs are meant to stop cyberattacks, they’re usually placed inline, meaning that all traffic has to pass through the IPS before it can reach the rest of the network.

Some organizations implement an IDS and an IPS as separate solutions. More often, IDS and IPS are combined in a single intrusion detection and prevention system (IDPS) which detects intrusions, logs them, alerts security teams and automatically responds.

IDSs and firewalls are complementary. Firewalls face outside the network and act as barriers by using predefined rulesets to allow or disallow traffic. IDSs often sit near firewalls and help catch anything that slips past them. Some firewalls, especially next-generation firewalls, have built-in IDS and IPS functions.