Detection capability and operational usability of network IDS Snort: A study based on MITRE ATT&CK framework

Signature-based intrusion detection systems (SIDS) are key security components in most organizations, and open-source detectors such as Snort are routinely used to assess how exposed a network is to attack. What a SIDS detects, however, is largely determined by which signatures it has enabled—a configuration that published evaluations seldom document, even as they report detection rates as characteristics of the tool. This paper quantifies how much of Snort’s detection capability is in fact governed by its configuration, and at what operational cost. We built and publicly release a real-traffic attack dataset covering all network-visible techniques of the MITRE ATT&CK Enterprise matrix, and assessed false positives against nearly 79 million legitimate flows from a public dataset. Snort’s two main rule families (Talos and Emerging Threats) were evaluated in their default and fully-activated forms and in combination. In their default form, Talos detects 73% of the attack instances and Emerging Threats 46%. Both families, however, ship a large share of their signatures commented out; uncommenting them raises coverage to 95% for Talos and 100% for Emerging Threats, reversing their relative standing. The techniques missed by the default configurations are therefore, for the most part, not undetectable but merely unconfigured. That coverage is nonetheless unusable as distributed: every untuned configuration misclassifies between 3.4 and 9.5 million benign flows, and raises from 34 to over 1000 false alarms per true positive. Most of that noise, however, comes from a few dozen signatures: removing those that fire most often on benign traffic inverts the ratio to one false alarm per fifteen true positives, and turns the combination of both rule families, redundant when untuned, into the widest-coverage option. That gain is paid in detection: 15%–26% of the techniques for which a signature exists go undetected. Snort with free rule sets is therefore not a protection layer that can be deployed as distributed: it must be pruned to the traffic it will monitor, and the resulting loss of detection accepted as part of the trade-off. All figures correspond to Snort 2; the methodology is engine- and feed-independent.