Real-Time Network Intrusion Detection System (RealTimeNIDS)

This project is a Real-Time Network Intrusion Detection System (NIDS) that monitors network traffic at the flow level using Zeek, extracts relevant features, and detects threats in real-time using both:

• Signature-based detection (e.g., port scans)
• Anomaly-based detection using a trained IsolationForest model

The system is trained on the CIC-IDS 2018 dataset for realistic attack patterns and supports continuous monitoring via Zeek's conn.log.

Important: This project is for educational and research purposes. Always obtain permission before deploying on real or institutional networks.

---

Table of Contents

• Features
• Known Limitations
• Tech Stack
• Installation
• Zeek Configuration
• Running the System
• Project Structure
• Future Improvements
• Acknowledgements
• License

---

Features

• Real-time network monitoring using Zeek
• Anomaly detection with IsolationForest (trained on CIC-IDS 2018)
• Signature-based detection for rule-based threats like port scans
• Flow-level feature extraction (e.g., byte rates, packet rates, flags)
• Threat detection console output with structured threat data

---

Known Limitations

This project is currently shelved following a live deployment test that exposed a gap between offline evaluation and real-world performance.

What happened: the IsolationForest model reported strong metrics during offline evaluation on the CIC-IDS 2018 test split, but failed to flag a controlled Denial-of-Service (DoS) attack when tested live against the running Zeek pipeline.

Likely causes:

• Distribution shift between the static CIC-IDS 2018 flows used for training and the live flow features generated by Zeek in real time
• Feature scaling/normalization computed on the offline dataset not generalizing to live traffic ranges
• Anomaly threshold calibrated on offline scores, not re-validated against live score distributions

Why this matters: offline metrics (precision, recall, F1) on a static test set do not guarantee real-time detection performance. This gap is the main reason the project is paused rather than actively developed — it needs a proper live-traffic validation and re-calibration pass before the anomaly detector can be trusted in production.

This is left in the repo intentionally as a documented lesson on the difference between offline model evaluation and production-grade deployment testing.

---

Tech Stack

• Python 3.10
• Zeek – Real-time network traffic analyzer
• scikit-learn – IsolationForest for anomaly detection
• joblib – For loading pre-trained ML models
• WSL (for Windows) – Zeek runs in Ubuntu via WSL
• Matplotlib / Pandas / NumPy (used during training, optional for runtime)

---

Installation

1. Clone the repository

git clone https://github.com/otuemre/RealTimeNIDS.git
cd RealTimeNIDS

2. Install Python dependencies

pip install -r requirements.txt

3. Install Zeek

For Ubuntu (WSL or native):

sudo apt update
sudo apt install zeek

---

Zeek Configuration

To start Zeek and monitor your interface:

sudo /opt/zeek/bin/zeek -i eth0 -C

• eth0 is your interface (check with ifconfig inside WSL)
• -C disables checksum validation (useful in WSL)

Zeek will generate a conn.log file containing flow records.

---

Running the System

Change the path to conn.log in src/realtime_nids/zeek_monitor.py:

LOG_FILE = 'PATH_TO_YOUR_CONN_FILE'

Start your monitor in another terminal:

python src/realtime_nids/zeek_monitor.py

You'll see real-time detection logs like:

[*] Starting real-time Zeek log monitor...
[!] 0 Live Threat Detect:
    -> {'type': 'signature', 'rule': 'port_scan', 'confidence': 1.0}
    -> {'type': 'anomaly', 'score': -0.72, 'confidence': 0.72}

Works for live tests (e.g., hping3, simulated attacks) for signature-based detection. See Known Limitations for anomaly-based detection gaps.

---

Project Structure

File	Description
src/realtime_nids/zeek_monitor.py	Reads and parses Zeek conn.log for real-time flow monitoring
src/realtime_nids/detection_engine.py	Contains both signature-based and IsolationForest-based anomaly detection
model/isolation_model.pkl	Pre-trained IsolationForest model (from CIC-IDS 2018)
src/realtime_nids/zeek_parser.py	(Optional helper) Parses logs and maps fields cleanly
notebooks/	Jupyter notebooks for model training and threshold tuning
datasets/	Location for downloaded training datasets
.env	Configuration (optional, not required)

---

Future Improvements

• Validate and re-calibrate the anomaly detector against live traffic distributions, not just the offline CIC-IDS 2018 test set
• Add support for a model retraining pipeline using live-captured flow data
• Dynamic threshold tuning via quantile calibration on live scores
• Web dashboard for real-time alert visualization
• Support for other models (e.g., One-Class SVM, Autoencoders)
• Add more signature-based rules derived from the CIC-IDS 2018 dataset

---

Acknowledgements

• Based on FreeCodeCamp's Real-Time IDS Tutorial
• IDS 2018 Intrusion CSVs (CSE-CIC-IDS2018) – Source: Kaggle: IDS Intrusion CSVs

---

License

Licensed under the MIT License. You're free to use and modify responsibly.