Cortex VM Analyzer

An AI-based observability demo for a common fleet-operations problem: synthetic virtual-machine and per-process metrics are generated in-process, analyzed by an LLM (with a deterministic rule-based fallback) to find resource bottlenecks, memory leaks, and other infra-level issues, and compared across hosts inside an interactive Streamlit dashboard.

This is a personal portfolio project. No real infrastructure is ever queried -- every metric, every hostname, and every "incident" is synthetically generated so the app can be cloned and demoed by anyone with zero setup.

Why I built this

The idea comes from a real operational pattern I've dealt with running large fleets of servers, all running the same set of processes: every so often a handful of hosts start behaving abruptly -- higher error rates, increased latency -- without an obvious single cause. Past incidents like that kept coming back to the same root causes: a resource bottleneck, a memory leak in one specific process, or some other infra-level issue -- never something you'd catch by immediately diving into application logs.

The core idea here is to compare an unhealthy host's VM-level and process-level metrics against a healthy peer running the identical workload, so the difference already shows up at the infrastructure level -- and points at a specific process -- before anyone starts application-level debugging. That's why Compare VMs is a first-class tab here rather than an afterthought, and why process-level metrics (CPU, resident memory, file descriptors) sit alongside VM-level ones throughout: the goal was never just "analyze one VM," it was "is this host actually different from its healthy peers, and if so, which process explains it."

The other goal was pairing that comparison with real AI-based analysis: an LLM reads the VM- and process-level findings together and reasons about whether something is a bottleneck, a leak, or a different infra issue, on top of a custom set of detection rules built from the same kind of operational knowledge (sustained thresholds, spike ratios, RSS-growth signatures) that past incidents like these actually taught. The rule engine is also what keeps the app fully explainable and demoable without live API access -- but the point of the project is the AI-based diagnosis, not the fallback underneath it.

How it works

Every analysis runs both a VM-level and a process-level pipeline in parallel:

1. Rule-based detection. anomaly_detector.py scans synthetic VM-level metrics (CPU, memory, disk, network) and, separately, every process's CPU/RSS/file-descriptor series -- using sustained-threshold rules, spike detection, and (for memory) direct RSS-growth detection, which is a much sharper leak signal than a static threshold since it doesn't care what a process's baseline memory happens to be.
2. AI-based analysis. ai_analyzer.py hands both the VM-level and process-level findings to an LLM, which is asked to determine whether a degradation is a resource bottleneck, a memory leak, or another infra issue -- and, whenever the data supports it, to name the specific process responsible rather than making a generic VM-level statement. If no API key is configured, the call fails, or the response isn't valid JSON, the app transparently falls back to a deterministic narrative built directly from the same rule-engine findings, so the dashboard always produces a complete answer.

The UI shows a small "source: AI-generated / rule-based fallback" badge, but doesn't otherwise change behavior based on which path produced the narrative -- both use the identical rule-engine findings as their factual basis.

Architecture

                 +----------------------+
                 |  metrics_generator   |   synthetic VM metrics (CPU/mem/
                 |  (VM + per-process)  |   disk/network) AND per-process
                 +----------+-----------+   metrics (cpu/rss/fd/threads),
                            |               same process set on every host
                            v
                 +----------------------+
                 |   data_processor     |   VM + per-process summary stats,
                 |  (summarize/compare) |   plus cross-host process comparison
                 +----------+-----------+   (sorted by biggest spread)
                            |
                            v
                 +----------------------+
                 |   anomaly_detector    |  VM-level thresholds/spikes +
                 |   (no AI, no network) |  process-level thresholds/spikes/
                 +----------+-----------+  RSS-growth (leak) detection
                            |
                            v
                 +----------------------+
                 |     ai_analyzer       |  AI-based analysis across VM +
                 |  (AI-based, with a   |  process findings, naming a
                 |   rule-based fallback|  specific process where possible
                 |   underneath)        |  (falls back to rule-based
                 +----------+-----------+   narrative if AI is unavailable)
                            |
                            v
                 +----------------------+
                 |     app.py            |  Streamlit dashboard:
                 |  (Streamlit UI)       |  AI Analysis / Metrics Overview /
                 |                       |  Raw Data / Compare VMs / History
                 +----+------------+----+
                      |            |
                      v            v
          +--------------+   +----------------+
          | cache_manager |   |  history_db    |   .cache/ (10 min TTL)
          | (disk cache)  |   |  (SQLite)      |   .data/analyzer_history.db
          +--------------+   +----------------+
                      |
                      v
            +--------------------+
            |  report_generator   |   downloadable PDF (VM + per-process
            |  (PDF export)       |   tables, ReportLab + Matplotlib)
            +--------------------+

Features

• Synthetic VM metrics for CPU, memory, per-mountpoint disk usage, disk I/O utilization, and per-interface network throughput.
• Synthetic per-process metrics (CPU, resident memory, file descriptors, thread count) for the same fictional process set on every host -- what makes comparing "the same process" across a healthy and an unhealthy host meaningful.
• Reproducible anomalies, some of which have a matching process-level signature: a CPU spike or a memory-leak ramp at the VM level is driven by one specific runaway/leaking process, deterministically seeded per hostname.
• AI-based analysis that reasons across VM- and process-level findings together to name a likely resource bottleneck, memory leak, or other infra issue -- with a deterministic rule-based fallback so the dashboard never produces a blank or broken answer.
• VM comparison, including a process-level comparison table across two or more hosts, sorted by the biggest spread per process/metric, so the standout process surfaces first.
• Disk-based caching (10-minute TTL) and a SQLite history of past runs.
• PDF report export, including per-process tables, for single-VM and comparative analyses.
• Zero external dependencies to demo -- everything runs in-process.

Getting started

Option 1: local Python

git clone <this-repo>
cd cortex-vm-analyzer
pip install -r requirements.txt
cp .env.example .env   # optional -- add your OpenAI key here, or leave as-is
streamlit run app.py

Then open http://localhost:8501 in your browser.

Option 2: Docker Compose

cp .env.example .env   # optional
docker compose up --build

Then open http://localhost:8501 in your browser.

Option 3: Bare-metal (systemd)

For a more "production ops" style deployment on a plain Linux VM (no containers), a systemd unit and install helper are included under deploy/:

sudo ./deploy/install.sh

This creates a dedicated unprivileged app user, installs the app to /opt/cortex-vm-analyzer with its own virtualenv, and registers it as a systemd service (systemctl status cortex-vm-analyzer) so it survives reboots and restarts on failure. See deploy/cortex-vm-analyzer.service for the unit definition -- adjust User/WorkingDirectory/paths to match your target host.

Enabling AI analysis

Put a real key in your .env to enable AI-based analysis:

OPENAI_API_KEY=sk-...
OPENAI_MODEL=gpt-4o-mini

Without a key, the app still runs correctly using the rule-based fallback narrative (shown with a "rule-based fallback" badge) -- useful for demoing the UI and the detection logic without needing an API key on hand, but the AI-based path is the intended way to use this.

Project layout

File	Purpose
metrics_generator.py	Synthetic VM and per-process metrics time-series generator
data_processor.py	Summary statistics, downsampling, and cross-host process comparison
anomaly_detector.py	Rule-based VM-level and process-level detection (thresholds, spikes, RSS-growth)
ai_analyzer.py	AI-based analysis across VM + process findings, with a rule-based fallback
cache_manager.py	Disk-based JSON cache with TTL
history_db.py	SQLite-backed history of past analyses
report_generator.py	PDF report generation, incl. per-process tables (ReportLab + Matplotlib)
app.py	Streamlit dashboard tying everything together
deploy/cortex-vm-analyzer.service	systemd unit for a bare-metal deployment
deploy/install.sh	Install helper for the systemd deployment path

Disclaimer

This project is a clean-room, from-scratch implementation built purely as a personal portfolio piece to demonstrate an architectural pattern. It does not connect to, represent, or reproduce any employer's internal systems, code, or infrastructure. All data is synthetic and generated in-process.