SentinelView

A cybersecurity OSINT threat intelligence web application.

Screenshots

Overview — severity breakdown, quick stats, and recent critical threat feed

Geo View — interactive 3D threat globe with live attack arc visualisation

Threat Feed — filterable intelligence queue with full threat detail panel

Analytics — threat trend timeline, severity-over-time, sector risk, and score distribution

Analytics — per-source ingestion health table (AlienVault OTX, NIST NVD, MITRE ATT&CK, abuse.ch, RSS feeds)

Overview

SentinelView aggregates, enriches, stores, and visualizes threat intelligence from public and API-backed sources. The backend runs FastAPI, PostgreSQL stores threat data, and the frontend provides dashboards for source health, trends, geography, and threat details.

Technologies

Project Structure

SentinelView/
|-- backend/             # FastAPI backend
|   |-- scrapers/        # OSINT data collection modules
|   |-- nlp/             # NLP and severity enrichment
|   `-- routers/         # API routes (threats + stats + admin)
|-- frontend/            # React frontend
|-- tests/               # Backend tests
|-- docker-compose.yml
|-- .env                 # Local secrets/config, not committed
`-- pytest.ini

Schema is created and lightly migrated at startup by Base.metadata.create_all() and ensure_database_schema() in backend/database.py — there is no Alembic migration history.

Configuration

Copy .env.example to .env and fill in the required values.

Important variables:

POSTGRES_USER=
POSTGRES_PASSWORD=
POSTGRES_DB=
DATABASE_URL=
OTX_API_KEY=
ALIENVAULT_OTX_API_KEY=
ABUSECH_AUTH_KEY=
THREATFOX_AUTH_KEY=
MALWAREBAZAAR_AUTH_KEY=

Inside Docker, docker-compose.yml overrides DATABASE_URL so the backend uses the db service.

Docker Startup

Start the stack:

docker compose up -d --build

Open:

Frontend: http://localhost:3000
Backend:  http://localhost:8000
Docs:     http://localhost:8000/docs

Stop the stack without deleting database data:

docker compose down

Fresh Database Ingestion

When the backend starts and the threats table is empty, it automatically runs:

run_full_pipeline()
_run_mitre_pipeline()

run_full_pipeline() fetches:

• RSS feeds
• AlienVault OTX
• NIST NVD
• abuse.ch ThreatFox
• abuse.ch MalwareBazaar

_run_mitre_pipeline() fetches MITRE ATT&CK.

AlienVault OTX is capped at 30 pages with 50 pulses per page, so one run can fetch up to 1500 OTX pulses.

Verify Data

Total stored threats:

docker compose exec -T db psql -U sentinelview -d sentinelview -c "select count(*) from threats;"

Threats by source:

docker compose exec -T db psql -U sentinelview -d sentinelview -c "select source_name, count(*) from threats group by source_name order by count desc;"

Scrape logs:

docker compose exec -T db psql -U sentinelview -d sentinelview -c "select source_name, status, articles_found, new_articles, error_message from scrape_logs order by id;"

Source health API:

curl http://localhost:8000/api/stats/source-health

Manual Ingestion Commands

Run the normal full pipeline:

docker compose exec -T backend python -c "from backend.scheduler import run_full_pipeline; run_full_pipeline()"

Run MITRE only:

docker compose exec -T backend python -c "from backend.scheduler import _run_mitre_pipeline; _run_mitre_pipeline()"

Fetch AlienVault OTX only without saving to the database:

docker compose exec -T backend python -c "from backend.scrapers.otx_scraper import OtxScraper; print(len(OtxScraper().scrape()))"

Tests

When Python dependencies are installed locally:

python -m pytest

License

MIT — see LICENSE.