Vpedia - Wikipedia RAG System

Originally Created by Martin Adrian Sebastian

Semantic search over Wikipedia using local vector embeddings and LanceDB.

---

Features

• Full Wikipedia Processing - Download and process complete Wikipedia dumps
• Semantic Search - Natural language queries with cosine similarity
• GPU Accelerated - CUDA support for fast vectorization
• Thermal Protection - Automatic cooling breaks for sustained processing
• Resume Support - Continue interrupted processing from last checkpoint
• Compression - Zlib-compressed text storage for space efficiency

---

Quick Start

Installation

# Clone repository
git clone https://github.com/rootedlab-code/vpedia.git
cd vpedia

# Install package
pip install -e .

# Or install with GPU monitoring
pip install -e ".[gpu]"

Usage

Run the unified CLI application:

vpedia

This will launch an interactive menu where you can:

1. Download Wikipedia dump
2. Process and vectorize content
3. Query the database
4. Monitor system status

Python API

from vpedia.processor import ProcessingPipeline
from vpedia.storage import VectorDatabase
from vpedia.vectorizer import EmbeddingGenerator

# Process Wikipedia
pipeline = ProcessingPipeline()
pipeline.run()

# Query
db = VectorDatabase()
embedder = EmbeddingGenerator()
query_vec = embedder.encode_single("theory of relativity")
results = db.search(query_vec, top_k=5)

---

Project Structure

vpedia/
├── src/vpedia/              # Main package (SRP-compliant)
│   ├── parser/              # XML parsing, text cleaning, chunking
│   ├── vectorizer/          # Embedding generation
│   ├── storage/             # LanceDB operations
│   ├── processor/           # Pipeline, thermal, workers
│   ├── cli/                 # Command-line interface
│   └── monitor/             # Real-time monitoring
├── tests/                   # pytest test suite
│   ├── unit/                # Unit tests
│   └── integration/         # Integration tests
├── scripts/                 # Utility scripts
├── docs/                    # Documentation (SRS, URS, SysRS, etc.)
├── pyproject.toml           # Modern Python packaging
└── requirements.txt         # Dependencies

---

System Requirements

Component	Minimum	Recommended
CPU	4 cores	8+ cores
RAM	16 GB	32 GB
Storage	80 GB SSD	150 GB NVMe
GPU	-	CUDA-capable (6GB+ VRAM)

---

Documentation

• Software Requirements (SRS)
• User Requirements (URS)
• System Requirements (SysRS)
• Design Document
• Architecture
• Test Suite

---

Configuration

Vpedia uses Environment Variables for configuration. Defaults are tuned for stability on standard hardware (e.g., 4 workers, batch size 192).

Variable	Default	Description
VPEDIA_DUMP_FILE	./enwiki-latest-pages-articles.xml.bz2	Path to Wikipedia XML dump
VPEDIA_DB_URI	./wiki_lancedb	Path to LanceDB directory
VPEDIA_TABLE_NAME	en_wikipedia	LanceDB table name
VPEDIA_MODEL	sentence-transformers/all-mpnet-base-v2	Embedding model name

For advanced tuning (workers, batch size, thermal limits), modify src/vpedia/config.py directly.

---

Testing

# Run all tests
pytest tests/ -v

# Run with coverage
pytest tests/ -v --cov=vpedia

# Skip slow tests (model loading)
pytest tests/ -v -m "not slow"

---

License

This project is licensed under the LGPL-3.0 License with an Attribution Requirement.

All forks and derivative works MUST retain prominent attribution to the original author:

Originally Created by Martin Adrian Sebastian

See LICENSE for full terms.

---

Author

Martin Adrian Sebastian

Wikipedia RAG System for offline semantic search.