Motivation: Biological databases offer structured, curated knowledge extracted from the scientific literature, yet extracting meaningful insights from them becomes increasingly challenging as queries grow in complexity, particularly when the subject of inquiry spans entire sets of biological entities rather than a single gene or protein, requiring both domain expertise and familiarity with database-specific querying conventions. Large language models (LLMs) and retrieval-augmented generation (RAG) offer significant potential to address these challenges by enabling natural language-driven interaction with biological databases.
Results: We present ProQuest, a modular natural language interface to the UniProt Knowledgebase, the most comprehensive protein sequence and annotation database. ProQuest integrates three components: ProteinSearch, an LLM-based module that translates natural language queries into executable database queries; SeqSim, a protein sequence similarity search module leveraging language model embeddings for rapid, biologically meaningful retrieval; and ProteinChat, a hybrid RAG pipeline that aggregates evidence from multiple complementary retrieval strategies to generate accurate, grounded natural language responses. Systematic evaluation demonstrated that hybrid retrieval substantially improved biological correctness, semantic alignment, and completeness over direct LLM generation, confirming the value of grounding model outputs in curated biological evidence.
Availability: The ProQuest demo is publicly accessible at https://proquest.ngrok.app, and the complete source code and data are available at https://github.com/HUBioDataLab/ProQuest.
Comprehensive platform for protein search, retrieval, and analysis using semantic, vector, and sequence-based technologies. Includes a FastAPI backend, React/Streamlit frontend, and extensive monitoring and analysis tools.
- Project Structure
- Prerequisites
- Installation & Setup
- Running the Application
- API Keys
- Monitoring & Analysis Tools
- Testing
- Assets & Data
- Directory Reference
- backend/: FastAPI server, core logic, models, and data processing scripts
- frontend/: React app (with Streamlit integration) for user interface
- config/: Setup, database, and environment scripts
- test/: Test scripts, evaluation, and analysis notebooks
- memory_monitoring/: Resource and performance monitoring
- backend/asset/: All major data files, embeddings, vector DBs, and reference files
- Python 3.10+
- Node.js (for frontend)
- pip (Python package manager)
- (Recommended) Virtual environment (venv, conda, etc.)
- Permissions to install packages and run scripts
cd backend
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
pip install -r ../config/requirements.txt # Or install manually as neededpython ../config/setUpDatabase.py
python ../config/implementVectorDatabase.py
# (Optional) python ../config/implementVectorDatabaseFromFlatFiles.pycd ../frontend
npm installcd backend
uvicorn main:app --reload --port 8000Endpoints:
- POST /llm_query
- POST /vector_search
cd ../frontend
npm startpython -m streamlit run main.py # (if using Streamlit UI)To use all LLM features, obtain free API keys from:
- Google Generative AI: Get API key
- NVIDIA Models: Get API key
- Mistral AI: Get API key
- memory_monitoring/: Scripts and Excel files for memory usage
- test/: Extensive test and analysis scripts (e.g.,
testRAG.py,evaluateRAGResult.py,visualizePerformanceBarChart.py)
Test scripts are in the test/ directory. Example:
cd test
python testRAG.pyThe project data is available on Figshare: https://doi.org/10.6084/m9.figshare.32885441
Instructions for running the application using the provided data files or rebuilding the required assets from scratch are available in DATA_AND_REPRODUCIBILITY_MANIFEST.md.
All major data files are in backend/asset/:
- protein_index2.db: Main SQLite database
- protein_embeddings_2.ann: Annoy index for vector search
- chroma_uniprot_nomic/: Chroma vector DB
- docs_sp.joblib: Preprocessed docs for BM25
- uniprot_sprot.fasta and related files: Sequence data
- queryfields.txt, result-fields.json, search-fields.json: Field definitions
- backend/
main.py: FastAPI serversrc/: Core logic (prompting, retrieval, embedding, etc.)asset/: All data files, embeddings, and vector DBsmodeldir/: HuggingFace model cache
- frontend/
src/components/: React UI componentssrc/pages/: Page-level React componentssrc/services/: API service modulespublic/: Static files (HTML, icons, manifest)
- config/: Setup and environment scripts
- test/: Test, evaluation, and visualization scripts
- vector_search_monitoring/, memory_monitoring/: Monitoring scripts and logs
ProQuest is an advanced, modular platform designed to make the vast knowledge in the UniProt protein database accessible through natural language. As biological databases grow in size and complexity, extracting meaningful insights—especially for queries involving sets of proteins or complex biological relationships—becomes increasingly challenging. ProQuest addresses this by combining large language models (LLMs) and retrieval-augmented generation (RAG) to enable intuitive, accurate, and evidence-grounded exploration of protein data.
Key Features:
- Natural Language Querying: Users can ask complex biological questions in plain language. ProQuest translates these into structured database queries, removing the need for specialized syntax or deep database expertise.
- Hybrid Retrieval Pipeline: The system integrates multiple retrieval strategies—semantic vector search (using protein language model embeddings), keyword-based BM25 ranking, full-text substring search, and sequence similarity search—ensuring that answers are both comprehensive and biologically relevant.
- Evidence-Grounded Answers: ProQuest’s RAG module synthesizes information from curated UniProt records, providing responses that are not only accurate but also traceable to their biological sources, minimizing hallucination and maximizing scientific reliability.
- Scalable and Extensible: The architecture is modular, supporting new retrieval strategies and easy adaptation to other biological databases.
System Components:
- ProteinSearch: Converts natural language queries into UniProt-compatible Solr queries using schema-informed prompt engineering and multi-model LLM sampling.
- SeqSim: Performs fast, biologically meaningful protein sequence similarity searches using ProtT5 embeddings and Annoy-based nearest-neighbor indexing.
- ProteinChat: Aggregates evidence from all retrieval modules and generates natural language answers, grounded in curated protein knowledge.
Why ProQuest? Traditional database interfaces require users to know both the biological context and the technical details of query languages. ProQuest lowers this barrier, making it possible for researchers of all backgrounds to access, interpret, and integrate protein knowledge at scale. Systematic evaluation shows that ProQuest’s hybrid retrieval and RAG approach delivers more accurate, complete, and biologically correct answers than LLMs or database queries alone.
For advanced usage, see individual scripts in config/ and test/.
For troubleshooting, check logs and monitoring outputs in the respective folders.