Migrate docsite embed/search to Postgres + pgvector (WIP) - #606

Closed

IZO-Ong wants to merge 48 commits into

OpenFn:release/nextfrom

IZO-Ong:postgres

Commits on Aug 7, 2026

feat: parameterize chunk size in DocsiteProcessor
IZO-Ong
committed
707398d
fix: revert unauthorized nltk import and _accumulate_chunks changes to match brief spec
IZO-Ong
committed
a4379e9
feat: rewrite DocsiteIndexer for Postgres batch lifecycle
IZO-Ong
committed
6f8600a
feat: rewire main() to the Postgres batch lifecycle
IZO-Ong
committed
a0964d5
feat: rewrite DocsiteSearch for Postgres hybrid search, preserve legacy Pinecone path
IZO-Ong
committed
6cfc4c4
fix: order keyword CTE by rank before limiting in hybrid search
IZO-Ong
committed
41d7e10
feat: backend-flagged docsite search with Postgres shadow-mode comparison
IZO-Ong
committed
340a2ec
fix: correct shadow-mode docstrings and test import placement
IZO-Ong
committed
3919172
feat: add offline golden-query recall/latency eval script
IZO-Ong
committed
0bb3ade
fix: remove orphaned threshold arg and revert unrequested noqa suppressions
IZO-Ong
committed
1d780db
fix: Restore back threshold to 0.8
IZO-Ong
committed
34d47cd
fix: apply the 0.7 relevance gate to both search_documentation backends
IZO-Ong
committed
1050006
refactor: remove shadow-mode Postgres comparison from job_chat
IZO-Ong
committed
b28b5fc
fix: cast hybrid RRF score to float8 so results stay JSON-serializable
IZO-Ong
committed
ba0769b
fix: use semantic strategy on Postgres so the relevance gate survives cutover
IZO-Ong
committed
73e8442
feat: add per-request backend selection to search_docsite
IZO-Ong
committed
5fd8ed8
feat: restore Pinecone write path behind embed_docsite target param
IZO-Ong
committed
cbf3ae9
fix: add type annotations to embed_docsite's new helper functions
IZO-Ong
committed
ed07783
feat: add versioned migration runner for the docsite schema
IZO-Ong
committed
131830e
feat: report backend agreement in the offline docsite eval
IZO-Ong
committed
ab6b223
docs: correct stale LegacyPineconeDocsiteSearch docstring
IZO-Ong
committed
378aeab
refactor: revert unrelated churn in docsite_processor
IZO-Ong
committed
1643730
refactor: revert import and formatting churn in docsite consumers

IZO-Ong
and
claude
committed
8a99866
refactor: drop type annotations from internal helpers
IZO-Ong
committed
0a021a2
refactor: extract shared docsite backend resolver
IZO-Ong
committed
f2f540f
docs: focus docsite comments on invariants
IZO-Ong
committed
62b77db
fix: run docsite migrations from the indexer, not every connection
IZO-Ong
committed
b8d753b
Remove openai key test
IZO-Ong
committed
27ab6ac
fix: return a clear 503 when the docsite schema is not initialised
IZO-Ong
committed
c2476c0
fix: bind docsite embeddings as pgvector Vector
IZO-Ong
committed
5d56511
fix: commit before toggling autocommit, and mark failed batches
IZO-Ong
committed
0725262
test: add Postgres docsite roundtrip integration suite
IZO-Ong
committed
b5946bd
fix: don't let a pruning failure invalidate the batch just promoted
IZO-Ong
committed
947e585
fix: change integration test to use 127.0.0.1 instead of localhost to speedup
IZO-Ong
committed
a303af8
fix: update env example to support docsite backend and postgres test url for integration testing
IZO-Ong
committed
74889bf
fix: run_eval compares postgres semantic, and loads dotenv
IZO-Ong
committed
483de4e
Add results from running golden queries
IZO-Ong
committed
590b808
fix: import ordering and streamline comments
IZO-Ong
committed
9fb9f11
lint: fix B008 and B904 linting issues
IZO-Ong
committed
6cbaec1
lint: fix B905, RUF059, edited docs in helpers
IZO-Ong
committed
ad6e2e8
fix: lint F401 and docsite name accepting in pinecone
IZO-Ong
committed
4bf19d4
feat: add results for each query for run_eval.py
IZO-Ong
committed
3b78106
refactor: use a timestamped migration filename
IZO-Ong
committed
b994b6f
test: expand golden query set to 30 categorised queries
IZO-Ong
committed
1b401d8
feat: report eval recall per query category
IZO-Ong
committed
4ae1fb1
feat: resolve docsite eval batches by chunk size
IZO-Ong
committed
2fed53a
feat: compare semantic and hybrid across chunk sizes in the eval
IZO-Ong
committed
f9e30d0

Commits on Aug 11, 2026

docs: update comments in roundtrip test
IZO-Ong
committed
10263d7