docs: fix Vector Store / embeddings API docs - #449
Open
ygd58 wants to merge 1 commit into
Open
Conversation
The Vector Store feature page and its LogIndexer example were stale against the current SDK in four ways reported in genlayerlabs#437: 1. Wrong import path. Docs showed `from backend.node.genvm.std.vector_store import VectorStore` (an internal path, not a contract-facing API) and `import genlayermodelwrappers` — neither exists in the published SDK. Verified against genvm-executor's actual runners/genlayer-py-std/src-emb/genlayer_embeddings/__init__.py: VecDB, VecDBElement, the Distance classes, and SentenceTransformer are all exported from `genlayer_embeddings`, imported as `import genlayer_embeddings as gle`. 2. VecDBElement was completely undocumented. Added a reference table (.key, .id, .value [settable], .distance, .remove()) sourced directly from vecdb.py's docstrings/property definitions, not guessed. 3. No complete, runnable example existed anywhere reachable from the docs. Replaced both pages' example with the real, tested LogIndexer contract from genlayer-studio/examples/contracts/log_indexer.py — byte-for-byte identical (diffed to confirm), not a paraphrase. This also fixes the missing 4th VecDB type parameter (the Distance metric) that the old docs example omitted. 4. No guidance for debugging "Could not load contract schema". Traced this to frontend/src/components/Simulator/ ConstructorParameters.vue in genlayer-studio: it's a generic Vue Alert with no detail. The real Python traceback IS captured server-side (backend/node/base.py's get_contract_schema bundles stdout/stderr/genvm_log into the raised exception, logged at INFO level per _execution_finished's comment: GenVM failures are user contract errors, not infra errors). Documented `docker compose logs jsonrpc -f` as the way to see it, verified against that source. Also fixed: the old docs used @allow_storage / gl.Contract (the pre-v0.3 API). The verified LogIndexer example uses the current @gl.storage.allow / gl.contract.Contract style per genlayer/__init__.py's own "recommended import pattern" docstring. Scoped this fix to only the two Vector Store pages the issue is about — the rest of the docs site still uses the older style throughout, which is a separate, much larger doc-consistency issue beyond this one's scope. Verification (no local mdx dev server, so checked what's checkable standalone): - Extracted every Python code block from both files and ran compile(..., 'exec') on each — all syntax-valid. - Diffed the LogIndexer example against genlayer-studio/examples/contracts/log_indexer.py directly — identical. - Installed @mdx-js/mdx standalone (no other project deps needed) and ran the actual MDX compiler against both changed files — both compile cleanly, catching the JSX/Callout-component syntax that a plain markdown lint wouldn't.
✅ Deploy Preview for genlayer-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Contributor
📝 WalkthroughWalkthroughThe PR updates Vector Store documentation and examples for the v0.3.0 APIs. It adds explicit Euclidean distance, vector ID tracking, tombstone removal, stale-result filtering, updated log operations, and schema-load troubleshooting. ChangesVector Store documentation
Estimated code review effort: 3 (Moderate) | ~25 minutes Possibly related issues
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
pages/developers/intelligent-contracts/features/vector-storage.mdx (1)
75-126: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winClarify that
similarityis a non-normalized heuristic.The example types
gle.EuclideanDistanceat line 108, but line 137 labelsstr(1 - result.distance)as"similarity". Euclidean distance is unbounded, so this can produce negative or uninterpretable values; rename the key or add wording that this is a rough heuristic, not a bounded cosine-style similarity score.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pages/developers/intelligent-contracts/features/vector-storage.mdx` around lines 75 - 126, The LogIndexer.get_closest_vector example labels 1 - result.distance as “similarity” even though EuclideanDistance is unbounded. Rename the returned key to reflect distance or explicitly describe it as a rough, non-normalized heuristic rather than a bounded similarity score.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pages/developers/intelligent-contracts/features/vector-storage.mdx`:
- Around line 143-167: Clear the tombstone when re-adding an already indexed
log: in add_log and update_log, set removed_log_ids[key] to False immediately
before the early return in the existing-entry branch. Apply this identical
change in pages/developers/intelligent-contracts/features/vector-storage.mdx
lines 143-167 and
pages/developers/intelligent-contracts/examples/vector-store-log-indexer.mdx
lines 76-106.
- Line 17: Update the `knn()` description in the vector-storage documentation to
state that callers must select a distance metric, using `gle.EuclideanDistance`
as a common choice. Remove the claim that Euclidean distance is the default, and
retain the surrounding explanation of nearest-neighbor and semantic-search use
cases.
- Around line 127-141: Clarify the returned score from the vector-search
snippets as a rough inverse Euclidean-distance/rank score rather than a
normalized similarity. Update both
`pages/developers/intelligent-contracts/features/vector-storage.mdx` lines
127-141 and
`pages/developers/intelligent-contracts/examples/vector-store-log-indexer.mdx`
lines 60-74 consistently by renaming the `similarity` field or documenting that
`1 - result.distance` may be negative and decreases as distance grows.
---
Outside diff comments:
In `@pages/developers/intelligent-contracts/features/vector-storage.mdx`:
- Around line 75-126: The LogIndexer.get_closest_vector example labels 1 -
result.distance as “similarity” even though EuclideanDistance is unbounded.
Rename the returned key to reflect distance or explicitly describe it as a
rough, non-normalized heuristic rather than a bounded similarity score.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: abb6187c-bc0b-4ea9-ab11-8cc564cbaf36
📒 Files selected for processing (2)
pages/developers/intelligent-contracts/examples/vector-store-log-indexer.mdxpages/developers/intelligent-contracts/features/vector-storage.mdx
|
|
||
| #### 2. Similarity Calculation | ||
| The Vector Store allows you to calculate the similarity between a given text and stored vectors using cosine similarity. This is useful for finding the most semantically similar texts, enabling applications like recommendation systems or text-based search. | ||
| The Vector Store lets you find the nearest neighbors of a query vector via `knn()`, using a configurable distance metric (Euclidean by default). This is useful for finding the most semantically similar texts, enabling applications like recommendation systems or text-based search. |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Locate and inspect the referenced docs page and related context.
fd -a 'vector-storage\.mdx$' .
echo '---'
if [ -f pages/developers/intelligent-contracts/features/vector-storage.mdx ]; then
wc -l pages/developers/intelligent-contracts/features/vector-storage.mdx
sed -n '1,130p' pages/developers/intelligent-contracts/features/vector-storage.mdx
fi
echo '--- search_vecdb ---'
rg -n "type VecDB|interface VecDB|struct VecDB|VecDB<|explicit metric|Euclidean|distance metric|knn<" . --glob '!node_modules' --glob '!dist' --glob '!build' || trueRepository: genlayerlabs/genlayer-docs
Length of output: 8020
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- check git diff for the referenced page ---'
git diff -- pages/developers/intelligent-contracts/features/vector-storage.mdx | sed -n '1,180p' || true
echo '--- related example page occurrences ---'
if [ -f pages/developers/intelligent-contracts/examples/vector-store-log-indexer.mdx ]; then
sed -n '1,130p' pages/developers/intelligent-contracts/examples/vector-store-log-indexer.mdx
fiRepository: genlayerlabs/genlayer-docs
Length of output: 5439
Clarify that the distance metric is required.
Update knn() wording to say callers must choose a distance metric, with gle.EuclideanDistance as the common choice. Avoid “Euclidean by default,” because VecDB[T, S, V, D] requires the metric type parameter.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pages/developers/intelligent-contracts/features/vector-storage.mdx` at line
17, Update the `knn()` description in the vector-storage documentation to state
that callers must select a distance metric, using `gle.EuclideanDistance` as a
common choice. Remove the claim that Euclidean distance is the default, and
retain the surrounding explanation of nearest-neighbor and semantic-search use
cases.
Comment on lines
+127
to
+141
| for result in self.vector_store.knn(emb, len(self.vector_store)): | ||
| log_id = result.value.log_id | ||
| if log_id in self.removed_log_ids and self.removed_log_ids[log_id]: | ||
| continue | ||
| if log_id not in self.log_vector_ids: | ||
| continue | ||
| if self.log_vector_ids[log_id] != result.id: | ||
| continue | ||
| return { | ||
| "vector": list(str(x) for x in result.key), | ||
| "similarity": str(1 - result.distance), | ||
| "id": result.value.log_id, | ||
| "text": result.value.text, | ||
| } | ||
| return None |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== Locate files =="
for f in \
pages/developers/intelligent-contracts/features/vector-storage.mdx \
pages/developers/intelligent-contracts/examples/vector-store-log-indexer.mdx
do
if [ -f "$f" ]; then
echo "FOUND $f"
wc -l "$f"
else
echo "MISSING $f"
fi
done
echo
echo "== Relevant snippets =="
python3 - <<'PY'
from pathlib import Path
for f in [
Path("pages/developers/intelligent-contracts/features/vector-storage.mdx"),
Path("pages/developers/intelligent-contracts/examples/vector-store_log-indexer.mdx"),
]:
pass
PY
for f in \
pages/developers/intelligent-contracts/features/vector-storage.mdx \
pages/developers/intelligent-contracts/examples/vector-store-log-indexer.mdx
do
echo "--- $f"
if [ -f "$f" ]; then
sed -n '1,180p' "$f"
fi
done
echo
echo "== Search for EuclideanDistance and distance names =="
rg -n "EuclideanDistance|distance|similarity|vector_store|VectorStore|knn" pages/developers/intelligent-contracts -S || true
echo
echo "== Search repository for EuclideanDistance definition/usages =="
rg -n "class EuclideanDistance|EuclideanDistance|def knn|knn\\(" -S . | head -200Repository: genlayerlabs/genlayer-docs
Length of output: 26348
Clarify that "similarity" is an inverse Euclidean distance
1 - result.distance is not a normalized similarity score for gle.EuclideanDistance; it can be negative or read unintuitively as distance grows. Rename/adjust the returned field, or document both snippets as a rough inverse/rank score.
📍 Affects 2 files
pages/developers/intelligent-contracts/features/vector-storage.mdx#L127-L141(this comment)pages/developers/intelligent-contracts/examples/vector-store-log-indexer.mdx#L60-L74
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pages/developers/intelligent-contracts/features/vector-storage.mdx` around
lines 127 - 141, Clarify the returned score from the vector-search snippets as a
rough inverse Euclidean-distance/rank score rather than a normalized similarity.
Update both `pages/developers/intelligent-contracts/features/vector-storage.mdx`
lines 127-141 and
`pages/developers/intelligent-contracts/examples/vector-store-log-indexer.mdx`
lines 60-74 consistently by renaming the `similarity` field or documenting that
`1 - result.distance` may be negative and decreases as distance grows.
Comment on lines
143
to
+167
| @gl.public.write | ||
| def add_log(self, log: str, log_id: int) -> None: | ||
| key = log_id | ||
| if key in self.log_vector_ids: | ||
| self.vector_store.get_by_id(self.log_vector_ids[key]).value = StoreValue( | ||
| text=log, log_id=key | ||
| ) | ||
| return | ||
|
|
||
| emb = self.get_embedding(log) | ||
| self.vector_store.insert(emb, StoreValue(text=log, log_id=u256(log_id))) | ||
| vector_id = self.vector_store.insert(emb, StoreValue(text=log, log_id=key)) | ||
| self.log_vector_ids[key] = vector_id | ||
|
|
||
| @gl.public.write | ||
| def update_log(self, log_id: int, log: str) -> None: | ||
| key = log_id | ||
| if key in self.log_vector_ids: | ||
| self.vector_store.get_by_id(self.log_vector_ids[key]).value = StoreValue( | ||
| text=log, log_id=key | ||
| ) | ||
| return | ||
|
|
||
| emb = self.get_embedding(log) | ||
| for elem in self.vector_store.knn(emb, 2): | ||
| if elem.value.text == log: | ||
| elem.value.log_id = u256(log_id) | ||
| vector_id = self.vector_store.insert(emb, StoreValue(text=log, log_id=key)) | ||
| self.log_vector_ids[key] = vector_id |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Tombstone never clears when a removed log_id is re-added. remove_log sets removed_log_ids[key] = True without removing key from log_vector_ids. add_log/update_log take the "update existing entry" branch when key in log_vector_ids and update the VecDBElement.value in place, but never reset removed_log_ids[key]. A log_id that is removed and then re-added stays permanently invisible to get_closest_vector, since the tombstone check still filters it out. This is a functional bug in the reference LogIndexer contract duplicated identically in both files.
pages/developers/intelligent-contracts/features/vector-storage.mdx#L143-L167: inadd_logandupdate_log, addself.removed_log_ids[key] = Falseright before the earlyreturnin the "already indexed" branch.pages/developers/intelligent-contracts/examples/vector-store-log-indexer.mdx#L76-L106: apply the identical fix toadd_logandupdate_log, since this code is byte-for-byte duplicated from the feature page.
🐛 Proposed fix (apply to both files)
def add_log(self, log: str, log_id: int) -> None:
key = log_id
if key in self.log_vector_ids:
self.vector_store.get_by_id(self.log_vector_ids[key]).value = StoreValue(
text=log, log_id=key
)
+ self.removed_log_ids[key] = False
return
...
def update_log(self, log_id: int, log: str) -> None:
key = log_id
if key in self.log_vector_ids:
self.vector_store.get_by_id(self.log_vector_ids[key]).value = StoreValue(
text=log, log_id=key
)
+ self.removed_log_ids[key] = False
return📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| @gl.public.write | |
| def add_log(self, log: str, log_id: int) -> None: | |
| key = log_id | |
| if key in self.log_vector_ids: | |
| self.vector_store.get_by_id(self.log_vector_ids[key]).value = StoreValue( | |
| text=log, log_id=key | |
| ) | |
| return | |
| emb = self.get_embedding(log) | |
| self.vector_store.insert(emb, StoreValue(text=log, log_id=u256(log_id))) | |
| vector_id = self.vector_store.insert(emb, StoreValue(text=log, log_id=key)) | |
| self.log_vector_ids[key] = vector_id | |
| @gl.public.write | |
| def update_log(self, log_id: int, log: str) -> None: | |
| key = log_id | |
| if key in self.log_vector_ids: | |
| self.vector_store.get_by_id(self.log_vector_ids[key]).value = StoreValue( | |
| text=log, log_id=key | |
| ) | |
| return | |
| emb = self.get_embedding(log) | |
| for elem in self.vector_store.knn(emb, 2): | |
| if elem.value.text == log: | |
| elem.value.log_id = u256(log_id) | |
| vector_id = self.vector_store.insert(emb, StoreValue(text=log, log_id=key)) | |
| self.log_vector_ids[key] = vector_id | |
| `@gl.public.write` | |
| def add_log(self, log: str, log_id: int) -> None: | |
| key = log_id | |
| if key in self.log_vector_ids: | |
| self.vector_store.get_by_id(self.log_vector_ids[key]).value = StoreValue( | |
| text=log, log_id=key | |
| ) | |
| self.removed_log_ids[key] = False | |
| return | |
| emb = self.get_embedding(log) | |
| vector_id = self.vector_store.insert(emb, StoreValue(text=log, log_id=key)) | |
| self.log_vector_ids[key] = vector_id | |
| `@gl.public.write` | |
| def update_log(self, log_id: int, log: str) -> None: | |
| key = log_id | |
| if key in self.log_vector_ids: | |
| self.vector_store.get_by_id(self.log_vector_ids[key]).value = StoreValue( | |
| text=log, log_id=key | |
| ) | |
| self.removed_log_ids[key] = False | |
| return | |
| emb = self.get_embedding(log) | |
| vector_id = self.vector_store.insert(emb, StoreValue(text=log, log_id=key)) | |
| self.log_vector_ids[key] = vector_id |
📍 Affects 2 files
pages/developers/intelligent-contracts/features/vector-storage.mdx#L143-L167(this comment)pages/developers/intelligent-contracts/examples/vector-store-log-indexer.mdx#L76-L106
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pages/developers/intelligent-contracts/features/vector-storage.mdx` around
lines 143 - 167, Clear the tombstone when re-adding an already indexed log: in
add_log and update_log, set removed_log_ids[key] to False immediately before the
early return in the existing-entry branch. Apply this identical change in
pages/developers/intelligent-contracts/features/vector-storage.mdx lines 143-167
and pages/developers/intelligent-contracts/examples/vector-store-log-indexer.mdx
lines 76-106.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #437 — the Vector Store / embeddings docs were stale against the current SDK in all four ways reported.
What was wrong, and how I verified the fix
1. Wrong import path. Docs showed
from backend.node.genvm.std.vector_store import VectorStore(an internal path) andimport genlayermodelwrappers— neither exists. Verified againstgenvm-executor's actualrunners/genlayer-py-std/src-emb/genlayer_embeddings/__init__.py:VecDB,VecDBElement, theDistanceclasses, andSentenceTransformerare all exported fromgenlayer_embeddings, imported asimport genlayer_embeddings as gle.2.
VecDBElementwas completely undocumented. Added a reference table (.key,.id,.value[settable],.distance,.remove()) sourced directly fromvecdb.py's docstrings/property definitions — not guessed.3. No complete, runnable example existed anywhere reachable from the docs. Replaced both pages' example with the real, tested
LogIndexercontract fromgenlayer-studio/examples/contracts/log_indexer.py— byte-for-byte identical (diffed to confirm, not a paraphrase). This also fixes the missing 4thVecDBtype parameter (theDistancemetric) that the old docs example omitted entirely.4. No guidance for debugging "Could not load contract schema". Traced this to
frontend/src/components/Simulator/ConstructorParameters.vueingenlayer-studio: it's a generic VueAlertwith no detail. The real Python traceback IS captured server-side —backend/node/base.py'sget_contract_schemabundlesstdout/stderr/genvm_loginto the raised exception, logged at INFO level (per_execution_finished's own comment: GenVM failures are user contract errors, not infra errors). Documenteddocker compose logs jsonrpc -fas the way to see it.Also fixed: the old docs used
@allow_storage/gl.Contract(pre-v0.3 API). The verifiedLogIndexerexample uses@gl.storage.allow/gl.contract.Contract, matchinggenlayer/__init__.py's own "recommended import pattern" docstring. I scoped this fix to only the two Vector Store pages the issue is about — the rest of the docs site still uses the older style throughout, which is a separate, much larger doc-consistency issue beyond this one's scope. Happy to open that as a follow-up issue if useful.Verification
No local
next dev/mdx server in my environment (the fullnpm run buildpipeline needs several pre-build scripts), so I verified what's checkable standalone instead of skipping verification:compile(..., 'exec')on each — all syntax-valid.LogIndexerexample againstgenlayer-studio/examples/contracts/log_indexer.pydirectly — identical.@mdx-js/mdxstandalone (no other project deps needed) and ran the actual MDX compiler against both changed files — both compile cleanly, which catches JSX/Callout-component syntax errors a plain markdown lint wouldn't.Summary by CodeRabbit