Skip to content

feat(collection): optional progress bar for Collection.upsert (#10) - #118

Open
srijanarya wants to merge 2 commits into
supabase:mainfrom
srijanarya:feat/upsert-progress
Open

srijanarya wants to merge 2 commits into
supabase:mainfrom
srijanarya:feat/upsert-progress

Conversation

@srijanarya

Copy link
Copy Markdown

Closes #10, scoped to upsert; create_index progress is left for a follow-up as discussed in the issue.

Adds an opt-in show_progress: bool = False parameter to Collection.upsert. When enabled:

  • Uses tqdm.auto, so it picks a notebook widget or a terminal bar automatically (the notebook-vs-shell problem raised in the issue).
  • Reports a known total when records is Sized (list, tuple, ...), and falls back to an indeterminate bar for generators and other unsized iterables.
  • tqdm is an optional extra (pip install vecs[progress]), not a new hard dependency, so pip install vecs is unchanged. Calling with show_progress=True without tqdm installed raises an ImportError with the install hint.
  • Default behaviour (show_progress=False) is unchanged.

Tests added in src/tests/test_upsert_progress.py for: default creates no progress object, sized input reports the total and updates per 500-record chunk, generator input is indeterminate and still updates per chunk, and the ImportError path when tqdm is absent. Full suite run locally against supabase/postgres:15.1.1.78: 44 passed; the 3 test_adapters.py text-embedding tests fail identically on main without the sentence-transformers extra installed.

…se#10)

Adds opt-in show_progress to Collection.upsert. Uses tqdm.auto so it
picks a notebook widget or terminal bar; reports a known total when
records is Sized, otherwise an indeterminate bar. tqdm is an optional
extra (vecs[progress]); default behaviour is unchanged.

Scoped to upsert; create_index progress is left for a follow-up.

@marcosbeta23 marcosbeta23 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Went through the diff against what we discussed on #10 — solid implementation. Left one inline note about progress-bar cleanup on an exception path; everything else looks good to me.

Comment thread src/vecs/collection.py
Comment on lines 389 to +394
sess.execute(stmt)
if progress is not None:
progress.update(len(chunk))

if progress is not None:
progress.close()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work overall — the show_progress scoping is clean, and tqdm.auto resolves the notebook/terminal split exactly as discussed in the issue.

One correctness note on this hunk: progress.close() (and the update() calls) only run if the per-chunk loop finishes without error. If sess.execute(stmt) raises partway through, the function exits via the exception and the bar never closes — harmless in a plain terminal, but tqdm.auto's notebook widget can be left half-rendered if that happens in Jupyter.

Since the fix needs to wrap the whole per-chunk loop rather than just these lines, something like a try/finally around the loop body (with progress.close() in the finally), or using tqdm as a context manager, would make this exception-safe. Happy to sketch it out further if useful — not blocking, but worth tightening before merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature Request: Progress Bars

2 participants