diff --git a/README.md b/README.md index da7cab0..165025a 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,7 @@ your workload is synchronous and fits `node:sqlite`, use `node:sqlite`** — nothing needs installing and nothing needs compiling. This package exists for the parts it does not cover. -Verified against `@appthreat/sqlite3` 9.0.1 on **Node v24.18.0 and +Verified against `@appthreat/sqlite3` 9.0.2 on **Node v24.18.0 and v26.7.0**, which expose an identical `node:sqlite` surface and behave identically on every point below — so this table holds across the whole range this package supports. `node:sqlite` did grow quickly during 24.x @@ -225,8 +225,13 @@ await using stmt = db2.prepare("SELECT 1"); // finalized the same way ``` `each()` stays callback-only — the async iterator is its promise-based -replacement. `db.prepare()` and `db.backup()` keep their synchronous return -in every form. +replacement. `db.backup()` keeps its synchronous return in every form. +`db.prepare()` returns the statement synchronously in its callback form; the +no-callback form returns the statement wrapped so that `await db.prepare(sql)` +resolves only once the prepare has completed and the introspection accessors +(`columns`, `parameterCount`, `parameterNames`, `readonly`) are populated — +and yields the statement itself. The wrapper still forwards every statement +method, so `db.prepare(sql).run(...)` chaining is unchanged. ## Performance options @@ -421,7 +426,13 @@ A JS function called per row is the wrong tool for bulk filtering — fetch and filter in JS (or write the predicate in SQL). A JS collation is even sharper: sorting 100k rows costs O(N log N) round trips (~17 s). Where they shine is pushing _logic_ into a query — a regexp, a domain -checksum, a custom aggregate over a bounded group. +checksum, a custom aggregate over a bounded group. SQLite invokes a +scalar function per row and needs each result before the next row is +read, so there is no batched/array form: the per-call cost is +structural. [docs/performance.md](docs/performance.md#javascript-functions-the-crossover-to-fetch-and-filter) +works out the crossover — a few thousand candidate rows is where +fetch-and-filter overtakes a UDF predicate, measured at ~60× on a real +16k-row scan. Two deliberate restrictions follow from the threading model: @@ -433,6 +444,17 @@ Two deliberate restrictions follow from the threading model: to run entirely (remove it with `removeCollation()` or use the async API): a comparison would need the blocked JS thread, and unlike functions, a collation callback has no way to report an error. + `db.withCollation(name, cmp, fn)` scopes a registration to the awaited + body, registering before it runs and removing after — even when the + body throws — so the sync methods are only gated for the block: + + ```js + const rows = await db.withCollation( + "locale", + (a, b) => a.localeCompare(b, "de"), + () => db.all("SELECT name FROM t ORDER BY name COLLATE locale"), + ); // here the collation is removed and getSync() works again + ``` Errors: a throwing callback surfaces as a `SQLITE_ERROR` whose message names the function, with the original JS error attached as `err.cause`; @@ -528,10 +550,12 @@ service it). The token form has no such restriction. ### Statement and connection introspection ```js -const stmt = await db.prepare("SELECT name AS who FROM users WHERE id = ?"); +const stmt = await db.prepare("SELECT name AS who FROM users WHERE id = $id", { $id: 1 }); stmt.readonly; // true — sqlite3_stmt_readonly stmt.parameterCount; // 1 -stmt.parameterNames; // ['?1'] (null entries for positional `?`) +stmt.parameterNames; // ['$id'] — a fully positional statement (`?`) has +// no names at all and reports undefined; mixed statements keep null at +// every positional index so indices stay aligned stmt.columns; // [{ name: 'who', declaredType: 'TEXT', // database: 'main', table: 'users', origin: 'name' }] stmt.status(sqlite3.STMTSTATUS_FULLSCAN_STEP); // >0: the query scanned @@ -545,7 +569,9 @@ await db.dbConfig(sqlite3.DBCONFIG_DEFENSIVE, true); // safe db_config switches The statement accessors serve a snapshot taken when the statement was prepared, so reading them never touches the sqlite handle and cannot -race a running query; fields SQLite reports as absent (an expression +race a running query; `await db.prepare(sql)` resolves only after that +snapshot is published, so the accessors are populated at the first read. +Fields SQLite reports as absent (an expression column has no origin, a typeless column no declared type) are omitted rather than nulled. Integer modes apply to `changes`/`totalChanges` as everywhere else. `tableInfo` runs a `PRAGMA table_info`, so a @@ -616,12 +642,18 @@ const copy = await sqlite3.deserializeFromBytes(bytes, { ``` `serializeToBytes` returns the exact bytes a file copy would contain -(the FIFO-ordering `db.serialize()` keeps its old meaning). The bytes -are named deliberately: overloading `serialize()` would be the worst -API decision available. `deserializeFromBytes` **copies** into -SQLite-owned memory — handing a JS buffer to SQLite directly is a -use-after-free waiting to happen — and rejects corrupt input with -`SQLITE_NOTADB` rather than crashing later. +(the FIFO-ordering `db.serialize()` keeps its old meaning). The +snapshot includes every committed transaction — serialization reads +through the pager, and the pager reads through the WAL — and for a WAL +database the returned bytes are rewritten to rollback-journal format, +so the output always round-trips through `deserializeFromBytes` (a +deserialized copy has no `-wal` file and could not open a WAL-format +image demanding recovery). The live database's journal mode is +untouched. The bytes are named deliberately: overloading `serialize()` +would be the worst API decision available. `deserializeFromBytes` +**copies** into SQLite-owned memory — handing a JS buffer to SQLite +directly is a use-after-free waiting to happen — and rejects corrupt +input with `SQLITE_NOTADB` rather than crashing later. ## Incremental blob I/O (v9) diff --git a/docs/install.md b/docs/install.md index 923f9ad..fbc4442 100644 --- a/docs/install.md +++ b/docs/install.md @@ -42,7 +42,7 @@ dependent allowlists it. This package declares `"install": "node-gyp-build"`, so pnpm prints a notice like: ``` -[ERR_PNPM_IGNORED_BUILDS] Ignored build scripts: @appthreat/sqlite3@9.0.1 +[ERR_PNPM_IGNORED_BUILDS] Ignored build scripts: @appthreat/sqlite3@9.0.2 ``` **You can ignore that notice.** Verified empirically (pnpm 11.23.0, macOS, diff --git a/docs/performance.md b/docs/performance.md index c1e19c2..f606b08 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -324,7 +324,76 @@ README quotes both. autocommit writes. Removing a listener returns exactly to baseline. - **JS functions/aggregates/collations**: ~19 µs per invocation, O(n log n) comparisons for a collation sort. Use them for glue, never per row - over large scans. + over large scans — the next section is the full picture. + +## JavaScript functions: the crossover to fetch-and-filter + +A scalar UDF registered with `db.function()` pays one blocking +cross-thread round trip per invocation: the query steps on a libuv +worker, and each call marshals its arguments to the JS thread, waits for +your function, and carries the result back — **~19 µs per call** on +Apple Silicon (19.6 µs measured; ~28 µs in the Linux container), against +~0.5 µs for the same work as a SQL expression. + +That cost is invisible when the function filters a bounded candidate +set and decisive when it filters a large scan. The crossover is easy to +estimate: a UDF predicate wins while + +``` +rows × 19 µs < (rows × per-row read cost) + (your filtering in JS) +``` + +— with a ~0.5 µs/row read, **fetch-and-filter overtakes the UDF at a +few thousand candidate rows**. A real integration (a version-comparison +predicate over a 16,046-row candidate set) measured the two shapes +end to end: + +| strategy | time | result | +| --------------------------------------------------- | ------- | ------------------ | +| the predicate as a JS UDF inside SQL | 331 ms | identical hit set | +| narrow projection over the candidates, filter in JS | 5.5 ms | identical hit set | + +— ~60× apart for identical results. The UDF shape is *correct*, just +slow, which is what makes it dangerous: nothing fails, the query simply +costs one thread-park per row. + +Rules of thumb: + +- **Push logic, not filtering.** A UDF shines when SQL needs your + domain logic on a *bounded* set — a regexp over candidate rows, a + checksum on a handful of values, a custom aggregate over a grouped + result. Marking the function `deterministic` lets SQLite use indexes; + `directOnly` (the default) keeps it out of attacker-reachable schema + SQL. +- **Bulk filtering belongs in SQL or after `all()`.** Write the + predicate in SQL, or fetch a narrow projection and filter in JS. Both + are shown in the table above; filtering in JS after `all()` costs + ~1.25 µs/row in the README's 100k-row example, against ~19 µs/row for + the per-row UDF. +- **There is no batched UDF.** SQLite invokes a scalar function per row + at step time and needs each result before the next row is read, so + calls cannot be coalesced into one array-in/array-out round trip + without changing results. The per-call cost is structural, which is + why the crossover above is the tool to reason with. + +The same arithmetic applies to aggregates (`step` is one round trip per +row) and collations (O(n log n) round trips for a sort — sorting in JS +after `all()` is faster for anything but small or one-off sorts). + +### Future direction: UDFs on the synchronous fast path + +The refusal of JS functions on the sync methods is policy, not the +structural limit above. On `getSync`/`runSync`/`allSync` the JS thread +is already the one executing SQL, so a callback could be invoked +re-entrantly — a direct call with no cross-thread round trip, the way +`node:sqlite` runs UDFs inline — and unlike collations, a function has +an error channel (`sqlite3_result_error`), so failures are reportable +mid-query. The current build refuses instead with an explicit error +(`src/function.cc`, `SyncRefusalMessage`). Landing inline sync-path +invocation would make the natural `... AND vers_compare(?, vers)` +shape genuinely fast, closing the gap the README's comparison table +attributes to `node:sqlite`. Deliberate follow-up work, out of scope +for this release. ## Where this package loses diff --git a/lib/augment.d.ts b/lib/augment.d.ts index 9b28ac4..c3c059e 100644 --- a/lib/augment.d.ts +++ b/lib/augment.d.ts @@ -277,10 +277,22 @@ declare module './native.js' { callback: (this: Database, err: SqliteError | null) => void, ): this; - /** Prepares a statement; returns it synchronously in every form. */ + /** + * Prepares a statement. The no-callback form returns the statement + * wrapped in a thenable: awaiting it settles only once the native + * prepare (and any bind) has completed and the introspection + * accessors — `columns`, `parameterCount`, `parameterNames`, + * `readonly` — are populated, and yields the statement itself. + * Before the await, the wrapper forwards every statement member, + * so `db.prepare(sql).run(...)` keeps its synchronous surface. A + * prepare failure rejects the await. + * @since 9.0.2 the await gates on prepare completion. + */ + prepare(sql: string): Statement & Promise; + /** Prepares a statement; the callback is an error-only errback. */ prepare( sql: string, - callback?: (this: Statement, err: SqliteError | null) => void, + callback: (this: Statement, err: SqliteError | null) => void, ): Statement; /** Prepares a statement with one array/named bind object. */ prepare( @@ -288,8 +300,15 @@ declare module './native.js' { params: BindParams, callback?: (this: Statement, err: SqliteError | null) => void, ): Statement; - /** Prepares a statement with variadic bind values. */ - prepare(sql: string, ...params: [...BindValue[]]): Statement; + /** + * Prepares a statement with variadic bind values; awaiting the + * result gates on the prepare and the bind both completing. + * @since 9.0.2 the await gates on prepare + bind completion. + */ + prepare( + sql: string, + ...params: [...BindValue[]] + ): Statement & Promise; /** Prepares a statement with variadic bind values and a callback. */ prepare( sql: string, @@ -474,6 +493,27 @@ declare module './native.js' { */ removeCollation(name: string): this; + /** + * Runs `fn` with a JavaScript collation registered, removing it + * again afterwards: the blast radius of the registration is the + * awaited block, not the connection's lifetime. Inside the block + * the synchronous methods refuse to run, as with `collation()`; + * an error thrown by `fn` still removes the collation before the + * rejection propagates. Interleaved or nested scopes for the same + * name are last-wins — use distinct names for concurrent scopes. + * + * @since 9.0.2 + * @example + * const rows = await db.withCollation('locale', + * (a, b) => a.localeCompare(b, 'de'), + * () => db.all('SELECT name FROM t ORDER BY name COLLATE locale')); + */ + withCollation( + name: string, + cmp: (a: string, b: string) => number, + fn: (db: Database) => unknown, + ): Promise; + // ---- Hooks, authorizer, progress, WAL, introspection (v9). /** diff --git a/lib/native.d.ts b/lib/native.d.ts index 2242b27..da81824 100644 --- a/lib/native.d.ts +++ b/lib/native.d.ts @@ -1284,8 +1284,10 @@ export declare class Statement extends EventEmitter { /** * The bind parameter names in 1-based order: `':a'`, `'@b'`, `'$c'`, - * `'?1'`. Positional `?` parameters have no name; their entries are - * null so indices stay aligned. + * `'?1'`. `undefined` for a fully positional statement (every + * parameter a bare `?`): there are no names to report. A mixed + * statement keeps one array entry per parameter, with null at every + * positional index so indices stay aligned. * * @since 9.0.0 */ diff --git a/lib/promises.js b/lib/promises.js index 5c3e97f..709f9dc 100644 --- a/lib/promises.js +++ b/lib/promises.js @@ -5,8 +5,10 @@ // Every wrapped method is dual-mode: when the last argument is a function // the call behaves exactly like the callback API (and returns `this`, so // chaining keeps working); otherwise it returns a promise. Database#prepare -// and Database#backup deliberately keep their synchronous return in both -// forms — see the handoff notes for 03. +// is not dual-mode: its callback form returns the statement synchronously, +// and its no-callback form returns the statement wrapped in a thenable +// that settles once the prepare has landed — see lib/sqlite3.js. +// Database#backup keeps its synchronous return in both forms. import { AsyncLocalStorage } from 'node:async_hooks'; import { Readable, Writable } from 'node:stream'; diff --git a/lib/sqlite3.js b/lib/sqlite3.js index fb541d9..223ebf9 100644 --- a/lib/sqlite3.js +++ b/lib/sqlite3.js @@ -984,9 +984,57 @@ function extractErrBack(args) { return undefined; } -// Captured before the promise API wraps Statement#bind: prepare()'s -// no-callback form must keep returning the statement synchronously, and a -// dual-mode bind would hand back a promise instead. +/** + * Splits a call's trailing callback across the two native slots that can + * both observe one failed prepare: the statement's error-only prepare + * errback and the queued statement method's completion callback (which + * also carries the call's results). When the prepare fails, both slots + * fire with the same error; the shared token here keeps the user's + * callback to a single invocation, whatever route delivered first. + * Success calls flow through untouched. + * + * Mutates `args`, replacing the trailing callback with the guarded + * completion wrapper. + * + * @param {unknown[]} args the call arguments, ending in a callback. + * @returns {{ + * errback: (this: import('./sqlite3-binding.js').Statement, err: import('./native.js').SqliteError | null) => void, + * completion: (this: import('./sqlite3-binding.js').Statement, ...call: unknown[]) => void, + * } | null} the two slot handlers, or null with no trailing callback. + * @private + */ +function splitPrepareCallback(args) { + const errBack = extractErrBack(args); + if (errBack === undefined) return null; + const userCallback = /** @type {(...call: unknown[]) => void} */ ( + args[args.length - 1] + ); + let delivered = false; + /** + * @param {import('./native.js').SqliteError | null} err + * @this {import('./sqlite3-binding.js').Statement} + */ + const errback = function (err) { + if (!err || delivered) return; + delivered = true; + errBack.call(this, err); + }; + /** + * @param {...unknown} call + * @this {import('./sqlite3-binding.js').Statement} + */ + const completion = function (...call) { + if (call[0] && delivered) return; + if (call[0]) delivered = true; + userCallback.apply(this, call); + }; + args[args.length - 1] = completion; + return { errback, completion }; +} + +// Captured before the promise API wraps Statement#bind: prepare()'s bind +// path must keep its synchronous statement return, and a dual-mode bind +// would hand back a promise instead. /** @type {(...args: any[]) => any} */ const nativeStatementBind = Statement.prototype.bind; // Internal fire-and-forget finalizes must not allocate a promise per call: @@ -1180,6 +1228,15 @@ Database.prototype.applyChangeset = function (changeset, options, callback) { * work so the snapshot cannot interleave with writes. Feed the result to * {@link sqlite3.deserializeFromBytes}. * + * The snapshot carries every committed transaction: serialization reads + * each page through the pager, and the pager reads through the WAL, so + * frames not yet checkpointed are included. The returned bytes are + * rewritten to rollback-journal format — a WAL-format image would demand + * WAL recovery that a deserialized copy cannot perform (it has no `-wal` + * file) — so the output is always valid input to + * `deserializeFromBytes()`. The live database's journal mode is + * untouched. + * * @this {import('./sqlite3-binding.js').Database} * @param {string | ((err: import('./native.js').SqliteError | null, bytes: Uint8Array) => void)} [dbName] * the attached database name (default `'main'`), or the callback. @@ -1354,27 +1411,261 @@ sqlite3.deserializeFromBytes = async function deserializeFromBytes( }; // Database#prepare stays uncached: the caller owns the returned statement. -// It also keeps its synchronous return in every form (see the promise API -// notes in lib/promises.js): `await db.prepare(sql)` still yields the -// statement, but a prepare error surfaces on its error event rather than -// as a rejection. +// +// The no-callback form returns a thenable wrapper around the statement. +// Awaiting it settles only once the worker has completed the prepare (and +// any bind), so the introspection accessors — `columns`, `parameterCount`, +// `parameterNames`, `readonly` — are populated at the first read after the +// await. The wrapper forwards every statement member, so pre-await +// chaining (`db.prepare(sql).run(...)`, `.finalize()`) keeps the +// historical synchronous surface; after the await, callers hold the +// statement itself. The callback form is unchanged: it returns the +// statement synchronously and a prepare error surfaces through its +// error-only errback rather than as a rejection. + +/** + * Wraps a statement whose prepare (and optional bind) is still queued: + * awaiting the wrapper settles only once that work has landed, while every + * statement member stays reachable for pre-await chaining. After the + * await, callers hold the statement itself, not the wrapper. + * + * @param {import('./sqlite3-binding.js').Statement} statement the statement. + * @param {Promise} ready the completion promise. + * @param {() => void} addAwaiter called for every read of `then`, so the + * failure routing can tell awaited wrappers from fire-and-forget ones. + * @returns {import('./sqlite3-binding.js').Statement & Promise} the thenable wrapper. + * @private + */ +function prepareThenableWrapper(statement, ready, addAwaiter) { + /** @type {import('./sqlite3-binding.js').Statement & Promise} */ + let wrapper; + wrapper = + /** @type {import('./sqlite3-binding.js').Statement & Promise} */ ( + /** @type {unknown} */ ( + new Proxy(statement, { + /** + * @param {import('./sqlite3-binding.js').Statement} target + * @param {string | symbol} prop + */ + get(target, prop) { + if (prop === 'then') { + addAwaiter(); + return /** @type {(onFulfilled?: (value: unknown) => unknown, onRejected?: (err: Error) => unknown) => unknown} */ ( + resolve, + reject, + ) => + ready.then( + /** + * @param {import('./sqlite3-binding.js').Statement} value + */ + (value) => + /** @type {(value: unknown) => unknown} */ ( + resolve + )(value), + /** + * @param {Error} err + */ + (err) => + /** @type {(err: Error) => unknown} */ ( + reject + )(err), + ); + } + if (prop === 'catch') { + return ( + /** @param {(err: Error) => unknown} onRejected */ + (onRejected) => ready.catch(onRejected) + ); + } + if (prop === 'finally') { + return ( + /** @param {() => void} onFinally */ + (onFinally) => ready.finally(onFinally) + ); + } + const value = Reflect.get(target, prop, target); + if (typeof value !== 'function') return value; + // Native methods must run with the statement as + // the receiver: a proxy receiver cannot be + // unwrapped back to the ObjectWrap. + return /** @type {(...args: unknown[]) => unknown} */ ( + ...args + ) => { + const result = value.apply(target, args); + // Methods that return `this` (the cores + // chain on it) hand back the wrapper, so + // pre-await chaining keeps one identity. + return result === target ? wrapper : result; + }; + }, + }) + ) + ); + return wrapper; +} + +/** + * The no-callback form of {@link Database#prepare}: schedules the prepare + * (and any bind) and returns the statement wrapped so awaiting it settles + * only once that work has landed and the introspection metadata is + * published. + * + * @param {import('./sqlite3-binding.js').Database} db the connection. + * @param {string} sql the SQL statement to prepare. + * @param {unknown[]} bindArgs the bind parameters (no trailing callback). + * @returns {import('./sqlite3-binding.js').Statement & Promise} the statement, awaiting it gates on completion. + * @private + */ +/** + * The no-callback form of {@link Database#prepare}: schedules the prepare + * (and any bind) and returns the statement wrapped so awaiting it settles + * only once that work has landed and the introspection metadata is + * published. + * + * Failure routing keeps both documented surfaces alive: a statement whose + * `'error'` event has a listener hears the failure there (so callback-style + * code holding the wrapper keeps working), an awaiter gets the rejection + * alone, and code doing neither keeps the historical loudness — an + * `'error'` event with no listener still throws. + * + * @param {import('./sqlite3-binding.js').Database} db the connection. + * @param {string} sql the SQL statement to prepare. + * @param {unknown[]} bindArgs the bind parameters (no trailing callback). + * @returns {import('./sqlite3-binding.js').Statement & Promise} the statement, awaiting it gates on completion. + * @private + */ +function prepareAsync(db, sql, bindArgs) { + let settled = false; + let awaiters = 0; + /** @type {import('./sqlite3-binding.js').Statement | undefined} */ + let statement; + /** @type {(value: import('./sqlite3-binding.js').Statement) => void} */ + let resolveReady; + /** @type {(err: Error) => void} */ + let rejectReady; + /** @type {Promise} */ + const ready = new Promise((resolve, reject) => { + resolveReady = resolve; + rejectReady = reject; + }); + // A rejection nobody awaits must not surface as an unhandled + // rejection: the wrapper's `then` re-exposes it to actual awaiters, + // and the statement's 'error' event stays the other surface. + ready.catch(() => { + // Deliberately empty; see above. + }); + + /** + * @param {Error} err + */ + const fail = (err) => { + if (settled) return; + settled = true; + if ( + statement !== undefined && + (statement.listenerCount('error') > 0 || awaiters === 0) + ) { + statement.emit('error', err); + } + rejectReady(err); + }; + const succeed = () => { + if (settled) return; + settled = true; + resolveReady( + /** @type {import('./sqlite3-binding.js').Statement} */ (statement), + ); + }; + + statement = associateStatement( + db, + new Statement( + db, + sql, + /** + * @param {import('./native.js').SqliteError | null} err + */ + (err) => { + if (err) fail(err); + else if (bindArgs.length === 0) succeed(); + }, + ), + ); + + if (bindArgs.length > 0) { + try { + const bindVariadic = + /** @type {(...args: unknown[]) => unknown} */ ( + /** @type {unknown} */ (nativeStatementBind) + ); + // The native bind runs behind the prepare (the statement + // queues it), so its callback fires once both have completed. + bindVariadic.call( + statement, + ...bindArgs, + /** + * @param {import('./native.js').SqliteError | null} err + */ + (err) => { + if (err) fail(err); + else succeed(); + }, + ); + } catch (err) { + // Strict binding throws synchronously (the historical + // contract); the freshly prepared statement is orphaned, so + // finalize it — close() could otherwise end up SQLITE_BUSY. + nativeStatementFinalize.call(statement); + throw err; + } + } + + return prepareThenableWrapper(statement, ready, () => { + awaiters++; + }); +} + /** * Prepares a statement for the caller to own. * - * Always returns the statement synchronously, even in the callback form - * (`await db.prepare(sql)` yields the statement): a prepare failure - * surfaces on the statement's `'error'` event instead of a rejection, and - * a bind failure throws synchronously after finalizing the orphan. + * The no-callback form returns the statement wrapped in a thenable: + * `await db.prepare(sql)` (with or without bind parameters) resolves only + * once the worker has completed the prepare and the introspection + * accessors — `columns`, `parameterCount`, `parameterNames`, `readonly` — + * are populated, and yields the statement itself. The wrapper still + * forwards every statement member, so pre-await chaining + * (`db.prepare(sql).run(...)`) keeps the synchronous surface. A prepare + * failure rejects the await (and is reported on the statement's `'error'` + * event when one is registered); a bind failure throws synchronously + * after finalizing the orphan. + * + * The callback form is unchanged: it returns the statement synchronously + * and the trailing callback is an error-only errback, so a prepare + * failure surfaces there rather than as a rejection. * * @this {import('./sqlite3-binding.js').Database} * @param {string} sql the SQL statement to prepare. * @param {...unknown} args bind parameters, then optionally a callback. - * @returns {import('./sqlite3-binding.js').Statement} the prepared statement. + * @returns {any} the statement — wrapped in a completion gate in + * promise mode; the precise overload set lives in lib/augment.d.ts. */ Database.prototype.prepare = function (sql, ...args) { - const statement = new Statement(this, sql, extractErrBack(args)); + // No trailing function: promise mode. A function is never a legal bind + // value, so the trailing-argument test is the same one the dual-mode + // methods use to detect callback mode. + if (args.length === 0 || typeof args[args.length - 1] !== 'function') { + return prepareAsync(this, sql, args); + } + // The trailing callback sits in two native slots — the prepare + // errback and the queued bind's completion — so a failed prepare + // would otherwise reach it twice. The split guards that. + const split = splitPrepareCallback(args); + const statement = new Statement( + this, + sql, + split === null ? undefined : split.errback, + ); associateStatement(this, statement); - if (!args.length) return statement; try { const bindVariadic = /** @type {(...args: unknown[]) => import('./sqlite3-binding.js').Statement} */ ( @@ -1525,6 +1816,12 @@ Database.prototype.map = cachedMethod( function cachedMethod(fn) { return function (sql, ...args) { const errBack = extractErrBack(args); + // In the two paths below a fresh statement is prepared, so the + // trailing callback sits in two native slots (prepare errback and + // the queued method's completion) and a failed prepare would + // reach it twice. The hit path has no prepare and pays nothing. + const split = splitPrepareCallback(args); + const prepareErrback = split?.errback ?? errBack; const cache = this._stmtCache; // Native state, read per field: while serialized, closing, or with @@ -1554,7 +1851,7 @@ function cachedMethod(fn) { if (!err) return; // Failed to prepare: drop it so the next call retries. cache.delete(sql); - if (errBack) errBack(err); + if (split) split.errback.call(fresh, err); else fresh.emit('error', err); }; const fresh = new Statement(this, sql, onPrepareError); @@ -1590,7 +1887,7 @@ function cachedMethod(fn) { } } - const statement = new Statement(this, sql, errBack); + const statement = new Statement(this, sql, prepareErrback); associateStatement(this, statement); try { return fn.call(this, statement, args, false); @@ -2065,7 +2362,8 @@ Database.prototype.aggregate = function (name, spec) { * While a JavaScript collation is registered, the synchronous methods * (`getSync`/`runSync`/`allSync`) refuse to run: a comparison would need * the JS thread that is blocked inside SQLite, and unlike functions a - * collation cannot report an error mid-comparison. + * collation cannot report an error mid-comparison. {@link Database#withCollation} + * scopes a registration to an awaited block, removing it again afterwards. * * @this {import('./sqlite3-binding.js').Database} * @param {string} name the collation name (1..255 bytes). @@ -2146,6 +2444,46 @@ Database.prototype.removeCollation = function (name) { return this; }; +/** + * Runs `fn` with a JavaScript collation registered, removing it again + * afterwards — the blast radius of the registration is the awaited block, + * not the connection's lifetime. Equivalent to + * `db.collation(name, cmp)` / `try { ... } finally { db.removeCollation(name) }`. + * + * While the collation is registered the synchronous methods refuse to run + * (see {@link Database#collation}); inside the block use the asynchronous + * API. An error thrown by `fn` still removes the collation before the + * rejection propagates. Interleaved or nested `withCollation` calls for + * the *same* name are last-wins — use distinct names for concurrent scopes. + * + * @this {import('./sqlite3-binding.js').Database} + * @param {string} name the collation name (1..255 bytes). + * @param {(a: string, b: string) => number} cmp the comparator. + * @param {(db: import('./sqlite3-binding.js').Database) => unknown} fn the + * body to run with the collation registered; awaited before removal. + * @returns {Promise} whatever `fn` resolves to. + * @throws {TypeError} when the name, comparator or body is missing. + * @since 9.0.2 + * @example + * const rows = await db.withCollation('locale', (a, b) => a.localeCompare(b, 'de'), + * () => db.all('SELECT name FROM t ORDER BY name COLLATE locale')); + * // here the collation is removed and the sync methods work again + */ +Database.prototype.withCollation = function (name, cmp, fn) { + if (typeof fn !== 'function') { + throw new TypeError('withCollation() requires a body function'); + } + // collation() validates the name and comparator synchronously. + this.collation(name, cmp); + return (async () => { + try { + return await fn(this); + } finally { + this.removeCollation(name); + } + })(); +}; + // --- Hooks, authorizer, progress, WAL and introspection (Deliverable 07) -- // // The commit/rollback/wal hooks are installed by the on()/removeListener() diff --git a/package.json b/package.json index 37a4038..272c574 100644 --- a/package.json +++ b/package.json @@ -1,7 +1,7 @@ { "name": "@appthreat/sqlite3", "description": "Asynchronous, non-blocking SQLite3 bindings. Modern rewrite of TryGhost/node-sqlite3", - "version": "9.0.1", + "version": "9.0.2", "homepage": "https://github.com/AppThreat/node-sqlite3", "author": "Team AppThreat ", "binary": { diff --git a/src/session.cc b/src/session.cc index fe40de5..31594bf 100644 --- a/src/session.cc +++ b/src/session.cc @@ -1252,6 +1252,19 @@ void Database::Work_BeginSerializeToBytes(Baton* baton) { void Database::Work_SerializeToBytes(napi_env e, void* data) { auto* baton = static_cast(data); + // sqlite3_serialize copies every page through the pager, and the + // pager reads through the WAL, so the snapshot carries every + // committed transaction whether or not its frames have been + // checkpointed into the main file. The one thing it cannot carry is + // the WAL format itself: a WAL database's header (bytes 18/19 = + // 0x02) demands WAL recovery on open, and a deserialized copy has no + // -wal file, so sqlite3_deserialize fails it with SQLITE_CANTOPEN at + // first use. Flip the image's file-format versions to 0x01 (rollback + // journal) — the workaround the sqlite3_deserialize documentation + // itself names. Only the copy is touched: with no NOCOPY flag sqlite + // never reads this buffer again, and the live database's journal + // mode is unaffected. + sqlite3_int64 size = 0; unsigned char* bytes = sqlite3_serialize(baton->db->_handle, baton->database.c_str(), &size, 0); @@ -1264,6 +1277,17 @@ void Database::Work_SerializeToBytes(napi_env e, void* data) { baton->message = std::string(sqlite3_errmsg(baton->db->_handle)); } else { + // A WAL-mode image carries header bytes 18/19 = 0x02, which demands + // WAL recovery on open. A deserialized copy has no -wal file, so + // sqlite3_deserialize rejects it with SQLITE_CANTOPEN at first use; + // its documentation names this exact workaround: flip the image's + // file-format versions to 0x01 (rollback journal). Only a copy is + // touched — with no NOCOPY flag sqlite never reads this buffer + // again, and the live database's journal mode is unaffected. + if (size >= 20 && bytes[18] == 0x02 && bytes[19] == 0x02) { + bytes[18] = 0x01; + bytes[19] = 0x01; + } baton->data = bytes; baton->size = size; } diff --git a/src/statement.cc b/src/statement.cc index 04d8a02..752a8cb 100644 --- a/src/statement.cc +++ b/src/statement.cc @@ -339,35 +339,41 @@ void Statement::Work_AfterPrepare(napi_env e, napi_status status, void* data) { // Who hears about a failed prepare? Normally the prepare's own // callback, which every callback-style entry point supplies. // - // When there is none -- the promise API's iterate()/fetch() path - // -- the error used to go to the statement's 'error' event while - // the calls queued behind the prepare were dropped in silence, so - // nothing ever settled their promises and the caller hung. That - // is the abort-during-prepare hang: sqlite3_interrupt() aborts a - // prepare just as readily as a step, so any abort landing in that - // window wedged the connection. Fail those calls instead; they - // are what the caller is actually waiting on. - // - // CleanQueue falls back to the 'error' event itself when the - // queue turns out to be empty, so nothing goes unreported. + // Calls already queued behind the prepare settle no matter what: + // they are what the caller is actually awaiting, and dropping them + // in silence hangs exactly the abort-during-prepare window this + // path exists for (sqlite3_interrupt() aborts a prepare just as + // readily as a step). The prepare callback runs before the queue + // so the error reaches its call-site context first — the JS layer + // shares a once-token between the two slots, so the queued + // delivery of the same error is a no-op there — and when there is + // no callback the queue is settled before the failure is reported + // on the statement's 'error' event, the documented surface for a + // prepare given no callback of its own. + EXCEPTION(stmt->message, stmt->status, exception); + // A user-defined function that threw during the step kept its JS + // error on the database as the pending cause of exactly this + // failure (Error() did this for the callback-only path). + stmt->db->AttachPendingJsError(exception_obj); Napi::Function prepare_cb = baton->callback.Value(); - if ((IS_FUNCTION(prepare_cb)) || stmt->queue.empty()) { - Error(baton.get()); - } else { - // Build the error before firing anything: a callback that - // throws leaves a pending exception, and constructing an - // Error while one is pending is a fatal napi error. - EXCEPTION(stmt->message, stmt->status, exception); - // Settle the queued calls first -- they are what the caller - // awaits -- then still report the failure on the statement - // itself, which is the documented surface for a prepare that - // was given no callback of its own. + if (IS_FUNCTION(prepare_cb)) { + Napi::Value argv[] = { exception }; + TRY_CATCH_CALL(stmt->Value(), prepare_cb, 1, argv); + stmt->FailQueue(exception, false); + } + else if (!stmt->queue.empty()) { stmt->FailQueue(exception, false); Napi::Value info[] = { Napi::String::New(env, "error"), exception }; EMIT_EVENT(stmt->Value(), 2, info); } + else { + Napi::Value info[] = { + Napi::String::New(env, "error"), exception + }; + EMIT_EVENT(stmt->Value(), 2, info); + } stmt->Finalize_(); } else { @@ -1807,6 +1813,20 @@ Napi::Value Statement::ParameterCountGetter(const Napi::CallbackInfo& info) { Napi::Value Statement::ParameterNamesGetter(const Napi::CallbackInfo& info) { if (!meta_valid) return info.Env().Undefined(); auto env = info.Env(); + // A fully positional statement (every parameter a bare `?`) carries no + // names at all: `undefined` says so plainly, instead of an array of + // nulls that reads like a bug at the call site. Mixed statements + // (named and positional) keep the array, with null at every positional + // index so the index mapping stays honest. Zero parameters is simply + // an empty array. + bool any_named = false; + for (const auto& name : meta.param_names) { + if (!name.empty()) { + any_named = true; + break; + } + } + if (!meta.param_names.empty() && !any_named) return env.Undefined(); Napi::Array result = Napi::Array::New(env, meta.param_names.size()); for (size_t i = 0; i < meta.param_names.size(); i++) { // Positional `?` parameters have no name: null keeps the index diff --git a/test/collation.test.js b/test/collation.test.js index ad0e3b4..0f953e8 100644 --- a/test/collation.test.js +++ b/test/collation.test.js @@ -198,4 +198,85 @@ describe('user-defined collations', function () { ['c', 'b', 'a'], ); }); + + // 9.0.2: a scoped registration — the blast radius of one collation() + // call used to be the connection's whole lifetime, silently removing + // the sync fast path until an explicit removeCollation(). + it('withCollation registers for the body and removes it afterwards', async function () { + await db.exec( + 'CREATE TABLE t (w TEXT);\n' + + "INSERT INTO t VALUES ('b'), ('a'), ('c')", + ); + const rows = await db.withCollation( + 'scoped', + (a, b) => a.localeCompare(b), + (scope) => { + assert.strictEqual(scope, db); + // The sync gate is active inside the block. + assert.throws(() => db.getSync('SELECT 1'), /collation/); + return db.all('SELECT w FROM t ORDER BY w COLLATE scoped'); + }, + ); + assert.deepStrictEqual( + rows.map((r) => r.w), + ['a', 'b', 'c'], + ); + // Outside the block the collation is gone and the sync methods + // work again. + assert.strictEqual(db.getSync('SELECT 1 AS v').v, 1); + }); + + it('withCollation removes the collation when the body throws', async function () { + const boom = new Error('body explosion'); + await assert.rejects( + () => + db.withCollation( + 'scoped', + (a, b) => a.localeCompare(b), + () => { + throw boom; + }, + ), + (err) => err === boom, + ); + assert.strictEqual(db.getSync('SELECT 1 AS v').v, 1); + }); + + it('withCollation awaits an async body before removing', async function () { + const rows = await db.withCollation( + 'scoped', + (a, b) => a.localeCompare(b), + async () => { + await db.exec('CREATE TABLE IF NOT EXISTS w (v TEXT)'); + return db.all( + "SELECT 'b' AS v UNION ALL SELECT 'a' ORDER BY 1 COLLATE scoped", + ); + }, + ); + assert.deepStrictEqual( + rows.map((r) => r.v), + ['a', 'b'], + ); + assert.strictEqual(db.getSync('SELECT 1 AS v').v, 1); + }); + + it('withCollation validates its arguments', function () { + assert.throws( + () => db.withCollation('x', (_a, _b) => 0), + /body function/, + ); + assert.throws( + () => + db.withCollation( + '', + (_a, _b) => 0, + () => 'body', + ), + /non-empty name/, + ); + assert.throws( + () => db.withCollation('x', 'nope', () => 'body'), + /comparator function/, + ); + }); }); diff --git a/test/introspection.test.js b/test/introspection.test.js index d6877fd..e7740ee 100644 --- a/test/introspection.test.js +++ b/test/introspection.test.js @@ -77,6 +77,18 @@ describe('statement introspection', function () { assert.deepStrictEqual(numbered.parameterNames, ['?1', '?2']); }); + it('parameterNames is undefined for fully positional statements', async function () { + // 9.0.2: a statement with only bare `?` parameters has no names to + // report — `undefined` says so plainly where an array of nulls + // read like a bug at the call site. + const positional = await prepare('SELECT ?, ?'); + assert.strictEqual(positional.parameterNames, undefined); + assert.strictEqual(positional.parameterCount, 2); + // Zero parameters keeps the (empty) array. + const none = await prepare('SELECT 1'); + assert.deepStrictEqual(none.parameterNames, []); + }); + it('columns carries declaredType, database, table and origin', async function () { const stmt = await prepare('SELECT name, score FROM user'); assert.deepStrictEqual(stmt.columns, [ diff --git a/test/prepare.test.js b/test/prepare.test.js index 4a16542..00ba756 100644 --- a/test/prepare.test.js +++ b/test/prepare.test.js @@ -568,4 +568,98 @@ describe('prepare', function () { db.close(done); }); }); + + // 9.0.2: the no-callback form resolves its await only once the worker + // has completed the prepare (and any bind). Before, `await + // db.prepare(sql)` settled one microtask later — long before the + // prepare landed — so `columns`, `parameterCount`, `parameterNames` + // and `readonly` read as `undefined` and only recovered after an + // arbitrary turn of the event loop. + describe('prepare() completion gate', function () { + let db; + before(function (_t, done) { + db = new sqlite3.Database(':memory:', done); + }); + + it('populates the introspection accessors right after the await', async function () { + await db.exec('CREATE TABLE t (v TEXT, n INT)'); + const stmt = await db.prepare('SELECT v, n FROM t WHERE n = ?'); + assert.strictEqual(stmt.parameterCount, 1); + assert.deepStrictEqual( + stmt.columns.map((c) => c.name), + ['v', 'n'], + ); + assert.strictEqual(stmt.parameterNames, undefined); + assert.strictEqual(stmt.readonly, true); + await stmt.finalize(); + }); + + it('gates the bind form on prepare and bind completing', async function () { + const stmt = await db.prepare('SELECT v, n FROM t WHERE n = ?', 1); + assert.strictEqual(stmt.parameterCount, 1); + const rows = await stmt.all(); + assert.deepStrictEqual(rows, []); + await stmt.finalize(); + }); + + it('keeps the synchronous chaining surface before the await', async function () { + const stmt = db.prepare('INSERT INTO t VALUES (?, ?)'); + // Same object identity across chained callback methods, and + // native methods run against the statement itself. + assert.ok(stmt instanceof sqlite3.Statement); + const out = stmt.run('a', 1, function (err) { + if (err) throw err; + }); + assert.strictEqual(out, stmt); + await stmt.finalize(); + const after = await db.get('SELECT count(*) AS c FROM t'); + assert.strictEqual(after.c, 1); + }); + + it('yields the statement itself after the await', async function () { + const awaited = await db.prepare('SELECT 40 + 2 AS answer'); + assert.ok(awaited instanceof sqlite3.Statement); + assert.strictEqual(awaited.sql, 'SELECT 40 + 2 AS answer'); + assert.strictEqual(awaited.parameterCount, 0); + assert.strictEqual((await awaited.get()).answer, 42); + await awaited.finalize(); + }); + + it('rejects the await on invalid SQL', async function () { + await assert.rejects( + () => db.prepare('SELECT * FROM no_such_table'), + (err) => { + assert.strictEqual(err.code, 'SQLITE_ERROR'); + return true; + }, + ); + await db.wait(); + }); + + it('reports a failure once: the await rejects, the event fires for listeners', async function () { + const events = []; + const stmt = db.prepare('SELECT * FROM no_such_table_either'); + stmt.on('error', (err) => events.push(err)); + await assert.rejects(() => stmt, /no such table/); + // One 'error' event for the listener, one rejection for the + // await — the same failure, not two. + assert.strictEqual(events.length, 1); + await db.wait(); + }); + + it('still returns the statement synchronously in the callback form', async function () { + const stmt = db.prepare('SELECT 1 AS one', function (err) { + if (err) throw err; + }); + assert.ok(stmt instanceof sqlite3.Statement); + // Accessors are undefined until the callback fires: the + // callback form keeps its historical asynchronous prepare. + assert.strictEqual(stmt.parameterCount, undefined); + await new Promise((resolve) => stmt.finalize(resolve)); + }); + + after(async function () { + await db.close(); + }); + }); }); diff --git a/test/serialize_bytes.test.js b/test/serialize_bytes.test.js index 6bb0a21..51f36b1 100644 --- a/test/serialize_bytes.test.js +++ b/test/serialize_bytes.test.js @@ -4,9 +4,19 @@ // readonly/resizable options, and the naming discipline (serialize means // FIFO ordering, the byte form is serializeToBytes). import assert from 'node:assert'; +import { rmSync } from 'node:fs'; +import { join } from 'node:path'; import { describe, it } from 'node:test'; import sqlite3 from '../lib/sqlite3.js'; +import { TMP_DIR } from './support/db.js'; + +/** Removes a database file and its journal/WAL siblings. */ +function removeDb(file) { + for (const suffix of ['', '-wal', '-shm', '-journal']) { + rmSync(`${file}${suffix}`, { force: true }); + } +} describe('serializeToBytes / deserializeFromBytes', function () { it('round-trips a database with every value shape', async function () { @@ -227,4 +237,85 @@ describe('serializeToBytes / deserializeFromBytes', function () { assert.ok(bytes instanceof Uint8Array); await db.close(); }); + + // 9.0.2: a WAL snapshot used to carry a WAL-format header (bytes + // 18/19 = 0x02), which demands WAL recovery a deserialized copy + // cannot perform — it has no -wal file — so the open failed with + // SQLITE_CANTOPEN at the far end. The serialization itself already + // includes every committed frame (sqlite3_serialize copies pages + // through the pager, and the pager reads through the WAL); the fix + // is purely the header rewrite. The reader-snapshot scenario below + // pins the frame-inclusion claim independently: a checkpoint could + // not copy the held frame even if serializeToBytes tried one. + it('round-trips a WAL database, rewriting the image to rollback-journal format', async function () { + const file = join(TMP_DIR, 'serialize-wal-round-trip.db'); + removeDb(file); + const db = await sqlite3.open(file); + let reader; + try { + await db.exec('PRAGMA journal_mode = WAL'); + await db.exec('CREATE TABLE t (a); INSERT INTO t VALUES (1)'); + + // Hold a reader snapshot open on a second connection, then + // commit another row on the writer: that frame sits beyond + // any checkpoint the reader prevents, and the snapshot below + // must still carry it. + reader = await sqlite3.open(file); + await reader.exec('BEGIN'); + await reader.get('SELECT count(*) AS c FROM t'); + await db.exec("INSERT INTO t VALUES ('held')"); + + const bytes = await db.serializeToBytes(); + // The image is rollback-journal format (header bytes 18/19), + // openable without a -wal file. + assert.strictEqual(bytes[18], 0x01); + assert.strictEqual(bytes[19], 0x01); + + const copy = await sqlite3.deserializeFromBytes(bytes, { + readOnly: true, + }); + assert.deepStrictEqual(await copy.all('SELECT a FROM t'), [ + { a: 1 }, + { a: 'held' }, + ]); + await copy.close(); + + // The live database is untouched: still WAL, still complete. + assert.strictEqual( + (await db.get('PRAGMA journal_mode')).journal_mode, + 'wal', + ); + assert.strictEqual( + (await db.get("SELECT count(*) AS c FROM t WHERE a = 'held'")) + .c, + 1, + ); + } finally { + if (reader) { + // The read transaction may already have ended if the body + // failed partway; either way the close below is what matters. + await reader.exec('ROLLBACK').catch(() => { + /* already rolled back */ + }); + await reader.close(); + } + await db.close(); + removeDb(file); + } + }); + + it('a non-WAL snapshot keeps its bytes exactly (no header rewrite)', async function () { + const db = await sqlite3.open(':memory:'); + await db.exec('CREATE TABLE t (v)'); + const bytes = await db.serializeToBytes(); + // A rollback-journal image already reads 0x01/0x01; nothing to + // change, and the magic is intact. + assert.strictEqual( + Buffer.from(bytes.buffer, bytes.byteOffset, 16).toString('latin1'), + 'SQLite format 3\u0000', + ); + assert.strictEqual(bytes[18], 0x01); + assert.strictEqual(bytes[19], 0x01); + await db.close(); + }); }); diff --git a/types/sqlite3.check.ts b/types/sqlite3.check.ts index 44738ba..4c18fb3 100644 --- a/types/sqlite3.check.ts +++ b/types/sqlite3.check.ts @@ -121,10 +121,17 @@ expectTypeOf( expectTypeOf(e).toEqualTypeOf(); }), ).toEqualTypeOf(); -expectTypeOf(db.prepare('SELECT 1')).toEqualTypeOf(); +// The no-callback form is the statement AND a promise of it: awaiting +// gates on the native prepare completing (9.0.2). +expectTypeOf(db.prepare('SELECT 1')).toEqualTypeOf< + Statement & Promise +>(); expectTypeOf( db.prepare('SELECT ?', 1, () => undefined), ).toEqualTypeOf(); +expectTypeOf(db.prepare('SELECT ?', 1)).toEqualTypeOf< + Statement & Promise +>(); // --- Database: promise mode with bound parameters (D03 follow-up) --------