diff --git a/.changeset/28977-sql-dry-run-refuses-date-bucket-over-non-date.md b/.changeset/28977-sql-dry-run-refuses-date-bucket-over-non-date.md index 9f61de6af0e..b55f8fb7647 100644 --- a/.changeset/28977-sql-dry-run-refuses-date-bucket-over-non-date.md +++ b/.changeset/28977-sql-dry-run-refuses-date-bucket-over-non-date.md @@ -12,7 +12,7 @@ Metadata-protocol: none **BREAKING** accept-set narrowing on the `/analytics/sql` dry run (`@objectstack/service-analytics`). `/analytics/query` is unchanged. - **Before.** Measured on real SQLite on `main` `ab085a04`. `/analytics/query` refused a month bucket over a text, select or boolean column, a year bucket over a number and a day bucket over a `time` column with the engine's `INVALID_FIELD` / 400 at `groupBy.N.dateGranularity`. The dry run rendered `strftime('%Y-%m', region)` and its kin, on the direct and the cross-object path, and for a cube dimension's one declared granularity asked raw. At `America/New_York` it answered `NOT_IMPLEMENTED` / 501. -- **Now.** The dry run judges the keys `execute()` hands the engine (its cross-object path's when the selection routes there) with the reading the draft preview already used (#28852), now `dataset-date-bucket-door.ts`. It runs after the dry run's `dateRange` read and its column-asked-twice refusal, as `query()` refuses, and gives the engine's refusal: the same message, `object`, `param`, `path`, `field` and `fields`. A field with no declared type, or a `formula`, is not judged, as in the engine. +- **Now.** The dry run judges the keys `execute()` hands the engine (its cross-object path's when the selection routes there) with the reading the draft preview already used (#28852), now `dataset-date-bucket-door.ts`. It runs after the dry run's `dateRange` read and its column-asked-twice refusal, as `query()` refuses, and gives the engine's refusal: the same message, `object`, `param`, `path`, `field` and `fields`. A field with no declared type is not judged, as in the engine; a `formula` is refused with the engine's virtual-field refusal on every face (#29167). - **Unchanged.** A date or datetime bucket renders byte for byte, the statement the query echoes. - **Who sees it.** A caller of the dry run for a selection the query refuses: the query's refusal, instead of SQL for a statement that never runs. diff --git a/.changeset/29167-formula-date-bucket-refused-alike.md b/.changeset/29167-formula-date-bucket-refused-alike.md new file mode 100644 index 00000000000..383786a7cb3 --- /dev/null +++ b/.changeset/29167-formula-date-bucket-refused-alike.md @@ -0,0 +1,20 @@ +--- +'@objectstack/objectql': patch +'@objectstack/service-analytics': patch +--- + +fix(objectql, service-analytics)!: a date bucket over a formula field is refused alike on every analytics face, instead of a driver error, one null group or a rendered statement + +Clause-②: no (narrowing) — `engine.aggregate` now refuses `INVALID_FIELD` / 400 at `groupBy.N.dateGranularity` a `dateGranularity` over a virtual (`formula`) field, whatever its `returnType`, before any driver is asked; `/analytics/sql` (`AnalyticsService.generateSql`) and the draft preview of `queryDataset` give the same refusal, field for field. An engine that aggregates in memory served one `null` group for it, the dry run rendered `strftime('%Y-%m', calc)`, and the preview served one `null` group. Nothing is widened: no export, public member, answer member, error code or status is added. +Metadata-protocol: none + + + +**BREAKING** accept-set narrowing on `engine.aggregate` (`@objectstack/objectql`) and on the `/analytics/sql` dry run and the draft preview (`@objectstack/service-analytics`), for a date bucket over a `formula` field only. + +- **Before.** Measured on `main` `7788eedb`, a month bucket over a `formula` with `returnType: 'date'`. On a SQL driver, `/analytics/query` (both compositions) answered the driver's missing-column backstop: `INVALID_FIELD` / 400 with no `path` and a "run schema sync" prescription that cannot help. On an engine whose driver has no native `aggregate` (the in-memory aggregation), the query was served: one `null` group holding every row. The dry run rendered `strftime('%Y-%m', calc)`, and the draft preview served one `null` group holding the total. +- **Now.** The engine's date-bucket door (`assertDateBucketTargetsAreTemporal`) judges a virtual field first, by the spec's `isVirtualSearchField` (`SEARCH_VIRTUAL_TYPES`), the predicate the sort, filter and search axes already refuse a formula by. It refuses with `INVALID_FIELD` / 400 at `groupBy.N.dateGranularity`, naming the field, its type and the granularity. The service's copy of that door (`dataset-date-bucket-door.ts`) refuses the same way, so the dry run and the preview give the query's refusal: the same message, `object`, `param`, `path`, `field` and `fields`. +- **Unchanged.** A date or datetime bucket is served on every face, byte for byte. A plain `groupBy` over a formula (no `dateGranularity`) is not this change's. +- **Who sees it.** A caller bucketing a formula field: a located refusal with a remedy, instead of a driver error, an empty-looking group or SQL for a statement that never runs. + +**The remedy.** Denormalise the value onto the object (a stored date or datetime field, written when the source changes) and bucket that. diff --git a/packages/objectql/src/aggregate-query-shape-door.ts b/packages/objectql/src/aggregate-query-shape-door.ts index d1955d43f51..84e537e90e9 100644 --- a/packages/objectql/src/aggregate-query-shape-door.ts +++ b/packages/objectql/src/aggregate-query-shape-door.ts @@ -46,7 +46,28 @@ * day by. A `date` (`CALENDAR_DATE_TYPES`) and a `datetime` (`INSTANT_TYPES`) * pass. **Not judged:** a field the declaration cannot type — undeclared (the * ingress door answers an unknown one first; a system column the map does not - * declare is bucketed as before), a `formula`, or a registry-less host. + * declare is bucketed as before), or a registry-less host. + * + * ## A virtual field (`formula`) is refused, on its storage ground [#29167] + * + * A `formula` names no stored value class (its value is the expression's), so + * the class reading above cannot judge it. It is refused on the ground the + * sort, filter, search and value-aggregate axes already refuse it on: it is + * VIRTUAL — computed on read, no column — judged by the spec's + * `isVirtualSearchField` (`SEARCH_VIRTUAL_TYPES`), never a local type list, + * and whatever its `returnType` says. Measured on `origin/main` `7788eedb` + * before this arm, a month bucket over a `formula` with `returnType: 'date'`: + * + * | caller | answer | + * |:--|:--| + * | a SQL driver | `INVALID_FIELD` / 400 from the driver's missing-column backstop: no `path`, and a "run schema sync" prescription that cannot help | + * | a driver with no native `aggregate` (the in-memory aggregation) | `200`, one `null` group holding every row | + * + * Both now answer {@link virtualDateBucketMessage}, `INVALID_FIELD` / 400 at + * `groupBy.N.dateGranularity`, with the remedy those axes give: store the + * value on a stored field and use that. `@objectstack/service-analytics`'s + * `dataset-date-bucket-door.ts` gives the same refusal on the two faces that + * reach no engine (the `/analytics/sql` dry run and the draft preview). * * ## The column names * @@ -65,6 +86,7 @@ */ import { StandardErrorCode } from '@objectstack/spec/api'; +import { isVirtualSearchField } from '@objectstack/spec/data'; import { declaredFieldClasses } from './having-filter.js'; type ShapeRefusal = Error & { @@ -87,10 +109,29 @@ function refusal( return err; } +/** + * [#29167] The refusal of a date bucket over a VIRTUAL field (see the module + * header): the position, the field, its type and the granularity it was + * bucketed by, then the store-the-value remedy the sort and filter axes give. + * `@objectstack/service-analytics`'s `dataset-date-bucket-door.ts` carries the + * one copy, compared field for field by its tests. + */ +function virtualDateBucketMessage( + object: string, path: string, field: string, type: string, dateGranularity: string, +): string { + return `aggregate('${object}'): ${path} buckets '${field}' by '${dateGranularity}', but '${field}' ` + + `is a ${type} field — its value is computed on read and never stored, so no driver materialises a ` + + `column to bucket. The query was NOT run. Denormalise the value onto '${object}' (a stored date or ` + + 'datetime field, written when the source changes) and bucket that. A formula field is virtual: with ' + + 'no column behind it the bucket reached the driver as a column that does not exist, and an engine ' + + 'that aggregates in memory put every row into one null group while the answer looked served.'; +} + /** * Refuse a `groupBy` entry whose `dateGranularity` buckets a field the object - * declares neither a date nor a datetime — `INVALID_FIELD` / 400, naming the - * position, the field and its declared type. See the module header. + * declares neither a date nor a datetime, or a virtual (`formula`) field — + * `INVALID_FIELD` / 400, naming the position, the field and its declared type. + * See the module header. */ export function assertDateBucketTargetsAreTemporal( object: string, @@ -105,11 +146,23 @@ export function assertDateBucketTargetsAreTemporal( if (entry === null || typeof entry !== 'object' || Array.isArray(entry)) continue; const { field, dateGranularity } = entry as { field?: unknown; dateGranularity?: unknown }; if (dateGranularity == null || typeof field !== 'string') continue; + const path = `groupBy.${i}.dateGranularity`; + const declared = Object.prototype.hasOwnProperty.call(fields, field) + ? (fields as Record)[field] + : undefined; + // [#29167] A virtual field first: the class reading below has no class + // for it, so it would otherwise be skipped unjudged. + if (isVirtualSearchField(declared as never)) { + throw refusal( + 'INVALID_FIELD', + virtualDateBucketMessage(object, path, field, String(declared?.type), String(dateGranularity)), + { field, fields: [field], object, param: 'groupBy', path }, + ); + } classes ??= declaredFieldClasses(fields); const cls = classes.get(field); if (cls === undefined || cls === 'date' || cls === 'datetime') continue; const type = String((fields as Record)[field]?.type); - const path = `groupBy.${i}.dateGranularity`; throw refusal( 'INVALID_FIELD', `aggregate('${object}'): ${path} buckets '${field}' by '${String(dateGranularity)}', but '${field}' ` diff --git a/packages/objectql/src/engine-aggregate-query-shape-door.test.ts b/packages/objectql/src/engine-aggregate-query-shape-door.test.ts index 399609a3c19..7633565a6e4 100644 --- a/packages/objectql/src/engine-aggregate-query-shape-door.test.ts +++ b/packages/objectql/src/engine-aggregate-query-shape-door.test.ts @@ -39,6 +39,8 @@ const PROBE = { opened_on: { name: 'opened_on', type: 'date' }, seen_at: { name: 'seen_at', type: 'datetime' }, doubled: { name: 'doubled', type: 'formula', expression: 'amount * 2' }, + // [#29167] A formula whose declared result IS a date: refused all the same. + opened_calc: { name: 'opened_calc', type: 'formula', expression: 'opened_on', returnType: 'date' }, }, }; @@ -106,6 +108,26 @@ describe('[#25818] the aggregate query shapes that answered a plausible wrong ta expect(reads, 'every refusal precedes the driver').toHaveLength(0); }); + it('[#29167] refuses a dateGranularity over a virtual (formula) field, whatever its returnType: INVALID_FIELD / 400 at groupBy.N.dateGranularity — no read', async () => { + // Before this arm the in-memory aggregation served one null group here: + // the formula has no stored value for it to bucket. + for (const field of ['doubled', 'opened_calc']) { + const err = await refusalOf(aggregate({ + groupBy: ['status', { field, dateGranularity: 'month' }], + aggregations: [COUNT], + })); + expect(envelopeOf(err), field).toEqual({ code: 'INVALID_FIELD', status: 400, httpStatus: 400 }); + expect({ path: err?.path, param: err?.param, field: err?.field, fields: err?.fields, object: err?.object }, field) + .toEqual({ path: 'groupBy.1.dateGranularity', param: 'groupBy', field, fields: [field], object: OBJECT }); + expect(err!.message, field).toContain( + `groupBy.1.dateGranularity buckets '${field}' by 'month', but '${field}' is a formula field — its value is computed on read and never stored`, + ); + // The store-the-value remedy the sort and filter axes give a virtual field. + expect(err!.message, field).toContain(`Denormalise the value onto '${OBJECT}' (a stored date or datetime field, written when the source changes) and bucket that.`); + } + expect(reads, 'every refusal precedes the driver').toHaveLength(0); + }); + it('refuses a repeated aggregation alias: INVALID_QUERY / 400 at the second aggregations.N.alias — no read', async () => { const err = await refusalOf(aggregate({ aggregations: [COUNT, { function: 'sum', field: 'amount', alias: 'n' }], @@ -144,13 +166,13 @@ describe('[#25818] the aggregate query shapes that answered a plausible wrong ta expect(reads).toHaveLength(0); }); - it('CONTROL a date or datetime bucket, a formula or undeclared bucket, and distinct columns are served', async () => { + it('CONTROL a date or datetime bucket, an undeclared bucket, and distinct columns are served', async () => { await expect(aggregate({ groupBy: [{ field: 'opened_on', dateGranularity: 'month' }], aggregations: [COUNT] })) .resolves.toEqual([{ opened_on: '2026-01', n: 1 }, { opened_on: '2026-02', n: 2 }]); await expect(aggregate({ groupBy: [{ field: 'seen_at', dateGranularity: 'month' }], aggregations: [COUNT] })) .resolves.toEqual([{ seen_at: '2026-01', n: 1 }, { seen_at: '2026-02', n: 2 }]); - // The declaration cannot type a formula or an undeclared column: no verdict. - await expect(aggregate({ groupBy: [{ field: 'doubled', dateGranularity: 'month' }], aggregations: [COUNT] })).resolves.toBeDefined(); + // The declaration cannot type an undeclared column: no verdict. [#29167] A + // formula under a dateGranularity is refused (above), no longer a control. await expect(aggregate({ groupBy: [{ field: 'created_at', dateGranularity: 'month' }], aggregations: [COUNT] })).resolves.toBeDefined(); // An alias moves the group column, so the field's own name is free. await expect(aggregate({ groupBy: [{ field: 'status', alias: 's' }], aggregations: [{ function: 'count', alias: 'status' }] })) @@ -160,7 +182,7 @@ describe('[#25818] the aggregate query shapes that answered a plausible wrong ta groupBy: [{ field: 'opened_on', dateGranularity: 'year', alias: 'y' }, { field: 'opened_on', dateGranularity: 'month', alias: 'm' }], aggregations: [COUNT, { function: 'sum', field: 'amount', alias: 'total' }], })).resolves.toEqual([{ y: '2026', m: '2026-01', n: 1, total: 10 }, { y: '2026', m: '2026-02', n: 2, total: 50 }]); - expect(reads, "each served query read once").toHaveLength(6); + expect(reads, "each served query read once").toHaveLength(5); }); it('the REST door answers the engine\'s refusal, not a 200 — through findData and the wire mapping', async () => { diff --git a/packages/services/service-analytics/src/__tests__/objectql-echo-non-date-bucket.test.ts b/packages/services/service-analytics/src/__tests__/objectql-echo-non-date-bucket.test.ts index 4750711b64d..665e63aa880 100644 --- a/packages/services/service-analytics/src/__tests__/objectql-echo-non-date-bucket.test.ts +++ b/packages/services/service-analytics/src/__tests__/objectql-echo-non-date-bucket.test.ts @@ -22,6 +22,11 @@ * | a cube dimension over `region` declaring one granularity, asked raw | the bucket | * | text `region` by month at `America/New_York` | `NOT_IMPLEMENTED` / 501 saying the query is served | * + * [#29167] …and a bucket over a `formula` (virtual: no stored column), which + * the dry run rendered as `strftime('%Y-%m', calc)` while the query refused it + * from the SQL driver's missing-column backstop: both now answer the engine's + * virtual-field refusal, field for field. + * * It now judges the keys `execute()` hands the engine with the service's one * reading of it (`dataset-date-bucket-door.ts`, #28852), after the `dateRange` * read and the column-asked-twice door as `query()` refuses, so it answers the @@ -99,6 +104,8 @@ describe('[#28977] the /analytics/sql dry run refuses a date bucket over a colum closed_at: { name: 'closed_at', type: 'datetime' }, closed_on: { name: 'closed_on', type: 'date' }, account: { name: 'account', type: 'lookup', reference: ACCOUNT }, + // [#29167] A formula the table stores no column for, whose result is a date. + calc: { name: 'calc', type: 'formula', expression: 'closed_on', returnType: 'date' }, }, } as never); await engine.syncSchemas(); @@ -155,6 +162,25 @@ describe('[#28977] the /analytics/sql dry run refuses a date bucket over a colum expect(asSql, 'the dry run vs the query').toEqual(asQuery); }); + it.each<[string, Query, string]>([ + ['by month', onObject({ timeDimensions: [td('calc', 'month')] }), 'groupBy.0.dateGranularity'], + ['after a datetime bucket', onObject({ timeDimensions: [td('closed_at', 'year'), td('calc', 'month')] }), 'groupBy.1.dateGranularity'], + ['at a non-UTC zone', onObject({ timeDimensions: [td('calc', 'month')], timezone: 'America/New_York' }), 'groupBy.0.dateGranularity'], + ])('[#29167] refuses a bucket over a formula (virtual: no stored column) %s with the query\'s refusal, where the dry run rendered strftime', async (_label, query, path) => { + const { asQuery, asSql } = await bothRefusals(query); + expect(asQuery).toEqual({ + message: + `aggregate('${DEAL}'): ${path} buckets 'calc' by 'month', but 'calc' is a formula field — its value is computed ` + + 'on read and never stored, so no driver materialises a column to bucket. The query was NOT run. Denormalise ' + + `the value onto '${DEAL}' (a stored date or datetime field, written when the source changes) and bucket that. ` + + 'A formula field is virtual: with no column behind it the bucket reached the driver as a column that does not ' + + 'exist, and an engine that aggregates in memory put every row into one null group while the answer looked served.', + code: 'INVALID_FIELD', status: 400, httpStatus: 400, + object: DEAL, param: 'groupBy', path, field: 'calc', fields: ['calc'], + }); + expect(asSql, 'the dry run vs the query').toEqual(asQuery); + }); + it.each<[string, Query, string]>([ ["beside a cross-object 'account.name' (its FK column first)", onObject({ dimensions: ['account.name'], timeDimensions: [td('region', 'month')] }), 'groupBy.1.dateGranularity'], ['after a raw date column and a datetime bucket', onObject({ dimensions: ['closed_on'], timeDimensions: [td('closed_at', 'month'), td('region', 'month')] }), 'groupBy.2.dateGranularity'], diff --git a/packages/services/service-analytics/src/__tests__/preview-date-bucket-type.test.ts b/packages/services/service-analytics/src/__tests__/preview-date-bucket-type.test.ts index e31bb73d33b..2c85fd83465 100644 --- a/packages/services/service-analytics/src/__tests__/preview-date-bucket-type.test.ts +++ b/packages/services/service-analytics/src/__tests__/preview-date-bucket-type.test.ts @@ -23,9 +23,11 @@ * (`preview-date-bucket-door.ts`) and gives the same refusal, field for field. * A date and a datetime bucket are served alike on every face, as before. * - * [#28977] Pinned too: a `time` column, a dataset dimension's declared - * `dateGranularity` over a text column, and the `formula` skip (the engine - * does not judge a formula, so neither does the door). And the preview now + * [#28977] Pinned too: a `time` column and a dataset dimension's declared + * `dateGranularity` over a text column. [#29167] A `formula` (virtual, no + * stored column) is refused on every face with the engine's virtual-field + * refusal; #28977 pinned it as skipped, and the preview served one `null` + * group for it. And the preview now * lowers its row filters before the column-asked-twice and date-bucket doors, * where the live strategy lowers them, so an unrecognised `dateRange` beside * either fault is refused about the `dateRange` on every face; the preview @@ -61,6 +63,7 @@ const DATASET = DatasetSchema.parse({ { name: 'opened_time', field: 'opened_time', type: 'date' }, { name: 'calc', field: 'calc', type: 'date' }, { name: 'region_month', field: 'region', type: 'date', dateGranularity: 'month' }, + { name: 'calc_month', field: 'calc', type: 'date', dateGranularity: 'month' }, ], measures: [{ name: 'total', aggregate: 'sum', field: 'amount' }], }); @@ -188,16 +191,29 @@ describe('[#28852] a date bucket over a column that is not a date, on every face expect(refusal).toMatchObject({ path: 'groupBy.0.dateGranularity', field: 'region', fields: ['region'] }); }); - it('[#28977] does not judge a bucket over a formula, as the engine does not', async () => { - // The engine skips a `formula` (its value class is the expression's), so - // both live faces answer the next refusal, the driver's for a column the - // table does not store. No face answers the date-bucket refusal. (The - // preview serves the selection over its drafted rows, a separate gap.) - for (const face of FACES) { - const outcome = await ask(face, { measures: ['total'], timeDimensions: [{ dimension: 'calc', granularity: 'month' }] }) - .then(() => 'served', (e: Error & { path?: string }) => e.path ?? e.message); - expect(outcome, face).not.toBe('groupBy.0.dateGranularity'); - } + it('[#29167] refuses a bucket over a formula (virtual: no stored column) with the engine\'s refusal, where the preview served one null group', async () => { + // [#28977] pinned the skip this replaces: the engine judged no formula, so + // the live faces answered the SQL driver's missing-column backstop (no + // path, a "run schema sync" prescription) and the preview served one + // `null` group holding the total. The engine's door now refuses a virtual + // field, and the service's copy gives the same refusal on the preview. + const refusal = await refusedAlike({ measures: ['total'], timeDimensions: [{ dimension: 'calc', granularity: 'month' }] }); + expect(refusal).toEqual({ + message: + `aggregate('${OBJECT}'): groupBy.0.dateGranularity buckets 'calc' by 'month', but 'calc' is a formula field — ` + + 'its value is computed on read and never stored, so no driver materialises a column to bucket. The query was ' + + `NOT run. Denormalise the value onto '${OBJECT}' (a stored date or datetime field, written when the source ` + + 'changes) and bucket that. A formula field is virtual: with no column behind it the bucket reached the driver ' + + 'as a column that does not exist, and an engine that aggregates in memory put every row into one null group ' + + 'while the answer looked served.', + code: 'INVALID_FIELD', status: 400, httpStatus: 400, + object: OBJECT, param: 'groupBy', path: 'groupBy.0.dateGranularity', field: 'calc', fields: ['calc'], + }); + }); + + it("[#29167] refuses a dataset dimension's declared dateGranularity over a formula", async () => { + const refusal = await refusedAlike({ dimensions: ['calc_month'], measures: ['total'] }); + expect(refusal).toMatchObject({ path: 'groupBy.0.dateGranularity', field: 'calc', fields: ['calc'] }); }); /** Every face refuses `selection` with one refusal of `code`, message included. */ diff --git a/packages/services/service-analytics/src/dataset-date-bucket-door.ts b/packages/services/service-analytics/src/dataset-date-bucket-door.ts index 9f2c607cee6..8b623cdaaf1 100644 --- a/packages/services/service-analytics/src/dataset-date-bucket-door.ts +++ b/packages/services/service-analytics/src/dataset-date-bucket-door.ts @@ -25,7 +25,17 @@ * * The reading is the engine's: the field's declared type, classed by the * spec's value-class sets the engine's `classOfDeclaredType` reads. A field - * with no declared type, or a `formula`, is not judged, as there. + * with no declared type is not judged, as there. + * + * [#29167] A VIRTUAL field (`formula`, the spec's `isVirtualSearchField` / + * `SEARCH_VIRTUAL_TYPES`) is refused first, as the engine's door now refuses + * it: computed on read, no column to bucket, whatever its `returnType` says. + * Measured on `origin/main` `7788eedb` before this arm, a month bucket over a + * `formula` with `returnType: 'date'`: the dry run rendered + * `strftime('%Y-%m', calc)` and the preview served one `null` group holding + * the total, while the live faces refused it from the SQL driver's + * missing-column backstop. All three now answer the engine's + * `INVALID_FIELD` / 400 at `groupBy.N.dateGranularity`, field for field. * * Why a copy of the sentence: `@objectstack/objectql` is a dev dependency of * this package, never a runtime one, and its root does not export the door. @@ -36,7 +46,7 @@ */ import { StandardErrorCode } from '@objectstack/spec/api'; -import { CALENDAR_DATE_TYPES, INSTANT_TYPES } from '@objectstack/spec/data'; +import { CALENDAR_DATE_TYPES, INSTANT_TYPES, isVirtualSearchField } from '@objectstack/spec/data'; import type { AggregateGroupKey } from './strategies/objectql-strategy.js'; type DateBucketRefusal = Error & { @@ -44,10 +54,20 @@ type DateBucketRefusal = Error & { field?: string; fields?: string[]; object?: string; param?: string; path?: string; }; +/** The refusal `assertDatasetDateBucketsTemporal` throws, with the engine's keys. */ +function dateBucketRefusal(message: string, object: string, field: string, path: string): DateBucketRefusal { + const err = new Error(message) as DateBucketRefusal; + err.code = StandardErrorCode.enum.INVALID_FIELD; + err.status = 400; + err.httpStatus = 400; + Object.assign(err, { field, fields: [field], object, param: 'groupBy', path }); + return err; +} + /** * Refuse the first bucketed key in `groupBy` (built for a query on `object`) - * whose field `declaredType` reads as declared, not a `formula`, and neither a - * date nor a datetime, with the engine's refusal for it. + * whose field `declaredType` reads as a virtual field, or as declared and + * neither a date nor a datetime, with the engine's refusal for it. */ export function assertDatasetDateBucketsTemporal( object: string, @@ -58,19 +78,28 @@ export function assertDatasetDateBucketsTemporal( if (typeof key === 'string') continue; const { field, dateGranularity } = key; const type = declaredType(field); - if (type === undefined || type === 'formula' || CALENDAR_DATE_TYPES.has(type) || INSTANT_TYPES.has(type)) continue; const path = `groupBy.${i}.dateGranularity`; - const err = new Error( + // [#29167] The engine's virtual-field arm (`aggregate-query-shape-door.ts` + // `virtualDateBucketMessage`), judged first as there. + if (type !== undefined && isVirtualSearchField({ type } as never)) { + throw dateBucketRefusal( + `aggregate('${object}'): ${path} buckets '${field}' by '${dateGranularity}', but '${field}' ` + + `is a ${type} field — its value is computed on read and never stored, so no driver materialises a ` + + `column to bucket. The query was NOT run. Denormalise the value onto '${object}' (a stored date or ` + + 'datetime field, written when the source changes) and bucket that. A formula field is virtual: with ' + + 'no column behind it the bucket reached the driver as a column that does not exist, and an engine ' + + 'that aggregates in memory put every row into one null group while the answer looked served.', + object, field, path, + ); + } + if (type === undefined || CALENDAR_DATE_TYPES.has(type) || INSTANT_TYPES.has(type)) continue; + throw dateBucketRefusal( `aggregate('${object}'): ${path} buckets '${field}' by '${dateGranularity}', but '${field}' ` + `is a declared ${type} field — only a date or datetime field can be date-bucketed. The query was ` + `NOT run. Group by '${field}' without 'dateGranularity', or bucket a date or datetime field. A ` + 'value that is not a date has no calendar position: every row fell into one null bucket (or, ' + 'for a number, an epoch bucket) while the answer looked served.', - ) as DateBucketRefusal; - err.code = StandardErrorCode.enum.INVALID_FIELD; - err.status = 400; - err.httpStatus = 400; - Object.assign(err, { field, fields: [field], object, param: 'groupBy', path }); - throw err; + object, field, path, + ); } }