feat(objectql,platform-objects): keep a by-id delete's rows in the recycle bin (ADR-0148)

.changeset/30237-recycle-bin-snapshot-on-delete.md

@@ -0,0 +1,29 @@
+---
+'@objectstack/objectql': minor
+'@objectstack/platform-objects': minor
+'@objectstack/spec': minor
+'@objectstack/cli': minor
+'@objectstack/plugin-audit': minor
+---
+
+feat(objectql,platform-objects): a by-id delete keeps its rows in the recycle bin (ADR-0148): the `sys_recycle_bin` object, the snapshot on delete, the retention stamp and the TTL reap
+
+Clause-②: yes (widening)
+Metadata-protocol: added
+Skills-docs: follow-up owed — skills/objectstack-api/SKILL.md :268, :417 (they teach that there is no recycle bin)
+
+The recycle bin's write side. A delete is still a physical delete, and no read path changed: the rows are gone from every query the moment the delete commits. What is new is the copy kept beside them. Restoring, purging and emptying the bin, the two grants that govern them and their REST doors are not in this release.
+
+- **`@objectstack/platform-objects`:** registers `sys_recycle_bin`, built from the spec's `RECYCLE_BIN_ENTRY_FIELDS`. One entry per delete operation. It is engine-owned, with no generic data-API method, because the snapshot holds every field of the deleted rows. Its tenant column cascades, so deleting an organization deletes its entries. The lifecycle sweep reaps an entry once its stamped `purge_after` has passed (`ttl` on `purge_after`, window `0h`), and asks no resolver.
+- **`@objectstack/objectql`:** a by-id delete of a business record writes one entry in the delete's own unit of work, before the record's row is removed. The entry holds the record and its master-detail cascade, the master first. Each row keeps its declared columns, and `secret` columns are left out, so a restored record's secret fields come back empty. The entry also records the `set_null` references the delete cleared, and `deleted_by`, `deleted_at`, `retention_days` and `purge_after`. If removing the row fails, the entry goes too: the transaction rolls it back, or the engine removes it when no transaction covered it.
+- **Retention:** `ObjectQLPlugin`'s new `recycleBinRetention: { resolve, token }` option takes the host's retention resolver, called once per delete with the token the host handed over. When the option is absent or the resolver answers `undefined`, the retention is `OS_RECYCLE_BIN_RETENTION_DAYS`, else 30 days. A value that is not a positive whole number of days is refused: from the resolver, the delete is refused before it writes anything; in the variable, the boot is refused, because the plugin reads it in `init()`.
+- **Which deletes do NOT enter the bin:**
+  - a delete by predicate (`multi: true`), which includes every lifecycle sweep reap, even though the sweep names each reaped row by its id;
+  - a child the cascade deletes, which is a row of its root's entry instead;
+  - any platform object, meaning a `sys_*`, `cloud_*` or `ai_*` object, whose rows are also left out of a business record's snapshot;
+  - any delete on a composition that does not register `PlatformObjectsPlugin`, which logs one `info` line saying so.
+- **Forward-only:** records deleted before this release have no entry.
+- **Storage:** every by-id delete of a business record now also writes one `sys_recycle_bin` row holding the deleted rows. The row is reaped once its retention has passed.
+- **`@objectstack/spec`:** `PLATFORM_OBJECTS_BY_PACKAGE['platform-objects']` lists `sys_recycle_bin`, so a reference to it lints as a real platform object. The comment above `RECYCLE_BIN_OBJECT` now names the object file that builds the table; no exported name, shape or description changed.
+- **`@objectstack/cli`:** the organization-ownership inventory has a row for `sys_recycle_bin` (fate `attribution`).
+- **`@objectstack/plugin-audit`:** `sys_recycle_bin` is excluded from the generic audit and activity writer, like the other engine-owned plumbing, so reaping an entry, clearing its `deleted_by` or cascading an organization's delete into its entries never copies a snapshot into `sys_audit_log` or `sys_activity`. The business delete an entry was taken for is audited exactly as before.

packages/cli/src/utils/organization-ownership-inventory.ts

@@ -811,6 +811,13 @@ export const ORGANIZATION_OWNERSHIP_INVENTORY: readonly InventoryEntry[] = [
   // Exchange rates (`@objectstack/platform-objects`, #27185): an organization's
   // administrators author its own rate table; no parent row anchors a rate.
   { object: 'sys_exchange_rate', census: PMAIN, fate: 'attribution', citation: `${CENSUS_MAIN} (in reach, added since the first census); ${D10_SINGLE}`, anchors: [] },
+  // The recycle bin (`@objectstack/platform-objects`, ADR-0148, #30237): the
+  // engine writes each entry with the organization of the record it holds,
+  // else the deleting context's. A NULL row comes only from a delete with
+  // neither (a tenancy-disabled object deleted by a context with no active
+  // organization). The record an entry names is gone, so no parent row
+  // anchors an entry.
+  { object: 'sys_recycle_bin', census: PMAIN, fate: 'attribution', citation: `${CENSUS_MAIN} (in reach, added since the first census); ${D10_SINGLE}`, anchors: [] },
   // A reader's own named list views (`@objectstack/platform-objects`, ruling
   // 6570568131 on #28971): user data on the preference template, each row in
   // its owner's organization; no parent row anchors a view.

packages/objectql/src/engine-cascade-planning.ts

@@ -511,8 +511,14 @@ export class CascadePlanningFacet {
    * A nested `delete()` the walk makes for a member joins the ambient set
    * ({@link cascadeDeleteSets}); any other delete reaching here — a hook's,
    * say — is the root of its own cascade, as before.
+   *
+   * [#30237, ADR-0148 D1] `onCollected` is the recycle bin's moment: called
+   * once with the freshly collected set, after the #26718 restricts are judged
+   * and before the walk deletes or clears anything, so the snapshot reads every
+   * row as it stood. Never called for a member that joins a cascade already in
+   * progress: such a row belongs to its root's entry.
    */
-  static async cascadeDeleteRelations(this: Pick<CascadePlanningHost, '_registry' | 'cascadeDeleteSets' | 'collectCascadeDeleteSet' | 'refuseRestrictsInsideOwnCascade' | 'walkReferencingRelations'>, object: string, ids: ReadonlyArray<string | number>, context?: ExecutionContext, depth = 0): Promise<void> {
+  static async cascadeDeleteRelations(this: Pick<CascadePlanningHost, '_registry' | 'cascadeDeleteSets' | 'collectCascadeDeleteSet' | 'refuseRestrictsInsideOwnCascade' | 'walkReferencingRelations'>, object: string, ids: ReadonlyArray<string | number>, context?: ExecutionContext, depth = 0, onCollected?: (set: CascadeDeleteSet) => Promise<void>): Promise<void> {
     const roots: Array<string | number> = [];
     const seen = new Set<string>();
     for (const id of ids) {
@@ -555,6 +561,7 @@ export class CascadePlanningFacet {
     // [#26718] Every restrict a member's own cascade would override is judged
     // here, before the first write.
     await this.refuseRestrictsInsideOwnCascade(cascadeSet, fresh.map((id) => ({ object, id })), context, objects);
+    if (onCollected) await onCollected(cascadeSet);
     await this.cascadeDeleteSets.run(cascadeSet, () =>
       this.walkReferencingRelations(object, fresh, context, objects, cascadeSet),
     );
@@ -1163,6 +1170,13 @@ export class CascadePlanningFacet {
               } else {
                 await this.update(childName, { id: depId, [fieldName]: null }, { context: referentialCtx } as any);
               }
+              // [#30237, ADR-0148 D2] The reference a restore re-links while
+              // the field still lacks it: the live row, the field, and the id
+              // this delete removed from it. Only a delete that enters the
+              // recycle bin carries the list.
+              cascadeSet.clearedReferences?.push({
+                object: childName, id: String(depId), field: fieldName, value: String(id),
+              });
             }
           }
           // [#27004] Every row outside the set has had `id` cleared from it.

packages/objectql/src/engine-top-level.ts

@@ -12,7 +12,7 @@
 // importer of `./engine.js` reads what it read.
 
 import { AUDIT_PROVENANCE_FIELDS, ENGINE_UPDATE_UPSERT_REMOVED, FILE_REFERENCE_TYPES, FullTextSearchSchema, QUERY_CURSOR_REMOVED, QUERY_DISTINCT_REMOVED, REFERENCE_VALUE_TYPES, RPC_QUERY_ALIAS_SLOTS, VALID_AST_OPERATORS, foldQueryAliasSlots, isFilterAST, isMultiValueField, lowerFilterCondition, normalizeFilterComparandTypes, parseFilterAST, readAutonumberCounter, referenceCarrierOf, referenceTargetOf, resolveInjectedColumnProvenance } from '@objectstack/spec/data';
-import type { DroppedFieldsEvent, FilterCondition, FilterLoweringOptions, HookContext, QueryAST, QueryAliasSlot, ServiceObject } from '@objectstack/spec/data';
+import type { DroppedFieldsEvent, FilterCondition, FilterLoweringOptions, HookContext, QueryAST, QueryAliasSlot, RecycleBinClearedReference, ServiceObject } from '@objectstack/spec/data';
 import type { EngineFilterJudgement, ISecurityService } from '@objectstack/spec/contracts';
 import { assertFilterIsMaterializable, assertListComparandShapes, collectComposableReadableIds, invalidFilterError } from './filter-comparand-shape.js';
 import { assertResolvedTemporalTokensInRange, assertTemporalComparandsInterpretable } from './temporal-comparand-door.js';
@@ -4372,6 +4372,16 @@ export interface CascadeDeleteSet {
    * one has removed from it; the remainder written leaves those out too.
    */
   readonly cleared: Map<string, Set<string>>;
+  /**
+   * [#30237, ADR-0148 D2] Every `set_null` reference the walk cleared, one per
+   * live row, field and removed id: the referencing row, the field, and the id
+   * the field held (a row this delete removes). Present only on the set of a
+   * delete that enters the recycle bin, which assigns it when the set is
+   * collected; the walk appends to it, nested deletes of the same cascade
+   * included, and the entry keeps it so a restore can re-link what is still
+   * empty.
+   */
+  clearedReferences?: RecycleBinClearedReference[];
 }
 
 /** [#26718] One member of a {@link CascadeDeleteSet}, as a single key. */

packages/objectql/src/engine.ts

@@ -81,6 +81,10 @@ import {
   type SecretHandleSlot,
   type StoredRowsReader,
 } from './engine-secret-internal-fields.js';
+// [#30237, ADR-0148] The recycle bin's write side: the capture body, and the
+// retention seam the host hands over (`RecycleBinCaptureHost`).
+import { openRecycleBinCapture, type RecycleBinCapture } from './recycle-bin/capture.js';
+import type { RecycleBinRetentionSettings } from './recycle-bin/retention.js';
 // [#25522 cut 6] The predicate-write hook dispatch bodies; the methods of the
 // same names below hand each body the members it reads (`HookDispatchHost`).
 import {
@@ -9019,9 +9023,55 @@ export class ObjectQL implements IObjectQLEngine {
     ids: ReadonlyArray<string | number>,
     context?: ExecutionContext,
     depth = 0,
+    onCollected?: (set: CascadeDeleteSet) => Promise<void>,
   ): Promise<void> {
     // [#29986 codemod] The body moved to `engine-cascade-planning.ts`, its docblock with it (see `CascadePlanningHost`).
-    return CascadePlanningFacet.cascadeDeleteRelations.call(cascadePlanningHost.of(this), object, ids, context, depth);
+    return CascadePlanningFacet.cascadeDeleteRelations.call(cascadePlanningHost.of(this), object, ids, context, depth, onCollected);
+  }
+
+  /**
+   * [#30237, ADR-0148 D3] What the host handed over for the recycle bin's
+   * retention: its resolver and the token to call it with. `undefined` until
+   * {@link setRecycleBinRetention}: `OS_RECYCLE_BIN_RETENTION_DAYS`, else 30.
+   */
+  private recycleBinRetention: RecycleBinRetentionSettings | undefined;
+
+  /** [#30237] The recycle bin's once-per-engine notices already given. */
+  private readonly recycleBinNotices = new Set<string>();
+
+  /**
+   * [#30237, ADR-0148 D3] Install the host's retention resolver and the opaque
+   * token it is called with (the one its `resolveFeatures` receives). The
+   * engine asks once per delete operation that enters the bin and stamps the
+   * answer on the entry; `ObjectQLPlugin` calls this from its
+   * `recycleBinRetention` option.
+   */
+  setRecycleBinRetention(settings: RecycleBinRetentionSettings | undefined): void {
+    this.recycleBinRetention = settings;
+  }
+
+  private openRecycleBinCapture(
+    object: string,
+    id: string | number,
+    context: ExecutionContext | undefined,
+    predicate: boolean,
+  ): RecycleBinCapture | undefined {
+    // [#30237] The body lives in `recycle-bin/capture.ts`, its docblock with it (see `RecycleBinCaptureHost`).
+    const readRegistry = () => this._registry;
+    const readLogger = () => this.logger;
+    const readCascadeDeleteSets = () => this.cascadeDeleteSets;
+    const readRetention = () => this.recycleBinRetention;
+    const readNotices = () => this.recycleBinNotices;
+    const host: ThisParameterType<typeof openRecycleBinCapture> = {
+      get _registry() { return readRegistry(); },
+      get logger() { return readLogger(); },
+      get cascadeDeleteSets() { return readCascadeDeleteSets(); },
+      get recycleBinRetention() { return readRetention(); },
+      get recycleBinNotices() { return readNotices(); },
+      getDriver: (...a) => this.getDriver(...a),
+      buildDriverOptions: (...a) => this.buildDriverOptions(...a),
+    };
+    return openRecycleBinCapture.call(host, object, id, context, predicate);
   }
 
   /**
@@ -9689,14 +9739,30 @@ export class ObjectQL implements IObjectQLEngine {
               // A dependent's stale-roll-up refusal stops the walk before this
               // row is removed, so it is not this delete's answer:
               // `cascadeStoppedRefusal` says what it is instead.
+              //
+              // [#30237, ADR-0148 D1] The recycle bin rides the same unit: the
+              // capture opens before the cascade (a retention defect refuses
+              // here, before any write), reads the collected set before the
+              // walk touches it, and writes the one entry after the walk and
+              // before this row goes. A removal that throws takes the entry
+              // with it: the transaction's rollback when one covers the bin,
+              // the capture's compensation otherwise. A caller's `multi: true`
+              // opens none, even here: the lifecycle sweep reaps by id with it.
               const runByIdDelete = async (writeContext?: ExecutionContext) => {
-                await this.cascadeDeleteRelations(object, [hookContext.input.id as string | number], writeContext)
+                const bin = this.openRecycleBinCapture(object, hookContext.input.id as string | number, writeContext, options?.multi === true);
+                await this.cascadeDeleteRelations(object, [hookContext.input.id as string | number], writeContext, 0, bin?.onCollected)
                   .catch((e: unknown) => { throw cascadeStoppedRefusal(e, object); });
-                return await driver.delete(
-                  object,
-                  hookContext.input.id as string,
-                  this.buildDriverOptions(object, writeContext, hookContext.input.options as any),
-                );
+                const entry = await bin?.write();
+                try {
+                  return await driver.delete(
+                    object,
+                    hookContext.input.id as string,
+                    this.buildDriverOptions(object, writeContext, hookContext.input.options as any),
+                  );
+                } catch (e) {
+                  await entry?.compensate();
+                  throw e;
+                }
               };
               // WHEN to open one is `planCascadeAtomicity`'s call — see there.
               // `'none'` (nothing references this object) keeps the pre-#7413

packages/objectql/src/federated-injected-column-readers.test.ts

@@ -309,6 +309,11 @@ const READERS: Record<string, Row> = {
     disposition: 'row-value',
     why: 'reads the tenant column off the written row; a federated row has none, so the event omits it',
   },
+  // [#30237] A recycle-bin entry belongs to the organization of the record it holds.
+  'recycle-bin/capture.ts#organizationOf :: resolveTenantFieldName()': {
+    disposition: 'row-value',
+    why: "reads the tenant column off the snapshotted root row; a federated row has none, so the entry takes the deleting context's organization",
+  },
   'relation-filter-lowering.ts#admitRelationCondition :: referenceTargetOf()': {
     disposition: 'caller-predicate',
     why: "lowers a relation condition the caller wrote; it names an injected column only when the caller did",

packages/objectql/src/index.ts

@@ -628,6 +628,8 @@ export type {
   LifecycleFloorViolation,
 } from './lifecycle/lifecycle-service.js';
 export { parseLifecycleDuration } from './lifecycle/duration.js';
+// [#30237, ADR-0148 D3] What a host hands `ObjectQLPlugin`'s `recycleBinRetention`.
+export type { RecycleBinRetentionSettings } from './recycle-bin/retention.js';
 export { lifecycleSettingsManifest } from './lifecycle/lifecycle-settings.js';
 
 // [#4551] Read-only referential-integrity audit — the reporting half of the