fix(objectql): a cleared optional multi-value field stores null, so a second record cleared on a unique one is no longer refused 409

.changeset/29343-multi-value-blank-null.md

@@ -0,0 +1,22 @@
+---
+'@objectstack/objectql': patch
+'@objectstack/core': patch
+---
+
+fix(objectql): a cleared optional multi-value field stores `null`, so a second record cleared on a unique one is no longer refused `409 UNIQUE_VIOLATION`
+
+Clause-②: yes (widening)
+Metadata-protocol: none
+
+This executes ruling A on #29343 (record 6591577985), which amends the #9447 sentence "an emptied multi-value lookup reads back as `[]`, never `null`" for optional columns: a cleared optional multi-value field is stored as `null`, the same as one never set. The spec half (the `multiple` and `required` doc blocks of `FieldSchema`) landed separately; this is the runtime half. It is a widening (Clause-②): a write that used to be refused `409` now lands, and no write that used to land is refused.
+
+A cleared `multiselect`, `checkboxes` or `tags` field, or a `select`, `radio`, `lookup`, `user`, `file` or `image` field with `multiple: true`, sent as `[]` was stored as the empty list (the JSON text `[]` on SQL), and sent as `''` was stored as the JSON text `""`. Neither holds a member, yet each was a value to the store: on a unique column, or a column in a unique index, the second record cleared was refused "A record with this value already exists".
+
+- **Now.** The engine's write door (`insert`, `insertMany`, `update` by id and by predicate, and the `validate` dry run; every REST, batch, upsert and import door reaches it) stores `[]`, or exactly `''`, as `null` on every optional column the `$empty` table puts on its `multi_value` row. Unique or not. A non-empty list is stored as sent.
+- **Read change.** An optional multi-value field cleared from now on reads back `null`, not `[]`, whoever clears it: an API or form write of `[]` or `''`, and the `set_null` cascade repair that removes the last member of a multi-value reference (it used to leave `[]`). A reader that branched on `Array.isArray` alone reads `null` and `[]` alike as empty. It still matches `is_empty` / `$empty: true` (the table reads `null` and `[]` as empty on that row), and now also matches `$null: true` and `field = null`.
+- **Readonly columns.** A `readonly: true` multi-value column is platform-stamped and never cleared by a writer, so a `[]` written there keeps `[]` (ruling record 6598733139): `sys_activity.changed_fields` still stores `[]` for "stamped, nothing changed" and `null` for "unstamped".
+- **Unchanged.** The `$empty` table itself. A `required` multi-value field given `[]` or `''` is refused `required` as before, also when it declares a `defaultValue`. Whitespace is a value. A real duplicate is still `409 UNIQUE_VIOLATION`.
+- **Defaults.** An optional multi-value field that declares a `defaultValue` now takes it on insert when sent `[]` or `''`, exactly as when sent `null`.
+- **Rows already stored.** Existing `[]` and `""` values are not migrated by this change. They keep their stored form, still answer `$empty: true`, and still collide with each other on a unique column until a data step cleans them. `os migrate blank-single-values` does not touch these columns.
+- **Seed replay.** A seed that writes `[]` or `''` on such a column is compared in the form the door stores it, `null`. A row that already stores `[]` is rewritten to `null` once on the next replay, then no longer changes.
+- **Export.** `@objectstack/core`'s `BlankAsNullReach` gains the member `'emptyList'`, which `blankAsNullColumns` returns for these columns and `blankStorageForm` reads.

content/docs/api/data-api.mdx

@@ -275,10 +275,11 @@ ROW: the delete is refused only over rows whose set the member removal would
 empty, and `dependentCount` counts just those rows — a row that keeps another
 member is updated and does not block the delete. On a
 `multiple: true` reference where `set_null` does run, it
-removes just the deleted id from the array and keeps the rest, and a reference set
-emptied that way reads back as `[]` — never `null`, so a client that branches on
-`null` for "no link" misses the emptied case (the `multiple` doc block in
-`packages/spec/src/data/field.zod.ts`, rendered in the
+removes just the deleted id from the array and keeps the rest. On an optional field
+a reference set emptied that way reads back as `null`, the same as one never set; a
+row stored before that rule can still read back `[]`, so a client treats `null` and
+`[]` alike as "no link", as the `is_empty` filter (`$empty`) does (the `multiple` doc
+block in `packages/spec/src/data/field.zod.ts`, rendered in the
 [Field reference](/docs/references/data/field)).
 
 A record that the same delete removes never blocks it. Before any refusal is

content/docs/data-modeling/field-types.mdx

@@ -496,7 +496,7 @@ Reference to a record in another object (foreign key).
 |:---|:---|:---|:---|
 | `reference` | `string` | **required** | Target object name (snake_case). A stored-metadata table (`sys_metadata`, `sys_metadata_history`) is refused — see [below](#a-relationship-never-targets-a-stored-metadata-table) |
 | `referenceFilters` | `string[]` | — | **Removed** (#2377, ADR-0049) — no longer a recognized field property. `FieldSchema` is a strict object, so an unknown key is **rejected with guidance**, not silently stripped (ADR-0078): the error echoes the offending key and prescribes the replacement. Use structured `lookupFilters` + `dependsOn` instead; see [Relationships](/docs/data-modeling/relationships) |
-| `deleteBehavior` | `'restrict' \| 'cascade' \| 'set_null'` | `'set_null'` | Behavior when referenced record is deleted. On a *required* lookup `set_null` is escalated to `restrict`, since a NOT NULL foreign key cannot be cleared — **whether the `set_null` was defaulted or written out explicitly**. On a `multiple: true` required lookup the escalation is judged per referencing row: only a row the member removal would leave EMPTY is refused. `cascade` and `restrict` are the values honored as written. Where `set_null` does run, a `multiple: true` lookup loses only the deleted **member** — the other members are kept, and a set emptied that way is stored as `[]`, never `null` |
+| `deleteBehavior` | `'restrict' \| 'cascade' \| 'set_null'` | `'set_null'` | Behavior when referenced record is deleted. On a *required* lookup `set_null` is escalated to `restrict`, since a NOT NULL foreign key cannot be cleared — **whether the `set_null` was defaulted or written out explicitly**. On a `multiple: true` required lookup the escalation is judged per referencing row: only a row the member removal would leave EMPTY is refused. `cascade` and `restrict` are the values honored as written. Where `set_null` does run, a `multiple: true` lookup loses only the deleted **member** — the other members are kept, and on an optional field a set emptied that way is stored as `null`, the same as one never set |
 
 ```typescript
 { name: 'company', label: 'Company', type: 'lookup', reference: 'account' }
@@ -1022,7 +1022,7 @@ These properties are available on **all** field types:
 | `type` | `FieldType` | **required** | One of the supported field types |
 | `required` | `boolean` | `false` | Whether the field is required |
 | `unique` | `boolean \| 'global' \| 'organization'` | `false` (`'organization'` on `autonumber`) | Unique constraint **and its scope** (ADR-0120). `'organization'` = one holder per organization; `true` = that same per-organization scope (positional synonym — prefer the explicit spelling in new code); `'global'` = one holder across the whole installation; `false` = no constraint. Omitted ⇒ `false`, except on an `autonumber` field, where omitted ⇒ `'organization'` — an auto-number is a business identifier, so the platform makes it unique per organization by default; write `unique: false` explicitly to opt out. `'tenant'`/`'org'` are rejected — the word is `'organization'`. Full rule in the `unique` doc block in [`FieldSchema`](/docs/references/data/field) |
-| `multiple` | `boolean` | `false` | Allow array of values (multi-record lookup, multi-select, multi-file). An emptied multi-value lookup reads back as `[]`, never `null` — the `multiple` doc block in [`FieldSchema`](/docs/references/data/field) |
+| `multiple` | `boolean` | `false` | Allow array of values (multi-record lookup, multi-select, multi-file). An emptied optional multi-value field is stored as `null`, the same as one never set (`[]` and `""` written to it store `null`); a row stored before that rule can still hold `[]`, and `is_empty` matches both — the `multiple` doc block in [`FieldSchema`](/docs/references/data/field) |
 | `searchable` | `boolean` | `false` | Search this field when the object declares no `searchableFields` — any type but `secret` / `password` / `vector`; a declared `searchableFields` is the whole set |
 | `sortable` | `boolean` | `true` | Allow sorting by this field |
 | `hidden` | `boolean` | `false` | Hide from default views |

content/docs/data-modeling/fields.mdx

@@ -292,9 +292,9 @@ Clearing a relationship is not a dangling reference: `null`, `""` and `[]` all m
 "no link". Which of them `deleteBehavior: 'set_null'` writes when the referenced
 record goes away depends on the field: a single-value lookup is set to `null`, while
 a `multiple: true` lookup keeps its other members and loses only the deleted one —
-and a set emptied that way is written as `[]`, never `null` (the `multiple` doc block
-in `packages/spec/src/data/field.zod.ts`, rendered in the
-[Field reference](/docs/references/data/field)).
+and on an optional field a set emptied that way is stored as `null`, the same as one
+never set (the `multiple` doc block in `packages/spec/src/data/field.zod.ts`, rendered
+in the [Field reference](/docs/references/data/field)).
 
 <Callout type="info">
 `deleteBehavior` governs what happens to *this* record when the **referenced**

content/docs/data-modeling/validation-rules.mdx

@@ -261,13 +261,14 @@ These properties apply to **all** field types and are validated by the base `Fie
 
 **Default constraints:** Validates that referenced record exists. Foreign key integrity enforced.
 
-**Multi-value lookups (`multiple: true`).** The empty set is representable: an emptied
-multi-value lookup is stored and read back as `[]`, never `null` — including when
-`deleteBehavior: 'set_null'` removes the last remaining member. The declared contract
-lives in the `multiple` and `required` doc blocks of
-`packages/spec/src/data/field.zod.ts` (rendered in the
-[Field reference](/docs/references/data/field)), and it makes `required` on a
-multi-value lookup mean **non-empty array**.
+**Multi-value lookups (`multiple: true`).** An emptied optional multi-value lookup is
+stored and read back as `null`, the same as one never set — `[]` and `""` written to it
+store `null`, including when `deleteBehavior: 'set_null'` removes the last remaining
+member. A row stored before that rule can still hold `[]`; the `is_empty` filter
+(`$empty`) reads `null` and `[]` alike as empty. The declared contract lives in the
+`multiple` and `required` doc blocks of `packages/spec/src/data/field.zod.ts`
+(rendered in the [Field reference](/docs/references/data/field)), and it makes
+`required` on a multi-value lookup mean **non-empty array**.
 
 <Callout type="info">
 The `required`-means-non-empty half is **enforced by the record validator** (#9476, per

content/docs/deployment/troubleshooting.mdx

@@ -250,11 +250,13 @@ client.data.find('project_task', { /* query */ });
 
 <Callout type="info">
 On a `multiple: true` lookup, `set_null` clears only the **member** that was deleted —
-the child's other references survive — and a set emptied that way is stored as `[]`,
-never `null` (the `multiple` doc block in `packages/spec/src/data/field.zod.ts`,
-rendered in the [Field reference](/docs/references/data/field)). A cleanup query or
-client branch that looks for `null` to find cleared references will not match those
-rows; test for an empty array instead.
+the child's other references survive — and on an optional field a set emptied that
+way is stored as `null`, the same as one never set (the `multiple` doc block in
+`packages/spec/src/data/field.zod.ts`, rendered in the
+[Field reference](/docs/references/data/field)). A row stored before that rule can
+still hold `[]`, so a cleanup query or client branch that finds cleared references
+uses the `is_empty` filter (`$empty: true`), which matches both, rather than testing
+for `null` or for an empty array alone.
 </Callout>
 
 ---

content/docs/protocol/objectql/types.mdx

@@ -726,12 +726,14 @@ const opportunities = await engine.find('opportunity', {
 - `restrict`: Prevent deletion if references exist
 - `cascade`: Delete this record when referenced record is deleted
 
-> **Residual shape of a multi-value reference.** A `multiple: true` lookup emptied by
-> member removal is written as `[]`, **never `null`**. That is not a cascade
-> convention: the representation binds every writer — cascade repair, form clears and
-> API writes alike — so a reader of an array field never needs a null branch. The
-> guarantee is pinned by `FieldSchema`, in the `multiple` doc block of
-> `packages/spec/src/data/field.zod.ts` (rendered in the
+> **Residual shape of a multi-value reference.** An optional `multiple: true` lookup
+> emptied by member removal is stored as `null`, the same as one never set. That is
+> not a cascade convention: the representation binds every writer — cascade repair,
+> form clears and API writes alike, where `[]` and `""` written to an optional
+> multi-value field store `null`. A reader of an array field reads `null` and `[]`
+> alike as empty, as the `is_empty` filter (`$empty`) does, since a row stored before
+> this rule can still hold `[]`. The rule is pinned by `FieldSchema`, in the
+> `multiple` doc block of `packages/spec/src/data/field.zod.ts` (rendered in the
 > [Field reference](/docs/references/data/field)).
 
 > **Required foreign keys.** A `required: true` lookup cannot be nulled, so

packages/cli/src/commands/migrate/blank-single-values.integration.test.ts

@@ -21,6 +21,7 @@
 import { describe, it, expect, beforeEach, afterEach } from 'vitest';
 import { ObjectQL } from '@objectstack/objectql';
 import { SqlDriver } from '@objectstack/driver-sql';
+import { blankAsNullColumns } from '@objectstack/core';
 import { blankSingleValueColumns, cleanBlankSingleValues } from './blank-single-values.js';
 
 const OPTS = [{ value: 'gold', label: 'Gold' }, { value: 'silver', label: 'Silver' }];
@@ -100,6 +101,11 @@ describe('os migrate blank-single-values: the column set', () => {
       ['audio', 'avatar', 'file', 'image', 'lookup', 'master_detail', 'radio', 'select', 'tree', 'user', 'video'],
     );
   });
+
+  it('[#29343] a multi_value-row column is in the door\'s map on its own reach, and out of this walk', () => {
+    expect(blankAsNullColumns(OBJ as any).get('m_select')).toBe('emptyList');
+    expect(blankSingleValueColumns(OBJ).map((c) => c.name)).not.toContain('m_select');
+  });
 });
 
 describe('os migrate blank-single-values: the walk on sqlite (#29431)', () => {

packages/cli/src/commands/migrate/blank-single-values.ts

@@ -106,16 +106,18 @@ const SYSTEM_CTX = { isSystem: true } as const;
 /**
  * The columns of one object this run cleans: the write door's #29277 arm, read
  * from `blankAsNullColumns` (`@objectstack/core`, the one rule the door and
- * the seed replay read). That map holds three arms; this run takes the two
+ * the seed replay read). That map holds four arms; this run takes the two
  * `'empty'` ones minus the text-like columns of the #29176 unique arm, which
  * leaves exactly the optional single-value columns on the `$empty` table's
  * `null_only` row: options, references and file references today. The text
  * types are the spec's `STRING_VALUE_TYPES`, the set the map itself reads for
  * that arm: no type name is listed here.
  *
- * Out, by the map's own rule: `required` columns, lifecycle columns,
- * `multiple: true` columns (the `multi_value` row), typed columns (the #20308
- * `'blank'` arm, whose door already refuses to store a blank as a value).
+ * Out, by the map's own rule: `required` columns and lifecycle columns. Out
+ * because this run takes only `reach === 'empty'`: `multiple: true` columns
+ * (the `multi_value` row), which the map holds on the #29343 `'emptyList'`
+ * arm, and typed columns (the #20308 `'blank'` arm, whose door already refuses
+ * to store a blank as a value).
  */
 export function blankSingleValueColumns(schema: unknown): Array<{ name: string; type: string }> {
   const fields = (schema as { fields?: Record<string, BlankAsNullFieldDef | undefined> } | undefined)?.fields;

packages/core/src/utils/blank-storage-form.test.ts

@@ -24,7 +24,7 @@ const schema = {
   indexes: [{ fields: ['idx_a'], unique: true }, { fields: ['code'] }],
 };
 
-describe('blankAsNullColumns: the three arms of the write door\'s blank-as-null rule', () => {
+describe('blankAsNullColumns: the four arms of the write door\'s blank-as-null rule', () => {
   const cols = blankAsNullColumns(schema);
 
   it('[#20308] a typed column takes any blank, required and lifecycle included', () => {
@@ -44,7 +44,45 @@ describe('blankAsNullColumns: the three arms of the write door\'s blank-as-null
     expect(cols.get('stage')).toBe('empty');
     expect(cols.get('ref')).toBe('empty');
     expect(cols.get('pic')).toBe('empty');
-    for (const out of ['stage_req', 'tags', 'blob', 'num', 'created_by']) expect(cols.has(out), out).toBe(false);
+    for (const out of ['stage_req', 'tags', 'blob', 'num', 'created_by']) expect(cols.get(out), out).not.toBe('empty');
+  });
+
+  it('[#29343] "emptyList" on every optional column on the multi_value row, unique or not', () => {
+    const list = blankAsNullColumns({
+      fields: {
+        tags: { type: 'select', multiple: true, options: [{ value: 'a' }] },
+        ms: { type: 'multiselect', options: [{ value: 'a' }] },
+        ms_u: { type: 'multiselect', unique: true, options: [{ value: 'a' }] },
+        cb: { type: 'checkboxes', options: [{ value: 'a' }] },
+        tg: { type: 'tags' },
+        refs: { type: 'lookup', multiple: true },
+        pics: { type: 'image', multiple: true },
+        ms_req: { type: 'multiselect', required: true, options: [{ value: 'a' }] },
+        created_by: { type: 'user', multiple: true },
+        ref: { type: 'lookup' },
+      },
+    });
+    for (const f of ['tags', 'ms', 'ms_u', 'cb', 'tg', 'refs', 'pics']) expect(list.get(f), f).toBe('emptyList');
+    expect(list.has('ms_req')).toBe(false);
+    expect(list.has('created_by')).toBe(false);
+    expect(list.get('ref')).toBe('empty');
+  });
+
+  it('[#29343] a readonly optional tags column written [] keeps [] — platform-stamped, never cleared (record 6598733139)', () => {
+    const cols = blankAsNullColumns({
+      fields: {
+        changed_fields: { type: 'tags', required: false, readonly: true },
+        tg: { type: 'tags' },
+        stage_ro: { type: 'select', readonly: true, options: [{ value: 'a' }] },
+      },
+    });
+    expect(cols.has('changed_fields')).toBe(false);
+    const stamped: string[] = [];
+    expect(blankStorageForm(stamped, cols.get('changed_fields'))).toBe(stamped);
+    // Controls: the same type without readonly still takes the arm, and the
+    // 'empty' arm is untouched by readonly.
+    expect(cols.get('tg')).toBe('emptyList');
+    expect(cols.get('stage_ro')).toBe('empty');
   });
 
   it('lifecycle names are the five engine-owned columns', () => {
@@ -66,6 +104,17 @@ describe('blankStorageForm', () => {
     expect(blankStorageForm('', undefined)).toBe('');
   });
 
+  it('"emptyList" nulls [] and exactly ""; whitespace and a non-empty list stay', () => {
+    expect(blankStorageForm([], 'emptyList')).toBeNull();
+    expect(blankStorageForm('', 'emptyList')).toBeNull();
+    const list = ['a'];
+    expect(blankStorageForm(list, 'emptyList')).toBe(list);
+    for (const v of ['  ', 'a', 0, false, null, undefined]) expect(blankStorageForm(v, 'emptyList')).toBe(v);
+    // An empty list on any other reach is not a blank of that column.
+    const empty: unknown[] = [];
+    for (const r of ['blank', 'empty', undefined] as const) expect(blankStorageForm(empty, r)).toBe(empty);
+  });
+
   it('a non-blank value or a non-string comes back as it is', () => {
     const obj = { a: 1 };
     for (const v of ['a', 0, false, null, undefined, obj]) {