es4

Embedded server-side state store with pluggable Recovery — for when you want durable in-process state without bringing in Redis/Valkey.

日本語版

Product knowledge lives under docs/ (OKF). This root README is the canonical English entry for install and usage; deeper specs and plans stay in docs/.

Current monorepo line: Git tag v0.10.0 (Go packages/go + Node @b4moss/es4 stay in lockstep). BREAKING vs v0.9.0: key hierarchy delimiter / → : and Scan(prefix).

Purpose

Keep a JSON key/value State in-process (Memory or on-disk SQLite)
Persist recovery points through a Recovery adapter (file / libSQL / S3-compatible object)
Drive Snapshot → Save → Restore on Open without exposing Snapshot/Recovery as a public API
Use as an embedded library or as a standalone HTTP Es4 Server (packages/go/cmd/es4-server)

Scope

In scope (Go + Node at v0.10.0):

Layer	Choices
State	Memory · on-disk SQLite (`state_path`) · Redis／Valkey (`state_backend` + `state_redis_url`) · Firestore (`state_backend=firestore`)
Recovery	`file` · `libsql` (local) · `object` (S3-compatible) — internal only
Keys	Hierarchy delimiter `:` (`/` is an ordinary character). Reject empty / leading / trailing `:` / `::`
Public API	State: `Set` / `Get` / `Delete` / `Exists` / `Clear` / `Scan(prefix)` · Tx: `BeginTx` → Commit/Rollback (Scan on Tx)
Server	HTTP Es4 Server + Docker — keys as one path segment (`/v1/keys/a:b:c`)
E2E	Object (RustFS) · File SQLite · Memory · libSQL File/Memory · Redis · Valkey · Firestore Emulator (`workflow_dispatch`; redis／valkey／firestore not in `layer=all`)

Out of scope / not yet:

Remote Turso (libsql://) in product E2E
Production GCP-required Firestore E2E (Emulator only)
Public SnapshotNow API (explicit flush exists only as a test helper)

Install

Requirements: Go 1.24+ (module declares go 1.24; toolchain go1.26.8 in packages/go/go.mod).

Module path (current, until a rename is approved):

github.com/b4moss/es4/packages/go

Go module tag: packages/go/v0.10.0 (see packages/go/VERSION).

go get github.com/b4moss/es4/packages/go@v0.10.0

Import:

import (
    "github.com/b4moss/es4/packages/go/pkg/es4"
    "github.com/b4moss/es4/packages/go/pkg/options"
)

Optional HTTP server binary (from a clone):

cd packages/go
go run ./cmd/es4-server

Server options load from ES4_* env (same snake_case keys as library Options). See docs/specs/es4-server/.

Node.js / TypeScript (`packages/node`)

Requirements: Node.js 20+. Package: @b4moss/es4 (see packages/node/README.md).

npm install @b4moss/es4

From a clone:

cd packages/node
npm ci
npm test
npm run test:e2e:memory   # file|libsql|object|redis|valkey|firestore

import { open, defaults } from "@b4moss/es4";

const db = await open({ ...defaults(), memory_only: true });
await db.set("hello", JSON.stringify({ n: 1 }));
await db.close();

Quick start

Minimal memory-only store (no Recovery):

package main

import (
    "context"
    "encoding/json"
    "fmt"
    "log"

    "github.com/b4moss/es4/packages/go/pkg/es4"
    "github.com/b4moss/es4/packages/go/pkg/options"
)

func main() {
    ctx := context.Background()
    db, err := es4.Open(ctx, options.Options{
        MemoryOnly: true,
    })
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()

    if err := db.Set(ctx, "users/1", json.RawMessage(`{"name":"ada"}`)); err != nil {
        log.Fatal(err)
    }
    raw, err := db.Get(ctx, "users/1")
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(string(raw))
}

Local SQLite State + file Recovery (periodic Snapshot; default interval 30s when unset):

db, err := es4.Open(ctx, options.Options{
    StatePath:        "./data/state.db",
    RecoveryBackend:  options.RecoveryBackendFile, // or omit if RecoveryPath is set
    RecoveryPath:     "./data/recovery.snap",
    RestoreOnStartup: true,
})

Keys are hierarchical strings separated by / (no leading/trailing /, no //). Values are arbitrary JSON documents (json.RawMessage). Missing keys: Get/Delete → es4.ErrNotFound; Exists → false, nil.

Transactions (separate Tx surface):

tx, err := db.BeginTx(ctx)
if err != nil {
    log.Fatal(err)
}
_ = tx.Set(ctx, "users/2", json.RawMessage(`{"name":"grace"}`))
if err := tx.Commit(); err != nil {
    log.Fatal(err)
}

Assemble Options from Defaults → optional YAML → ES4_* env with options.Load(path, nil).

Usage — Recovery backends & product E2E

Library Open selects backends from options.Options.Effective() (see docs/specs/options/ and packages/go/pkg/options). Snapshot/Recovery are internal; production code relies on snapshot_interval (default 30s) and restore_on_startup (default true). E2E tests set snapshot_interval=0 and call the test-only SnapshotNow helper to flush explicitly.

Library wiring (minimal)

Memory (no persistence)

es4.Open(ctx, options.Options{MemoryOnly: true})

File Recovery (recovery_backend=file or non-empty recovery_path)

es4.Open(ctx, options.Options{
    StatePath:       "/var/es4/state.db", // empty → Memory State
    RecoveryBackend: options.RecoveryBackendFile,
    RecoveryPath:    "/var/es4/recovery.snap",
})

Object Recovery (S3-compatible; needs credentials in the process environment / AWS default chain)

es4.Open(ctx, options.Options{
    RecoveryBackend:    options.RecoveryBackendObject,
    RecoveryS3Bucket:   "my-bucket",
    RecoveryS3Prefix:   "es4/prod/",
    RecoveryS3Region:   "us-east-1",
    RecoveryS3Endpoint: "", // set for MinIO / RustFS / GCS-compatible endpoints
    RestoreOnStartup:   true,
})

libSQL Recovery (local file or :memory:; remote Turso not covered by product E2E)

// File-backed Recovery DB
es4.Open(ctx, options.Options{
    RecoveryBackend:   options.RecoveryBackendLibSQL,
    RecoveryLibSQLURL: "file:/var/es4/recovery.db",
    RestoreOnStartup:  true,
})

// Process-local in-memory Recovery
es4.Open(ctx, options.Options{
    RecoveryBackend:   options.RecoveryBackendLibSQL,
    RecoveryLibSQLURL: ":memory:",
})

Redis / Valkey State (same adapter; valkey is an alias)

es4.Open(ctx, options.Options{
    StateBackend:        options.StateBackendRedis, // or StateBackendValkey
    StateRedisURL:       "redis://127.0.0.1:6379/0",
    StateRedisKeyPrefix: "myapp/prod/", // optional; empty → HASH es4:state
    SnapshotInterval:    0,             // or leave default if also using Recovery
})

Firestore State (Emulator: set FIRESTORE_EMULATOR_HOST)

es4.Open(ctx, options.Options{
    StateBackend:               options.StateBackendFirestore,
    StateFirestoreProjectID:    "demo-es4",
    StateFirestoreCollection:   "es4-state",
    // StateFirestoreDatabaseID: "", // Effective default: (default)
    SnapshotInterval:           0,
})

Optional: recovery_ttl (Go duration; 0 keeps only the latest generation), recovery_libsql_auth_token for authenticated DSN forms.

Product E2E — run the workflow layers

E2E is manual only (.github/workflows/e2e.yml, workflow_dispatch). Input layer:

`layer`	What runs	Extra service
`all`	`go test -tags=e2e ./...` (every `//go:build e2e` package)	RustFS required; Redis/Valkey/Firestore skip unless env set
`object`	`TestE2E_ObjectRecovery_*`	RustFS
`file`	`TestE2E_FileSQLite_*`	No
`memory`	`TestE2E_Memory_*`	No
`libsql`	`TestE2E_LibSQL*` (File C1–C5 + Memory C2/C4/C5)	No
`redis`	`TestE2E_RedisState_*`	Compose `redis` (not in `all`)
`valkey`	`TestE2E_ValkeyState_*`	Compose `valkey` (not in `all`)
`firestore`	`TestE2E_FirestoreState_*`	Compose `firestore` (not in `all`)

Local commands (from repo root / packages/go):

# Object needs RustFS first
docker compose -f docker/e2e/docker-compose.yml up -d --wait rustfs
export AWS_ACCESS_KEY_ID=es4e2eaccess
export AWS_SECRET_ACCESS_KEY=es4e2esecretkey
export ES4_E2E_S3_ENDPOINT=http://127.0.0.1:9000
export ES4_E2E_S3_REGION=us-east-1

# all (CI sets ES4_E2E_REQUIRE_RUSTFS=1 so missing RustFS fails)
cd packages/go && go test -tags=e2e ./... -count=1 -timeout 10m

# object only
go test -tags=e2e ./pkg/es4 -run 'TestE2E_ObjectRecovery_' -count=1 -timeout 10m

# file SQLite (no RustFS)
go test -tags=e2e ./pkg/es4 -run 'TestE2E_FileSQLite_' -count=1 -timeout 10m

# memory (no RustFS)
go test -tags=e2e ./pkg/es4 -run 'TestE2E_Memory_' -count=1 -timeout 10m

# Redis State
docker compose -f docker/e2e/docker-compose.yml up -d --wait redis
export ES4_E2E_REDIS_URL=redis://127.0.0.1:6379/0
go test -tags=e2e ./pkg/es4 -run 'TestE2E_RedisState_' -count=1 -timeout 10m

# Valkey State
docker compose -f docker/e2e/docker-compose.yml up -d --wait valkey
export ES4_E2E_VALKEY_URL=redis://127.0.0.1:6380/0
go test -tags=e2e ./pkg/es4 -run 'TestE2E_ValkeyState_' -count=1 -timeout 10m

# Firestore Emulator (not part of layer=all)
docker compose -f docker/e2e/docker-compose.yml up -d --wait firestore
export FIRESTORE_EMULATOR_HOST=127.0.0.1:8080
export ES4_E2E_FIRESTORE_PROJECT_ID=demo-es4
go test -tags=e2e ./pkg/es4 -run 'TestE2E_FirestoreState_' -count=1 -timeout 10m

Compose defaults and cleanup: docker/e2e/README.md. Behavior catalog: docs/tests/e2e/e2e-spec.md.

Harness files (all //go:build e2e):

Layer	File
Object	`packages/go/pkg/es4/object_recovery_e2e_test.go`
File SQLite	`packages/go/pkg/es4/file_sqlite_e2e_test.go`
Memory	`packages/go/pkg/es4/memory_e2e_test.go`
libSQL File / Memory	`libsql_file_e2e_test.go` · `libsql_memory_e2e_test.go`
Redis State	`redis_state_e2e_test.go`
Valkey State	`valkey_state_e2e_test.go`
Firestore State	`firestore_state_e2e_test.go`

Writing / running libSQL E2E

Spec: docs/tests/e2e/e2e-spec.md §L · plan: docs/plans/v0.7.3/.

Sub-layer	Catalog	Options pattern
File-backed	C1–C5	`recovery_backend=libsql`, `recovery_libsql_url=file:<temp>/recovery.db`, empty `state_path` (Memory State), `restore_on_startup=true`, `snapshot_interval=0`
InMemory	C2 / C4 / C5 only	same backend, `recovery_libsql_url=:memory:` (no restart/restore scenarios)

Run:

cd packages/go && go test -tags=e2e ./pkg/es4 -run 'TestE2E_LibSQL' -count=1 -timeout 10m

GitHub Actions: run workflow e2e with layer=libsql (no RustFS). layer=all also includes these tests via ./....

When authoring new libSQL cases, follow the existing harness: temp-dir isolation + start/t.Cleanup clear, OpenWith + SkipAsyncRestore: true for deterministic restore in tests, and test-only SnapshotNow after writes when proving Save→re-Open. Do not rely on interval waits. Remote libsql:// is explicitly out of E2E scope.

Docs map

Path	Role
`docs/README.md`	Product pillar (purpose, architecture diagrams, usage modes)
`docs/index.md`	OKF index (`okf_version`)
`docs/roadmap.md`	Phase / provisional SemVer hub (may lag tags slightly)
`docs/plans/`	Upcoming / completed plan notes
`docs/specs/`	Normative specs: options · state · tx · snapshot · recovery · es4-server
`docs/tests/`	Test specs, including `docs/tests/e2e/`
`b4moss/charter` (v1)	Engineering charter (OKF, TDD, versioning, git). See AGENTS.md 「開発憲章（charter）」
`.github/CI.md`	CI / E2E / release policy

Prefer code + specs over older roadmap wording when they disagree.

Name		Name	Last commit message	Last commit date
.github		.github
docker		docker
docs		docs
packages		packages
scripts		scripts
.gitignore		.gitignore
AGENTS.md		AGENTS.md
LICENSE		LICENSE
README.md		README.md
README_ja.md		README_ja.md
codecov.yml		codecov.yml

Navigation Menu

Repository navigation

About

Repository files navigation

es4

Purpose

Scope

Install

Node.js / TypeScript (`packages/node`)

Quick start

Usage — Recovery backends & product E2E

Library wiring (minimal)

Product E2E — run the workflow layers

Writing / running libSQL E2E

Docs map

License

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

es4

Purpose

Scope

Install

Node.js / TypeScript (packages/node)

Quick start

Usage — Recovery backends & product E2E

Library wiring (minimal)

Product E2E — run the workflow layers

Writing / running libSQL E2E

Docs map

License

About

Resources

Code of conduct

Contributing

Security policy

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

Node.js / TypeScript (`packages/node`)