[adenhertog]

Closes #347

Summary

Adds RedisTransport as a new @node-ts/bus-redis: each service's queue is a Redis Stream read by a consumer group, on Redis 7+ or Valkey 7.2+. It replaces the 0.x Redis lists transport that had the same name.

Background

#347 asks for a Redis Streams transport in the monorepo, to replace node-ts/bus-redis (bus-core 1.x, inversify, unchanged since 2021). The maintainer accepted the design first; it's in this comment. It differs from the issue in three places: node-redis 6 rather than 5, no Sentinel in v1, and a dead letter stream per queue rather than a shared one.

Problem

There's no Redis transport for the current bus. Teams that already run Redis or Valkey need another broker, or have to stay on the unmaintained 0.x package.

Approach

• Package: packages/bus-redis, with redis (node-redis) ^6 as a peer dependency. The transport makes its own clients with RESP2, so the reply shapes don't depend on node-redis' defaults. It reads on a second, duplicated connection that blocks for at most 1 s.

• Frozen connections: node-redis' command timeout stops covering a command once it's written to the socket. Without more, a frozen or half-open connection would leave a read, stop() and dispose() waiting forever. So:

  • every connection gets a socket timeout (15 s, and at least the block time + 10 s on the reader) and a 5 s PING, and reconnects after a socket timeout;
  • writes count as socket activity, so a watchdog also sends each ready connection a PING every 5 s. Without a reply in 15 s, it destroys the connection, which fails what's pending, and replaces it. It runs from connect() to disconnect() and never overlaps itself;
  • stop() gives a read in flight 3 s, then destroys the reader, and gives handing back messages another 3 s. The bus then waits for messages being handled, whose settling on a frozen connection fails once the watchdog replaces it, up to 20 s later;
  • disconnect() gives leaving the group and close() 3 s each, then destroys the connection;
  • integration tests freeze a TCP proxy during a blocked read, and while sending every 2 s. They check that receiving recovers after the proxy is unfrozen, and that stop() and dispose() finish.

• Keys: <prefix>:{<queue>}:queue (a stream, with consumer group <queue>), :delayed (a sorted set), :dead-letter (a stream), and <prefix>:subscriptions:<$name> (a set). The default prefix is bus.

  • Every queue key shares the {<queue>} hash tag, and every script touches one queue's keys, so Redis Cluster stays possible later.
  • queueName and keyPrefix can't contain { or } (InvalidRedisKeyName).

• Sending:

  • Fan-out is SMEMBERS on the subscription set, then one MULTI with an XADD … NOMKSTREAM per subscriber, so every subscriber gets the message or none does.
  • sendToAddress is XADD NOMKSTREAM, which throws EndpointNotFound when the stream doesn't exist, and creates nothing.

• Receiving:

  • Each process has one reader, which fetches for all of its waiting workers.
  • A script runs before each read. It moves due retries back to the stream. Once a second, it takes over entries left pending past visibilityTimeoutMs (default 30 s), with XPENDING … IDLE + XCLAIM. Once a minute, it removes stale consumers that have nothing pending.
  • failedAttempts = the failedAttempts field + deliveries − 1, so a receipt that timed out counts, as on SQS and Postgres.
  • If converting or dead-lettering one entry of a batch fails, the entries after it are given back at once rather than waiting out the visibility timeout. The failing entry is left to its visibility timeout, so one that fails every time is counted rather than looping.
  • Delete, return and fail are each one Lua script. Each first checks the entry is still pending to this consumer with its delivery count, which works as the lease. Otherwise it settles nothing and logs a warning.
  • stop() hands back entries it read but never gave to a worker, without counting an attempt. disconnect() leaves the consumer group, but only when nothing is pending to it.

• Retries and dead letters:

  • Returned messages wait in the sorted set until they're due.
  • fail moves the message to the queue's dead letter stream with a bus-failure field, trimmed with MINID ~ to deadLetterRetentionMs (default 14 days; 0/Infinity keep everything).
  • Unparseable messages are dead-lettered when read.
  • The docs give a Lua redrive script, and the integration test runs it.

• Commands used: only Redis 6.2/7.0 commands (no XACKDEL/XNACK/XREADGROUP CLAIM), so Valkey works.

• Provisioning:

  • provision() runs XGROUP CREATE … 0 MKSTREAM (BUSYGROUP counts as success) and SADDs the subscriptions. A send-only bus or a dry run makes no calls.
  • initialize() only checks, with XINFO GROUPS and SISMEMBER, and throws ResourcesNotProvisioned.
  • The plan's runtime permissions are a redis-acl document. The integration test runs a bus as an ACL user with exactly those rules.

• Infra:

  • docker-compose redis:8 on host port 6380 and valkey/valkey:8 on 6381, with REDIS_URL in test.env.
  • CircleCI adds both as secondary containers, and runs the Redis integration file a second time against Valkey.
  • Tests use a random key prefix and only UNLINK their own keys. They never flush.

• Docs:

  • A new /transports/redis page: keys, provisioning and ACLs, keeping messages safe (noeviction, AOF, async replication), redrive, and migrating from 0.x.
  • Elsewhere: the sidebar, a card on the transports page, rows in the provisioning and recoverability tables, a MIGRATING.md entry, and the root README.
  • Package README and CLAUDE.md files.

• Changeset: a minor changeset with a **Breaking:** note for 0.x users.

• Tests: transportTests passes on Redis 8 and Valkey 8. The other integration tests cover:

  • resources missing at startup;
  • a dry run that creates nothing;
  • provisioning twice;
  • an expired receipt taken over, with the attempt counted;
  • Redis freezing during a blocked read, and while messages are sent every 2 s;
  • a batch whose second entry can't be dead-lettered;
  • a delayed retry;
  • leaving the consumer group;
  • redrive;
  • unparseable messages;
  • EndpointNotFound;
  • the ACL user.

  Unit tests cover the plan, ACL escaping and config validation, including rejecting fractional durations (Redis needs whole milliseconds).

Maintainer actions after release (not done here):

• Release it together with the other packages. It starts at 0.0.0, and npm already has @node-ts/bus-redis 0.1.1–0.1.9. Released with the others under the linked versions, with master's pending major changesets, its first version is 2.0.0. Released on its own, it would fall below or clash with 0.1.x. Check the version packages PR.

• Deprecate only the old range: npm deprecate @node-ts/bus-redis@"<1.0.0" "Replaced by @node-ts/bus-redis 2.x, a Redis Streams transport: https://node-ts.github.io/bus/transports/redis". A bare npm deprecate @node-ts/bus-redis would deprecate the new versions too. The old versions are two lines: 0.1.8 (latest, bus-core 1.x, new RedisTransport({ connectionString })) and 0.1.1–0.1.7 and 0.1.9 (the 0.x tag, bus-core 0.6 with inversify's BusRedisModule). The migration docs cover both.

• Archive the old repo: node-ts/bus-redis.

• Badge: the docs page's npm badge shows 0.1.8 until the release.

• This is original work under the clean-room policy, not ported, translated or copied from another messaging framework

• Added a changeset (pnpm changeset) for user-facing changes to published packages, or none is needed

• Docs: updated docs/ for user-facing changes, or none needed

🤖 Generated with Claude Code