Self-hosting guidance for Storage v4 and S3

maintenance-ops/self-hosting/deployment-architecture.mdx

@@ -1,144 +1,129 @@
 ---
 title: "Deployment Architecture"
-description: "Infrastructure and sizing recommendations for self-hosted PowerSync in staging and production."
+description: "Infrastructure and sizing recommendations for self-hosted PowerSync."
 sidebarTitle: "Deployment Architecture"
 ---
 
 ## Minimal Setup
 
-A minimal "development" setup (e.g. for a staging or a QA environment) is:
+For a runnable development setup, use the [self-hosting examples](https://github.com/powersync-ja/self-host-demo), which include Docker Compose files and configuration.
 
-1. A single PowerSync "compute" container (API + replication) with 512MB memory, 1 vCPU.
-2. A single MongoDB node in replica set mode, 2GB memory, 1 vCPU. M10+ when using Atlas.
-3. Load balancer for TLS.
+For development and testing, use:
 
-This setup has no redundancy. If the replica set fails, you may need to recreate it from scratch which will re-sync all clients.
+1. One PowerSync container running API and replication: 1 vCPU, 512 MiB memory.
+2. A MongoDB or Postgres [bucket storage](#storage-requirements) database. For MongoDB, use one node in replica set mode: 1 vCPU, 2 GiB memory (M10+ on Atlas).
+3. A load balancer for TLS.
+
+This setup has no redundancy. If bucket storage data is lost, [re-replicate from the source database](#backups). All clients must then download their data again.
 
 ## Production
 
-For production, we recommend running a high-availability setup with the following baseline requirements:
+Run API and replication in separate containers so you can scale them independently.
 
-1. 1x PowerSync replication container, 1GB memory, 1 vCPU
-2. 2+ PowerSync API containers, 1GB memory each, 1vCPU each.
-3. A 3-node MongoDB replica set, 2+GB memory each. Refer to the MongoDB documentation for deployment requirements. M10+ when using Atlas.
-4. A load balancer with redundancy.
-5. Run a daily compact job.
+### Storage Requirements
 
-For further robustness, in cases where you have larger rows and higher load, we recommend increasing the memory and CPU requirements to 2GB memory and 2 vCPU. This applies to both the replication and API containers.
+You can use MongoDB or Postgres for bucket storage, independently of your source database:
 
-When scaling up, add 1x PowerSync API container per 100 connections. The MongoDB replica set should be scaled based on CPU and memory usage.
+- [MongoDB](/configuration/powersync-service/self-hosted-instances#mongodb-storage) supports storage version 4 and optional S3 object storage. Use a three-node replica set for production.
+- [Postgres](/configuration/powersync-service/self-hosted-instances#postgres-storage) uses storage version 2 and does not support storage version 4 or S3 object storage.
 
-### Replication Container
+With [S3 object storage](/sync/advanced/storage-version-4#self-hosted-s3-setup), S3 stores larger blocks of sync data while MongoDB retains smaller blocks and metadata.
 
-The replication container handles replicating from the source database to PowerSync's bucket storage.
+### Compute Sizing
 
-The replication process is run using the docker command `start -r sync`, for example `docker run powersync start -r sync`.
+The tested configuration uses ARM64 compute, MongoDB bucket storage, and storage version 4 with S3, available since PowerSync Service v1.26.0. Use these CPU and memory settings as a starting point:
 
-Only one process can replicate at a time. If multiple are running concurrently, you may see an error `[PSYNC_S1003] Sync rules have been locked by another process for replication`. If you use rolling deploys, it is normal to see this error for a short duration while multiple processes are running.
+| Role | vCPU | Memory |
+| --- | --- | --- |
+| API | 1 | 2 GiB |
+| Replication | 2 | 4 GiB |
 
-Memory and CPU usage of the replication container is primarily driven by write load on the source database. A good starting point is 1GB memory and 1 vCPU for the container, but this may be scaled down depending on the load patterns.
+### Connection Capacity and Scaling
 
-Set the environment variable `NODE_OPTIONS=--max-old-space-size-percentage=80` to allocate 80% of the container's assigned memory to the Node.js heap. This avoids manually retuning when scaling resources up or down.
+This configuration supports up to **1,000 concurrent connections per API process**. Larger datasets, more buckets per client, and simultaneous initial syncs can require fewer connections per process.
 
-### API Containers
+After load testing, raise [`max_concurrent_connections`](/configuration/powersync-service/self-hosted-instances#param-max-concurrent-connections) from its default of `200` up to `1000`. The setting limits HTTP streams and WebSockets separately; monitor their combined count.
 
-The API container handles streaming sync connections, as well as any other API calls.
+Scale on both CPU utilization and connections per process. Idle connections can keep CPU low, while initial syncs can saturate CPU with fewer connections. Keep autoscaling targets below the connection limit to allow for bursts while new containers start.
 
-The API process is run using the docker command `start -r api`, for example `docker run powersync start -r api`.
+Add API containers before many clients run an initial sync at once or before deployments that make clients download their data again. Wait for the new containers to become healthy before those initial syncs or deployments begin. Adding containers does not immediately redistribute existing long-lived connections. Reduce capacity gradually after the download load subsides.
 
-Each API container is limited to 200 concurrent connections, but we recommend targeting 100 concurrent connections or less per container. This may change as we implement additional performance optimizations.
+### Replication Container
 
-Memory and CPU usage of API containers are driven by:
+Run `start -r sync` to replicate source changes into bucket storage. Additional replication containers provide failover, not more throughput.
 
-1. Number of concurrent connections.
-2. Number of buckets per connection.
-3. Amount of data synced to each connection.
+During rolling deploys or failover, [`PSYNC_S1003`](/debugging/error-codes#psync_s1xxx-replication-issues) is expected briefly while another process holds the replication lock.
 
-A good starting point is 1GB memory and 1 vCPU per container, but this may be scaled up or down depending on the specific load patterns.
+Size replication for source write volume, row size, and buckets per row. A row in ten buckets creates more replication work than a row in one bucket. Initial replication and catch-up after a large batch update can need more capacity than steady-state replication.
 
-Set the environment variable `NODE_OPTIONS=--max-old-space-size-percentage=80` to allocate 80% of the container's assigned memory to the Node.js heap. This avoids manually retuning when scaling resources up or down.
+Use [replication lag](/maintenance-ops/replication-lag), replication throughput, CPU, memory, and database metrics to assess capacity.
+
+### API Containers
+
+Run at least two API containers in production for availability. Scale horizontally by adding containers as client demand grows.
+
+Run `start -r api` for client connections. Collect [metrics](/maintenance-ops/self-hosting/monitoring) separately from API and replication containers.
 
 ### Compact Job
 
-We recommend running a compact job daily as a cron job, or after any large maintenance jobs. For details, see the documentation on [Compacting Buckets](/maintenance-ops/compacting-buckets).
+Run `compact` as a separate scheduled job, hourly for storage version 4. Each run decides which buckets need chunk merging or full compaction, rather than fully compacting every bucket each hour.
 
-Run the compact job using the docker command `compact`, for example `docker run powersync compact`.
+Compaction resources depend on bucket size, accumulated changes, and concurrency. Monitor completion, duration, CPU, and peak memory to size the job for your dataset. If jobs fail or overlap, check the compact job's CPU and memory usage and bucket storage performance before changing resources or the schedule. See [Compacting Buckets](/maintenance-ops/compacting-buckets#self-hosted-powersync).
 
-The compact job uses up to 1GB memory for compacting, if available. Set the environment variable `NODE_OPTIONS=--max-old-space-size-percentage=80` to allocate 80% of the container's assigned memory to the Node.js heap. This avoids manually retuning when scaling resources up or down.
+### Memory Configuration
 
-### Load Balancer
+On PowerSync Service v1.26.1, start with `NODE_OPTIONS=--max-old-space-size-percentage=80` for API, replication, and compact containers. This caps the V8 old-generation heap, not total process memory. Monitor total container memory under load and lower the percentage if needed to leave room for non-heap allocations.
 
-A load balancer is required in front of the API containers to provide TLS support and load balancing. Most cloud providers have built-in options for load balancing, such as ALB on AWS.
+### Load Balancer
 
-It is currently required to host the API container on a dedicated subdomain - we do not support running it on the same subdomain as another service.
+Use a redundant load balancer to terminate TLS and distribute connections across API containers. Host the API on a dedicated subdomain.
 
-For self-hosting, [nginx](https://nginx.org/en/) is always a good option. A basic nginx configuration could look like this:
+For HTTP streaming with [nginx](https://nginx.org/en/), add this proxy block to your TLS server configuration. Replace `powersync:8080` with your PowerSync API hostname and port. Disabling response buffering lets sync data reach clients as it arrives.
 
-```yaml
-server {
-    listen 443 ssl;
-    server_name powersync.example.org;
-
-    # SSL configuration here
-
-    # Reverse proxy settings
-    location / {
-        proxy_pass http://powersync_server_ip:powersync_port;  # Replace with your powersync details
-        proxy_set_header Host $host;
-        proxy_set_header X-Real-IP $remote_addr;
-        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-
-        # Disable proxy response buffering.
-        # This is not relevant for websocket connections, but is important when using
-        # HTTP streaming connections (configured in the PowerSync Client SDK).
-        proxy_buffering off;
-    }
+```nginx
+location / {
+    proxy_pass http://powersync:8080;
+    proxy_set_header Host $host;
+    proxy_set_header X-Real-IP $remote_addr;
+    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+    proxy_buffering off;
 }
 ```
 
-When using nginx as a Kubernetes ingress, set the proxy buffering option as an annotation on the ingress:
+For nginx Kubernetes ingress, use this annotation:
 
 ```yaml
 nginx.ingress.kubernetes.io/proxy-buffering: "off"
 ```
 
 ### Health Checks
 
-If the load balancer supports health checks, it may be configured to poll the API container at `/probes/liveness`. This endpoint is expected to have a 200 response when the container is healthy. See [Health Checks](/maintenance-ops/self-hosting/healthchecks) for details.
+Configure load-balancer health checks to poll `/probes/liveness`. See [Health Checks](/maintenance-ops/self-hosting/healthchecks).
 
 ### Migrations
 
-Occasionally, new versions of the PowerSync Service image may require migrations on the underlying storage database. This is also specifically required the first time the service starts up on a new storage database.
-
-By default, migrations are run as part of the replication and API containers. In some cases, a migration may add significant delay to the container startup.
-
-To avoid this startup delay, the migrations may be run as a separate job on each update, before replacing the rest of the containers. To run the migrations, run the docker command `migrate up`, for example `docker run powersync migrate up`.
-
-In this case, disable automatic migrations in the config:
-
-```yaml
-# service.yaml
+API and replication containers run storage migrations on startup. Large migrations can delay startup during an update. To run migrations as a separate job before replacing the containers, disable automatic migrations:
 
+```yaml service.yaml
 migrations:
-    # Setting this to false (default) enables automatic migrations on startup.
-    # When set to true, migrations must be triggered manually by modifying the container `command`.
-    disable_auto_migration: true
+  disable_auto_migration: true
 ```
 
-Note that if you disable automatic migrations, and do not run the migration job manually, the service may run with an outdated storage schema version. This may lead to unexpected and potentially difficult-to-debug errors in the service.
+Then run `migrate up` as a separate job before each deployment, including the first.
 
 ## Backups
 
-We recommend using Git to back up your configuration files.
+Keep non-secret configuration in version control and credentials in a secrets manager. PowerSync containers do not persist sync data locally.
 
-None of the containers use any local storage, so no backups are required there.
-
-The bucket storage database may be backed up using the recommendations for the storage database system. This is not a strong requirement, since this data can be recovered by re-replicating from the source database.
+Recover PowerSync's sync data by re-replicating from the source database. Allow for the time and capacity needed for full replication and for clients to download their data again.
 
 ## Self-Hosted Architecture Diagram
 
+The diagram omits S3 object storage.
+
 <Frame caption="PowerSync Deployment Architecture">
   <img
     src="/images/architecture/powersync-architecture-diagram-self-host.png"
+    alt="Source database changes flow through replication and bucket storage to API containers, a TLS load balancer, and clients. Client writes go through the app backend."
   />
 </Frame>

maintenance-ops/self-hosting/multiple-instances.mdx

@@ -9,17 +9,17 @@ description: "Run multiple self-hosted PowerSync instances for load distribution
   Multiple instances are not required in most cases. See the [Deployment Architecture](/maintenance-ops/self-hosting/deployment-architecture) for details on standard horizontal scaling setups.
 </Warning>
 
-When exceeding a couple thousand concurrent clients, the standard PowerSync setup may not scale sufficiently to handle the load. In this case, we recommend you [contact us](/resources/contact-us) to discuss the options. However, we give a basic overview of using multiple PowerSync instances to scale here.
+Scale API containers within one instance before introducing multiple instances. There is no single connection-count threshold for splitting an instance: capacity depends on the sync workload, replication throughput, and bucket storage performance. Use the [capacity and scaling guidance](/maintenance-ops/self-hosting/deployment-architecture#connection-capacity-and-scaling) to evaluate your deployment, and [contact us](/resources/contact-us) if you need help planning beyond it.
 
 Each PowerSync "instance" is a single endpoint (URL), that is backed by:
 
-1. One replication container.
+1. One active replication process.
 2. Multiple API containers, scaling horizontally.
 3. One bucket storage database.
 
 This setup is described in the [Deployment Architecture](/maintenance-ops/self-hosting/deployment-architecture).
 
-To scale further, multiple copies of this setup can be run, using the same source database.
+To scale further, you can run multiple copies of this setup against the same source database. Each instance needs its own bucket storage database and, if enabled, an isolated [S3 bucket or prefix](/sync/advanced/storage-version-4#self-hosted-s3-setup).
 
 ## Mapping Users to PowerSync Endpoints