From ea4befdeb213645c6b246ea829dfd82bdcfb8522 Mon Sep 17 00:00:00 2001 From: joshuabrink Date: Thu, 8 Oct 2026 15:09:20 +0200 Subject: [PATCH 1/6] Update self-hosted sizing for storage v4 and S3 --- .../self-hosting/deployment-architecture.mdx | 121 +++++++----------- .../self-hosting/multiple-instances.mdx | 6 +- 2 files changed, 49 insertions(+), 78 deletions(-) diff --git a/maintenance-ops/self-hosting/deployment-architecture.mdx b/maintenance-ops/self-hosting/deployment-architecture.mdx index baca90508..af32a49f6 100644 --- a/maintenance-ops/self-hosting/deployment-architecture.mdx +++ b/maintenance-ops/self-hosting/deployment-architecture.mdx @@ -6,97 +6,75 @@ sidebarTitle: "Deployment Architecture" ## Minimal Setup -A minimal "development" setup (e.g. for a staging or a QA environment) is: +For development and testing, use: -1. A single PowerSync "compute" container (API + replication) with 512MB memory, 1 vCPU. -2. A single MongoDB node in replica set mode, 2GB memory, 1 vCPU. M10+ when using Atlas. -3. Load balancer for TLS. +1. One PowerSync container running API and replication: 1 vCPU, 512 MB memory. +2. One MongoDB node in replica set mode: 1 vCPU, 2 GB memory (M10+ on Atlas). +3. A load balancer for TLS. -This setup has no redundancy. If the replica set fails, you may need to recreate it from scratch which will re-sync all clients. +This setup has no redundancy. ## Production -For production, we recommend running a high-availability setup with the following baseline requirements: +Separate API and replication workloads so you can scale them independently. -1. 1x PowerSync replication container, 1GB memory, 1 vCPU -2. 2+ PowerSync API containers, 1GB memory each, 1vCPU each. -3. A 3-node MongoDB replica set, 2+GB memory each. Refer to the MongoDB documentation for deployment requirements. M10+ when using Atlas. -4. A load balancer with redundancy. -5. Run a daily compact job. +### Storage Requirements -For further robustness, in cases where you have larger rows and higher load, we recommend increasing the memory and CPU requirements to 2GB memory and 2 vCPU. This applies to both the replication and API containers. +The sizing below uses ARM64 compute, PowerSync Service v1.26.0 or later, and [storage version 4 with S3](/sync/advanced/storage-version-4#self-hosted-s3-setup). Use a three-node MongoDB replica set for bucket storage. -When scaling up, add 1x PowerSync API container per 100 connections. The MongoDB replica set should be scaled based on CPU and memory usage. +Apply the same S3 configuration and permissions to replication, API, and compact jobs. -### Replication Container +### Compute Sizing -The replication container handles replicating from the source database to PowerSync's bucket storage. +Use these tested allocations as a starting point: -The replication process is run using the docker command `start -r sync`, for example `docker run powersync start -r sync`. +| Role | vCPU | Memory | Deployment | +| --- | --- | --- | --- | +| API | 1 | 2 GiB | At least two containers; scale horizontally | +| Replication | 2 | 4 GiB | One active replication process | +| Compaction | 2 | 4 GiB | Separate scheduled job | -Only one process can replicate at a time. If multiple are running concurrently, you may see an error `[PSYNC_S1003] Sync rules have been locked by another process for replication`. If you use rolling deploys, it is normal to see this error for a short duration while multiple processes are running. +Allocate additional CPU and memory for any sidecars. -Memory and CPU usage of the replication container is primarily driven by write load on the source database. A good starting point is 1GB memory and 1 vCPU for the container, but this may be scaled down depending on the load patterns. +### Connection Capacity and Scaling -Set the environment variable `NODE_OPTIONS=--max-old-space-size-percentage=80` to allocate 80% of the container's assigned memory to the Node.js heap. This avoids manually retuning when scaling resources up or down. +This profile supports up to 1,000 concurrent connections per API process. Larger datasets, more buckets per client, and simultaneous initial syncs can require fewer connections per process. -### API Containers +After load testing, raise [`max_concurrent_connections`](/configuration/powersync-service/self-hosted-instances#param-max-concurrent-connections) from its default of `200` up to `1000`. The setting limits HTTP streams and WebSockets separately; monitor their combined count. -The API container handles streaming sync connections, as well as any other API calls. +Scale on both CPU utilization and connections per process. Keep autoscaling targets below the connection limit to allow for bursts while new containers start. -The replication process is run using the docker command `start -r api`, for example `docker run powersync start -r api`. +Add API containers before large initial-sync waves or deployments that make clients download their data again. Wait for the new containers to become healthy before starting the workload. -Each API container is limited to 200 concurrent connections, but we recommend targeting 100 concurrent connections or less per container. This may change as we implement additional performance optimizations. +### Replication Container -Memory and CPU usage of API containers are driven by: +Run `start -r sync` to replicate source changes into bucket storage. Additional replication containers provide failover, not more throughput. -1. Number of concurrent connections. -2. Number of buckets per connection. -3. Amount of data synced to each connection. +Size replication for source write volume, document size, and buckets per row. Use [replication lag](/maintenance-ops/replication-lag) and compute and database metrics to assess capacity. -A good starting point is 1GB memory and 1 vCPU per container, but this may be scaled up or down depending on the specific load patterns. +### API Containers -Set the environment variable `NODE_OPTIONS=--max-old-space-size-percentage=80` to allocate 80% of the container's assigned memory to the Node.js heap. This avoids manually retuning when scaling resources up or down. +Run `start -r api` for client connections. Collect [metrics](/maintenance-ops/self-hosting/monitoring) separately from API and replication containers. ### Compact Job -We recommend running a compact job daily as a cron job, or after any large maintenance jobs. For details, see the documentation on [Compacting Buckets](/maintenance-ops/compacting-buckets). +Run `compact` hourly for storage version 4. Monitor completion, duration, and peak memory; avoid overlapping runs. See [Compacting Buckets](/maintenance-ops/compacting-buckets#self-hosted-powersync). -Run the compact job using the docker command `compact`, for example `docker run powersync compact`. +### Memory Configuration -The compact job uses up to 1GB memory for compacting, if available. Set the environment variable `NODE_OPTIONS=--max-old-space-size-percentage=80` to allocate 80% of the container's assigned memory to the Node.js heap. This avoids manually retuning when scaling resources up or down. +On Service `1.26.1`, start with `NODE_OPTIONS=--max-old-space-size-percentage=80` for all three roles. Adjust this heap limit using total container memory usage, which also includes non-heap allocations. ### Load Balancer -A load balancer is required in front of the API containers to provide TLS support and load balancing. Most cloud providers have built-in options for load balancing, such as ALB on AWS. - -It is currently required to host the API container on a dedicated subdomain - we do not support running it on the same subdomain as another service. +Use a redundant load balancer to terminate TLS and distribute connections across API containers. Host the API on a dedicated subdomain. -For self-hosting, [nginx](https://nginx.org/en/) is always a good option. A basic nginx configuration could look like this: +For [nginx](https://nginx.org/en/), disable response buffering in the proxy `location` block for HTTP streaming: -```yaml -server { - listen 443 ssl; - server_name powersync.example.org; - - # SSL configuration here - - # Reverse proxy settings - location / { - proxy_pass http://powersync_server_ip:powersync_port; # Replace with your powersync details - proxy_set_header Host $host; - proxy_set_header X-Real-IP $remote_addr; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - - # Disable proxy response buffering. - # This is not relevant for websocket connections, but is important when using - # HTTP streaming connections (configured in the PowerSync Client SDK). - proxy_buffering off; - } -} +```nginx +proxy_buffering off; ``` -When using nginx as a Kubernetes ingress, set the proxy buffering option as an annotation on the ingress: +For nginx Kubernetes ingress, use this annotation: ```yaml nginx.ingress.kubernetes.io/proxy-buffering: "off" @@ -104,41 +82,34 @@ nginx.ingress.kubernetes.io/proxy-buffering: "off" ### Health Checks -If the load balancer supports health checks, it may be configured to poll the API container at `/probes/liveness`. This endpoint is expected to have a 200 response when the container is healthy. See [Health Checks](/maintenance-ops/self-hosting/healthchecks) for details. +Configure load-balancer health checks to poll `/probes/liveness`. See [Health Checks](/maintenance-ops/self-hosting/healthchecks). ### Migrations -Occasionally, new versions of the PowerSync Service image may require migrations on the underlying storage database. This is also specifically required the first time the service starts up on a new storage database. - -By default, migrations are run as part of the replication and API containers. In some cases, a migration may add significant delay to the container startup. - -To avoid this startup delay, the migrations may be run as a separate job on each update, before replacing the rest of the containers. To run the migrations, run the docker command `migrate up`, for example `docker run powersync migrate up`. - -In this case, disable automatic migrations in the config: - -```yaml -# service.yaml +API and replication containers run storage migrations on startup. To move this work to a separate deployment job, disable automatic migrations: +```yaml service.yaml migrations: - # Setting this to false (default) enables automatic migrations on startup. - # When set to true, migrations must be triggered manually by modifying the container `command`. - disable_auto_migration: true + disable_auto_migration: true ``` -Note that if you disable automatic migrations, and do not run the migration job manually, the service may run with an outdated storage schema version. This may lead to unexpected and potentially difficult-to-debug errors in the service. +Then run `migrate up` as a separate job before each deployment, including the first. ## Backups -We recommend using Git to back up your configuration files. +Keep non-secret configuration in version control and credentials in a secrets manager. -None of the containers use any local storage, so no backups are required there. +With S3 enabled, plan recovery for both MongoDB and S3. Follow [S3 retention requirements](/sync/advanced/storage-version-4#self-hosted-s3-setup) rather than expiring objects yourself. -The bucket storage database may be backed up using the recommendations for the storage database system. This is not a strong requirement, since this data can be recovered by re-replicating from the source database. +PowerSync can also rebuild sync data from the source database, but this requires full replication and client re-downloads. ## Self-Hosted Architecture Diagram +The diagram omits S3 object storage. + Source database changes flow through replication and bucket storage to API containers, a TLS load balancer, and clients. Client writes go through the app backend. diff --git a/maintenance-ops/self-hosting/multiple-instances.mdx b/maintenance-ops/self-hosting/multiple-instances.mdx index cbb59fe52..472a115a2 100644 --- a/maintenance-ops/self-hosting/multiple-instances.mdx +++ b/maintenance-ops/self-hosting/multiple-instances.mdx @@ -9,17 +9,17 @@ description: "Run multiple self-hosted PowerSync instances for load distribution Multiple instances are not required in most cases. See the [Deployment Architecture](/maintenance-ops/self-hosting/deployment-architecture) for details on standard horizontal scaling setups. -When exceeding a couple thousand concurrent clients, the standard PowerSync setup may not scale sufficiently to handle the load. In this case, we recommend you [contact us](/resources/contact-us) to discuss the options. However, we give a basic overview of using multiple PowerSync instances to scale here. +Scale API containers within one instance before introducing multiple instances. There is no single connection-count threshold for splitting an instance: capacity depends on the sync workload, replication throughput, and bucket storage performance. Use the [capacity and scaling guidance](/maintenance-ops/self-hosting/deployment-architecture#connection-capacity-and-scaling) to evaluate your deployment, and [contact us](/resources/contact-us) if you need help planning beyond it. Each PowerSync "instance" is a single endpoint (URL), that is backed by: -1. One replication container. +1. One active replication process. 2. Multiple API containers, scaling horizontally. 3. One bucket storage database. This setup is described in the [Deployment Architecture](/maintenance-ops/self-hosting/deployment-architecture). -To scale further, multiple copies of this setup can be run, using the same source database. +To scale further, you can run multiple copies of this setup against the same source database. Each instance needs its own bucket storage database and, if enabled, an isolated [S3 bucket or prefix](/sync/advanced/storage-version-4#self-hosted-s3-setup). ## Mapping Users to PowerSync Endpoints From 90a898764825e51d69451b9166e25cc18489da3e Mon Sep 17 00:00:00 2001 From: joshuabrink Date: Thu, 8 Oct 2026 15:19:38 +0200 Subject: [PATCH 2/6] docs: restore deployment sizing and operations context --- .../self-hosting/deployment-architecture.mdx | 36 ++++++++++++------- 1 file changed, 24 insertions(+), 12 deletions(-) diff --git a/maintenance-ops/self-hosting/deployment-architecture.mdx b/maintenance-ops/self-hosting/deployment-architecture.mdx index af32a49f6..93488644b 100644 --- a/maintenance-ops/self-hosting/deployment-architecture.mdx +++ b/maintenance-ops/self-hosting/deployment-architecture.mdx @@ -22,7 +22,9 @@ Separate API and replication workloads so you can scale them independently. The sizing below uses ARM64 compute, PowerSync Service v1.26.0 or later, and [storage version 4 with S3](/sync/advanced/storage-version-4#self-hosted-s3-setup). Use a three-node MongoDB replica set for bucket storage. -Apply the same S3 configuration and permissions to replication, API, and compact jobs. +S3 stores larger blocks of sync data; MongoDB retains smaller blocks and metadata. The MongoDB requirement applies to bucket storage, not the source database. + +Apply the same S3 configuration and permissions to replication, API, and compact jobs. Clients connect to the API, which reads from both stores. ### Compute Sizing @@ -42,15 +44,17 @@ This profile supports up to 1,000 concurrent connections per API process. Larger After load testing, raise [`max_concurrent_connections`](/configuration/powersync-service/self-hosted-instances#param-max-concurrent-connections) from its default of `200` up to `1000`. The setting limits HTTP streams and WebSockets separately; monitor their combined count. -Scale on both CPU utilization and connections per process. Keep autoscaling targets below the connection limit to allow for bursts while new containers start. +Scale on both CPU utilization and connections per process. Idle connections can keep CPU low, while initial syncs can saturate CPU with fewer connections. Keep autoscaling targets below the connection limit to allow for bursts while new containers start. -Add API containers before large initial-sync waves or deployments that make clients download their data again. Wait for the new containers to become healthy before starting the workload. +Add API containers before large initial-sync waves or deployments that make clients download their data again. Wait for the new containers to become healthy before starting the workload. Adding containers does not immediately redistribute existing long-lived connections. Reduce capacity gradually after the download load subsides. ### Replication Container Run `start -r sync` to replicate source changes into bucket storage. Additional replication containers provide failover, not more throughput. -Size replication for source write volume, document size, and buckets per row. Use [replication lag](/maintenance-ops/replication-lag) and compute and database metrics to assess capacity. +Size replication for source write volume, document size, and buckets per row. A row in ten buckets creates more replication work than a row in one bucket. Initial replication and catch-up after a large batch update can need more capacity than steady-state replication. + +Use [replication lag](/maintenance-ops/replication-lag), replication throughput, CPU, memory, and database metrics to assess capacity. ### API Containers @@ -58,20 +62,28 @@ Run `start -r api` for client connections. Collect [metrics](/maintenance-ops/se ### Compact Job -Run `compact` hourly for storage version 4. Monitor completion, duration, and peak memory; avoid overlapping runs. See [Compacting Buckets](/maintenance-ops/compacting-buckets#self-hosted-powersync). +Run `compact` hourly for storage version 4. Each run decides which buckets need chunk merging or full compaction, rather than fully compacting every bucket each hour. + +Compaction resources depend on bucket size, accumulated changes, and concurrency. Monitor completion, duration, and peak memory to establish a baseline for your dataset. If jobs fail or overlap, check compute and bucket storage performance before changing resources or the schedule. See [Compacting Buckets](/maintenance-ops/compacting-buckets#self-hosted-powersync). ### Memory Configuration -On Service `1.26.1`, start with `NODE_OPTIONS=--max-old-space-size-percentage=80` for all three roles. Adjust this heap limit using total container memory usage, which also includes non-heap allocations. +On Service `1.26.1`, start with `NODE_OPTIONS=--max-old-space-size-percentage=80` for API, replication, and compact containers. This caps the V8 old-generation heap, not total process memory. Monitor total container memory under load and lower the percentage if needed to leave room for non-heap allocations. ### Load Balancer Use a redundant load balancer to terminate TLS and distribute connections across API containers. Host the API on a dedicated subdomain. -For [nginx](https://nginx.org/en/), disable response buffering in the proxy `location` block for HTTP streaming: +For HTTP streaming with [nginx](https://nginx.org/en/), add this proxy block to your TLS server configuration. Replace `powersync:8080` with your API upstream. Disabling response buffering lets sync data reach clients as it arrives. ```nginx -proxy_buffering off; +location / { + proxy_pass http://powersync:8080; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_buffering off; +} ``` For nginx Kubernetes ingress, use this annotation: @@ -86,7 +98,7 @@ Configure load-balancer health checks to poll `/probes/liveness`. See [Health Ch ### Migrations -API and replication containers run storage migrations on startup. To move this work to a separate deployment job, disable automatic migrations: +API and replication containers run storage migrations on startup. Large migrations can delay startup during an update. To run migrations as a separate job before replacing the containers, disable automatic migrations: ```yaml service.yaml migrations: @@ -97,11 +109,11 @@ Then run `migrate up` as a separate job before each deployment, including the fi ## Backups -Keep non-secret configuration in version control and credentials in a secrets manager. +Keep non-secret configuration in version control and credentials in a secrets manager. PowerSync containers do not persist sync data locally. -With S3 enabled, plan recovery for both MongoDB and S3. Follow [S3 retention requirements](/sync/advanced/storage-version-4#self-hosted-s3-setup) rather than expiring objects yourself. +With S3 enabled, a MongoDB backup alone does not include the objects it references. Plan recovery for both stores and follow [S3 retention requirements](/sync/advanced/storage-version-4#self-hosted-s3-setup) rather than expiring objects yourself. -PowerSync can also rebuild sync data from the source database, but this requires full replication and client re-downloads. +PowerSync can also rebuild sync data from the source database. Allow for the time and capacity needed for full replication and for clients to download their data again. ## Self-Hosted Architecture Diagram From 413dc0a9305c99bf0de5d269a52ed4dbf559b5d7 Mon Sep 17 00:00:00 2001 From: joshuabrink Date: Thu, 8 Oct 2026 15:41:21 +0200 Subject: [PATCH 3/6] Size compaction from workload measurements --- maintenance-ops/self-hosting/deployment-architecture.mdx | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/maintenance-ops/self-hosting/deployment-architecture.mdx b/maintenance-ops/self-hosting/deployment-architecture.mdx index 93488644b..bc34b704e 100644 --- a/maintenance-ops/self-hosting/deployment-architecture.mdx +++ b/maintenance-ops/self-hosting/deployment-architecture.mdx @@ -34,7 +34,6 @@ Use these tested allocations as a starting point: | --- | --- | --- | --- | | API | 1 | 2 GiB | At least two containers; scale horizontally | | Replication | 2 | 4 GiB | One active replication process | -| Compaction | 2 | 4 GiB | Separate scheduled job | Allocate additional CPU and memory for any sidecars. @@ -62,9 +61,9 @@ Run `start -r api` for client connections. Collect [metrics](/maintenance-ops/se ### Compact Job -Run `compact` hourly for storage version 4. Each run decides which buckets need chunk merging or full compaction, rather than fully compacting every bucket each hour. +Run `compact` as a separate scheduled job, hourly for storage version 4. Each run decides which buckets need chunk merging or full compaction, rather than fully compacting every bucket each hour. -Compaction resources depend on bucket size, accumulated changes, and concurrency. Monitor completion, duration, and peak memory to establish a baseline for your dataset. If jobs fail or overlap, check compute and bucket storage performance before changing resources or the schedule. See [Compacting Buckets](/maintenance-ops/compacting-buckets#self-hosted-powersync). +Compaction resources depend on bucket size, accumulated changes, and concurrency. Monitor completion, duration, CPU, and peak memory to size the job for your dataset. If jobs fail or overlap, check compute and bucket storage performance before changing resources or the schedule. See [Compacting Buckets](/maintenance-ops/compacting-buckets#self-hosted-powersync). ### Memory Configuration From 897d7a5f850800ad07fea7e13d5db448160bc47b Mon Sep 17 00:00:00 2001 From: joshuabrink Date: Mon, 12 Oct 2026 11:39:57 +0200 Subject: [PATCH 4/6] Clarify bucket storage support and recovery --- .../self-hosting/deployment-architecture.mdx | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/maintenance-ops/self-hosting/deployment-architecture.mdx b/maintenance-ops/self-hosting/deployment-architecture.mdx index bc34b704e..720c55193 100644 --- a/maintenance-ops/self-hosting/deployment-architecture.mdx +++ b/maintenance-ops/self-hosting/deployment-architecture.mdx @@ -9,7 +9,7 @@ sidebarTitle: "Deployment Architecture" For development and testing, use: 1. One PowerSync container running API and replication: 1 vCPU, 512 MB memory. -2. One MongoDB node in replica set mode: 1 vCPU, 2 GB memory (M10+ on Atlas). +2. A MongoDB or Postgres [bucket storage](#storage-requirements) database. For MongoDB, use one node in replica set mode: 1 vCPU, 2 GB memory (M10+ on Atlas). 3. A load balancer for TLS. This setup has no redundancy. @@ -20,15 +20,16 @@ Separate API and replication workloads so you can scale them independently. ### Storage Requirements -The sizing below uses ARM64 compute, PowerSync Service v1.26.0 or later, and [storage version 4 with S3](/sync/advanced/storage-version-4#self-hosted-s3-setup). Use a three-node MongoDB replica set for bucket storage. +You can use MongoDB or Postgres for bucket storage, independently of your source database: -S3 stores larger blocks of sync data; MongoDB retains smaller blocks and metadata. The MongoDB requirement applies to bucket storage, not the source database. +- [MongoDB](/configuration/powersync-service/self-hosted-instances#mongodb-storage) supports storage version 4 and optional S3 object storage. Use a three-node replica set for production. +- [Postgres](/configuration/powersync-service/self-hosted-instances#postgres-storage) uses storage version 2 and does not support storage version 4 or S3 object storage. -Apply the same S3 configuration and permissions to replication, API, and compact jobs. Clients connect to the API, which reads from both stores. +With [S3 object storage](/sync/advanced/storage-version-4#self-hosted-s3-setup), S3 stores larger blocks of sync data; MongoDB retains smaller blocks and metadata. Apply the same S3 configuration and permissions to replication, API, and compact jobs. Clients connect to the API, which reads from both stores. ### Compute Sizing -Use these tested allocations as a starting point: +The tested profile uses ARM64 compute, MongoDB bucket storage, and storage version 4 with S3, available since PowerSync Service v1.26.0. Use these allocations as a starting point: | Role | vCPU | Memory | Deployment | | --- | --- | --- | --- | @@ -110,9 +111,7 @@ Then run `migrate up` as a separate job before each deployment, including the fi Keep non-secret configuration in version control and credentials in a secrets manager. PowerSync containers do not persist sync data locally. -With S3 enabled, a MongoDB backup alone does not include the objects it references. Plan recovery for both stores and follow [S3 retention requirements](/sync/advanced/storage-version-4#self-hosted-s3-setup) rather than expiring objects yourself. - -PowerSync can also rebuild sync data from the source database. Allow for the time and capacity needed for full replication and for clients to download their data again. +Recover PowerSync's sync data by re-replicating from the source database. Allow for the time and capacity needed for full replication and for clients to download their data again. ## Self-Hosted Architecture Diagram From e13102d260576e163cb75745f77acc465e0977d2 Mon Sep 17 00:00:00 2001 From: joshuabrink Date: Mon, 12 Oct 2026 13:35:26 +0200 Subject: [PATCH 5/6] Improve wording --- .../self-hosting/deployment-architecture.mdx | 34 +++++++++---------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/maintenance-ops/self-hosting/deployment-architecture.mdx b/maintenance-ops/self-hosting/deployment-architecture.mdx index 720c55193..fea124ade 100644 --- a/maintenance-ops/self-hosting/deployment-architecture.mdx +++ b/maintenance-ops/self-hosting/deployment-architecture.mdx @@ -1,22 +1,22 @@ --- title: "Deployment Architecture" -description: "Infrastructure and sizing recommendations for self-hosted PowerSync in staging and production." +description: "Infrastructure and sizing recommendations for self-hosted PowerSync." sidebarTitle: "Deployment Architecture" --- ## Minimal Setup +For a runnable development setup, use the [self-hosting examples](https://github.com/powersync-ja/self-host-demo), which include Docker Compose files and configuration. + For development and testing, use: 1. One PowerSync container running API and replication: 1 vCPU, 512 MB memory. 2. A MongoDB or Postgres [bucket storage](#storage-requirements) database. For MongoDB, use one node in replica set mode: 1 vCPU, 2 GB memory (M10+ on Atlas). 3. A load balancer for TLS. -This setup has no redundancy. - ## Production -Separate API and replication workloads so you can scale them independently. +Run API and replication in separate containers so you can scale them independently. ### Storage Requirements @@ -25,46 +25,46 @@ You can use MongoDB or Postgres for bucket storage, independently of your source - [MongoDB](/configuration/powersync-service/self-hosted-instances#mongodb-storage) supports storage version 4 and optional S3 object storage. Use a three-node replica set for production. - [Postgres](/configuration/powersync-service/self-hosted-instances#postgres-storage) uses storage version 2 and does not support storage version 4 or S3 object storage. -With [S3 object storage](/sync/advanced/storage-version-4#self-hosted-s3-setup), S3 stores larger blocks of sync data; MongoDB retains smaller blocks and metadata. Apply the same S3 configuration and permissions to replication, API, and compact jobs. Clients connect to the API, which reads from both stores. +With [S3 object storage](/sync/advanced/storage-version-4#self-hosted-s3-setup), S3 stores larger blocks of sync data while MongoDB retains smaller blocks and metadata. ### Compute Sizing -The tested profile uses ARM64 compute, MongoDB bucket storage, and storage version 4 with S3, available since PowerSync Service v1.26.0. Use these allocations as a starting point: +The tested configuration uses ARM64 compute, MongoDB bucket storage, and storage version 4 with S3, available since PowerSync Service v1.26.0. Use these CPU and memory settings as a starting point: -| Role | vCPU | Memory | Deployment | -| --- | --- | --- | --- | -| API | 1 | 2 GiB | At least two containers; scale horizontally | -| Replication | 2 | 4 GiB | One active replication process | - -Allocate additional CPU and memory for any sidecars. +| Role | vCPU | Memory | +| --- | --- | --- | +| API | 1 | 2 GiB | +| Replication | 2 | 4 GiB | ### Connection Capacity and Scaling -This profile supports up to 1,000 concurrent connections per API process. Larger datasets, more buckets per client, and simultaneous initial syncs can require fewer connections per process. +This configuration supports up to **1,000 concurrent connections per API process**. Larger datasets, more buckets per client, and simultaneous initial syncs can require fewer connections per process. After load testing, raise [`max_concurrent_connections`](/configuration/powersync-service/self-hosted-instances#param-max-concurrent-connections) from its default of `200` up to `1000`. The setting limits HTTP streams and WebSockets separately; monitor their combined count. Scale on both CPU utilization and connections per process. Idle connections can keep CPU low, while initial syncs can saturate CPU with fewer connections. Keep autoscaling targets below the connection limit to allow for bursts while new containers start. -Add API containers before large initial-sync waves or deployments that make clients download their data again. Wait for the new containers to become healthy before starting the workload. Adding containers does not immediately redistribute existing long-lived connections. Reduce capacity gradually after the download load subsides. +Add API containers before many clients run an initial sync at once or before deployments that make clients download their data again. Wait for the new containers to become healthy before those initial syncs or deployments begin. Adding containers does not immediately redistribute existing long-lived connections. Reduce capacity gradually after the download load subsides. ### Replication Container Run `start -r sync` to replicate source changes into bucket storage. Additional replication containers provide failover, not more throughput. -Size replication for source write volume, document size, and buckets per row. A row in ten buckets creates more replication work than a row in one bucket. Initial replication and catch-up after a large batch update can need more capacity than steady-state replication. +Size replication for source write volume, row size, and buckets per row. A row in ten buckets creates more replication work than a row in one bucket. Initial replication and catch-up after a large batch update can need more capacity than steady-state replication. Use [replication lag](/maintenance-ops/replication-lag), replication throughput, CPU, memory, and database metrics to assess capacity. ### API Containers +Run at least two API containers in production for availability. Scale horizontally by adding containers as client demand grows. + Run `start -r api` for client connections. Collect [metrics](/maintenance-ops/self-hosting/monitoring) separately from API and replication containers. ### Compact Job Run `compact` as a separate scheduled job, hourly for storage version 4. Each run decides which buckets need chunk merging or full compaction, rather than fully compacting every bucket each hour. -Compaction resources depend on bucket size, accumulated changes, and concurrency. Monitor completion, duration, CPU, and peak memory to size the job for your dataset. If jobs fail or overlap, check compute and bucket storage performance before changing resources or the schedule. See [Compacting Buckets](/maintenance-ops/compacting-buckets#self-hosted-powersync). +Compaction resources depend on bucket size, accumulated changes, and concurrency. Monitor completion, duration, CPU, and peak memory to size the job for your dataset. If jobs fail or overlap, check the compact job's CPU and memory usage and bucket storage performance before changing resources or the schedule. See [Compacting Buckets](/maintenance-ops/compacting-buckets#self-hosted-powersync). ### Memory Configuration @@ -74,7 +74,7 @@ On Service `1.26.1`, start with `NODE_OPTIONS=--max-old-space-size-percentage=80 Use a redundant load balancer to terminate TLS and distribute connections across API containers. Host the API on a dedicated subdomain. -For HTTP streaming with [nginx](https://nginx.org/en/), add this proxy block to your TLS server configuration. Replace `powersync:8080` with your API upstream. Disabling response buffering lets sync data reach clients as it arrives. +For HTTP streaming with [nginx](https://nginx.org/en/), add this proxy block to your TLS server configuration. Replace `powersync:8080` with your PowerSync API hostname and port. Disabling response buffering lets sync data reach clients as it arrives. ```nginx location / { From b4babc36731906210a724e4f101f3d7321f65e73 Mon Sep 17 00:00:00 2001 From: joshuabrink Date: Mon, 12 Oct 2026 15:39:43 +0200 Subject: [PATCH 6/6] Address deployment architecture review feedback --- .../self-hosting/deployment-architecture.mdx | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/maintenance-ops/self-hosting/deployment-architecture.mdx b/maintenance-ops/self-hosting/deployment-architecture.mdx index fea124ade..3cbad278d 100644 --- a/maintenance-ops/self-hosting/deployment-architecture.mdx +++ b/maintenance-ops/self-hosting/deployment-architecture.mdx @@ -10,10 +10,12 @@ For a runnable development setup, use the [self-hosting examples](https://github For development and testing, use: -1. One PowerSync container running API and replication: 1 vCPU, 512 MB memory. -2. A MongoDB or Postgres [bucket storage](#storage-requirements) database. For MongoDB, use one node in replica set mode: 1 vCPU, 2 GB memory (M10+ on Atlas). +1. One PowerSync container running API and replication: 1 vCPU, 512 MiB memory. +2. A MongoDB or Postgres [bucket storage](#storage-requirements) database. For MongoDB, use one node in replica set mode: 1 vCPU, 2 GiB memory (M10+ on Atlas). 3. A load balancer for TLS. +This setup has no redundancy. If bucket storage data is lost, [re-replicate from the source database](#backups). All clients must then download their data again. + ## Production Run API and replication in separate containers so you can scale them independently. @@ -50,6 +52,8 @@ Add API containers before many clients run an initial sync at once or before dep Run `start -r sync` to replicate source changes into bucket storage. Additional replication containers provide failover, not more throughput. +During rolling deploys or failover, [`PSYNC_S1003`](/debugging/error-codes#psync_s1xxx-replication-issues) is expected briefly while another process holds the replication lock. + Size replication for source write volume, row size, and buckets per row. A row in ten buckets creates more replication work than a row in one bucket. Initial replication and catch-up after a large batch update can need more capacity than steady-state replication. Use [replication lag](/maintenance-ops/replication-lag), replication throughput, CPU, memory, and database metrics to assess capacity. @@ -68,7 +72,7 @@ Compaction resources depend on bucket size, accumulated changes, and concurrency ### Memory Configuration -On Service `1.26.1`, start with `NODE_OPTIONS=--max-old-space-size-percentage=80` for API, replication, and compact containers. This caps the V8 old-generation heap, not total process memory. Monitor total container memory under load and lower the percentage if needed to leave room for non-heap allocations. +On PowerSync Service v1.26.1, start with `NODE_OPTIONS=--max-old-space-size-percentage=80` for API, replication, and compact containers. This caps the V8 old-generation heap, not total process memory. Monitor total container memory under load and lower the percentage if needed to leave room for non-heap allocations. ### Load Balancer