diff --git a/.github/vale/config/vocabularies/PowerSync/accept.txt b/.github/vale/config/vocabularies/PowerSync/accept.txt index 024557270..a23df6e3c 100644 --- a/.github/vale/config/vocabularies/PowerSync/accept.txt +++ b/.github/vale/config/vocabularies/PowerSync/accept.txt @@ -167,6 +167,8 @@ ECDSA E2EE ECS EdDSA +GTID +GTIDs HA HS256 IDE @@ -237,6 +239,7 @@ varchar # Products/Companies Auth0 +Aurora Avalonia Bavard Blazor diff --git a/configuration/source-db/connection.mdx b/configuration/source-db/connection.mdx index b0d895592..5f0336744 100644 --- a/configuration/source-db/connection.mdx +++ b/configuration/source-db/connection.mdx @@ -56,6 +56,31 @@ Select your Postgres hosting provider for steps to connect your newly-created Po If you get an error such as "IPs in this range are not supported", the instance is likely not configured to be publicly accessible. A DNS lookup on the host should give a public IP, and not for example `10.x.x.x` or `172.31.x.x`. + + 1. In the [PowerSync Dashboard](https://dashboard.powersync.com/), select your project and instance and go to the **Database Connection** view. + 2. Select the **Postgres** tab. + 3. Use the cluster endpoint as the "**Host**". AWS also calls it the writer endpoint: it always points at the cluster's primary instance, the writer. You can find it on the cluster's details page in the RDS console, or in the `Endpoint` field of `aws rds describe-db-clusters`. Do not use the reader endpoint, whose hostname contains `.cluster-ro-`, or an instance endpoint, whose hostname has no `.cluster-` segment. Aurora only supports logical replication from the writer, and the cluster endpoint follows the writer through a failover while an instance endpoint does not. + 4. Complete the remaining fields: + - "**Name**" can be any name for the connection. + - "**Port**" is 5432 for Postgres databases. + - "**Database name**" is the database in your cluster to replicate. + - "**Username**" and "**Password**" maps to the `powersync_role` created in [Source Database Setup](/configuration/source-db/setup#postgres). + - PowerSync has the AWS RDS CA certificates pre-configured, so the default `verify-full` SSL mode works without additional configuration. + 5. If the cluster is not publicly accessible, select your endpoint in the **Private Endpoint** dropdown. Keep the cluster endpoint as the "**Host**". PowerSync verifies the server certificate against it and sends the traffic through the Private Endpoint. With a Private Endpoint, **Test Connection** does not connect to the database, so confirm the connection after deploying as described in [Private Endpoints](/configuration/source-db/private-endpoints). + 6. Click **Test Connection** and fix any errors. + 7. Click **Save Connection**. + + PowerSync deploys and configures an isolated cloud environment for you, which can take a few minutes to complete. + + ### Troubleshooting + + If you get an error such as "IPs in this range are not supported", the host resolves to a private IP. Make the cluster publicly accessible, or connect through a [Private Endpoint](/configuration/source-db/private-endpoints). + + If Sync Config validation fails with `postgres query failed`, check the instance logs for the underlying error. + + If the logs show `Hostname/IP does not match certificate's altnames`, the server certificate does not cover the hostname in "**Host**". This can happen when "**Host**" is an instance endpoint after a failover, or when a Private Endpoint still forwards to a previous writer or cluster. Switch to the cluster endpoint, or update the target group behind the Private Endpoint. + + 1. In the [PowerSync Dashboard](https://dashboard.powersync.com/), select your project and instance and go to the **Database Connection** view. 2. Select the **Postgres** tab. @@ -212,37 +237,26 @@ To connect to **Azure DocumentDB** (formerly Azure Cosmos DB for MongoDB vCore), Select your MySQL hosting provider for steps to connect your newly-created PowerSync instance to your MySQL database: - To enable binary logging and GTID replication in AWS Aurora, you need to create a [DB Parameter Group](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Reference.ParameterGroups.html) and configure it with the necessary parameters. Follow these steps: - - 1. Navigate to [Amazon RDS console](https://console.aws.amazon.com/rds/) In the navigation pane, select **Parameter groups** and click **Create Parameter Group**: - - - - 2. Add all the required [binlog configuration](/configuration/source-db/setup#binlog-configuration) parameters. For example: - - - - 3. Associate your newly created parameter group with your Aurora cluster: - 1. In the navigation pane, select **Databases**. - 2. Select your Aurora cluster. - 3. Select **Modify**. - 4. In the **DB Parameter Group** section, select the parameter group you created. - 5. Click **Continue** and then **Apply** immediately. - 4. Whitelist PowerSync's IPs in your Aurora cluster's security group to allow access. See [Security and IP Filtering](/configuration/source-db/security-and-ip-filtering) for more details. - 5. In the [PowerSync Dashboard](https://dashboard.powersync.com/), select your project and instance and go to the **Database Connection** view. - 6. Select the **MySQL** tab. - 7. Fill in your MySQL connection details from AWS Aurora: - 1. "**Name**", "**Host**", "**Port**", "**Database name**", "**Username**", "**Password**" are required. - 2. "**Name**" can be any name for the connection. - 3. "**Host**" is the endpoint for your Aurora cluster. - 4. "**Database name**" is the default database to replicate. - 5. "**Username**" and "**Password**" maps to your database user. - 8. Click **Test Connection** and fix any errors. - 9. Click **Save Connection**. + Before connecting, complete [Aurora MySQL Setup](/configuration/source-db/setup#aws-aurora-mysql). + + + + In the [PowerSync Dashboard](https://dashboard.powersync.com/), select your project and instance and go to **Database Connection**. Select the **MySQL** tab. + + + Use the cluster (writer) endpoint as **Host**. Do not use the reader endpoint: Aurora MySQL binary logs are available only on the writer. The cluster endpoint follows the writer through a failover, while an instance endpoint does not. If the new writer has a different server UUID, PowerSync takes a new snapshot before resuming replication. + + Complete the remaining required fields: + + - **Name** can be any name for the connection. + - **Port** is your cluster's database port (`3306` by default). + - **Database name** is the default database to replicate. + - **Username** and **Password** are the credentials for the replication user created in [Database User Configuration](/configuration/source-db/setup#database-user-configuration). + + + Click **Test Connection** and fix any errors. Click **Save Connection**. + + PowerSync deploys and configures an isolated cloud environment for you, which can take a few minutes to complete. @@ -256,12 +270,12 @@ For other providers and self-hosted databases: 1. "**Name**", "**Host**", "**Port**", "**Database name**", "**Username**", "**Password**" are required. 2. "**Name**" can be any name for the connection. 3. "**Host**" the endpoint for your database. - 4. "**Database name**" is the default database to replicate. Additional databases are derived by qualifying the tables in your Sync Streams/Sync Rules. + 4. "**Database name**" is the default database to replicate. Additional databases are derived by qualifying the tables in your Sync Streams. 5. "**Username**" and "**Password**" maps to your database user. 4. Click **Test Connection** and fix any errors. 5. Click **Save Connection**. - PowerSync deploys and configures an isolated cloud environment for you, which can take a few minutes to complete. + PowerSync deploys and configures an isolated cloud environment for you, which can take a few minutes to complete. diff --git a/configuration/source-db/private-endpoints.mdx b/configuration/source-db/private-endpoints.mdx index 82569395d..e8c96a97f 100644 --- a/configuration/source-db/private-endpoints.mdx +++ b/configuration/source-db/private-endpoints.mdx @@ -62,11 +62,11 @@ The flow is the same for both supported source databases: configure an Endpoint To expose a Postgres database via PrivateLink, you need a Network Load Balancer that forwards traffic to the database. This works for Postgres running on EC2 or RDS. - For AWS RDS, the steps below do not handle dynamic IPs if the RDS instance's IP changes. This is specifically relevant when using an RDS cluster with failover support. See this [AWS blog post](https://aws.amazon.com/blogs/database/access-amazon-rds-across-vpcs-using-aws-privatelink-and-network-load-balancer/) for handling IP changes automatically. + The target group holds a fixed IP address. It does not follow the database when the IP changes, which happens on an Aurora failover, an instance replacement, or a migration to a new cluster. PowerSync then cannot connect, or reaches a different instance and rejects its certificate because the names in the certificate do not include the configured host. Update the target group when the writer changes, or automate it as described in this [AWS blog post](https://aws.amazon.com/blogs/database/access-amazon-rds-across-vpcs-using-aws-privatelink-and-network-load-balancer/). 1. **Create a Target Group**: - 1. Obtain the RDS instance's private IP address. Make sure this points to a writable instance. + 1. Obtain the private IP address of the instance that PowerSync must replicate from. For Aurora, resolve the cluster endpoint to get the current writer's IP. For RDS, use the instance's private IP. 2. Create a **Target Group** with IP addresses as the target type, using the IP from above. Use TCP protocol and the database port (typically `5432` for Postgres). 2. **Create a Network Load Balancer (NLB)**: 1. Select the same VPC as your RDS instance. @@ -135,12 +135,16 @@ The flow is the same for both supported source databases: configure an Endpoint 2. Select the **Postgres** or **MongoDB** tab. 3. In the **Private Endpoint** dropdown, select your endpoint. Only endpoints in the same region as the instance with status `Available` are selectable. 4. Fill in the rest of the connection details: - - **For Postgres**: enter your database connection details as usual. PowerSync routes traffic through the Private Endpoint to your load balancer. + - **For Postgres**: enter your database connection details as usual, with the database's own hostname as the "**Host**": the cluster endpoint for Aurora, or the instance endpoint for RDS. PowerSync uses this hostname to verify the server certificate and sends the traffic through the Private Endpoint to your load balancer. Do not enter the load balancer or Endpoint Service DNS name as the host, because it is not in the database's certificate. - **For MongoDB**: on the Atlas cluster, click **Connect**, choose **Private Endpoint** as the connection type, select the provisioned endpoint, choose **Drivers** as the connection method, and copy the resulting connection string. It should look something like `mongodb+srv://:@your-cluster-pl-0.abcde.mongodb.net/`. Paste it into the **URI** field in the Dashboard. 5. Click **Test Connection** and resolve any errors. 6. Click **Save Connection**. PowerSync deploys and configures an isolated cloud environment for you, which can take a few minutes. Monitor the logs to confirm the instance connects. + + + For a connection that uses a Private Endpoint, **Test Connection** checks that the configuration is complete but does not connect to the database. It reports success without verifying reachability, TLS, credentials, `wal_level`, or the `powersync` publication. Complete the [Source Database Setup](/configuration/source-db/setup) steps first, then confirm the connection in the instance's **Health** view and logs after deploying. + diff --git a/configuration/source-db/setup.mdx b/configuration/source-db/setup.mdx index 41ed2c78d..796347909 100644 --- a/configuration/source-db/setup.mdx +++ b/configuration/source-db/setup.mdx @@ -7,6 +7,7 @@ Jump to: [Postgres](#postgres) | [MongoDB](#mongodb) | [Azure DocumentDB](#azure import PostgresPowerSyncUser from '/snippets/postgres-powersync-user.mdx'; import PostgresPowerSyncPublication from '/snippets/postgres-powersync-publication.mdx'; +import PostgresRdsPowerSyncUser from '/snippets/postgres-rds-powersync-user.mdx'; import NeonDatabaseSetup from '/snippets/neon-database-setup.mdx'; import SupabaseGuidedSetup from '/snippets/supabase-guided-setup.mdx'; @@ -66,24 +67,47 @@ We have documented steps for some specific hosting providers: ### 2. Create a PowerSync database user - Create a PowerSync user on Postgres: + + + ### 3. Create `powersync` publication + + + + + + ### Supported Scaling Options + + PowerSync replicates only from the writer. Clusters can also include reader instances. + + | Scaling Option | Scaling Model | PowerSync Support | + | --- | --- | --- | + | Provisioned | Configured instance capacity | Supported | + | Aurora Serverless v2 | Automatically adjusts instance capacity | Supported | + | Aurora Postgres Limitless | Horizontally scales writes and reads across multiple nodes | [Not supported](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/limitless-reqs-limits.html#limitless-limitations) | + + ### Prerequisites + + PowerSync must be able to reach the writer. Either make the writer publicly accessible over IPv4 and restrict access to PowerSync's IPs (see [IP Filtering](/configuration/source-db/security-and-ip-filtering)), or connect through a [Private Endpoint](/configuration/source-db/private-endpoints) using AWS PrivateLink. If you use public access, apply the same network settings to reader instances that can become the writer. + + ### 1. Ensure logical replication is enabled + + In Aurora, `rds.logical_replication` is a setting of the **DB cluster parameter group**, not of a DB instance parameter group. + + 1. In the [Amazon RDS console](https://console.aws.amazon.com/rds/), open your cluster's **Configuration** tab and follow the **Parameter group** link of type **DB cluster parameter group**. If the cluster still uses a default parameter group, [create a custom DB cluster parameter group](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_WorkingWithDBClusterParamGroups.html) and assign it to the cluster first. Default parameter groups cannot be edited. + 2. Set `rds.logical_replication` to `1` and save. + 3. Reboot the writer. The change only takes effect after the reboot. + 4. Connect to the writer through the cluster endpoint and confirm the settings: ```sql - -- SQL to create powersync user - CREATE ROLE powersync_role WITH BYPASSRLS LOGIN PASSWORD 'myhighlyrandompassword'; - - -- Allow the role to perform replication tasks - GRANT rds_replication TO powersync_role; - - -- Set up permissions for the newly created role - -- Read-only (SELECT) access is required - GRANT SELECT ON ALL TABLES IN SCHEMA public TO powersync_role; - - -- Optionally, grant SELECT on all future tables (to cater for schema additions) - ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO powersync_role; + SHOW rds.logical_replication; -- on + SHOW wal_level; -- logical ``` - To restrict read access to specific tables, explicitly list allowed tables for both the `SELECT` privilege, and for the publication (as well as for any other publications that may exist). + For the related parameters `max_replication_slots`, `max_wal_senders` and `max_worker_processes`, see [Setting up logical replication for your Aurora PostgreSQL DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraPostgreSQL.Replication.Logical.Configure.html). PowerSync uses one replication slot per instance, plus a second slot while it processes a new Sync Config version. See [Postgres Maintenance](/configuration/source-db/postgres-maintenance). + + ### 2. Create a PowerSync database user + + ### 3. Create `powersync` publication @@ -228,7 +252,7 @@ For other providers and self-hosted databases: ### 1. Ensure logical replication is enabled - PowerSync reads the Postgres WAL using logical replication in order to create [buckets](/architecture/powersync-service#bucket-system) in accordance with your [Sync Streams](/sync/streams/overview) (or legacy [Sync Rules](/sync/rules/overview)). + PowerSync reads the Postgres WAL using logical replication in order to create [buckets](/architecture/powersync-service#bucket-system) in accordance with your [Sync Streams](/sync/streams/overview). If you are managing Postgres yourself, set `wal_level = logical` in your config file: @@ -332,8 +356,8 @@ To replicate data from MongoDB to PowerSync in a consistent manner, PowerSync us PowerSync supports three configuration options for post-images: 1. **Off**: (`post_images: off`): Uses `fullDocument: 'updateLookup'` for backwards compatibility. This was the default for older instances. However, this may lead to consistency issues, so we strongly recommend enabling post-images instead. -2. **Auto-Configure**: (`post_images: auto_configure`) The **default** for new instances: Automatically enables the `changeStreamPreAndPostImages` option on collections as needed. Requires the permissions/privileges mentioned above. If a collection is removed from [Sync Streams](/sync/streams/overview) (or legacy [Sync Rules](/sync/rules/overview)), you need to manually disable `changeStreamPreAndPostImages` on that collection. -3. **Read-only**: (`post_images: read_only`): Uses `fullDocument: 'required'` and requires `changeStreamPreAndPostImages: { enabled: true }` to be set on every collection referenced in your Sync Streams/Sync Rules. Replication will error if this is not configured. This option is ideal when permissions are restricted. +2. **Auto-Configure**: (`post_images: auto_configure`) The **default** for new instances: Automatically enables the `changeStreamPreAndPostImages` option on collections as needed. Requires the permissions/privileges mentioned above. If a collection is removed from [Sync Streams](/sync/streams/overview), you need to manually disable `changeStreamPreAndPostImages` on that collection. +3. **Read-only**: (`post_images: read_only`): Uses `fullDocument: 'required'` and requires `changeStreamPreAndPostImages: { enabled: true }` to be set on every collection referenced in your Sync Streams. Replication will error if this is not configured. This option is ideal when permissions are restricted. To manually configure collections for `read_only` mode, run this command on each collection: @@ -416,6 +440,70 @@ For implementation and design details, including how PowerSync handles checkpoin PowerSync reads from the MySQL [binary log](https://dev.mysql.com/doc/refman/8.4/en/binary-log.html) (binlog) to replicate changes. We use a modified version of the [Zongji MySQL](https://github.com/powersync-ja/powersync-mysql-zongji) binlog listener to achieve this. +For Aurora MySQL, follow the provider-specific steps below instead of the generic [Binlog Configuration](#binlog-configuration). + + + PowerSync supports provisioned Aurora MySQL clusters and Aurora Serverless v2 clusters with a single writer. The cluster can include reader instances, but PowerSync replicates only from the writer. + + ### Prerequisites + + For PowerSync Cloud, make the writer publicly accessible over IPv4 and allow inbound traffic from [PowerSync's IP addresses](/configuration/source-db/security-and-ip-filtering) on the database port (`3306` by default). Apply the same network settings to reader instances that can become the writer. [Private Endpoints](/configuration/source-db/private-endpoints#current-limitations) are not currently supported for MySQL. + + + + In the [Amazon RDS console](https://console.aws.amazon.com/rds/), open the **DB cluster parameter group** linked from your cluster's **Configuration** tab. If the cluster uses a default group, [create a custom DB cluster parameter group](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_WorkingWithDBClusterParamGroups.html) with the family that matches your Aurora MySQL version. Default parameter groups cannot be edited. + + + ![Amazon RDS creation form with Aurora MySQL, the aurora-mysql8.0 family, and DB Cluster Parameter Group selected](/images/installation/aurora-create-parameter-group.png) + + + Set these parameters to enable binary logging and global transaction identifiers (GTIDs): + + | Parameter in the AWS Console | Required Value | + | --- | --- | + | `binlog_format` | `ROW` | + | `binlog_row_image` | `FULL` | + | `enforce_gtid_consistency` | `ON` | + | `gtid-mode` | `ON` | + + + ![Amazon RDS parameter editor with enforce_gtid_consistency and gtid-mode set to ON](/images/installation/aurora-set-gtid-flags.png) + + + Aurora [enables binary logging](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_LogAccess.MySQL.BinaryFormat.html) through `binlog_format`. You do not need to configure `log_bin` or `server_id` separately. The AWS console calls the GTID parameter `gtid-mode`; its SQL variable is `gtid_mode`. + + + If you created a new parameter group, assign it to the cluster through **Modify** in the RDS console. Save your changes and reboot the writer for the settings to take effect. If reader instances do not restart with the writer, reboot them manually to apply the same settings. + + + Connect to the writer through the cluster endpoint and run this query: + + ```sql + SELECT + @@GLOBAL.log_bin AS log_bin, -- 1 + @@GLOBAL.gtid_mode AS gtid_mode, -- ON + @@GLOBAL.enforce_gtid_consistency AS enforce_gtid_consistency, -- ON + @@GLOBAL.binlog_format AS binlog_format, -- ROW + @@GLOBAL.binlog_row_image AS binlog_row_image; -- FULL + ``` + + + Set [binlog retention](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/mysql-stored-proc-configuring.html#mysql_rds_set_configuration-usage-notes.binlog-retention-hours) long enough to cover the initial snapshot and expected downtime or replication lag. If required binlogs are purged before PowerSync reads them, PowerSync must take a new snapshot. + + Replace `24` with the number of hours your workload requires: + + ```sql + CALL mysql.rds_set_configuration('binlog retention hours', 24); + ``` + + Monitor storage usage after increasing retention. + + + Follow [Database User Configuration](#database-user-configuration) to create a user with replication privileges and read access to the tables you want to sync. + + + + ### Binlog Configuration To ensure that PowerSync can read the binary log, you need to configure your MySQL server to enable binary logging and configure it with the following server command options: @@ -456,7 +544,7 @@ GRANT SELECT ON .* TO 'repl_user'@'%'; FLUSH PRIVILEGES; ``` -It is possible to constrain the MySQL user further and limit access to specific tables. Care should be taken to ensure that all the tables in your Sync Streams/Sync Rules are included in the grants. +It is possible to constrain the MySQL user further and limit access to specific tables. Care should be taken to ensure that all the tables in your Sync Streams are included in the grants. ```sql -- Grant select to the users and the invoices tables in the source database diff --git a/snippets/postgres-rds-powersync-user.mdx b/snippets/postgres-rds-powersync-user.mdx new file mode 100644 index 000000000..b7a0d68d5 --- /dev/null +++ b/snippets/postgres-rds-powersync-user.mdx @@ -0,0 +1,18 @@ +Create a PowerSync user on Postgres: + +```sql +-- SQL to create powersync user +CREATE ROLE powersync_role WITH BYPASSRLS LOGIN PASSWORD 'myhighlyrandompassword'; + +-- Allow the role to perform replication tasks +GRANT rds_replication TO powersync_role; + +-- Set up permissions for the newly created role +-- Read-only (SELECT) access is required +GRANT SELECT ON ALL TABLES IN SCHEMA public TO powersync_role; + +-- Optionally, grant SELECT on all future tables (to cater for schema additions) +ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO powersync_role; +``` + +To restrict read access to specific tables, explicitly list allowed tables for both the `SELECT` privilege, and for the publication (as well as for any other publications that may exist).