diff --git a/docs/en/solutions/How_to_Deploy_CloudBeaver_for_PostgreSQL_Management.md b/docs/en/solutions/How_to_Deploy_CloudBeaver_for_PostgreSQL_Management.md new file mode 100644 index 000000000..cbd1cb248 --- /dev/null +++ b/docs/en/solutions/How_to_Deploy_CloudBeaver_for_PostgreSQL_Management.md @@ -0,0 +1,161 @@ +--- +kind: + - How To +products: + - Alauda Application Services +ProductsVersion: + - 4.0,4.1,4.2,4.3 +id: KB260515008 +--- + +# How to Deploy CloudBeaver for PostgreSQL Management + +## Issue + +You need a web-based SQL client for browsing and querying PostgreSQL instances managed by Alauda Application Services, without installing a desktop tool on every operator's workstation. CloudBeaver is the open-source web edition of DBeaver and runs as a single-pod Deployment on Kubernetes. This how-to deploys CloudBeaver and connects it to a PostgreSQL cluster managed by the PostgreSQL Operator. + +## Environment + +- Any Kubernetes cluster reachable to operators (NodePort exposure is used below; Ingress / Route works equally well) +- A `StorageClass` capable of provisioning ReadWriteOnce PVCs — used to persist CloudBeaver workspace state across pod restarts +- Network reachability from the CloudBeaver pod to the target PostgreSQL Service (`` on port 5432) + +## Resolution + +### 1. Prepare the manifest + +Save the following to `cloudbeaver.yaml`. Adjust `storageClassName`, image registry, and resource requests to match your environment: + +```yaml +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: cloudbeaver +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: sc-topolvm # replace with any RWO StorageClass available in the cluster + volumeMode: Filesystem +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: cloudbeaver +spec: + replicas: 1 + selector: + matchLabels: + app: cloudbeaver + strategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 1 + maxUnavailable: 1 + template: + metadata: + labels: + app: cloudbeaver + spec: + containers: + - name: cloudbeaver + image: docker-mirrors.alauda.cn/dbeaver/cloudbeaver:latest + imagePullPolicy: Always + ports: + - name: web + containerPort: 8978 + protocol: TCP + resources: + limits: + cpu: "1" + memory: 1Gi + requests: + cpu: 100m + memory: 256Mi + volumeMounts: + - name: cloudbeaver-data + mountPath: /opt/cloudbeaver/workspace + volumes: + - name: cloudbeaver-data + persistentVolumeClaim: + claimName: cloudbeaver +--- +apiVersion: v1 +kind: Service +metadata: + name: cloudbeaver +spec: + type: NodePort + selector: + app: cloudbeaver + ports: + - name: web + port: 8978 + targetPort: 8978 + protocol: TCP +``` + +> CloudBeaver stores its administrator password, saved connections, and query history under `/opt/cloudbeaver/workspace`. Without a persistent volume, everything is lost on every pod restart. Confirm the chosen `storageClassName` exists in the target cluster before applying. + +> **Air-gapped / IPv6-only clusters:** the public `docker-mirrors.alauda.cn/dbeaver/cloudbeaver:latest` image may be unreachable from the cluster. Mirror it into the cluster's own registry first and reference that path in the Deployment, for example `skopeo copy docker://docker-mirrors.alauda.cn/dbeaver/cloudbeaver:latest docker:///dbeaver/cloudbeaver:latest`. + +### 2. Deploy + +```bash +kubectl -n apply -f cloudbeaver.yaml +kubectl -n rollout status deploy/cloudbeaver +``` + +### 3. Discover the access URL + +The Service uses NodePort, so any node IP plus the allocated NodePort exposes the UI: + +```bash +namespace= +HOST=$(kubectl -n "$namespace" get pod -l app=cloudbeaver \ + -o jsonpath='{.items[0].status.hostIP}') +PORT=$(kubectl -n "$namespace" get svc cloudbeaver \ + -o jsonpath='{.spec.ports[0].nodePort}') +echo "http://$HOST:$PORT" +``` + +Open the URL in a browser. + +### 4. Initial setup + +1. On first launch CloudBeaver prompts you to set the administrator password. Choose a strong password and store it safely — this account governs all subsequent server-side configuration. +2. Log in with the administrator user. +3. (Optional) Switch the UI language under the user menu in the top-right. + +### 5. Connect to a PostgreSQL instance + +1. Click **New Connection** and choose **PostgreSQL**. +2. Fill **Host** with the cluster Service name and **Port** `5432`. From inside the cluster the host is `.` (the read-write Service); the `-repl` Service points at replicas. From outside the cluster, retrieve the NodePort or LoadBalancer address: + + ```bash + kubectl -n get svc + ``` + +3. Enter the database user and password. The Operator stores each role's credentials in a Secret named `..credentials.postgresql.acid.zalan.do`. Retrieve the `postgres` superuser password from its Secret: + + ```bash + kubectl get secret postgres..credentials.postgresql.acid.zalan.do \ + -n -o jsonpath='{.data.password}' | base64 -d; echo + ``` + + The same Secret also carries `username`, `host` and `port` keys. + +4. (Optional) Set **Database** to the target database; otherwise CloudBeaver connects to the default `postgres` database. +5. Under **Access Management**, grant the current CloudBeaver user permission to use the new connection. +6. Save and test the connection. SQL editors can now be opened from the browser and queries executed against the PostgreSQL cluster. + +### 6. Uninstall + +```bash +kubectl -n delete -f cloudbeaver.yaml +``` + +The PVC is removed alongside the rest of the manifest. To preserve CloudBeaver state for a future redeploy, delete only the Deployment and Service and re-attach the existing PVC during the next install. diff --git a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md new file mode 100644 index 000000000..db2fb7e8b --- /dev/null +++ b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md @@ -0,0 +1,364 @@ +--- +kind: + - Solution +products: + - Alauda Application Services +ProductsVersion: + - 4.3 +id: KB260720001 +--- + +# PostgreSQL Instance Cross-Cluster Migration Guide (Operator v4.3.3) + +## Background + +### The Challenge + +Sometimes a running PostgreSQL instance must move to a different Kubernetes cluster — cluster decommissioning, hardware refresh, moving a workload between ACP platforms, or consolidating environments. Dump-and-restore migrations require downtime proportional to database size and lose writes made after the dump. + +### The Solution + +This guide migrates a PostgreSQL instance across clusters using the operator's hot standby (cross-cluster replication) feature: the target instance is created as a *standby cluster* that bootstraps directly from the source via streaming replication, stays continuously in sync, and is then promoted in a controlled two-phase switchover. Downtime is limited to the switchover itself (seconds to a couple of minutes), and data integrity is verified with checksums before and after cutover. + +This procedure was validated end-to-end with PostgreSQL Operator v4.3.3 on two separate ACP platforms (source and target in different platforms, connected via NodePort). It builds on the [PostgreSQL Hot Standby Cluster Configuration Guide](./How_to_Use_PostgreSQL_Hot_Standby_Cluster.md) (KB251000009); read that first for concept background. + +## Environment Information + +- PostgreSQL Operator: v4.3.3 on **both** source and target clusters (see [Important Limitations](#important-limitations)) +- ACP: any 4.x cluster able to run the v4.3.3 operator; source and target may be in different ACP platforms +- PostgreSQL: same major version on both sides (this guide uses 16) + +## Important Limitations + +- **The operator version must match on both sides.** Cross-version pairing (e.g. a v4.2+/v4.3 standby against a primary managed by v4.1.x) fails with `pq: column "external_ip" does not exist` and — dangerously — leaves the "standby" running as an empty independent primary (tracked as ECO-703). See [Troubleshooting](#troubleshooting). +- Source and target must run the same PostgreSQL major version. +- The standby cluster must initially be created with `numberOfInstances: 1`; scale it up after promotion. +- `replSvcType` must be identical on both clusters. +- The target cluster must be able to reach the source cluster's node IP + NodePort (standby pulls from primary). Verify this before starting — see Step 2. If there is **no network path between the clusters**, the streaming approach cannot work — use the [workstation-relayed logical migration](#alternative-migration-without-inter-cluster-connectivity) instead. +- On mixed-architecture target clusters, pin the instance (and ideally the operator) to nodes matching the source architecture. Streaming replication copies the data directory bit-for-bit; PostgreSQL does not support mixed-architecture replication. + +## Migration Overview + +``` +Step 1 Prepare source: enable clusterReplication, record NodePort, baseline checksums +Step 2 Preflight: network reachability, version/arch checks +Step 3 Create standby on target (bootstraps from source, stays streaming) +Step 4 Verify sync: identity, lag, checksums +Step 5 Cutover: two-phase switchover (demote source, promote target) +Step 6 Post-migration: repoint clients; keep reverse standby or dismantle +``` + +## Step 1: Prepare the Source Instance + +If the source instance does not yet have cluster replication enabled, patch it (this is an online change; the operator creates the replication metadata and exposes the master service): + +```bash +SRC_NS="pg-migrate" +SRC_CLUSTER="acid-mig" + +kubectl -n $SRC_NS patch postgresql $SRC_CLUSTER --type merge -p '{ + "spec": { + "clusterReplication": {"enabled": true, "replSvcType": "NodePort"}, + "postgresql": {"parameters": {"max_slot_wal_keep_size": "10GB"}} + } +}' +``` + +> `max_slot_wal_keep_size` bounds WAL retained for the replication slot if the standby disconnects. Size it to your volume: it must fit in the instance's free disk space, and it defines how long a standby outage you can tolerate before a re-bootstrap is needed. + +Record the connection coordinates the standby will use: + +```bash +# NodePort of the source master service +kubectl -n $SRC_NS get svc $SRC_CLUSTER -o jsonpath='{.spec.ports[0].nodePort}{"\n"}' + +# A node IP hosting the instance (any node IP of the cluster works for NodePort) +kubectl -n $SRC_NS get pod -l cluster-name=$SRC_CLUSTER -o jsonpath='{range .items[*]}{.status.hostIP}{"\n"}{end}' +``` + +Confirm the source registered itself in the replication metadata (role `primary`, correct `node_port`): + +```bash +kubectl -n $SRC_NS exec ${SRC_CLUSTER}-0 -c postgres -- psql -U postgres -x \ + -c "SELECT * FROM sys_operator.multi_cluster_info;" +``` + +Take a data integrity baseline. Adapt the checksum query to your schema — the point is a number you can compare after migration: + +```bash +kubectl -n $SRC_NS exec ${SRC_CLUSTER}-0 -c postgres -- psql -U postgres -d -tA -c " +SELECT relname, n_live_tup FROM pg_stat_user_tables ORDER BY relname;" +# Example per-table checksum: +# SELECT count(*), sum(hashtext(id::text || payload)) FROM your_table; +``` + +Finally, force a checkpoint so the standby's basebackup starts from a consistent point: + +```bash +kubectl -n $SRC_NS exec ${SRC_CLUSTER}-0 -c postgres -- psql -U postgres -c "CHECKPOINT;" +``` + +## Step 2: Preflight Checks on the Target + +**Network reachability** — the standby (operator pod *and* PostgreSQL pods) must reach the source NodePort. Test from a pod on the overlay network of the target cluster, on the node(s) where the instance will run: + +```bash +# From any pod with bash on the target cluster: +kubectl exec -- bash -c 'timeout 4 bash -c "echo > /dev/tcp//" && echo OPEN || echo CLOSED' +``` + +If this prints `CLOSED`, stop and fix connectivity first. Note that reachability can differ per node (broken egress on individual nodes has been observed); test from the nodes you will schedule onto. + +**Version check** — both operators must be v4.3.3 (or at least the same version): + +```bash +kubectl -n operators get csv | grep postgres-operator +``` + +**Architecture** — on mixed-architecture targets, decide the node set now and pin with `nodeAffinity` (shown in Step 3). + +## Step 3: Create the Standby on the Target Cluster + +Create the namespace and a bootstrap secret holding the **source** cluster's admin credentials: + +```bash +TGT_NS="pg-migrate" + +# Read the admin password from the source cluster: +kubectl --context -n $SRC_NS get secret \ + postgres.${SRC_CLUSTER}.credentials.postgresql.acid.zalan.do \ + -o jsonpath='{.data.password}' | base64 -d +``` + +```yaml +kind: Secret +apiVersion: v1 +metadata: + name: standby-bootstrap-secret + namespace: pg-migrate +type: kubernetes.io/basic-auth +stringData: + username: postgres + password: "" +``` + +Create the standby instance. Keep PostgreSQL version, parameters, and volume size aligned with the source; start with a single replica: + +```yaml +apiVersion: acid.zalan.do/v1 +kind: postgresql +metadata: + name: acid-mig # name may differ from the source + namespace: pg-migrate +spec: + teamId: acid + numberOfInstances: 1 # required initially for standby clusters + postgresql: + version: "16" # must match the source major version + parameters: + max_slot_wal_keep_size: '10GB' + volume: + size: 5Gi # same capacity as source + storageClass: + # Only needed on mixed-architecture clusters — match the source arch: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: kubernetes.io/arch + operator: In + values: ["arm64"] + clusterReplication: + enabled: true + isReplica: true + peerHost: "" + peerPort: + replSvcType: NodePort + bootstrapSecret: standby-bootstrap-secret +``` + +What happens on creation: the operator connects to the source, copies the database user credential secrets into the target namespace, creates a local `-xcr` service whose endpoints point at the source nodes, and bootstraps the pod with `pg_basebackup` from the source. `patronictl list` shows `creating replica` during the basebackup (duration depends on database size and link bandwidth), then `Standby Leader | streaming`. + +## Step 4: Verify Synchronization + +All three checks below must pass before cutover. + +**1. Streaming state and shared identity** — the cluster identifier printed by `patronictl list` must be *identical* on source and target (it is the PostgreSQL system identifier). If the target shows a different identifier, it bootstrapped as an independent cluster and contains none of your data — see [Troubleshooting](#troubleshooting). + +```bash +# Target — expect: Standby Leader | streaming +kubectl -n $TGT_NS exec acid-mig-0 -c postgres -- patronictl list + +# Source — expect the standby connected, plus the slot active: +kubectl -n $SRC_NS exec ${SRC_CLUSTER}-0 -c postgres -- psql -U postgres -c \ + "SELECT application_name, client_addr, state FROM pg_stat_replication; + SELECT slot_name, active FROM pg_replication_slots WHERE slot_name='xdc_hotstandby';" +``` + +**2. Replication lag** — write on the source, confirm it appears on the target within seconds, and compare LSNs: + +```bash +kubectl -n $SRC_NS exec ${SRC_CLUSTER}-0 -c postgres -- psql -U postgres -tA -c "SELECT pg_current_wal_lsn();" +kubectl -n $TGT_NS exec acid-mig-0 -c postgres -- psql -U postgres -tA -c "SELECT pg_last_wal_replay_lsn();" +``` + +**3. Data checksums** — rerun the Step 1 baseline queries on the target; every value must match. + +## Step 5: Cutover (Two-Phase Switchover) + +Perform the switchover in the documented order — demote first, then promote — so there is never a moment with two writable primaries. + +1. **Stop application writes** to the source (scale writers down, or hold traffic at the application layer). + +2. **Confirm zero lag** (Step 4, check 2 — the two LSNs must be equal once writes stop). + +3. **Phase 1 — demote the source** to a standby: + +```bash +kubectl --context -n $SRC_NS patch postgresql $SRC_CLUSTER --type merge \ + -p '{"spec":{"clusterReplication":{"isReplica":true},"numberOfInstances":1}}' +``` + +Wait until the source shows `Standby Leader` (it may pass through `stopped` briefly). The demoted source finds the target automatically through the replication metadata — no `peerHost` needs to be added to its spec. + +4. **Phase 2 — promote the target** and scale it to full size: + +```bash +kubectl --context -n $TGT_NS patch postgresql acid-mig --type merge \ + -p '{"spec":{"clusterReplication":{"isReplica":false},"numberOfInstances":2}}' +``` + +5. **Gate on the real promotion signal.** Do not rely on `pg_is_in_recovery()` alone — during promotion and scale-up the cluster status can read `Running` while Patroni is still converting roles. Wait until **all** of the following hold: + +```bash +# a) Patroni shows a Leader in state running (a timeline increase here is normal): +kubectl -n $TGT_NS exec acid-mig-0 -c postgres -- patronictl list +# b) the CR reports Running: +kubectl -n $TGT_NS get postgresql acid-mig -o jsonpath='{.status.PostgresClusterStatus}{"\n"}' +# c) the new replica is streaming (after scale-up): +kubectl -n $TGT_NS exec acid-mig-0 -c postgres -- psql -U postgres -c \ + "SELECT application_name, state FROM pg_stat_replication;" +``` + +6. **Verify writes and integrity** on the new primary: insert a marker row, rerun the checksum queries, compare with the baseline. + +> A leadership change between the target's pods during promote+scale-up (with an extra timeline bump) is normal operator rolling behavior and does not indicate a problem. + +## Step 6: Post-Migration + +- **Repoint clients** to the target cluster's service (and update any external access such as NodePort/LoadBalancer/ingress used by applications). +- The demoted source is now a live **reverse DR standby** of the target: writes on the target replicate back to it. Choose one: + - **Keep it** as disaster-recovery / rollback insurance (recommended for at least a soak period). Rolling back is the same two-phase switchover in the opposite direction. + - **Dismantle it** and remove the replication setup entirely — see below. +- Scale/tune the target (replica count, resources, backup schedule, monitoring) to match what the source had. + +### Dismantling the Source and Removing the Replication Configuration + +Perform these steps **in order** — the standby must be gone before the primary's replication configuration is removed, otherwise the standby loses its upstream while still streaming. + +**1. Delete the demoted source instance** (on the source cluster): + +```bash +kubectl --context -n $SRC_NS delete postgresql $SRC_CLUSTER +# PVC retention on CR deletion depends on operator configuration — check and +# remove any leftovers to reclaim storage: +kubectl --context -n $SRC_NS get pvc -l cluster-name=$SRC_CLUSTER +kubectl --context -n $SRC_NS delete pvc -l cluster-name=$SRC_CLUSTER --ignore-not-found +``` + +The operator deletes the credential secrets it owns along with the CR. + +**2. Convert the target to a normal (non-replicated) instance** by removing the `clusterReplication` block: + +```bash +kubectl --context -n $TGT_NS patch postgresql acid-mig --type json \ + -p '[{"op":"remove","path":"/spec/clusterReplication"}]' +``` + +The operator handles this transition: it re-applies normal-primary Patroni configuration, drops the `sys_operator` metadata schema from the database, and removes the `xdc_hotstandby` slot from the replication configuration. (The reverse — converting an existing normal instance *into* a standby — is not a supported transition; standbys must be created as standbys.) + +**3. Clean up leftover objects** that the operator does not remove: + +```bash +# Bootstrap secret (user-created, never operator-owned): +kubectl --context -n $TGT_NS delete secret standby-bootstrap-secret + +# The -xcr service lingers after the role change (it is only removed +# with the CR itself) — delete it: +kubectl --context -n $TGT_NS delete svc acid-mig-xcr --ignore-not-found + +# The physical replication slot can also linger even though it was removed +# from the Patroni configuration — drop it if the Step 4 check finds it: +kubectl --context -n $TGT_NS exec acid-mig-0 -c postgres -- psql -U postgres -c \ + "SELECT pg_drop_replication_slot('xdc_hotstandby') + WHERE EXISTS (SELECT 1 FROM pg_replication_slots WHERE slot_name='xdc_hotstandby' AND NOT active);" +``` + +**4. Verify the target is a clean standalone instance:** + +```bash +kubectl -n $TGT_NS exec acid-mig-0 -c postgres -- psql -U postgres -tA -c \ + "SELECT count(*) FROM pg_replication_slots WHERE slot_name='xdc_hotstandby'; + SELECT count(*) FROM pg_namespace WHERE nspname='sys_operator';" +# Both counts must be 0 +kubectl -n $TGT_NS exec acid-mig-0 -c postgres -- patronictl list +# Expect a plain Leader/Replica cluster, no Standby Leader +``` + +## Alternative: Migration Without Inter-Cluster Connectivity + +When the clusters have **no network path between them** but your workstation can reach both Kubernetes API servers, the migration can be relayed through the workstation as a logical dump/restore piped between two `kubectl exec` sessions — the clusters never talk to each other. Downtime equals the full copy duration (versus seconds for the streaming switchover), but there are no operator-version or CPU-architecture constraints, and the PostgreSQL major only needs to be the same or newer on the target. + +The full validated procedure is a separate solution: [How to Migrate a PostgreSQL Instance Between Network-Isolated Clusters](./How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md) (KB260721001). + +## Troubleshooting + +### Standby fails with `pq: column "external_ip" does not exist` — and runs as an empty independent primary + +**Cause:** operator version mismatch between source and target (e.g. source primary managed by v4.1.x, target operator v4.2+/v4.3). The replication metadata table schema differs between these lines and is never migrated (ECO-703). The standby create hook aborts midway and the pod falls through to a normal bootstrap: it comes up as a *fresh, empty, writable* primary while its CR still says `isReplica: true`. + +**Detection:** compare the cluster identifier in `patronictl list` on both sides — different identifier means independent cluster, not a standby. + +**Fix:** upgrade both operators to the same version (preferred). If the source operator cannot be upgraded immediately, add the missing column on the source primary (safe for both versions — all statements reference columns by name): + +```sql +ALTER TABLE sys_operator.multi_cluster_info ADD COLUMN IF NOT EXISTS external_ip CHAR(64) DEFAULT ''; +``` + +Then recreate the standby cleanly — see the next item. + +### Recreating a failed standby: create hook fails with secret "already exists" + +The operator copies the source's credential secrets into the target namespace *before* bootstrapping, and the copy step fails hard if they already exist from a previous attempt. To retry a standby creation from scratch, delete all of: + +```bash +kubectl -n $TGT_NS delete postgresql acid-mig +kubectl -n $TGT_NS delete pvc -l cluster-name=acid-mig # data from the failed attempt must go +kubectl -n $TGT_NS delete secret \ + postgres.acid-mig.credentials.postgresql.acid.zalan.do \ + standby.acid-mig.credentials.postgresql.acid.zalan.do +``` + +then re-apply the standby manifest. + +### Standby stuck in `creating replica` + +The basebackup is either still copying (large database — check network throughput) or cannot connect. Verify Step 2 reachability *from the node the standby pod landed on*; per-node egress differences are a real failure mode. Also confirm the bootstrap secret contains the current source admin password. + +### Replication slot errors (`TypeError ... 'int' and 'NoneType'`) in standby logs + +Known Patroni issue; replication generally continues. See the [Hot Standby guide's troubleshooting section](./How_to_Use_PostgreSQL_Hot_Standby_Cluster.md#troubleshooting) — drop the `xdc_hotstandby` slot if needed. + +## Verification Checklist + +| Check | When | Pass condition | +|---|---|---| +| Network preflight | before Step 3 | target overlay pod reaches `SRC_NODE_IP:NODEPORT` | +| Operator versions | before Step 3 | identical on both clusters | +| Shared system identifier | after Step 3 | `patronictl list` identifier equal on both sides | +| Streaming | after Step 3 | target `Standby Leader / streaming`; source slot `xdc_hotstandby` active | +| Checksums (pre-cutover) | Step 4 | all values match baseline | +| LSN parity | Step 5, writes stopped | `pg_current_wal_lsn()` == `pg_last_wal_replay_lsn()` | +| Promotion | Step 5 | Patroni Leader running + CR Running + replica streaming | +| Checksums (post-cutover) | Step 5 | all values match baseline; new write on target succeeds | +| Reverse replication (if source kept) | Step 6 | target write appears on demoted source | diff --git a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md new file mode 100644 index 000000000..1cbf4e643 --- /dev/null +++ b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md @@ -0,0 +1,608 @@ +--- +kind: + - Solution +products: + - Alauda Application Services +ProductsVersion: + - 4.x +id: KB260721001 +--- + +# How to Migrate a PostgreSQL Instance Between Network-Isolated Clusters + +## Background + +### The Challenge + +The streaming-replication migration described in the [PostgreSQL Instance Cross-Cluster Migration Guide](./How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md) requires the target cluster to reach the source cluster's network. In many real deployments the clusters are network-isolated — separate platforms, firewalled sites, disconnected security zones — and no such path exists or can be opened. + +### The Solution + +When an administrator workstation can reach **both** Kubernetes API servers, the migration can be relayed through the workstation: `pg_dump` runs in the source pod, `pg_restore` runs in the target pod, and the data flows between them through two `kubectl exec` channels piped together on the workstation. The clusters never exchange a packet. + +The procedure is four human operations: create the target instance (Step 1), stop application writes (Step 2), run the migration script (Step 3), and cut over (Steps 4–5). The script migrates **every** application database of the instance and verifies each one. + +Because the transfer is logical (SQL-level), this method has no same-version requirement: it works across different operator versions, across CPU architectures, and from an older PostgreSQL major to the same or a newer major on the target. Migrating to an **older** PostgreSQL major (a downgrade) is not supported. + +**Trade-off:** unlike streaming replication, this is a point-in-time copy. Writes made on the source after the dump starts are not transferred — application writes must be stopped for the whole dump+restore window, so downtime equals the full copy duration. + +### What Is and Is Not Migrated + +A logical migration carries less than a byte-level copy. Know the boundary before relying on it: + +| Carried by this procedure | NOT carried — handle deliberately | +|---|---| +| Tables, data, indexes, views, functions, sequences (including positions) | `GRANT`s (`-x` skips ACLs at restore; they remain inside the dump file — see Step 4) | +| Roles and their attributes (generated into the target CR in Step 1) | `COMMENT ON` metadata (`--no-comments`; a deliberate, lossy trade for a verifiable exit code) | +| Database encoding and locale (enforced by the script) | CR spec beyond `users`/`databases`/`volume`: `resources`, `postgresql.parameters`, `patroni.pg_hba`, `connectionPooler`, the pod security settings (`restrictedPsaEnabled`, `spiloPrivileged`, `spiloRunAsUser`, …), sidecars, load-balancer flags — port them in the target CR yourself | +| Database- and role-in-database-level settings (`ALTER DATABASE/ROLE ... SET`) | Objects requiring superuser: event triggers, publications/subscriptions, FDW servers and user mappings | +| Extensions (schema placement and version preserved) | Tablespace layout (`--no-tablespaces` maps everything to the default) | +| Planner statistics (regenerated via `ANALYZE` in the window) | Instances using `preparedDatabases` (its `_owner/_reader/_writer` role model and `user_management` schema are out of scope for this guide) | + +Applications connecting through a source-side `connectionPooler` service need the pooler declared in the target CR, or their connection endpoint changes at cutover. + +## Environment Information + +- PostgreSQL Operator: any 4.x version on each side (versions do **not** need to match) +- PostgreSQL: target major version equal to or newer than the source +- Workstation: `bash` and `kubectl` access (kubeconfig/context) to both clusters + +## Prerequisites + +- Two kubeconfig contexts on the workstation, one per cluster; verify both work: + +```bash +SRC_CTX=""; SRC_NS=""; SRC_CLUSTER="" +TGT_CTX=""; TGT_NS=""; TGT_CLUSTER="" + +kubectl --context $SRC_CTX -n $SRC_NS get postgresql $SRC_CLUSTER +kubectl --context $TGT_CTX -n $TGT_NS get postgresql $TGT_CLUSTER 2>/dev/null || echo "target instance not created yet" +``` + +- Enough workstation bandwidth to both API servers for the database size (all data flows through the workstation). +- A maintenance window sized from a rehearsal (below) — not guessed. +- A stable window on both instances: no pending node maintenance or evictions. The script resolves each master pod once at start; a Patroni failover mid-run sends the stream to a demoted pod. For long runs, consider `patronictl pause` on both sides for the duration, and redo any database that was in flight if a failover happens anyway. +- Sufficient Kubernetes permissions in both namespaces: `get` and `list` on `postgresqls.acid.zalan.do`, `pods`, and `secrets`; `create` on `pods/exec`; on the target side additionally `create` on `postgresqls` and `secrets` (credential pre-staging) and `patch` on `secrets` (troubleshooting); on the source side, for the final cleanup, `delete` on `postgresqls` and `persistentvolumeclaims`. +- The commands assume the operator's standard Spilo image: pod labels `spilo-role`/`cluster-name`, database container named `postgres`, and client binaries under `/usr/lib/postgresql//bin/` for the majors that image generation bundles (current images bundle 13–17; the script verifies the major it needs before starting). Custom or non-Spilo images require adapting these labels and paths. + +### Rehearse and Size the Window + +The stop-write window equals the full dump+restore+verify duration — do not open a maintenance window without knowing that number. Rehearse first: run Step 1 against a throwaway target instance and run the Step 3 script **without stopping source writes**. Expect `FAIL` verdicts (`SOURCE CHANGED DURING DUMP` is normal here) — the rehearsal measures *duration*, not correctness. Then delete the rehearsal target (CR and PVCs). + +Set a go/no-go rule before the real window: if verification has not passed with an agreed margin of window time remaining, roll back — re-enable source writes, keep the source untouched, and investigate offline. The target can be dropped and recreated at any time; the source is the asset. + +## Step 1: Create the Target Instance + +List the application databases and owners the source actually holds: + +```bash +SRC_POD=$(kubectl --context $SRC_CTX -n $SRC_NS get pod \ + -l spilo-role=master,cluster-name=$SRC_CLUSTER -o jsonpath='{.items[0].metadata.name}') + +kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ + psql -U postgres -tA -c \ + "SELECT datname, pg_get_userbyid(datdba) AS owner FROM pg_database + WHERE NOT datistemplate AND datname <> 'postgres' ORDER BY 1;" +``` + +Every CR-managed database in this list must be declared in the target CR (`spec.databases`, with its owner in `spec.users`) — the operator creates the roles and databases and manages their credentials; do not attempt to dump global objects (roles) with `pg_dumpall` from the source. The `postgres` maintenance database is managed by the operator/image on each side and is not migrated. Databases owned by `postgres` (created outside the CR spec) do not go into the CR — the migration script creates them on the target automatically. + +The `users:`/`databases:` sections of the target CR can be generated directly from the source instead of hand-written. The `users:` generator enumerates **all** application roles from `pg_roles` — not just database owners — because read-only, monitoring, and per-service accounts that own no database must also exist on the target, and it carries each role's attributes as `userFlags` (a role generated as `some_role: []` on the target would silently lose `NOLOGIN`, `CREATEDB`, etc.): + +```bash +{ + echo " users:" + kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ + psql -U postgres -tA -c \ + "SELECT ' ' || rolname || ': [' || array_to_string(ARRAY( + SELECT f FROM unnest(ARRAY[ + CASE WHEN rolsuper THEN 'superuser' END, + CASE WHEN rolcreatedb THEN 'createdb' END, + CASE WHEN rolcreaterole THEN 'createrole' END, + CASE WHEN NOT rolcanlogin THEN 'nologin' END]) f WHERE f IS NOT NULL), ',') || ']' + FROM pg_roles + WHERE rolname NOT LIKE 'pg\\_%' + AND rolname NOT IN ('postgres','standby','pooler','admin','zalandos','cron_admin','robot_zmon') + ORDER BY (rolname) COLLATE \"C\";" + echo " databases:" + kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ + psql -U postgres -tA -c \ + "SELECT ' ' || datname || ': ' || pg_get_userbyid(datdba) FROM pg_database + WHERE NOT datistemplate AND datname <> 'postgres' AND pg_get_userbyid(datdba) <> 'postgres' ORDER BY 1;" +} +``` + +Review the generated `users:` list before applying: the `NOT IN` filter excludes the standard operator/image system roles — if your deployment defines additional operator-managed roles, exclude those too rather than declaring them in the CR. + +### Preserve application credentials + +By default the target operator generates **new** passwords for the CR users, which would force every application to be re-configured at cutover. To keep the source passwords, copy each application user's credential secret into the target namespace **before creating the CR** — when the operator finds an existing credential secret, it adopts it and sets the role's password from it instead of generating a new one: + +```bash +SUFFIX="credentials.postgresql.acid.zalan.do" +for SEC in $(kubectl --context $SRC_CTX -n $SRC_NS get secret \ + -l application=spilo,cluster-name=$SRC_CLUSTER -o name); do + NAME=${SEC#secret/} + U=${NAME%.$SRC_CLUSTER.$SUFFIX} + [ "$U" = "$NAME" ] && continue # not a credential secret + case "$U" in postgres|standby|pooler) continue ;; esac # operator-managed system users + UB64=$(kubectl --context $SRC_CTX -n $SRC_NS get $SEC -o jsonpath='{.data.username}') + PB64=$(kubectl --context $SRC_CTX -n $SRC_NS get $SEC -o jsonpath='{.data.password}') + [ -n "$UB64" ] && [ -n "$PB64" ] \ + || { echo "WARNING: $NAME has an empty username/password field — skipping (fix on the source first)" >&2; continue; } + kubectl --context $TGT_CTX -n $TGT_NS apply -f - < +``` + +Restoring into a single instance keeps the full-restore WAL burst off streaming replication and the WAL archive; scale `numberOfInstances` up after verification (Step 4). Size `volume` with headroom — a logical restore holds data, indexes under construction, and the restore's own WAL at the same time. + +Wait until the instance is actually ready — `Running` reports pod health, while the operator creates roles, databases, and credential secrets **asynchronously after** that, so poll for the objects Step 3 depends on: + +```bash +until [ "$(kubectl --context $TGT_CTX -n $TGT_NS get postgresql $TGT_CLUSTER \ + -o jsonpath='{.status.PostgresClusterStatus}')" = "Running" ]; do sleep 5; done + +TGT_POD=$(kubectl --context $TGT_CTX -n $TGT_NS get pod \ + -l spilo-role=master,cluster-name=$TGT_CLUSTER -o jsonpath='{.items[0].metadata.name}') + +# every CR-declared user must exist as a role, with its credential secret: +until [ "$(kubectl --context $TGT_CTX -n $TGT_NS exec $TGT_POD -c postgres -- \ + psql -U postgres -tA -c "SELECT count(*) FROM pg_roles WHERE rolname = 'app_owner';")" = "1" ] \ + && kubectl --context $TGT_CTX -n $TGT_NS get secret \ + "app-owner.$TGT_CLUSTER.credentials.postgresql.acid.zalan.do" >/dev/null 2>&1 +do sleep 5; done +``` + +Note the secret's name: the operator sanitizes role names into RFC 1123 form, so a role like `app_owner` gets the secret `app-owner..credentials...` (`_` becomes `-`). Use the sanitized form wherever a secret is fetched by name. + +### Match the source's pod security settings + +Pod security is a property of the CR, not of the data — nothing in this procedure carries it over, so the target runs with whatever its own CR declares. Copy the whole set from the source: + +```bash +kubectl --context $SRC_CTX -n $SRC_NS get postgresql $SRC_CLUSTER -o jsonpath=' +restrictedPsaEnabled: {.spec.restrictedPsaEnabled} +spiloAllowPrivilegeEscalation: {.spec.spiloAllowPrivilegeEscalation} +spiloPrivileged: {.spec.spiloPrivileged} +spiloRunAsUser: {.spec.spiloRunAsUser} +spiloRunAsGroup: {.spec.spiloRunAsGroup} +{"\n"}' +``` + +**Treat these five as one setting, not five independent ones.** If the target namespace enforces the restricted Pod Security Standard, `restrictedPsaEnabled: true` on its own is *not* enough to get the instance running: + +- `restrictedPsaEnabled` gives the pods `runAsNonRoot: true`, a `RuntimeDefault` seccomp profile, `allowPrivilegeEscalation: false` and `capabilities.drop: [ALL]`. It does **not** clear `privileged`, which follows the operator's own configuration and can default to `true` — a privileged container is rejected under the restricted standard no matter what else the CR says. `spiloPrivileged: false` is what clears it. +- `runAsNonRoot: true` without a non-root uid is a runtime failure rather than an admission one: the pod is admitted, then kubelet refuses to start a container whose image would run as root. `spiloRunAsUser: 101` / `spiloRunAsGroup: 103` are the operator's own defaults for the Spilo image. + +Instances created through the platform console carry all five together; hand-applied CRs generally carry none. Check what the target namespace demands before creating the CR: + +```bash +kubectl --context $TGT_CTX get ns $TGT_NS \ + -o jsonpath='{.metadata.labels.pod-security\.kubernetes\.io/enforce}{"\n"}' +``` + +Both directions of mismatch cost you something. Set too little and the target never reaches `Running`, so Step 3 has no pod to exec into. Set nothing while the source was restricted and the migration quietly returns the application to a weaker security posture than it ran with before cutover. + +The migration itself is unaffected either way: every operation in Step 3 runs through `kubectl exec` and the pod-local socket, and the dump files are written on the workstation, not inside the pod. + +**These fields have a version floor.** They exist only from operator **4.2.0** onward — no 4.0.x or 4.1.x release has them, and they are carried by the CRD the operator bundle installs, not by the CRD in the Helm chart. Where a property is missing from the CRD, the API server **prunes it on apply**: no error, no warning, it simply is not there afterwards. So on an older target you can apply the CR above, see it accepted, and get pods with none of the security context you thought you asked for. Read the values back rather than trusting the apply: + +```bash +kubectl --context $TGT_CTX -n $TGT_NS get postgresql $TGT_CLUSTER \ + -o jsonpath='{.spec.restrictedPsaEnabled}{"\n"}' +``` + +An empty line here — immediately after applying a CR that set it — means this target cannot honour these settings at all. If the source is restricted and the target operator predates 4.2.0, upgrade the target operator before migrating, or accept the difference as a deliberate decision rather than discovering it after cutover. + +## Step 2: Stop Writes + +Stop application writes on the source — the transfer is a point-in-time copy, and anything written after the dump starts is lost. + +Do not rely on "the team said the app is stopped": **enforce** it. Blocking new connections and terminating leftovers turns a would-be silent data loss into an immediate, visible failure: + +```bash +# per application database: no new non-superuser connections, kill the stragglers +kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ + psql -U postgres -c "ALTER DATABASE CONNECTION LIMIT 0;" +kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ + psql -U postgres -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity + WHERE datname = '' AND usename NOT IN ('postgres','standby');" +``` + +(If you abort the migration, restore access with `ALTER DATABASE CONNECTION LIMIT -1;`.) As a second line of defense, the Step 3 script re-reads the source row counts after each transfer and fails that database with `SOURCE CHANGED DURING DUMP` if anything moved. + +The script records and compares exact per-table row counts automatically. For stronger guarantees on critical tables, additionally record per-table checksums now and re-check them in Step 4. Use a construct that is stable across PostgreSQL versions and explicit about NULLs (an aggregate over an expression that can be NULL silently skips those rows): + +```bash +kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ + psql -U postgres -d -tA -c \ + "SELECT count(*), md5(string_agg(id::text || ':' || coalesce(payload,''), '|' ORDER BY id)) + FROM ;" +``` + +## Step 3: Run the Migration Script + +The script performs the whole migration in one run: it enumerates the source's application databases, ensures each exists on the target **with the source's encoding and locale** (recreating empty mismatched ones — this also covers CR-created databases, which the operator builds with target defaults), pre-creates missing extensions preserving their schema and version, transfers each database, carries over database-level settings, runs `ANALYZE`, and verifies row counts, an object census, and sequence positions — reporting `PASS`/`FAIL` per database and exiting non-zero if anything failed. + +Fill in the six variables at the top, then run it with `bash`. Without arguments it migrates every database; pass database names (`bash migrate.sh appdb`) to migrate only those — used to redo a single database after a failure. Every transfer goes through the same script in one of two modes: + +- **File mode** (default; override the location with `DUMP_DIR=/path`): each database is dumped to `$DUMP_DIR/.dump` first, then restored from the file. This is the default because the long-lived `kubectl exec` streams are the fragile link in this relay — API-server load balancers and kubelet idle timeouts can cut a multi-hour stream, and a file makes that cost one database instead of the whole run. By default every run takes a **fresh** dump; set `REUSE_DUMPS=1` to reuse completed dumps (resumable). Reuse is only safe **within the same stopped-write maintenance window** — a dump taken before source writes resumed would silently migrate stale data. +- **Pipe mode** (`PIPE_MODE=1 bash migrate.sh`): source streams straight into the target — no intermediate storage on the workstation. Acceptable for small instances (roughly under 5 GB) where a cut stream just means a quick rerun; for anything that takes hours, use file mode. + +```bash +#!/usr/bin/env bash +# Whole-instance PostgreSQL migration relayed through the workstation. +# Migrates every application database of $SRC_CLUSTER into $TGT_CLUSTER. +set -u -o pipefail + +SRC_CTX=""; SRC_NS=""; SRC_CLUSTER="" +TGT_CTX=""; TGT_NS=""; TGT_CLUSTER="" + +# File mode is the default (resumable; a broken exec stream costs one database, +# not the run). PIPE_MODE=1 streams directly — for small instances only. +DUMP_DIR="${DUMP_DIR:-./pg-migrate-dumps}" + +SRC_POD=$(kubectl --context "$SRC_CTX" -n "$SRC_NS" get pod \ + -l spilo-role=master,cluster-name="$SRC_CLUSTER" -o jsonpath='{.items[0].metadata.name}') \ + && [ -n "$SRC_POD" ] || { echo "ERROR: cannot find source master pod for $SRC_CLUSTER" >&2; exit 1; } +TGT_POD=$(kubectl --context "$TGT_CTX" -n "$TGT_NS" get pod \ + -l spilo-role=master,cluster-name="$TGT_CLUSTER" -o jsonpath='{.items[0].metadata.name}') \ + && [ -n "$TGT_POD" ] || { echo "ERROR: cannot find target master pod for $TGT_CLUSTER" >&2; exit 1; } + +srcsql() { local db=$1; shift; kubectl --context "$SRC_CTX" -n "$SRC_NS" exec "$SRC_POD" -c postgres -- psql -U postgres -d "$db" -tA -v ON_ERROR_STOP=1 "$@"; } +tgtsql() { local db=$1; shift; kubectl --context "$TGT_CTX" -n "$TGT_NS" exec "$TGT_POD" -c postgres -- psql -U postgres -d "$db" -tA -v ON_ERROR_STOP=1 "$@"; } +tgtexec() { kubectl --context "$TGT_CTX" -n "$TGT_NS" exec "$TGT_POD" -c postgres -- "$@"; } + +# Multi-hour dumps/restores must not be killed by timeouts configured on the +# instance or its roles. +PGOPT='-c statement_timeout=0 -c lock_timeout=0 -c idle_in_transaction_session_timeout=0' + +# Client binaries matching the SOURCE server major; images bundle a bounded set +# of majors, so verify before starting instead of failing mid-restore. +SRC_MAJOR=$(srcsql postgres -c "SHOW server_version;" | cut -d. -f1) +case "$SRC_MAJOR" in ''|*[!0-9]*) echo "ERROR: cannot determine source PostgreSQL major (psql via $SRC_POD failed)" >&2; exit 1 ;; esac +PG_BIN=/usr/lib/postgresql/$SRC_MAJOR/bin +tgtexec test -x "$PG_BIN/pg_restore" \ + || { echo "ERROR: $PG_BIN/pg_restore not present in the target image" >&2; exit 1; } +echo "Source PostgreSQL major: $SRC_MAJOR (client binaries: $PG_BIN)" + +# Verification queries. The explicit COLLATE "C" pins the sort order — the two +# databases can legitimately have different collations, and a text comparison +# of differently-ordered output would report false differences. The object +# census excludes extension-owned objects (extension versions differ across +# PostgreSQL majors) and counts everything else by kind. The ::text cast on +# relkind is required: relkind is type "char", and from PostgreSQL 15 onward +# "char" || is ambiguous ("operator is not unique") — uncast, the +# census errors out on every modern target. +COUNT_QUERY="SELECT schemaname||'.'||relname, (xpath('/row/c/text()', query_to_xml(format( + 'SELECT count(*) AS c FROM %I.%I', schemaname, relname), false, true, '')))[1]::text::bigint + FROM pg_stat_user_tables + WHERE schemaname NOT IN ('metric_helpers','user_management') + ORDER BY (schemaname||'.'||relname) COLLATE \"C\";" +OBJ_QUERY="SELECT c.relkind::text||':'||count(*) FROM pg_class c + JOIN pg_namespace n ON n.oid = c.relnamespace + WHERE n.nspname NOT IN ('pg_catalog','information_schema','pg_toast','metric_helpers','user_management') + AND NOT EXISTS (SELECT 1 FROM pg_depend dep WHERE dep.objid = c.oid AND dep.deptype = 'e') + GROUP BY c.relkind ORDER BY (c.relkind::text) COLLATE \"C\";" +SEQ_QUERY="SELECT schemaname||'.'||sequencename||'='||coalesce(last_value,0) FROM pg_sequences + WHERE schemaname NOT IN ('metric_helpers','user_management') + ORDER BY (schemaname||'.'||sequencename) COLLATE \"C\";" + +FAILED=""; ATTEMPTED=0 +while read -r DB OWNER; do + [ -z "$DB" ] && continue + # With arguments, migrate only the named databases (e.g. to redo one FAIL). + if [ "$#" -gt 0 ]; then + case " $* " in *" $DB "*) ;; *) continue ;; esac + fi + ATTEMPTED=$((ATTEMPTED+1)) + echo "=== migrating database: $DB (owner: $OWNER) ===" + + # The target database must exist with the SOURCE's encoding and locale — + # plain CREATE DATABASE inherits the target template's defaults, silently + # changing sort order and LIKE behavior (or failing the restore outright on + # an encoding mismatch). TEMPLATE template0 is required to override them. + # CR-declared databases are covered too: the operator creates those with + # target defaults, so an empty database with the wrong locale is recreated. + read -r ENC COLL CTYPE < <(srcsql postgres -F' ' -c \ + "SELECT pg_encoding_to_char(encoding), datcollate, datctype FROM pg_database WHERE datname = '$DB';") + CREATE_SQL="CREATE DATABASE \"$DB\" OWNER \"$OWNER\" TEMPLATE template0 ENCODING '$ENC' LC_COLLATE '$COLL' LC_CTYPE '$CTYPE';" + TGT_LOC=$(tgtsql postgres -c "SELECT pg_encoding_to_char(encoding)||'/'||datcollate||'/'||datctype FROM pg_database WHERE datname = '$DB';") + if [ -z "$TGT_LOC" ]; then + echo " creating database $DB on target ($ENC/$COLL/$CTYPE)" + tgtsql postgres -c "$CREATE_SQL" >/dev/null || { FAILED="$FAILED $DB(create)"; continue; } + elif [ "$TGT_LOC" != "$ENC/$COLL/$CTYPE" ]; then + if [ "$(tgtsql "$DB" -c "SELECT count(*) FROM pg_stat_user_tables WHERE schemaname NOT IN ('metric_helpers','user_management');")" = "0" ]; then + echo " recreating database $DB on target with source locale ($ENC/$COLL/$CTYPE, was $TGT_LOC)" + tgtsql postgres -c "DROP DATABASE \"$DB\";" >/dev/null \ + && tgtsql postgres -c "$CREATE_SQL" >/dev/null \ + || { FAILED="$FAILED $DB(locale)"; continue; } + else + echo " FAIL: $DB exists on target with different locale ($TGT_LOC vs $ENC/$COLL/$CTYPE) and is not empty" + FAILED="$FAILED $DB(locale)"; continue + fi + fi + + # Pre-create only the extensions the target is missing (creation needs + # superuser), preserving the source's schema placement and version — a + # relocated or version-drifted extension breaks dependent objects at restore. + while read -r EXT ESCH EVER; do + [ -z "$EXT" ] && continue + [ "$(tgtsql "$DB" -c "SELECT 1 FROM pg_extension WHERE extname = '$EXT';")" = "1" ] && continue + tgtsql "$DB" -c "CREATE SCHEMA IF NOT EXISTS \"$ESCH\";" >/dev/null 2>&1 + tgtsql "$DB" -c "CREATE EXTENSION \"$EXT\" WITH SCHEMA \"$ESCH\" VERSION '$EVER';" >/dev/null 2>&1 \ + || tgtsql "$DB" -c "CREATE EXTENSION \"$EXT\" WITH SCHEMA \"$ESCH\";" >/dev/null 2>&1 \ + || echo " WARNING: could not create extension $EXT (schema $ESCH) on target — restore may fail" + done < <(srcsql "$DB" -F' ' -c \ + "SELECT e.extname, n.nspname, e.extversion FROM pg_extension e + JOIN pg_namespace n ON n.oid = e.extnamespace WHERE e.extname <> 'plpgsql';") + + # Baseline on the source (exact counts, object census, sequence positions). + # Every capture is guarded. A verification query that ERRORS must never be + # mistaken for one that returned nothing: psql exits non-zero, the script + # carries on (there is deliberately no `set -e` — one bad database must not + # abort a multi-hour run), and two failed queries would then compare equal + # and report PASS for a database nobody verified. An unverifiable database + # is a FAILED database. + VERIFY_ERR="" + SRC_COUNTS=$(srcsql "$DB" -c "$COUNT_QUERY") || VERIFY_ERR="source row counts" + SRC_OBJS=$(srcsql "$DB" -c "$OBJ_QUERY") || VERIFY_ERR="source object census" + SRC_SEQS=$(srcsql "$DB" -c "$SEQ_QUERY") || VERIFY_ERR="source sequence positions" + if [ -n "$VERIFY_ERR" ]; then + echo " FAIL: $DB (baseline query failed: $VERIFY_ERR — not migrating a database that cannot be verified)" + FAILED="$FAILED $DB(verify)"; continue + fi + + # Transfer. The restore connects as postgres over the pod-local socket and + # switches to the owner with --role: objects land owned by the owner, with + # no password handling and nothing secret in the kubectl argument list. + if [ "${PIPE_MODE:-0}" != "0" ]; then + kubectl --context "$SRC_CTX" -n "$SRC_NS" exec "$SRC_POD" -c postgres -- \ + env PGOPTIONS="$PGOPT" \ + "$PG_BIN/pg_dump" -U postgres -Fc --no-comments \ + -N metric_helpers -N user_management "$DB" \ + | kubectl --context "$TGT_CTX" -n "$TGT_NS" exec -i "$TGT_POD" -c postgres -- \ + env PGOPTIONS="$PGOPT" \ + "$PG_BIN/pg_restore" -U postgres --role="$OWNER" -d "$DB" --no-owner --no-tablespaces -x + RC=$? + else + install -d -m 700 "$DUMP_DIR" + DUMP_FILE="$DUMP_DIR/$DB.dump" + if [ "${REUSE_DUMPS:-0}" != "0" ] && [ -s "$DUMP_FILE" ]; then + echo " reusing existing dump $DUMP_FILE" + else + kubectl --context "$SRC_CTX" -n "$SRC_NS" exec "$SRC_POD" -c postgres -- \ + env PGOPTIONS="$PGOPT" \ + "$PG_BIN/pg_dump" -U postgres -Fc --no-comments \ + -N metric_helpers -N user_management "$DB" > "$DUMP_FILE.partial" \ + && mv "$DUMP_FILE.partial" "$DUMP_FILE" \ + || { rm -f "$DUMP_FILE.partial"; FAILED="$FAILED $DB(dump)"; continue; } + fi + kubectl --context "$TGT_CTX" -n "$TGT_NS" exec -i "$TGT_POD" -c postgres -- \ + env PGOPTIONS="$PGOPT" \ + "$PG_BIN/pg_restore" -U postgres --role="$OWNER" -d "$DB" --no-owner --no-tablespaces -x < "$DUMP_FILE" + RC=$? + fi + + # Carry over database- and role-in-database-level settings + # (ALTER DATABASE ... SET / ALTER ROLE ... IN DATABASE ... SET) — pg_dump + # without --create does not emit them, so they would silently disappear. + srcsql postgres -c \ + "SELECT 'ALTER '||CASE WHEN s.setrole = 0 THEN 'DATABASE '||quote_ident(d.datname) + ELSE 'ROLE '||quote_ident(r.rolname)||' IN DATABASE '||quote_ident(d.datname) END + ||' SET '||quote_ident(split_part(cfg,'=',1))||' = '||quote_literal(substr(cfg, strpos(cfg,'=')+1))||';' + FROM pg_db_role_setting s + JOIN pg_database d ON d.oid = s.setdatabase + LEFT JOIN pg_roles r ON r.oid = s.setrole + CROSS JOIN LATERAL unnest(s.setconfig) cfg + WHERE d.datname = '$DB';" \ + | kubectl --context "$TGT_CTX" -n "$TGT_NS" exec -i "$TGT_POD" -c postgres -- \ + psql -U postgres -v ON_ERROR_STOP=1 -f - >/dev/null \ + || echo " WARNING: could not carry over ALTER DATABASE/ROLE ... SET settings for $DB" + + # A restore ships no planner statistics — without ANALYZE the first minutes + # of production traffic run on empty stats (sequential scans everywhere). + # This belongs INSIDE the maintenance window, not after cutover. + tgtexec env PGOPTIONS="$PGOPT" "$PG_BIN/vacuumdb" -U postgres --analyze-in-stages -d "$DB" >/dev/null 2>&1 \ + || echo " WARNING: vacuumdb --analyze-in-stages failed for $DB — run ANALYZE manually before cutover" + + # Verify: rows, object census, and sequence positions must match — and the + # source must not have changed while the transfer ran (writes not stopped). + VERIFY_ERR="" + TGT_COUNTS=$(tgtsql "$DB" -c "$COUNT_QUERY") || VERIFY_ERR="target row counts" + TGT_OBJS=$(tgtsql "$DB" -c "$OBJ_QUERY") || VERIFY_ERR="target object census" + TGT_SEQS=$(tgtsql "$DB" -c "$SEQ_QUERY") || VERIFY_ERR="target sequence positions" + SRC_RECHECK=$(srcsql "$DB" -c "$COUNT_QUERY") || VERIFY_ERR="source re-check" + if [ -n "$VERIFY_ERR" ]; then + echo " FAIL: $DB (verification query failed: $VERIFY_ERR — the data may be intact, but this run did NOT verify it; do not cut over on this result)" + FAILED="$FAILED $DB(verify)" + elif [ "$SRC_RECHECK" != "$SRC_COUNTS" ]; then + echo " FAIL: $DB (SOURCE CHANGED DURING DUMP — enforce the write stop per Step 2, then redo this database)" + FAILED="$FAILED $DB" + elif [ "$RC" -eq 0 ] && [ "$SRC_COUNTS" = "$TGT_COUNTS" ] && [ "$SRC_OBJS" = "$TGT_OBJS" ] && [ "$SRC_SEQS" = "$TGT_SEQS" ]; then + echo " PASS: $DB (rows, objects, and sequences identical)" + else + echo " FAIL: $DB (transfer exit $RC; rows $( [ "$SRC_COUNTS" = "$TGT_COUNTS" ] && echo match || echo DIFFER ); objects $( [ "$SRC_OBJS" = "$TGT_OBJS" ] && echo match || echo DIFFER ); sequences $( [ "$SRC_SEQS" = "$TGT_SEQS" ] && echo match || echo DIFFER ))" + FAILED="$FAILED $DB" + fi +done < <(srcsql postgres -F' ' -c \ + "SELECT datname, pg_get_userbyid(datdba) FROM pg_database + WHERE NOT datistemplate AND datname <> 'postgres' ORDER BY (datname) COLLATE \"C\";") + +echo +if [ "$ATTEMPTED" -eq 0 ]; then + echo "ERROR: no databases migrated — source enumeration failed, or no database matched the arguments" >&2 + exit 1 +fi +if [ -n "$FAILED" ]; then echo "MIGRATION INCOMPLETE — failed:$FAILED"; exit 1; fi +echo "MIGRATION COMPLETE — $ATTEMPTED database(s) verified." +``` + +If a database reports `FAIL`, see [Troubleshooting](#troubleshooting); after fixing the cause, redo just that database by passing its name to the script (in file mode, add `REUSE_DUMPS=1` to skip re-dumping if writes have stayed stopped). + +### Security notes + +- This relay deliberately makes the administrator workstation a data path between two otherwise-isolated environments — exactly what an isolation policy exists to control. Confirm the data path is approved by your security/compliance owner before migrating; "each hop is TLS" is a transport property, not an authorization. +- The migration script itself handles no database passwords: dumps and restores connect as `postgres` over the pod-local socket (the restore switches to the owner with `--role`). The only password-bearing command in this guide is the Step 4 login test — see the caveat there. +- In file mode (the default), `$DUMP_DIR` holds full plaintext logical copies of the databases. The script creates it with mode `700`; if you point `DUMP_DIR` at an existing directory, restrict it yourself, encrypt at rest if your policy requires, and delete the dumps once the migration is verified. +- The data traverses the two TLS-protected `kubectl exec` channels; nothing is exposed on the network beyond the two Kubernetes API connections. + +## Step 4: Verify Application Access + +The script has already verified per-table row counts, an object census, and sequence positions, and has run `ANALYZE` on each database. Confirm in addition that each application user can actually query its data with the **target**-managed credentials (this catches ownership problems immediately), and re-check any Step 2 checksums. If you pre-staged the credential secrets in Step 1, this password is identical to the source one — applications keep their existing credentials and only the connection endpoint changes at cutover. Note the secret name uses the RFC 1123-sanitized role name (`app_owner` → `app-owner`): + +```bash +TGT_POD=$(kubectl --context $TGT_CTX -n $TGT_NS get pod \ + -l spilo-role=master,cluster-name=$TGT_CLUSTER -o jsonpath='{.items[0].metadata.name}') + +APP_PW=$(kubectl --context $TGT_CTX -n $TGT_NS get secret \ + .$TGT_CLUSTER.credentials.postgresql.acid.zalan.do \ + -o jsonpath='{.data.password}' | base64 -d) + +kubectl --context $TGT_CTX -n $TGT_NS exec $TGT_POD -c postgres -- \ + env PGPASSWORD="$APP_PW" psql -U -h localhost -d -tA -c \ + "SELECT current_user, count(*) FROM ;" +``` + +**Caveat:** `kubectl exec` places its command — including this `PGPASSWORD` value — in the API request URI, which Kubernetes audit logging records at `Metadata` level and above, and it is also visible in local `ps` output while the command runs. This is a deliberate, one-off login *test*; if your audit policy treats recorded credentials as exposed, rotate this password after cutover (or verify from an application pod instead). + +The restore skips ACL statements (`pg_restore -x`) because they reference roles managed by the source operator. The `GRANT`s themselves are still **inside the dump file** — if additional (non-owner) users need privileges on the migrated data, either re-grant manually now, or list the dump's ACL entries (`"$PG_BIN/pg_restore" -l .dump | grep ' ACL '`) and replay just those with `pg_restore -L` once the roles exist on the target. + +## Step 5: Cut Over and Clean Up + +- Scale the target back out (`numberOfInstances: 2` or your HA baseline) and wait for the replica to catch up. +- Repoint applications to the target instance's service and re-enable writes. +- Retire the source in stages rather than deleting it immediately — it is your only rollback for data problems discovered after cutover. Keep it read-only and scaled to zero for an observation period first: + +```bash +# freeze the source but keep its data (rollback insurance) +kubectl --context $SRC_CTX -n $SRC_NS patch postgresql $SRC_CLUSTER \ + --type merge -p '{"spec":{"numberOfInstances":0}}' +``` + +- After the observation period (days, per your policy), delete it. There is no replication configuration to tear down. If your operator is configured with delete-protection annotations (`delete_annotation_date_key`/`delete_annotation_name_key`), set those annotations on the CR first or the operator ignores the deletion: + +```bash +kubectl --context $SRC_CTX -n $SRC_NS delete postgresql $SRC_CLUSTER +# PVC retention depends on operator configuration — remove leftovers: +kubectl --context $SRC_CTX -n $SRC_NS delete pvc -l cluster-name=$SRC_CLUSTER --ignore-not-found +``` + +## Troubleshooting + +### Application user gets `permission denied for table ...` after a successful restore + +The restore ran without switching to the application user, so all objects are owned by `postgres` (the script connects as `postgres` but switches with `--role=`; this typically comes from a hand-run restore that omitted `--role`). Either redo the database — drop and recreate it, then rerun the script with the database name — or transfer ownership in place: + +```sql +DO $$ +DECLARE r record; +BEGIN + FOR r IN SELECT tablename FROM pg_tables WHERE schemaname = 'public' LOOP + EXECUTE format('ALTER TABLE public.%I OWNER TO app_owner', r.tablename); + END LOOP; + FOR r IN SELECT sequencename FROM pg_sequences WHERE schemaname = 'public' LOOP + EXECUTE format('ALTER SEQUENCE public.%I OWNER TO app_owner', r.sequencename); + END LOOP; +END $$; +``` + +This block covers tables and sequences in the `public` schema only. Views, functions, types, and objects in other schemas need analogous `ALTER ... OWNER TO` statements (enumerate them via `pg_views`, `pg_proc`, `pg_type`, or `\dn`/`\df` in psql). + +Related scope note: the migration's `--no-owner` model assumes the operator's standard layout — one database wholly owned by one CR-defined user. If a database has multiple owning roles, `SECURITY DEFINER` functions, or objects in non-`public` schemas with distinct owners, ownership collapses onto the restoring user; plan an explicit post-restore ownership and grant pass for those objects. + +### Restore reports `permission denied to create extension ...` + +The source database uses an extension that is not pre-installed on the target, and the application user cannot create it. The script warns about this (`WARNING: could not create extension`). Create the extension as `postgres` on the target database, then rerun the restore. If `CREATE EXTENSION` itself fails with a missing-file error, the extension's packages are not present in the target image at all — it must be added to the image/instance before this migration can carry that database. + +### Restore reports `must be owner of extension pg_stat_statements` (and similar) + +`--no-comments` was omitted from `pg_dump`; the extension comments require superuser. The data restores fine — the errors only break the clean exit code. Rerun with `--no-comments` for a verifiable result. + +### Restore reports `unrecognized configuration parameter "transaction_timeout"` + +The dump was taken with a `pg_dump` newer than the target server (typically the image's default binary — a hand-run dump without the explicit `PG_BIN` path; the script pins `$PG_BIN` to the source major, which it verifies exists on both sides before starting). Rerun with the version-matched binary path. If the script's own pre-check fails instead (`pg_restore not present in the target image` — very old or custom images bundle fewer majors), run the dump and restore with a workstation-side PostgreSQL client of the matching major rather than the in-pod binaries. + +### A database reports `FAIL ... SOURCE CHANGED DURING DUMP` + +Source writes were not fully stopped: the script re-reads the source row counts after each transfer and refuses to PASS a database whose source moved mid-copy. Enforce the write stop (Step 2 — `CONNECTION LIMIT 0` plus `pg_terminate_backend`), then redo that database (`bash migrate.sh ` after dropping it on the target). + +### Restore reports `schema "metric_helpers" already exists` (and similar) + +The `-N metric_helpers -N user_management` exclusions were omitted. These errors are harmless for the excluded schemas' objects, but rerun with the exclusions for a clean, verifiable exit code. + +### Rerunning after a failed restore + +Drop the target database first — a partial restore leaves objects that collide (`relation already exists`, duplicate-key COPY failures). Note `DROP DATABASE` must be executed as its own single statement: + +```bash +kubectl --context $TGT_CTX -n $TGT_NS exec $TGT_POD -c postgres -- psql -U postgres -c "DROP DATABASE appdb" +``` + +Then rerun the script with **only the failed database** as an argument (`bash migrate.sh `) — it recreates the database with the correct owner and locale; add `REUSE_DUMPS=1` to restore from the completed dump instead of re-dumping (only if source writes have stayed stopped). Do not rerun it without arguments after a partial success: restoring into the databases that already transferred produces `already exists` collisions that mark them as failed. + +### Application passwords changed after the migration + +The target CR was created **before** the credential secrets were pre-staged (Step 1), so the operator generated new passwords. To restore the source credentials after the fact, update both the secret and the role together — the secret alone is not enough, and an `ALTER ROLE` alone would be reverted by the operator's next sync from the secret. The commands below stay safe for passwords containing quotes, backslashes, or JSON metacharacters: the secret is patched with the base64 value (JSON-safe by construction), and the SQL uses psql's `:'pw'` literal quoting instead of string interpolation: + +```bash +PW_B64=$(kubectl --context $SRC_CTX -n $SRC_NS get secret \ + app-owner.$SRC_CLUSTER.credentials.postgresql.acid.zalan.do -o jsonpath='{.data.password}') + +kubectl --context $TGT_CTX -n $TGT_NS patch secret \ + app-owner.$TGT_CLUSTER.credentials.postgresql.acid.zalan.do \ + --type merge -p "{\"data\":{\"password\":\"$PW_B64\"}}" + +kubectl --context $TGT_CTX -n $TGT_NS exec -i $TGT_POD -c postgres -- \ + psql -U postgres -v pw="$(printf %s "$PW_B64" | base64 -d)" <<'SQL' +ALTER ROLE app_owner PASSWORD :'pw'; +SQL +``` + +Verify with a login test as in Step 4. + +### Transfer is slow + +Throughput is bounded by the workstation's link to both API servers (every byte traverses it twice: exec stream in, exec stream out). Run the migration from a machine with good connectivity to both platforms (e.g. a jump host), and keep the default file mode so progress is resumable per database. If a very large restore needs parallelism, note `pg_restore -j N` cannot read from stdin — copy the dump file into the target pod (`kubectl cp`) and restore from the local path there. If the measured rehearsal duration does not fit any acceptable window, this relay is the wrong tool for that instance — plan a network-connected migration path instead. diff --git a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_to_Another_Node.md b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_to_Another_Node.md new file mode 100644 index 000000000..efb7e789f --- /dev/null +++ b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_to_Another_Node.md @@ -0,0 +1,134 @@ +--- +kind: + - Solution +products: + - Alauda Application Services +ProductsVersion: + - 4.0,4.1,4.2,4.3 +id: KB260515007 +--- + +# How to Migrate a PostgreSQL Instance to Another Node + +## Issue + +A PostgreSQL cluster managed by the PostgreSQL Operator uses node-local storage +(for example TopoLVM). One or more instances must be moved off their current +node — because the node is being decommissioned, or an instance must run on a +specific compute node. Because the data lives in a node-local PersistentVolume, +the pod cannot simply be rescheduled; the volume is pinned to its node. + +## Environment + +- Alauda Application Services PostgreSQL Operator (Zalando-based, + `acid.zalan.do/v1` `postgresql` resource). +- Node-local storage such as TopoLVM (each PVC is bound to one node). +- At least one target node with enough free capacity. Because migration + re-clones data, the target node should have capacity for roughly **twice** the + instance's PVC size during the transition. + +## Resolution + +The migration relies on a property of the Operator: when an instance's PVC and +pod are deleted, the StatefulSet recreates the pod, a fresh PVC is provisioned +wherever the pod is scheduled, and Patroni re-clones the data from the current +leader. Data is preserved through streaming replication, not by moving the +volume. + +> Validated on ACP 4.2 and 4.3: after deleting a replica's PVC and pod, the +> member was recreated on a node, re-synced from the leader, and previously +> written rows were present on the resynced member. + +In the examples below set `$NAMESPACE` and `$CLUSTER_NAME` for the target +cluster. Replace placeholder node names with your own. + +### 1. Confirm the current placement + +```bash +kubectl get pod -n $NAMESPACE -o wide | grep $CLUSTER_NAME +kubectl get pvc -n $NAMESPACE -o wide | grep $CLUSTER_NAME +``` + +Note which member is the leader (do not delete the leader's volume first): + +```bash +kubectl exec -n $NAMESPACE $CLUSTER_NAME-0 -c postgres -- patronictl list +``` + +### 2. Restrict scheduling to the target node(s) + +So the recreated pod lands only on a desired node, cordon the other eligible +nodes (or use a `nodeSelector`/label that only the target nodes carry). Keep at +least the target node schedulable. + +```bash +# Label ONLY the target node(s) so the recreated pod can land there and nowhere else +kubectl label node target=true --overwrite +``` + +Do **not** label the source node — labeling both source and target would let the +pod reschedule back onto the source. If you prefer cordoning over labels, cordon +all non-target nodes instead and skip the `nodeSelector` step below. + +If you use a label-based selector, set it on the instance: + +```bash +kubectl patch postgresql -n $NAMESPACE $CLUSTER_NAME --type merge \ + -p '{"spec":{"nodeSelector":{"target":"true"}}}' +``` + +### 3. Migrate one member at a time + +Always migrate a **non-leader** member first. Delete its PVC and pod together — +the PVC deletion blocks until the pod that mounts it is gone, so delete the pod +in parallel: + +```bash +# Delete the data PVC (it will stay in Terminating until the pod is gone) +kubectl delete pvc pgdata-$CLUSTER_NAME-1 -n $NAMESPACE --wait=false + +# Delete the pod to release the PVC +kubectl delete pod $CLUSTER_NAME-1 -n $NAMESPACE +``` + +The StatefulSet recreates `$CLUSTER_NAME-1`; a new PVC is provisioned on the +scheduled node and Patroni re-clones from the leader. + +### 4. Verify the member rejoined with its data + +```bash +kubectl get pod -n $NAMESPACE -o wide | grep $CLUSTER_NAME-1 +kubectl exec -n $NAMESPACE $CLUSTER_NAME-0 -c postgres -- patronictl list +``` + +The migrated member should return to role `Replica`, state `running`/`streaming` +with `Lag in MB` `0`. Spot-check data on the member (it is read-only / in +recovery): + +```bash +kubectl exec -n $NAMESPACE $CLUSTER_NAME-1 -c postgres -- \ + psql -U postgres -tAc "SELECT pg_is_in_recovery();" # expect t +``` + +### 5. Migrate the (former) leader if required + +To move the leader, first perform a switchover so another member becomes leader, +then repeat steps 3–4 for the old leader: + +```bash +kubectl exec -n $NAMESPACE $CLUSTER_NAME-0 -c postgres -- \ + patronictl switchover $CLUSTER_NAME --force +``` + +### 6. Restore scheduling + +Uncordon any nodes you cordoned and remove temporary labels/`nodeSelector` once +all members are on their intended nodes. + +## Notes + +- Migrate members one at a time and wait for each to fully re-sync (`Lag = 0`) + before moving the next, so the cluster always retains a healthy quorum. +- For a single-instance cluster, temporarily scale to two instances, let the new + member sync on the target node, switch over, then scale back to one — this + avoids downtime that a delete-and-reclone of the sole instance would cause.