diff --git a/packages/ui/docs/supabase/migration-tooling.md b/packages/ui/docs/supabase/migration-tooling.md new file mode 100644 index 00000000..f1f2165f --- /dev/null +++ b/packages/ui/docs/supabase/migration-tooling.md @@ -0,0 +1,367 @@ +# Migration Tooling (Leaving Supabase) + +## Overview + +Chapter in the **Leaving Supabase** series: concrete CLI snippets used to back up data, export storage, restore databases, and rewrite image URLs while moving away from Supabase-managed services. + +The goal of this chapter is not to present generic examples, but to preserve the patterns that were actually useful during migration work in `pm-pics`. + +All code below is copied inline so the article stays self-contained. + +--- + +## 1. Common pattern: resolve the right `.env` + +Several commands use the same defensive pattern: try explicit `--env` first, then fall back through a small list of common locations and validate the file by checking whether it contains the expected variable. + +```ts +function resolveEnvPath(sourceDir: string, envFlag?: string): string { + if (envFlag) { + const explicit = resolve(sourceDir, envFlag); + if (!existsSync(explicit)) { + throw new Error(`Env file not found: ${explicit}`); + } + return explicit; + } + + const candidates = [ + ".env", + ".env.production", + "server/.env", + "server/.env.production", + ]; + + for (const name of candidates) { + const p = resolve(sourceDir, name); + if (!existsSync(p)) continue; + const content = readFileSync(p, "utf-8"); + if (!content.includes("DATABASE_URL")) continue; + return p; + } + + throw new Error(`No .env found in ${sourceDir}`); +} +``` + +Why this mattered: +- Migration scripts are often run from inconsistent working directories. +- A fast variable check avoids picking the wrong `.env`. +- The fallback order keeps local and production workflows predictable. + +--- + +## 2. Split database backups into structure and data + +The backup flow treats schema and data as separate artifacts. That makes restore and troubleshooting much easier than one giant dump. + +```ts +if (backupType === "structure" || backupType === "all") { + const outPath = join(dbDir, `structure-${date}.sql`); + await dumpSchema({ + pgUrl: env.dbUrl, + outPath, + schemas: ["public"], + excludeIndexes: argv.noIndexes || argv["no-indexes"] || false, + excludeTables, + }); +} + +if (backupType === "data" || backupType === "all") { + const outPath = join(dbDir, `data-${date}.sql`); + await dumpData({ + pgUrl: env.dbUrl, + outPath, + schemas: ["public"], + excludeTables: excludeDataTables, + }); +} +``` + +Why this mattered: +- Schema-only dumps are easier to replay during iterative migration. +- Data-only dumps let you re-import content without recreating everything. +- Large or derived tables can be excluded from data dumps while keeping schema intact. + +Also notable is the optional `auth.users` export: + +```ts +if (argv["include-auth"] || argv.includeAuth) { + const outPath = join(dbDir, `auth-users-${date}.sql`); + try { + await dumpTableData({ + pgUrl: env.dbUrl, + outPath, + table: "auth.users", + }); + } catch (err: any) { + log.warn(`auth.users dump failed (may not exist on target): ${err?.message ?? err}`); + } +} +``` + +This is useful when the schema still references Supabase auth rows during the transition. + +--- + +## 3. Recursively export Supabase Storage buckets + +The storage backup flow walks Supabase Storage recursively and downloads every object with bounded concurrency. + +```ts +interface StorageEntry { + name: string; + id?: string; + fullPath: string; +} + +async function listFilesRecursive(client: any, bucketId: string, path = ""): Promise { + const { data, error } = await client.storage.from(bucketId).list(path, { limit: 1000 }); + if (error || !data) return []; + + const out: StorageEntry[] = []; + for (const item of data) { + const fullPath = path ? `${path}/${item.name}` : item.name; + if (!item.id) { + const sub = await listFilesRecursive(client, bucketId, fullPath); + out.push(...sub); + } else { + out.push({ name: item.name, id: item.id, fullPath }); + } + } + return out; +} + +async function pMap( + items: T[], + worker: (item: T) => Promise, + concurrency: number, +): Promise { + let index = 0; + async function run(): Promise { + const current = index++; + if (current >= items.length) return; + await worker(items[current]); + await run(); + } + await Promise.all(Array.from({ length: Math.min(concurrency, items.length) }, () => run())); +} +``` + +Why this mattered: +- Supabase Storage listings are folder-like, so migrations need explicit recursion. +- Concurrency keeps exports fast without turning the migration into a thundering herd. +- This pattern works equally well for backup, verification, and cross-provider copy jobs. + +--- + +## 4. Mirror object storage into application-native paths + +One of the most useful ideas is not just downloading bucket objects, but writing additional alias files into the application's destination layout. + +```ts +if (bucket.id === "pictures") { + const aliases = pictureAliases.get(file.fullPath) || []; + if (aliases.length > 0) { + const ext = file.name.includes(".") + ? file.name.slice(file.name.lastIndexOf(".")) + : ""; + + for (const alias of aliases) { + const aliasPath = join(outRoot, bucket.id, "storage", alias.userId, `${alias.id}${ext}`); + await mkdir(dirname(aliasPath), { recursive: true }); + await writeFile(aliasPath, content); + } + } +} +``` + +Why this mattered: +- Supabase object keys are rarely the same shape as the target application's storage contract. +- Writing aliases during export reduces the amount of later rewrite logic. +- It turns backup into migration preparation instead of just archival. + +--- + +## 5. Restore into a plain Postgres target with a privileged URL + +The database import flow makes restore logic explicit: use a higher-privilege connection for DDL, optionally clear the public schema, then import schema and data separately. + +```ts +const envPath = resolveEnvPath(targetDir, argv.env); +const env = loadEnvFromFile(envPath); + +if (!env.serviceDbUrl && !env.dbUrl) { + throw new Error(`DATABASE_URL is empty in ${envPath}. Cannot connect to database.`); +} + +let importUrl = env.serviceDbUrl || env.dbUrl; + +if (argv.clear) { + await clearPublicSchema(importUrl); +} + +if (argv.schema) { + await restoreSchema({ + pgUrl: importUrl, + sqlPath: schemaPath, + }); +} + +if (argv.data) { + await restoreData({ + pgUrl: importUrl, + sqlPath: dataPath, + }); +} +``` + +Why this mattered: +- Restore usually needs more privileges than the application runtime pool should have. +- Resetting `public` explicitly makes repeated migration tests less fragile. +- Schema and data replay remain independently controllable. + +The same command also provisions a service role after restore: + +```ts +const createUser = argv.createUser || argv["create-user"]; +if (createUser) { + const createUserPassword = argv.createUserPassword || argv["create-user-password"]; + const pgbossSchema = + argv.pgbossSchema || + argv["pgboss-schema"] || + (dbname ? `pgboss_${dbname}` : "pgboss"); + + await createServiceUser(importUrl, createUser, createUserPassword, pgbossSchema); +} +``` + +This is a practical reminder that leaving Supabase is not only about data import; it also means recreating the operational roles the application expects. + +--- + +## 6. Rewrite image URLs from Supabase Storage to VFS + +The image migration flow is the most migration-specific command in the set. It identifies Supabase-based `pictures.image_url` values, derives the intended destination path, and updates rows to point at the new VFS endpoint. + +```ts +const q = ` + SELECT id, user_id, image_url + FROM pictures + WHERE image_url IS NOT NULL + AND ( + image_url LIKE '%/storage/v1/object/public/pictures/%' + OR image_url LIKE '%/api/images/render?%' + OR image_url LIKE '%storage%2Fv1%2Fobject%2Fpublic%2Fpictures%2F%' + ) + ORDER BY created_at ASC + ${limitSql} +`; + +const res = await pool.query(q); +``` + +```ts +const nextUrl = + `${serverBase}/api/vfs/get/${encodeURIComponent(vfsStore)}` + + `/${encodeURIComponent(row.user_id)}/${encodeURIComponent(filename)}`; + +await pool.query( + `UPDATE pictures SET image_url = $1, updated_at = NOW() WHERE id = $2 AND image_url = $3`, + [nextUrl, row.id, row.image_url] +); +``` + +Why this mattered: +- The migration had to support both direct Supabase URLs and wrapped render URLs. +- URL rewrites were done idempotently by checking the old value in the `WHERE` clause. +- The target path was derived from application semantics, not from Supabase storage keys alone. + +--- + +## 7. Hydrate missing files during URL migration + +The most robust part of the image migration flow is that it does not assume local storage is already complete. When the expected target file is missing, the command can fetch the old remote file and write it into the VFS layout on the fly. + +```ts +if (!existsSync(localFile)) { + const fetchRes = await fetch(row.image_url); + if (!fetchRes.ok) { + missingLocal++; + continue; + } + + const content = Buffer.from(await fetchRes.arrayBuffer()); + if (!dryRun) { + await mkdir(userDir, { recursive: true }); + await writeFile(localFile, content); + } + + hydratedMissing++; +} +``` + +Why this mattered: +- Real migrations are messy; backups are often incomplete. +- Hydration lets the URL rewrite and content recovery happen in one pass. +- This reduces the chance of ending up with database rows that point to files that do not exist. + +The command goes one step further and can also resolve files by content hash: + +```ts +async function resolveFilenameByContentMatch( + sourceUrl: string, + userDir: string, + cache: Map>, +): Promise { + let idx = cache.get(userDir); + if (!idx) { + idx = await indexUserFilesByHash(userDir); + cache.set(userDir, idx); + } + + const res = await fetch(sourceUrl); + if (!res.ok) return null; + const ab = await res.arrayBuffer(); + const h = sha256(new Uint8Array(ab)); + const candidates = idx.get(h); + return candidates?.[0] ?? null; +} +``` + +That is especially useful when filenames changed but file contents remained identical. + +--- + +## 8. A small but useful transport command + +The bucket transport command is short, but it captures a useful operational pattern: migrate selected buckets between two Supabase environments with a dry-run mode and bounded concurrency. + +```ts +const report = await migrateBuckets(source.client, target.client, { + buckets, + concurrency: argv.concurrency, + dryRun: argv["dry-run"], +}); +``` + +This command is less about the copy logic itself and more about orchestration: +- explicit source and target env files +- optional bucket filtering +- dry-run support before destructive or expensive copy work +- a summary report at the end + +--- + +## Practical takeaway + +The useful migration patterns were not “Supabase features”, but standard operational building blocks: + +- resolve environment files predictably +- split schema and data backups +- export object storage recursively with concurrency limits +- map provider-specific object paths into application-native paths +- restore using a privileged database URL +- rewrite URLs idempotently +- hydrate or recover missing files during migration + +That is another reason the long-term recommendation remains the same: **build on Postgres and application-owned storage conventions as early as possible**. diff --git a/packages/ui/docs/supabase/overview.md b/packages/ui/docs/supabase/overview.md new file mode 100644 index 00000000..1c1f1df4 --- /dev/null +++ b/packages/ui/docs/supabase/overview.md @@ -0,0 +1,83 @@ +# Leaving Supabase + +## Why We're Moving On + +Supabase is a compelling starting point — a managed Postgres layer with built-in auth, auto-generated REST APIs, and a slick dashboard. For prototyping, that's excellent. At scale, the cracks show. + +--- + +## The Problems + +### 1. Pricing Doesn't Scale + +At **$25–$30 per project instance**, Supabase is affordable for a single project but becomes a liability in any multi-tenant or multi-environment setup. Staging, production, and regional deployments each need their own instance. There is no meaningful tier between the free plan and a per-project paid plan — you either share one instance across everything (bad isolation) or pay per environment. + +By contrast, a self-hosted Postgres server on a VPS can serve many projects and environments for a flat monthly cost, with better hardware control and no artificial limits on database size, connections, or edge function invocations. + +### 2. Performance Is a Layer Problem + +Supabase does not give you raw Postgres access from the browser — it routes all queries through **PostgREST**, a REST-to-SQL translation layer. That layer adds latency and prevents certain query patterns entirely. Complex joins, custom functions, and anything that doesn't fit PostgREST's REST model requires workarounds (RPC calls, edge functions) that layer even more overhead on top. + +Direct `pg` access from the application server — which is the standard pattern in every non-Supabase stack — is simply faster and more flexible. + +### 3. "Open Source" + +Supabase markets itself as open source, and the core Postgres engine is. But the features that make Supabase useful in production — **branching**, **read replicas**, **PITR (point-in-time recovery)**, **connection pooling at scale via Supavisor** — are cloud-only or require the managed platform. The self-hosted version is a constellation of services (PostgREST, GoTrue, Realtime, Storage, Kong, Studio) that you are responsible for orchestrating, upgrading, and securing. It is not a simple binary. Running it locally for development is workable; running it reliably in production is a different project entirely. + +### 4. Storage and Image Processing Are a Paywall + +Supabase Storage is usable at the free tier for basic uploads, but anything beyond basic file hosting typically requires **Edge Functions** — and that is where the trade-offs become much more apparent. + +Image resizing (transforms) is **gated behind the Pro subscription**. There is no free or self-hosted path to on-the-fly image processing through the Supabase Storage API. For a product where image quality and delivery speed are core features, this is a non-starter. + +Even with Pro access, Edge Functions run in a **constrained Deno runtime** — not standard Node.js. Any library that doesn't work in Deno, or that assumes a standard filesystem or native bindings (most image processing libraries), requires a full workaround. The execution environment is also cold-started and **noticeably slow** for the first invocation, which is exactly when a user is waiting on a thumbnail. + +The practical outcome: image resizing that should be a single `sharp` call in a Node.js service becomes a Supabase Pro subscription, a Deno-compatible rewrite, and cold-start latency baked into every new session. + +### 5. Switching Is Deliberately Hard + +The most operationally costly aspect of Supabase is how tightly its abstractions couple your codebase to the platform: + +- **`auth.uid()`** is a GoTrue-backed function that doesn't exist outside Supabase. Every RLS policy that calls it breaks the moment you connect to a plain Postgres instance. +- **`auth.users`** is a Supabase-managed table. Foreign key constraints across your entire schema point at it. +- **`TO authenticated` / `TO anon`** roles in RLS policies don't exist on vanilla Postgres — PostgREST creates them at startup. +- **JWT signing** is done by Supabase's own GoTrue service. Your application is hard-coded to trust those tokens specifically. +- **Dump files** contain Supabase-custom `psql` meta-commands (`\restrict`, `\unrestrict`) that fail on standard `psql`. + +These constraints are easy to underestimate during onboarding, and they tend to become obvious only during migration work. + +--- + +## What We're Replacing It With + +The migration is covered in detail in the sibling chapters of this series. At a high level: + +| Supabase Component | Replacement | +|:---|:---| +| **Auth (GoTrue)** | [Zitadel](./auth-zitadel.md) — standalone OIDC IdP, Go binary, systemd | +| **PostgREST / RLS coupling** | [Application-level auth + session-variable RLS](./rls-leaving-supabase.md) | +| **Managed Postgres** | Self-hosted Postgres, direct `pg` pool from Node.js / Rust | +| **Dashboard** | Direct psql, self-hosted tools as needed | + +The migration is **non-destructive** — existing RLS policies are preserved by re-implementing the `auth.uid()` contract via a session variable, so no policies need to be rewritten or dropped. The auth schema is reduced to a minimal `auth.users` stub that satisfies foreign key constraints while GoTrue is removed entirely. + +--- + +## For "Vibe Coders" + +Supabase is attractive because it dramatically lowers the entry barrier. You can get authentication, storage, database access, and a dashboard with very little setup, which makes it especially appealing for fast prototypes and AI-assisted development. + +The problem is that the easy entry hides the expensive exit. Once the product grows, you eventually hit the edges: pricing per project, performance overhead from platform layers, storage/image-processing limits, and deep coupling to Supabase-specific auth and RLS conventions. At that point, leaving or scaling no longer feels beginner-friendly; it requires fairly specialized knowledge in Postgres, authentication, RLS, infrastructure, and migration planning. + +The practical conclusion is simple: if you already expect to build a serious product, **build on Postgres directly from day one**. Use standard tooling, own your database contract, and keep authentication and authorization portable. The initial setup is a bit more work, but the long-term architecture is cheaper, faster, and easier to control. + +--- + +## Articles in This Series + +- **[Auth with Zitadel](./auth-zitadel.md)** — OIDC flows, JWKS verification, Zitadel vs Keycloak vs Better Auth, configuration. +- **[RLS without Supabase](./rls-leaving-supabase.md)** — The five portability problems, workarounds, pool separation, minimal `auth` schema on plain Postgres. +- **[Migration Tooling](./migration-tooling.md)** — Reusable CLI patterns for DB backup/restore, storage export, and URL migration away from Supabase. +- **[ACL NodeJS Implementation](https://git.polymech.info/polymech/mono/src/branch/master/packages/acl)** — The application-side authorization layer that replaces dependence on platform-coupled policy helpers. +- **[In the making: home baked VFS](https://git.polymech.info/polymech/mono/src/branch/master/packages/vfs)** — The storage layer that replaces Supabase Storage object URLs with application-owned file serving. +