mono/packages/ui/docs/tiles.md

GridSearch Polygon Rendering & Server Freeze Mitigation

The Problem

When dealing with GridSearches containing over 5,000+ polygons (e.g., all towns in Spain), querying better-sqlite3, parsing GeoPackage WKB into JS objects, putting them into an array, and then running JSON.stringify() on a 100MB+ object tree freezes the V8 JavaScript engine. The garbage collector natively blocks the main event loop while traversing this massive JS object.

Even with asynchronous yields (e.g., setTimeout(resolve, 0)), constructing massive JavaScript arrays of multi-polygons will lock up the Node.js thread and cause other API requests to timeout.

Architectural Options

1. Raw String Streaming (Lowest Effort, High Impact)

Skip building the 100MB+ V8 object tree entirely.

• How: Query SQLite geometries and stream the raw JSON strings directly into the HTTP response (e.g. c.streamText()).
• Pros: Peak memory drops from 500MB+ to ~1MB. The V8 engine never builds the massive object tree, preventing the GC freeze.
• Cons: The browser still has to download and parse a massive JSON file at once, which may freeze the frontend map rendering momentarily.

2. Pre-generate Static .geojson Files (Best Performance)

Instead of asking the database for polygons every time a map requests them, generate the full polygons file once.

• How: When gs.enumerate() creates a search, it also writes a {basename}-polygons.json file to the searches/ directory. The UI fetches this static file directly.
• Pros: Perfectly zero-cost for the Node backend at request-time. NGINX or Hono streams the file instantly without touching the heavy event loop.
• Cons: Increases disk usage. The initial file generation still freezes the server briefly (unless offloaded to a background task like PgBoss).

3. Native Worker Threads (worker_threads)

Offload the synchronous SQLite querying to a separate Node.js thread.

• How: Spin up a piscina worker pool. The worker thread opens a separate better-sqlite3 connection, does the parsing, stringifies it, and passes the buffer back.
• Pros: Main event loop remains 100% responsive.
• Cons: Significant architectural overhead. Transferring 100MB strings via postMessage still incurs a minor memory/serialization hit.

4. Vector Tiles / .mvt & Lazy Loading (The "Proper" GIS Way)

Maplibre GL natively supports loading data in Vector Tiles (zoom + X + Y bounding boxes) rather than pulling all 5,000 geometries at once.

• How: The UI requests data via /api/locations/gridsearch/tiles/{z}/{x}/{y}. The backend dynamically queries better-sqlite3 strictly for polygons intersecting that tile envelope.
• Pros: Infinitely scalable. 5 million polygons won't freeze the server or the browser.
• Cons: Highest effort. Requires implementing an MVT generation proxy (using geojson-vt or PostGIS equivalents) and pagination in the client.

Current Mitigation

Currently, a temporary fix uses await new Promise(r => setTimeout(r, 0)) in the /api/locations/gridsearch/polygons endpoint every 50 iterations to yield to the event loop. However, moving towards Option 2 (Static generation) or Option 4 (Vector tiles) is strongly recommended for production stability.