36 lines
3.2 KiB
Markdown
36 lines
3.2 KiB
Markdown
# GridSearch Polygon Rendering & Server Freeze Mitigation
|
|
|
|
## The Problem
|
|
When dealing with GridSearches containing over 5,000+ polygons (e.g., all towns in Spain), querying `better-sqlite3`, parsing GeoPackage WKB into JS objects, putting them into an array, and then running `JSON.stringify()` on a 100MB+ object tree freezes the V8 JavaScript engine. The garbage collector natively blocks the main event loop while traversing this massive JS object.
|
|
|
|
Even with asynchronous yields (e.g., `setTimeout(resolve, 0)`), constructing massive JavaScript arrays of multi-polygons will lock up the Node.js thread and cause other API requests to timeout.
|
|
|
|
## Architectural Options
|
|
|
|
### 1. Raw String Streaming (Lowest Effort, High Impact)
|
|
Skip building the 100MB+ V8 object tree entirely.
|
|
- **How:** Query SQLite geometries and stream the raw JSON strings directly into the HTTP response (e.g. `c.streamText()`).
|
|
- **Pros:** Peak memory drops from 500MB+ to ~1MB. The V8 engine never builds the massive object tree, preventing the GC freeze.
|
|
- **Cons:** The browser still has to download and parse a massive JSON file at once, which may freeze the frontend map rendering momentarily.
|
|
|
|
### 2. Pre-generate Static `.geojson` Files (Best Performance)
|
|
Instead of asking the database for polygons *every time* a map requests them, generate the full polygons file once.
|
|
- **How:** When `gs.enumerate()` creates a search, it also writes a `{basename}-polygons.json` file to the `searches/` directory. The UI fetches this static file directly.
|
|
- **Pros:** Perfectly zero-cost for the Node backend at request-time. NGINX or Hono streams the file instantly without touching the heavy event loop.
|
|
- **Cons:** Increases disk usage. The initial file generation still freezes the server briefly (unless offloaded to a background task like PgBoss).
|
|
|
|
### 3. Native Worker Threads (`worker_threads`)
|
|
Offload the synchronous SQLite querying to a separate Node.js thread.
|
|
- **How:** Spin up a `piscina` worker pool. The worker thread opens a separate `better-sqlite3` connection, does the parsing, stringifies it, and passes the buffer back.
|
|
- **Pros:** Main event loop remains 100% responsive.
|
|
- **Cons:** Significant architectural overhead. Transferring 100MB strings via `postMessage` still incurs a minor memory/serialization hit.
|
|
|
|
### 4. Vector Tiles / .mvt & Lazy Loading (The "Proper" GIS Way)
|
|
Maplibre GL natively supports loading data in Vector Tiles (zoom + X + Y bounding boxes) rather than pulling all 5,000 geometries at once.
|
|
- **How:** The UI requests data via `/api/locations/gridsearch/tiles/{z}/{x}/{y}`. The backend dynamically queries `better-sqlite3` strictly for polygons intersecting that tile envelope.
|
|
- **Pros:** Infinitely scalable. 5 million polygons won't freeze the server or the browser.
|
|
- **Cons:** Highest effort. Requires implementing an MVT generation proxy (using `geojson-vt` or PostGIS equivalents) and pagination in the client.
|
|
|
|
## Current Mitigation
|
|
Currently, a temporary fix uses `await new Promise(r => setTimeout(r, 0))` in the `/api/locations/gridsearch/polygons` endpoint every 50 iterations to yield to the event loop. However, moving towards **Option 2** (Static generation) or **Option 4** (Vector tiles) is strongly recommended for production stability.
|