ADR: provider-owned large-layer editing

Status

Accepted. The provider foundation, browser and native transports, provider-backed GeoEditor selection, attribute paging, compact provider analysis, bounded large-geometry overlay, composed immutable recovery, exact segment snapping, targeted derived refresh, and operation-only project recovery are implemented. The limitations section records the geometry shapes and transport evidence that remain bounded or unavailable.

Context

A complete vector layer cannot be copied into project state, React state, GeoJSON, or Geoman when the layer may contain millions of features. Render tiles also contain clipped and simplified geometry, so they cannot be the source of an edit. Edit cost must follow the selected and dirty working set, not total layer size.

Decision

Provider contract and ownership

A provider-backed MeratynLayer persists a schema-decoded VectorProviderReference. Its canonical records remain in the provider. The core VectorDataset API deliberately has no complete-collection read. It offers metadata, exact stable-ID reads, cursor queries and streams, snap candidates, geometry complexity and windows, and atomic edit batches.

The ownership hierarchy is:

Scope	Owner and resources
Application	The browser vector control’s existing worker, or the desktop native DuckDB actor and protocol host
Project	`ProjectSession.providers`, memoized dataset handles, renderer registrations, and project subscriptions
Edit session	Exact active features, base-feature suppression, bounded overlay data, and the operation journal
Request	One query, tile, snap, validation, or commit Effect

Opening a dataset requires an Effect Scope. The project registry memoizes one handle per complete provider reference, races work against project disposal, and closes the project scope exactly once. Replacing or closing a project aborts requests, closes handles, releases renderer and protocol registrations, clears runtime layer data, and prevents late results from reaching the replacement session. A document reference binding releases a handle when its last authored reference disappears, while shared references remain live for sibling layers and can be acquired again after a project reopen. Promises are confined to foreign boundaries such as Tauri, MapLibre, the vector control, React Query, and Geoman.

Stable identity and canonical geometry

Provider feature IDs, revisions, dataset IDs, and cursors are non-empty Schema-branded strings. A stable ID is shared by provider queries, vector tiles, MapLibre feature state, selection, the attribute table, GeoEditor, operations, and conflict checks. Array position, tile order, and viewport order are not valid identities.

Browser vector layers are eligible whenever the vector control reports a non-streamed materialized layer. The patched vector engine validates an explicit source ID field when supplied; otherwise it creates an internal __meratyn_id from the materialized table row identity. In both cases the MapLibre source uses promoteId. Native import validates an optional source ID field for non-null uniqueness. Without one, it creates UUIDs once in the persistent workspace. Native MVT output also exposes __meratyn_id and uses promoteId.

MapLibre tiles are used only to render, hit-test, and discover the stable ID. GeoEditor first asks the provider for complexity. Features within the standard aggregate budget are then fetched by exact ID at full resolution, and only that bounded selected set is converted to GeoJSON for Geoman. Tile fragments are never reconstructed or committed as canonical geometry.

Browser backend

The browser adapter reuses maplibre-gl-vector’s existing long-lived DuckDB worker through a scoped openLayerDataHandle. It does not start a second WASM database when editing begins. The handle supports exact reads, bounded bbox and attribute queries with field projection, geometry omission, deterministic field-plus-ID ordering and ordered keyset cursors. Its atomic writable batches include insert, replace-geometry, replace-feature, patch-attributes, and delete; duplicate IDs in one batch are rejected before it starts the transaction. Each materialized table owns a revision-clock row seeded once at ingestion. Dirty operations allocate revisions with transactional UPDATE ... RETURNING, so commits do not rescan the feature table for its maximum revision. Control replay rebinds the logical dataset without changing its durable reference, and project/control teardown closes every live handle.

The checked-in Chromium harness runs the exact patched browser bundle against a real DuckDB-WASM worker and connection. It verifies generated IDs, geometry-free projection, filtered ordered cursor continuation, exact and bbox reads, atomic replace-feature, revision-conflict preservation, bounded geometry windows, indexed nearest-point snapping, sparse LineString vertex patches, and stale-revision rejection. It is functional evidence, not a browser performance run. DuckDB still downloads its Spatial extension in this harness unless the host supplies spatialExtensionPath; the repository does not bundle that extension binary.

The same harness has an analysis mode. Providers with a native aggregate path return compact exact statistics from DuckDB; providers without one use a one-field-at-a-time paginated fallback that yields between pages and never builds a complete client-side collection. Empty field selections use a count-only path, so Explorer and Statistics can report the population without retaining feature rows.

The existing scalable vector-tile source remains the renderer. Mutable browser layers use the worker’s persistent vertex and segment index for exact windows, snapping, and bounded root-LineString patches. Immutable browser inputs retain the durable composed-delta reference but use a browser-owned workspace provider: the worker keeps the indexed base and applies an operation-only workspace delta, while commitJournal records an idempotent receipt before applying the operation. A project-scoped IndexedDB receipt WAL is keyed by the raw content identity and replays pending receipts after worker loss or reload. The normal worker tile source is invalidated after a durable commit, so no full feature GeoJSON overlay is required for this path; edit-session suppression is released only after MapLibre idle.

Native backend and cold start

Desktop owns one file-backed DuckDB actor in Tauri application state. App setup starts one bounded synchronous interactive queue plus bounded maintenance and analysis workers. The interactive actor owns the primary connection; maintenance and analysis use cloned connections to the same database, so background work does not import or copy the dataset. Interactive writes remain serialized while compaction and aggregate analysis run on their own back-pressured lanes. Cancellation uses request IDs and DuckDB’s InterruptHandle, including cancellation that arrives before queued work starts. A bounded set of 4,096 completed-request tombstones makes a recent late cancel idempotent without allowing the cancellation registry to grow indefinitely. Blocking database work never runs on the async or WebView thread. Each maintenance job publishes with an atomic revision compare-and-swap, so a stale job cannot overwrite a newer cache. Analysis runs inside a transaction snapshot and checks cancellation before, during, and after its aggregate work.

Desktop Add Vector Layer counts first. At 100,000 features or more it selects the native provider before reading file bytes or constructing a complete FeatureCollection. The vector-panel picker, drag and drop, and desktop file tree all use this count-first route. The first provider open imports canonical geometry and properties into the application data workspace, creates a primary stable-ID key and R-tree, and persists dataset metadata. Later handles and process restarts reopen that workspace instead of re-importing an unchanged dataset. Exact reads, bbox pages, snapping, geometry windows, commits, and tiles all address the same actor handle.

Native datasets also own persistent normalized vertex and segment tables. Stable geometry/part/ring/index addresses, coordinates, and segment bounds are indexed separately from the canonical geometry. Complexity counts, bounded viewport windows, and snapping query these tables, so request cost is bounded without decoding a whole huge geometry. Existing workspaces backfill the side tables once when opened. A separate feature-state table owns current revisions, bounds, and cache-publication state; keyed vertex and segment delta tables hold dirty coordinates over the immutable normalized base. During a huge-geometry edit, pointer moves update only the local overlay journal. Save submits one vertex-patch batch that writes the addressed deltas, adjacent segments, state, and a durable latest-only compaction job. It does not rewrite or copy the canonical geometry blob. A revision-CAS compactor later publishes the rebuilt cache, while exact, bbox, window, snap, and tile reads either merge the delta or wait for the required cache revision. Cancel never writes canonical data.

Initial import performs one checkpoint before exposing the dataset handle. This pays import WAL debt as cold-open work rather than making the first small Save flush the imported geometry. No checkpoint runs in the vertex-patch transaction.

The actor encodes raw MVT PBF with ST_AsMVT, caps tiles at 4 MiB and zoom 22, and includes only stable ID plus geometry. A custom meratyn-native-vector:// MapLibre protocol returns those bytes directly from the already-open handle and propagates protocol aborts to the actor. Project documents store the durable provider reference, never the actor handle or the ephemeral protocol token. The tokenized tile URL lives only in ProjectRuntimeState.layerSource; save persists neither token nor runtime tile URL. Restore allocates a new token and re-registers the renderer.

Overview tiles bound projection and encoding to 3,072 features at z0-z4 and 24,576 at z5-z8; z9 and deeper remain uncapped. Candidate ordering is stable by feature ID, with the current and immediately previous dirty generations first so a committed edit remains visible. The query derives a conservative numeric WGS84 tile envelope once, prefilters on persisted feature bounds, and retains canonical ST_Intersects as the exact spatial predicate. This avoids carrying unrelated geometry through the overview sort while preserving the R-tree scan.

Memory and composed providers

The memory provider is the compatibility implementation for genuinely small GeoJSON. It owns a stable-ID index and applies revision-checked batches atomically, but its input is still a complete FeatureCollection and it is not the large-layer backend.

The core composed provider models an immutable indexed base plus an in-memory delta. The delta stores inserted records, replacements, and tombstones. Exact, page, stream, snap, geometry-window, and metadata reads merge those structures without rewriting the base. Cursors encode both base and delta revisions and are rejected after either side changes, preventing a page sequence from mixing snapshots. A conflicted batch publishes none of its operations.

Browser lifecycle persists a composed-delta reference for every immutable materialized input. Its base reference carries a bounded raw-source content identity (sha256-tree-v1); identities are computed from sequential 4 MiB chunks, not parsed or copied layer data. A project-scoped IndexedDB workspace WAL stores only strict-codec operation receipts (4,000,000 bytes, 256 receipts, 4,096 forward operations, and 200,000 vertex-patch coordinates). It stores no feature snapshots or replacement geometry and is keyed by content identity; mixed identities are quarantined and missing identity disables recovery. Export and full compaction remain explicit concerns; normal edits merge base and delta reads.

Edit journal, conflict, and render flow

Provider edits are explicit insert, replace-geometry, replace-feature, patch-attributes, patch-vertices, affine-transform, and delete operations. The Effect-owned journal stores bounded inverse operations, not layer snapshots. Before a Save, the project recovery encoder validates the operation, inverse, feature/vertex counts, and byte budget. After the provider transaction succeeds, the forward and inverse operations are persisted in project.providerRecovery with a project-wide sequence shared with document history. Undo and redo after Save replay a provider transaction, refresh its affected tiles, then advance the persisted cursor. A renderer retry never re-commits the provider operation. Revision mismatches return a typed conflict and leave canonical data and pending edits unchanged.

The standard provider geometry flow is:

Hit-test the provider layer and retain promoted string IDs.
Read complexity and fetch only the bounded exact canonical working set.
Load at most 64 features and 20,000 aggregate vertices into Geoman.
Suppress only those IDs in each owning style layer by wrapping opacity paint with feature-state, while leaving the base renderer visible.
Rebuild a local operation journal as Geoman edits, undo, and redo change the bounded working set. Combined geometry and attribute changes use atomic replace-feature operations.
On Save, commit the journal once. Pass affected bounds, IDs, and revisions to the layer renderer, regionally refresh the owning source, and wait for idle.
Clear Geoman and release feature-state suppression.

For a native large-geometry Save, the provider writes only dirty coordinates and enqueues a latest-revision compaction job. The project-owned publication fiber refreshes affected tiles and waits for idle; only then does it release the edit overlay and suppression. A second project-owned fiber drains native compaction and exposes publishing and compacting states with retry actions. The retry path publishes or compacts an already durable revision and never replays the edit journal.

Cancel rolls back the local journal and closes the edit scope without committing. A conflict keeps the session open. If a commit succeeds but renderer refresh fails, retry refreshes the committed revision and does not submit the batch again.

Regional refresh uses MapLibre 6.4’s public refreshTiles API plus an exact-version patch that evicts matching out-of-view cache entries and queues a reload even when a matching in-view tile is already loading. Affected bounds are expanded by the MVT buffer and converted to deduplicated canonical XYZ IDs over the source’s declared zoom range, including Mercator latitude clamping and antimeridian splitting. Invalid bounds or more than 4,096 IDs fall back to a safe full-source refresh. Suppression remains until MapLibre reports idle.

Separately, the GeoJSON-VT worker source queue keeps only the latest pending generation per layer. Older tile waiters follow its newest barrier instead of retaining obsolete reload work.

Attribute table and snapping

The provider-backed attribute table uses forward cursors with 200-row pages, at most five cached pages, and projections capped at 128 fields. Geometry is always omitted. Filtering and ordering are sent to providers only when their capability contract permits them. Selection uses stable feature IDs. Attribute drafts become one atomic revision-checked batch of at most 64 features; conflicts retain the drafts. Explorer and Statistics use compact exact provider aggregates when available, including filtered and selected scopes. A provider without an aggregate implementation uses the generic one-field paginated fallback, so the UI remains available without materializing a complete layer. Selection lookup reuses an index for an immutable feature collection, while join refresh walks only the changed layer’s transitive dependents.

The snapping controller queries a bounded pointer envelope, excludes dirty feature IDs, caps candidates, caches briefly, and interrupts an older request when the pointer advances. It is connected to Geoman’s production pointer path. Native and patched browser providers query persistent vertex/segment indexes and return the nearest point on a segment, not a segment midpoint. The provider boundary decodes and deterministically orders the worker result; a legacy handle without snapGeometry fails with a typed unsupported-operation error rather than loading complete geometries or manufacturing candidates. Finalizers clear only the fiber they own, so an old request cannot erase ownership of its replacement. GeoEditor keeps a prepared Flatbush index for local vertices, segments, and centroids. Scope or feature changes invalidate its source index; camera changes invalidate only the projection pass; provider-backed external candidates have an independent index and invalidation path. A cold view rebuild therefore remains proportional to the projected targets, while pointer queries visit only the tolerance envelope.

Very large individual geometries

The standard Geoman budget is 20,000 vertices. Above it, GeoEditor does not call exact and does not create every vertex marker. It requests a bounded geometry window from the provider and renders up to 512 nearby vertices and segments in a transient MapLibre overlay. Vertices have stable geometry/part/ring/index paths. A provider-side complexity preflight chooses this path before any full geometry is materialized. Window and vertex indexes are reused while revision and bounds are unchanged, so pointer and viewport updates pay only for the affected window. A pointer drag records a local stable-address patch, applies it to the bounded window, and updates the overlay source. moveend reloads the window for the new viewport. Local undo and redo replay that patch journal. Only Save submits one revision-checked patch-vertices operation; Cancel discards it. Closing the edit or project removes the overlay and suppression.

The native provider implements indexed complexity, windows, vertex patches, and affine transforms. Its vertex and segment delta tables keep dirty edits sparse, and a delta-aware tile path can render dirty 2D LineString features before compaction. The patched browser worker, including the immutable browser-owned workspace provider, implements bounded windows and indexed sparse patches for root pure 2D LineString features (with a 4,096-patch limit). The worker WAL records an idempotent receipt before applying each operation, so recovery never needs a full exact geometry. GeoEditor keeps vertex handles gated for polygons, multipart/nested geometries, Z/M-preserving shapes, and mixed transform-plus- vertex edits; worker-owned affine transforms remain available where advertised. Large drag, viewport, and snap work runs in scope-owned fibers. Closing the edit or project interrupts and awaits them, preventing a late pointer or viewport result from writing into a cancelled or replacement session.

Project serialization and migration

Provider adoption is additive. Existing small GeoJSON and Sketches layers keep their current geojson compatibility path and document undo behavior. There is no automatic migration that moves an arbitrary embedded FeatureCollection into a large provider.

Eligible browser vector-control layers persist a browser-vector reference for mutable workspaces, or a composed-delta reference whose browser base carries the immutable source identity. Reconciliation removes the reference if the layer becomes streamed, unavailable, or loses its identity. Native vector layers persist a native-duckdb reference and source path. Provider-backed layers do not persist canonical records in geojson or ProjectRuntimeState.layerData; existing save rules retain GeoJSON only when it is the small layer’s sole restorable copy. Invalid provider references are dropped while decoding a project rather than trusted as runtime handles.

Benchmark definition

The benchmark writer uses versioned JSONL schema meratyn.large-layer-editing.v2. Every metric retains raw samples, min, median, p95, and max summaries, plus process RSS samples and source/workspace bytes where available. Full-profile cold operations use zero warmups and three independent samples. Warm operations use five warmups and 30 samples. The smoke profile uses one warmup and three samples. The repeatable matrix is:

point datasets with 10,000, 100,000, 1,000,000, and 10,000,000 features;
100,000 dense lines with 64 vertices each;
100,000 dense polygons with 65 vertices each;
one line with 2,000,000 vertices;
1,000,000 points with 64 attributes;
one-feature and 64-feature edit batches.

It records engine cold initialization, cold import/open, warm reopen, exact-by-ID, a 1,000-row geometry-free bbox page, z4 PBF tile generation, single and 64-feature commits, huge-geometry window and vertex-patch latency, diagnostics, source/workspace bytes, and process RSS where available. The full profile reports the 100,000-to-10,000,000 median ratios. The functional browser-worker harness runs separately and does not fabricate native or UI latency.

The complete post-index artifact was captured immediately before the final numeric tile-envelope prefilter. On the 10,000,000-point fixture, median/p95 times were 26.03/27.28 ms for click-to-canonical-feature-ready, 7.04/7.80 ms for exact-by-ID, 599.52/614.62 ms for a 1,000-row geometry-free bbox page, 11,045.22/11,334.26 ms for a z4 PBF tile, 21.91/22.86 ms for a single-feature commit, 1,100.45/1,169.55 ms for a 64-feature commit, and 12,146.48/12,808.35 ms from commit through fresh PBF. Cold import/open was 330,785.90/358,766.71 ms with a 4.43 GiB process-lifetime peak.

The final prefilter has separate focused release evidence because rerunning the entire cold matrix would repeat hours of unaffected imports. The ignored 100,000-to-10,000,000 z4 probe measured 14.25 ms and 142.53 ms medians, respectively: a 10.00x ratio. EXPLAIN ANALYZE showed one R-tree scan, no hash join or second base-table scan, 863 and 85,769 exact candidates respectively, and at most 3,072 encoded features. This focused probe validates tile scaling, not the full commit-to-MapLibre-idle path; the complete post-prefilter commit-to-fresh-PBF ratio remains unmeasured.

For the 2,000,000-vertex line, median/p95 times were 2.73/3.00 ms for indexed complexity, 152.29/166.01 ms for a 1,000-element geometry window, 1,984.69/2,287.63 ms for the intentionally unbounded exact read. The final delta-overlay diagnostic measured the one-coordinate provider commit at 191.06 ms, including 111.22 ms of indexed delta/base lookup and 11.45 ms for transaction commit. The same fixture took 4,757.27 ms for the historical canonical compaction drain. Dirty 2D LineString tiles now read the indexed delta while that drain runs in the project-owned background fiber, so the 4.76 s compaction is not on the Save-to-visible-overlay path. This is a focused single sample rather than a median/p95 series. Dense-line and dense-polygon z4 tiles measured 275.50/297.65 ms and 452.74/540.08 ms respectively; the 1,000,000-row, 64-attribute fixture’s geometry-free bbox page measured 593.63/706.02 ms. The focused overlay artifact is artifacts/benchmarks/native-vertex-overlay-2m-20260830.jsonl (SHA-256 5361CA5AC0E63FCE9F0F92DAFB40A2F761F6CA2A7BB41E32D32525E1EBC3F01B).

The v2 smoke and earlier v1 full release artifacts remain historical pre-index baselines. A self-contained 100,000-point browser harness measured 33.5 ms medians from provider exact retrieval and provider commit to two animation frames, a 16.8 ms median animation-frame p95 gap during commit, and no observed main-thread long tasks in either operation. It does not bootstrap the real GeoEditor/MapLibre interaction, so rendered-handle latency and time from commit to MapLibre idle remain unmeasured. The native click-to-canonical-feature-ready and commit-to-fresh-PBF figures are provider proxies, not substitutes for those UI measurements. This ADR makes no QGIS parity claim.

The live patched-worker acceptance test additionally exercised a 20,001-coordinate LineString through bounded windows, off-midpoint nearest-point snapping, an atomic revisioned vertex patch, fresh MVT generation, close/reopen persistence, and stale-revision rejection. Its counters are diagnostic, not a browser UI latency benchmark.

Focused verification

The provider analysis path can be exercised against the real DuckDB-WASM worker with the following command:

$env:MERATYN_VECTOR_ENTRY=(Resolve-Path packages/map-plugins/src/vector/index.ts).Path
$env:MERATYN_BROWSER_PROVIDER_LIVE_MODE='analysis'
bun scripts/benchmarks/browser-provider-live.ts

One run analyzed a 120,000-row fixture in 250 ms. The response was 1,556 bytes with two aggregate fields and no feature rows. The probe observed no long tasks and a maximum 16.8 ms frame gap. The optional Chromium JavaScript heap reading was 31,200,000 bytes; it excludes WASM-worker memory and native RSS, and the MapStub harness does not measure map FPS. The native provider validation suite and four concurrency tests passed; five tests were intentionally ignored.

A separate 100,000-row dirty-query EXPLAIN probe improved from 186 ms to 34.8 ms after the indexed source and summary row per dirty ID were used. This is a focused query measurement rather than a full render or commit-to-idle benchmark.

A direct selection probe over 40,000 features reduced an empty-selection inversion from 3,538.36 ms in implementation 4b96d395c to 10.53 ms in one cold Bun 1.4 run on the actual machine. This measures the selection helper only, not map FPS. Pure snapping matched the previous findSnap behavior in 7,200 randomized comparisons across all seven modes. Cold identity-projection probes measured 105.6 ms for 100,000 geometry targets and 199.9 ms for 250,000 geometry targets; the initial rebuild remains a linear cost when a view first becomes dirty.

Commands:

npm run benchmark:large-layer-editing:smoke
npm run benchmark:large-layer-editing
npm run test:browser-provider-live

Consequences and remaining limitations

Browser support requires the patched maplibre-gl-vector version and a materialized layer. Explicit source IDs are validated for uniqueness; layers without one receive an internal identity for that materialized browser session. Streamed layers remain render-only. Immutable recovery requires a stable raw-source content identity; without it, reconciliation removes the provider reference rather than risking stale replay. The live harness needs network access to download DuckDB Spatial unless spatialExtensionPath is supplied.
Immutable browser edits use the browser-owned composed workspace provider. Its operation-only IndexedDB receipt WAL replays pending/committed receipts after worker loss or reload and uses worker tile invalidation, so no full geometry overlay or recommit is needed. The WAL is capped at 4,000,000 bytes, 256 receipts, 4,096 forward operations, and 200,000 vertex-patch coordinates; it stores no snapshots or replacement geometry.
The patched worker’s bounded geometry windows and indexed vertex patches are product-enabled for root pure 2D LineString features in both mutable and immutable browser workspaces. Polygon, multipart/nested, Z/M-preserving, and mixed transform-plus-vertex edits remain clearly gated; worker-owned affine transforms remain available where the provider advertises them.
Removing one native layer revokes its protocol token and renderer immediately. The project reference binding releases the shared actor handle when its last document reference disappears, while sibling layers continue to share it.
Provider commits are integrated into bounded project recovery metadata. Forward and inverse operations are persisted after Save, and undo/redo uses the same project-wide order as document edits. Recovery is capped at 4 MiB, 256 operations, 256 features, and 200,000 vertices; no full-layer snapshot is stored.
Binary transport is implemented for native tiles. Exact features and record pages still cross browser and Tauri boundaries as bounded JSON, not WKB, GeoArrow, or Arrow IPC.
Snapping is exact only through the native index or the patched worker’s snapGeometry operation. A legacy browser handle without that operation is rejected with a typed unsupported error rather than falling back to full geometry reads or midpoint candidates.
Native vertex patches validate changed segments against effective indexed neighbors, and native affine transforms validate the changed geometry plus indexed neighbors under explicit work budgets. Browser root-LineString vertex patches perform the same bounded indexed-neighbor check in the worker; unsupported browser shapes remain gated instead of taking a full-geometry fallback.
The browser harness measures provider-to-two-frame latency, long tasks, and frame cadence. Its MapStub does not measure map FPS, and its JavaScript heap counter excludes WASM-worker memory and native RSS. Real rendered-handle and MapLibre-idle latency, plus native 10,000,000-feature UI behavior, remain outstanding.