petrolyst-infra/stacks/fuel-monitoring/fuel-db/migrations/README.md

# fuel-db migrations

**Status: DRAFT, not applied.** Nothing here runs automatically: `deploy.sh` never touches
fuel-db, and the compose file mounts only `../init/01-roles.sh`. A person applies these files by
hand, once the owner gives the go-ahead (phase P3 of the design).

The design and its rationale live in the fuel-monitoring corpus:
`.context/plans/2026-10-24-fuel-db-baseline-dashboard.md`. The SQL comments cite it as
"design §n". This folder carries no site data.

## Files

| File | Creates | Readable by `fuel_ro` (Superset, Petrolyst server) |
|---|---|---|
| `0001_fm_core.sql` | schema `fm`, `btree_gist` in it, 25 tables, constraints, trigger-backed rules 1–13 | no |
| `0002_fm_derivations.sql` | internal views in `fm`: claims → visits → stock intervals → L/h and L/day intervals, dependency hashes, baseline validity | no |
| `0003_bi_views.sql` | `fm.v_site_baseline_status` and the 18 `public.v_*` views Superset reads | yes, read-only |

Each file is one transaction. Apply them in order.

## Applying (on the server)

1. Create a loader role as in the repository README, "Data loaders". Its sessions act as
   `fuel_owner`, so every object belongs to `fuel_owner`. `fuel_ro` can then read the `public`
   views through the existing default privileges, with no role or grant changes.
2. Open the tunnel and apply the three files as the loader:

   ```sh
   ssh -L 5433:127.0.0.1:5433 petrolyst
   for f in 0001_fm_core 0002_fm_derivations 0003_bi_views; do
     psql -h localhost -p 5433 -U <loader> -d fuel -v ON_ERROR_STOP=1 -f "$f.sql" || break
   done
   ```

3. Check as `fuel_ro`
   (`docker exec -it petrolyst-fuel-db-1 psql -U postgres -d fuel`, then `SET ROLE fuel_ro;`):
   - `\dv public.*` lists the 18 `v_*` views;
   - `SELECT count(*) FROM fm.site;` fails with "permission denied for schema fm".

The `public` views are owned by `fuel_owner` and run with its rights, so `fuel_ro` never needs,
and never gets, read access to `fm`. PostgreSQL's catalog still shows every schema's table and
column names to any role, so Superset's SQL Lab can list fm's names; names only, no rows.

## Data

- Nobody edits `fm.*` by hand. Rows arrive only through the corpus loader (`fmdb.py export` /
  `load`, phase P1, not yet written), in one transaction per load.
- Every value is a claim:
  - a correction marks the old row `superseded`, inserts the new row under the same `source_key`,
    and points `superseded_by` at it;
  - the old row may point at itself until the new row exists;
  - a removed record is `retracted`, never deleted.
- Review status, coordinate acceptance and assumption acceptance come only from an `fm.review` row
  that matches the exact `assertion_sha256` (rule 11).
- A baseline version is inserted as `proposed`. `approved` is a later update, refused unless the
  version meets its policy and its inputs are unchanged (rule 1).

## Trigger-backed rules (in 0001)

1. Baseline versions start as draft or proposed. Approval needs `meets_policy` and unchanged
   inputs. Approved rows and their inputs are immutable; the only change is approved →
   superseded. An input never moves to another version.
2. An internal transfer's tanks are placed at the movement's site on the visit date.
3. A claim's site is its visit's site, and a reading's meter belongs to that site.
4. A reading's epoch belongs to its meter.
5. A runtime assumption derived from fuel cannot be accepted. This is a CHECK constraint.
6. A baseline input's dates equal its endpoint visits. Included inputs carry the version's
   population.
7. A source document supersedes only one of its own family.
8. A confirmed or proposed meter mapping names a generator placed at the meter's site.
9. Meter-coverage members:
   - valid when written, and written in the assumption's own transaction;
   - immutable afterwards;
   - complete at commit: as many counters as covered sets.
10. The live claims of one `event_key` share visit, kind and tanks. A superseded or retracted
    claim keeps its own, so a correction can change them.
11. Status comes only from a matching review. Values are never corrected in place.
12. An idle assumption names a generator placed at its own site during its dates.
13. Identity and classification never change in place: a visit's key, site, date and sequence; a
    source document's hash, family and kind (its rank); an asset's uid and kind; a meter's id and
    site; an epoch's id and meter. A wrong visit date is corrected by a new visit, which the
    superseding claims point to.

## Rollback

Before any data matters, run `DROP SCHEMA fm CASCADE` as the loader. This also drops every
`public.v_*` view, and leaves fuel-db as it was before 0001.

Once data has been loaded, this deletes it. The corpus remains the source, so a re-load restores
it, except baseline versions and their approvals, which live only in fuel-db: export
`fm.baseline_version` and `fm.baseline_input` first.

## Verified locally (2026-10-24, scratch only)

With Trino, before its removal on 29 Oct 2026; `trino_ro` is now `fuel_ro`.

Setup:
- `timescale/timescaledb:2.31.0-pg17`, with this repository's `01-roles.sh`;
- a loader created by the README recipe;
- `trinodb/trino:484`, with this repository's `trino/etc` connected as `trino_ro`.

Results:
- **Apply:** all three files applied as the loader, under the loader's default `search_path`.
- **Fixture:** the operator workbook of 21 Oct 2026, the five site records, and synthetic sites
  for the runtime path.
- **Tests:** all 150 checks of the design's §9 pass: 98 written with the migrations, and 52
  regressions for the implementation inspection's findings. These cover:
  - figures and statuses;
  - claims, reviews and revisions;
  - movement events;
  - coverage, placement and idle scope;
  - baseline dependencies and approval;
  - the L/day path;
  - current periods;
  - flow-meter reconciliation (coverage, placement, visit times), runtime validity, modelled-L/h
    windows, disputed claims supplying no values, the L/day reference, S2's L/h rows following
    the version's validity, corrections under one `event_key`, fixed meter and epoch owners, and
    population keys that name the generators a coverage assumption covers;
  - every refusal listed above.
- **Trino:**
  - all 303 columns of the 18 views have identical names, order and types in PostgreSQL and
    Trino;
  - `fm` is unreadable: a query is refused, though its names are listed (see Applying);
  - writes are refused.
- **Not run on the server:** none of this has been run against the live fuel-db.