fuel-db: schema fm, derivations and BI views (draft, not applied); ATOMA dashboard spec

Adds stacks/fuel-monitoring/fuel-db/migrations/0001-0003 with their README, and
petrolyst-insights/dashboards/atoma/SPEC.md. Nothing applies the migrations:
deploy.sh never touches fuel-db, and a person applies them by hand in phase P3.

Verified in scratch Docker (timescaledb 2.31.0-pg17 with this repository's
01-roles.sh, trino 484 with its trino/etc): 150 SQL checks pass; the 18 views
have 303 columns with identical types in PostgreSQL and Trino; fm cannot be
read through Trino. Codex (gpt-6-astra) inspected the change in four rounds;
the round-4 fixes are committed without a further inspection, on the owner's
instruction.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Nasim SadatandClaude Opus 5.5 committed 2026-10-24 09:58:14 +02:00
1 parent 7d01f66f84
commit bf96edc828
5 files changed
+2274

No files matched your search

@@ -0,0 +1,122 @@
# fuel-db migrations
**Status: DRAFT, not applied.** Nothing here runs automatically: `deploy.sh` never touches
fuel-db, and the compose file mounts only `../init/01-roles.sh`. A person applies these files by
hand, once the owner gives the go-ahead (phase P3 of the design).
The design and its rationale live in the fuel-monitoring corpus:
`.context/plans/2026-10-24-fuel-db-baseline-dashboard.md`. The SQL comments cite it as
"design §n". This folder carries no site data.
## Files
| File | Creates | Visible to Trino |
|---|---|---|
| `0001_fm_core.sql` | schema `fm`, `btree_gist` in it, 25 tables, constraints, trigger-backed rules 1–13 | no |
| `0002_fm_derivations.sql` | internal views in `fm`: claims → visits → stock intervals → L/h and L/day intervals, dependency hashes, baseline validity | no |
| `0003_bi_views.sql` | `fm.v_site_baseline_status` and the 18 `public.v_*` views Superset reads | yes, read-only |
Each file is one transaction. Apply them in order.
## Applying (on the server)
1. Create a loader role as in the repository README, "Data loaders". Its sessions act as
`fuel_owner`, so every object belongs to `fuel_owner`. `trino_ro` can then read the `public`
views through the existing default privileges, with no role or grant changes.
2. Open the tunnel and apply the three files as the loader:
```sh
ssh -L 5433:127.0.0.1:5433 petrolyst
for f in 0001_fm_core 0002_fm_derivations 0003_bi_views; do
psql -h localhost -p 5433 -U <loader> -d fuel -v ON_ERROR_STOP=1 -f "$f.sql" || break
done
```
3. Check through Trino (`docker exec -it fuel-monitoring-trino-1 trino`):
- `SHOW TABLES FROM fuel.public` lists the 18 `v_*` views;
- `SELECT count(*) FROM fuel.fm.site` fails with "permission denied for schema fm".
The `public` views are owned by `fuel_owner` and run with its rights, so Trino never needs, and
never gets, read access to `fm`. Trino's metadata still lists fm's table and column names, as it
lists every schema in fuel-db (TimescaleDB's internal ones included); names only, no rows.
## Data
- Nobody edits `fm.*` by hand. Rows arrive only through the corpus loader (`fmdb.py export` /
`load`, phase P1, not yet written), in one transaction per load.
- Every value is a claim:
- a correction marks the old row `superseded`, inserts the new row under the same `source_key`,
and points `superseded_by` at it;
- the old row may point at itself until the new row exists;
- a removed record is `retracted`, never deleted.
- Review status, coordinate acceptance and assumption acceptance come only from an `fm.review` row
that matches the exact `assertion_sha256` (rule 11).
- A baseline version is inserted as `proposed`. `approved` is a later update, refused unless the
version meets its policy and its inputs are unchanged (rule 1).
## Trigger-backed rules (in 0001)
1. Baseline versions start as draft or proposed. Approval needs `meets_policy` and unchanged
inputs. Approved rows and their inputs are immutable; the only change is approved →
superseded. An input never moves to another version.
2. An internal transfer's tanks are placed at the movement's site on the visit date.
3. A claim's site is its visit's site, and a reading's meter belongs to that site.
4. A reading's epoch belongs to its meter.
5. A runtime assumption derived from fuel cannot be accepted. This is a CHECK constraint.
6. A baseline input's dates equal its endpoint visits. Included inputs carry the version's
population.
7. A source document supersedes only one of its own family.
8. A confirmed or proposed meter mapping names a generator placed at the meter's site.
9. Meter-coverage members:
- valid when written, and written in the assumption's own transaction;
- immutable afterwards;
- complete at commit: as many counters as covered sets.
10. The live claims of one `event_key` share visit, kind and tanks. A superseded or retracted
claim keeps its own, so a correction can change them.
11. Status comes only from a matching review. Values are never corrected in place.
12. An idle assumption names a generator placed at its own site during its dates.
13. Identity and classification never change in place: a visit's key, site, date and sequence; a
source document's hash, family and kind (its rank); an asset's uid and kind; a meter's id and
site; an epoch's id and meter. A wrong visit date is corrected by a new visit, which the
superseding claims point to.
## Rollback
Before any data matters, run `DROP SCHEMA fm CASCADE` as the loader. This also drops every
`public.v_*` view, and leaves fuel-db as it was before 0001.
Once data has been loaded, this deletes it. The corpus remains the source, so a re-load restores
it, except baseline versions and their approvals, which live only in fuel-db: export
`fm.baseline_version` and `fm.baseline_input` first.
## Verified locally (2026-10-24, scratch only)
Setup:
- `timescale/timescaledb:2.31.0-pg17`, with this repository's `01-roles.sh`;
- a loader created by the README recipe;
- `trinodb/trino:484`, with this repository's `trino/etc` connected as `trino_ro`.
Results:
- **Apply:** all three files applied as the loader, under the loader's default `search_path`.
- **Fixture:** the operator workbook of 21 Oct 2026, the five site records, and synthetic sites
for the runtime path.
- **Tests:** all 150 checks of the design's §9 pass: 98 written with the migrations, and 52
regressions for the implementation inspection's findings. These cover:
- figures and statuses;
- claims, reviews and revisions;
- movement events;
- coverage, placement and idle scope;
- baseline dependencies and approval;
- the L/day path;
- current periods;
- flow-meter reconciliation (coverage, placement, visit times), runtime validity, modelled-L/h
windows, disputed claims supplying no values, the L/day reference, S2's L/h rows following
the version's validity, corrections under one `event_key`, fixed meter and epoch owners, and
population keys that name the generators a coverage assumption covers;
- every refusal listed above.
- **Trino:**
- all 303 columns of the 18 views have identical names, order and types in PostgreSQL and
Trino;
- `fm` is unreadable: a query is refused, though its names are listed (see Applying);
- writes are refused.
- **Not run on the server:** none of this has been run against the live fuel-db.