fuel-db: schema fm, derivations and BI views (draft, not applied); ATOMA dashboard spec
Adds stacks/fuel-monitoring/fuel-db/migrations/0001-0003 with their README, and petrolyst-insights/dashboards/atoma/SPEC.md. Nothing applies the migrations: deploy.sh never touches fuel-db, and a person applies them by hand in phase P3. Verified in scratch Docker (timescaledb 2.31.0-pg17 with this repository's 01-roles.sh, trino 484 with its trino/etc): 150 SQL checks pass; the 18 views have 303 columns with identical types in PostgreSQL and Trino; fm cannot be read through Trino. Codex (gpt-6-astra) inspected the change in four rounds; the round-4 fixes are committed without a further inspection, on the owner's instruction. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
7d01f66f84
commit
bf96edc828
5 files changed
+2274
No files matched your search
@@ -0,0 +1,122 @@
|
||||
# fuel-db migrations
|
||||
|
||||
**Status: DRAFT, not applied.** Nothing here runs automatically: `deploy.sh` never touches
|
||||
fuel-db, and the compose file mounts only `../init/01-roles.sh`. A person applies these files by
|
||||
hand, once the owner gives the go-ahead (phase P3 of the design).
|
||||
|
||||
The design and its rationale live in the fuel-monitoring corpus:
|
||||
`.context/plans/2026-10-24-fuel-db-baseline-dashboard.md`. The SQL comments cite it as
|
||||
"design §n". This folder carries no site data.
|
||||
|
||||
## Files
|
||||
|
||||
| File | Creates | Visible to Trino |
|
||||
|---|---|---|
|
||||
| `0001_fm_core.sql` | schema `fm`, `btree_gist` in it, 25 tables, constraints, trigger-backed rules 1–13 | no |
|
||||
| `0002_fm_derivations.sql` | internal views in `fm`: claims → visits → stock intervals → L/h and L/day intervals, dependency hashes, baseline validity | no |
|
||||
| `0003_bi_views.sql` | `fm.v_site_baseline_status` and the 18 `public.v_*` views Superset reads | yes, read-only |
|
||||
|
||||
Each file is one transaction. Apply them in order.
|
||||
|
||||
## Applying (on the server)
|
||||
|
||||
1. Create a loader role as in the repository README, "Data loaders". Its sessions act as
|
||||
`fuel_owner`, so every object belongs to `fuel_owner`. `trino_ro` can then read the `public`
|
||||
views through the existing default privileges, with no role or grant changes.
|
||||
2. Open the tunnel and apply the three files as the loader:
|
||||
|
||||
```sh
|
||||
ssh -L 5433:127.0.0.1:5433 petrolyst
|
||||
for f in 0001_fm_core 0002_fm_derivations 0003_bi_views; do
|
||||
psql -h localhost -p 5433 -U <loader> -d fuel -v ON_ERROR_STOP=1 -f "$f.sql" || break
|
||||
done
|
||||
```
|
||||
|
||||
3. Check through Trino (`docker exec -it fuel-monitoring-trino-1 trino`):
|
||||
- `SHOW TABLES FROM fuel.public` lists the 18 `v_*` views;
|
||||
- `SELECT count(*) FROM fuel.fm.site` fails with "permission denied for schema fm".
|
||||
|
||||
The `public` views are owned by `fuel_owner` and run with its rights, so Trino never needs, and
|
||||
never gets, read access to `fm`. Trino's metadata still lists fm's table and column names, as it
|
||||
lists every schema in fuel-db (TimescaleDB's internal ones included); names only, no rows.
|
||||
|
||||
## Data
|
||||
|
||||
- Nobody edits `fm.*` by hand. Rows arrive only through the corpus loader (`fmdb.py export` /
|
||||
`load`, phase P1, not yet written), in one transaction per load.
|
||||
- Every value is a claim:
|
||||
- a correction marks the old row `superseded`, inserts the new row under the same `source_key`,
|
||||
and points `superseded_by` at it;
|
||||
- the old row may point at itself until the new row exists;
|
||||
- a removed record is `retracted`, never deleted.
|
||||
- Review status, coordinate acceptance and assumption acceptance come only from an `fm.review` row
|
||||
that matches the exact `assertion_sha256` (rule 11).
|
||||
- A baseline version is inserted as `proposed`. `approved` is a later update, refused unless the
|
||||
version meets its policy and its inputs are unchanged (rule 1).
|
||||
|
||||
## Trigger-backed rules (in 0001)
|
||||
|
||||
1. Baseline versions start as draft or proposed. Approval needs `meets_policy` and unchanged
|
||||
inputs. Approved rows and their inputs are immutable; the only change is approved →
|
||||
superseded. An input never moves to another version.
|
||||
2. An internal transfer's tanks are placed at the movement's site on the visit date.
|
||||
3. A claim's site is its visit's site, and a reading's meter belongs to that site.
|
||||
4. A reading's epoch belongs to its meter.
|
||||
5. A runtime assumption derived from fuel cannot be accepted. This is a CHECK constraint.
|
||||
6. A baseline input's dates equal its endpoint visits. Included inputs carry the version's
|
||||
population.
|
||||
7. A source document supersedes only one of its own family.
|
||||
8. A confirmed or proposed meter mapping names a generator placed at the meter's site.
|
||||
9. Meter-coverage members:
|
||||
- valid when written, and written in the assumption's own transaction;
|
||||
- immutable afterwards;
|
||||
- complete at commit: as many counters as covered sets.
|
||||
10. The live claims of one `event_key` share visit, kind and tanks. A superseded or retracted
|
||||
claim keeps its own, so a correction can change them.
|
||||
11. Status comes only from a matching review. Values are never corrected in place.
|
||||
12. An idle assumption names a generator placed at its own site during its dates.
|
||||
13. Identity and classification never change in place: a visit's key, site, date and sequence; a
|
||||
source document's hash, family and kind (its rank); an asset's uid and kind; a meter's id and
|
||||
site; an epoch's id and meter. A wrong visit date is corrected by a new visit, which the
|
||||
superseding claims point to.
|
||||
|
||||
## Rollback
|
||||
|
||||
Before any data matters, run `DROP SCHEMA fm CASCADE` as the loader. This also drops every
|
||||
`public.v_*` view, and leaves fuel-db as it was before 0001.
|
||||
|
||||
Once data has been loaded, this deletes it. The corpus remains the source, so a re-load restores
|
||||
it, except baseline versions and their approvals, which live only in fuel-db: export
|
||||
`fm.baseline_version` and `fm.baseline_input` first.
|
||||
|
||||
## Verified locally (2026-10-24, scratch only)
|
||||
|
||||
Setup:
|
||||
- `timescale/timescaledb:2.31.0-pg17`, with this repository's `01-roles.sh`;
|
||||
- a loader created by the README recipe;
|
||||
- `trinodb/trino:484`, with this repository's `trino/etc` connected as `trino_ro`.
|
||||
|
||||
Results:
|
||||
- **Apply:** all three files applied as the loader, under the loader's default `search_path`.
|
||||
- **Fixture:** the operator workbook of 21 Oct 2026, the five site records, and synthetic sites
|
||||
for the runtime path.
|
||||
- **Tests:** all 150 checks of the design's §9 pass: 98 written with the migrations, and 52
|
||||
regressions for the implementation inspection's findings. These cover:
|
||||
- figures and statuses;
|
||||
- claims, reviews and revisions;
|
||||
- movement events;
|
||||
- coverage, placement and idle scope;
|
||||
- baseline dependencies and approval;
|
||||
- the L/day path;
|
||||
- current periods;
|
||||
- flow-meter reconciliation (coverage, placement, visit times), runtime validity, modelled-L/h
|
||||
windows, disputed claims supplying no values, the L/day reference, S2's L/h rows following
|
||||
the version's validity, corrections under one `event_key`, fixed meter and epoch owners, and
|
||||
population keys that name the generators a coverage assumption covers;
|
||||
- every refusal listed above.
|
||||
- **Trino:**
|
||||
- all 303 columns of the 18 views have identical names, order and types in PostgreSQL and
|
||||
Trino;
|
||||
- `fm` is unreadable: a query is refused, though its names are listed (see Applying);
|
||||
- writes are refused.
|
||||
- **Not run on the server:** none of this has been run against the live fuel-db.
|
||||
Reference in new issue
Block a user