fuel-db: customer on every row, place and holder for site, licence tables
Draft migrations rewritten in place (PD-16, PD-18), with 0004 for the licence tables (PD-20) and rule tests in fuel-db/tests. Dashboards' column list regenerated; coverage_detail selects operator_uid. Checks: tests/run.sh 106 of 106; 18 views, 686 rows, 0 differences against a reference from the same corpus. Codex's fixes to its inspection are not re-inspected. Bands: persistence, Superset. Rules: 6, 8. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Nasim SadatandClaude Opus 5.5 committed
1 parent
e4806f5799
commit
ab553526f6
11 files changed
+1197
-398
No files matched your search
@@ -12,12 +12,23 @@ The design and its rationale live in the fuel-monitoring corpus:
|
||||
|
||||
| File | Creates | Readable by `fuel_ro` (Superset, Petrolyst server) |
|
||||
|---|---|---|
|
||||
| `0001_fm_core.sql` | schema `fm`, `btree_gist` in it, 25 tables, constraints, trigger-backed rules 1–13 | no |
|
||||
| `0001_fm_core.sql` | schema `fm`, `btree_gist` in it, 29 tables (customers, assets, places and holders, claims, reviews, baselines), constraints, trigger-backed rules 1–15 | no |
|
||||
| `0002_fm_derivations.sql` | internal views in `fm`: claims → visits → stock intervals → L/h and L/day intervals, dependency hashes, baseline validity | no |
|
||||
| `0003_bi_views.sql` | `fm.v_site_baseline_status` and the 18 `public.v_*` views Superset reads | yes, read-only |
|
||||
| `0004_licence.sql` | schema `licence`: plans, prices, entitlements, customer licences, overrides and their change log (PD-19, PD-20). Holds no customer's licence | no |
|
||||
|
||||
Each file is one transaction. Apply them in order.
|
||||
|
||||
**The shape (PD-16, PD-18).** Every row in `fm` carries its customer, `operator_uid`, except
|
||||
`asset_kind`. A child names its parent and the customer together in a composite foreign key, so it
|
||||
cannot point at another customer's parent. A site is a `place` with one fixed `holder`; holders keep
|
||||
their `SITE-0001` ids and places get `PLC-0001`. Internal `fm` speaks of holder and place. The public
|
||||
views keep their `site_*` column names, because dashboards, the app and the server read them; each
|
||||
gains `operator_uid`. They get renamed, or holder-named siblings, with the first fleet customer.
|
||||
|
||||
**Runtime is generator-only.** An engine can be a placed asset, but rules 8, 9 and 12 refuse a
|
||||
meter or idle assumption on one: the runtime and L/h calculations have not been checked on engines.
|
||||
|
||||
**Who depends on the views.** Superset's dashboards list every column (`petrolyst-insights/dashboards/atoma/columns.tsv`;
|
||||
the import reports drift). The Petrolyst server reads `public.v_site`; its integration test
|
||||
(petrolyst-platform `server/tests/test_integration.py`) names the columns and types it needs. After
|
||||
@@ -29,11 +40,11 @@ point 8, closed 30 Oct 2026).
|
||||
1. Create a loader role as in the repository README, "Data loaders". Its sessions act as
|
||||
`fuel_owner`, so every object belongs to `fuel_owner`. `fuel_ro` can then read the `public`
|
||||
views through the existing default privileges, with no role or grant changes.
|
||||
2. Open the tunnel and apply the three files as the loader:
|
||||
2. Open the tunnel and apply the four files as the loader:
|
||||
|
||||
```sh
|
||||
ssh -L 5433:127.0.0.1:5433 petrolyst
|
||||
for f in 0001_fm_core 0002_fm_derivations 0003_bi_views; do
|
||||
for f in 0001_fm_core 0002_fm_derivations 0003_bi_views 0004_licence; do
|
||||
psql -h localhost -p 5433 -U <loader> -d fuel -v ON_ERROR_STOP=1 -f "$f.sql" || break
|
||||
done
|
||||
```
|
||||
@@ -41,7 +52,8 @@ point 8, closed 30 Oct 2026).
|
||||
3. Check as `fuel_ro`
|
||||
(`docker exec -it petrolyst-fuel-db-1 psql -U postgres -d fuel`, then `SET ROLE fuel_ro;`):
|
||||
- `\dv public.*` lists the 18 `v_*` views;
|
||||
- `SELECT count(*) FROM fm.site;` fails with "permission denied for schema fm".
|
||||
- `SELECT count(*) FROM fm.holder;` fails with "permission denied for schema fm";
|
||||
- `SELECT count(*) FROM licence.plan;` fails with "permission denied for schema licence".
|
||||
|
||||
The `public` views are owned by `fuel_owner` and run with its rights, so `fuel_ro` never needs,
|
||||
and never gets, read access to `fm`. PostgreSQL's catalog still shows every schema's table and
|
||||
@@ -66,14 +78,15 @@ column names to any role, so Superset's SQL Lab can list fm's names; names only,
|
||||
1. Baseline versions start as draft or proposed. Approval needs `meets_policy` and unchanged
|
||||
inputs. Approved rows and their inputs are immutable; the only change is approved →
|
||||
superseded. An input never moves to another version.
|
||||
2. An internal transfer's tanks are placed at the movement's site on the visit date.
|
||||
3. A claim's site is its visit's site, and a reading's meter belongs to that site.
|
||||
2. An internal transfer's tanks are placed on the movement's holder on the visit date.
|
||||
3. A claim's holder is its visit's holder, and a reading's meter belongs to that holder.
|
||||
4. A reading's epoch belongs to its meter.
|
||||
5. A runtime assumption derived from fuel cannot be accepted. This is a CHECK constraint.
|
||||
6. A baseline input's dates equal its endpoint visits. Included inputs carry the version's
|
||||
population.
|
||||
7. A source document supersedes only one of its own family.
|
||||
8. A confirmed or proposed meter mapping names a generator placed at the meter's site.
|
||||
7. A source document supersedes only one of its own family. The same document under a second
|
||||
customer is refused until document sharing exists.
|
||||
8. A confirmed or proposed meter mapping names a generator placed on the meter's holder.
|
||||
9. Meter-coverage members:
|
||||
- valid when written, and written in the assumption's own transaction;
|
||||
- immutable afterwards;
|
||||
@@ -81,48 +94,34 @@ column names to any role, so Superset's SQL Lab can list fm's names; names only,
|
||||
10. The live claims of one `event_key` share visit, kind and tanks. A superseded or retracted
|
||||
claim keeps its own, so a correction can change them.
|
||||
11. Status comes only from a matching review. Values are never corrected in place.
|
||||
12. An idle assumption names a generator placed at its own site during its dates.
|
||||
13. Identity and classification never change in place: a visit's key, site, date and sequence; a
|
||||
source document's hash, family and kind (its rank); an asset's uid and kind; a meter's id and
|
||||
site; an epoch's id and meter. A wrong visit date is corrected by a new visit, which the
|
||||
superseding claims point to.
|
||||
12. An idle assumption names a generator placed on its own holder during its dates.
|
||||
13. Identity and classification never change in place: a visit's key, holder, date and sequence; a
|
||||
source document's hash, family and kind (its rank); an asset's uid and kind; a place's uid; a
|
||||
holder's uid, kind and asset; a meter's id and holder; an epoch's id and meter; any row's customer.
|
||||
A wrong visit date is corrected by a new visit, which the superseding claims point to. Naming a
|
||||
column a table does not have raises an error.
|
||||
14. A time zone is one PostgreSQL knows.
|
||||
15. A vehicle holder's asset is a vehicle; only a fixed installation stands at a place.
|
||||
|
||||
## Rollback
|
||||
|
||||
Before any data matters, run `DROP SCHEMA fm CASCADE` as the loader. This also drops every
|
||||
`public.v_*` view, and leaves fuel-db as it was before 0001.
|
||||
Before any data matters, run `DROP SCHEMA licence CASCADE; DROP SCHEMA fm CASCADE` as the loader. This
|
||||
also drops every `public.v_*` view, and leaves fuel-db as it was before 0001.
|
||||
|
||||
Once data has been loaded, this deletes it. The corpus remains the source, so a re-load restores
|
||||
it, except baseline versions and their approvals, which live only in fuel-db: export
|
||||
`fm.baseline_version` and `fm.baseline_input` first.
|
||||
|
||||
## Verified locally (2026-10-24, scratch only)
|
||||
## Verified locally (2026-10-30, scratch only)
|
||||
|
||||
The reads below ran through Trino, removed since on 29 Oct 2026, as today's `fuel_ro`.
|
||||
Not run against the live fuel-db. An earlier set of 150 checks was a scratch script, since removed;
|
||||
these replace it:
|
||||
|
||||
Setup:
|
||||
- `timescale/timescaledb:2.31.0-pg17`, with this repository's `01-roles.sh`;
|
||||
- a loader created by the README recipe;
|
||||
|
||||
Results:
|
||||
- **Apply:** all three files applied as the loader, under the loader's default `search_path`.
|
||||
- **Fixture:** the operator workbook of 21 Oct 2026, the five site records, and synthetic sites
|
||||
for the runtime path.
|
||||
- **Tests:** all 150 checks of the design's §9 pass: 98 written with the migrations, and 52
|
||||
regressions for the implementation inspection's findings. These cover:
|
||||
- figures and statuses;
|
||||
- claims, reviews and revisions;
|
||||
- movement events;
|
||||
- coverage, placement and idle scope;
|
||||
- baseline dependencies and approval;
|
||||
- the L/day path;
|
||||
- current periods;
|
||||
- flow-meter reconciliation (coverage, placement, visit times), runtime validity, modelled-L/h
|
||||
windows, disputed claims supplying no values, the L/day reference, S2's L/h rows following
|
||||
the version's validity, corrections under one `event_key`, fixed meter and epoch owners, and
|
||||
population keys that name the generators a coverage assumption covers;
|
||||
- every refusal listed above.
|
||||
- **Read-only role:**
|
||||
- `fm` is unreadable: a query is refused, though its names are listed (see Applying);
|
||||
- writes are refused.
|
||||
- **Not run on the server:** none of this has been run against the live fuel-db.
|
||||
- **Rules:** `../tests/run.sh` applies these files to a throw-away database and runs `tests/rules.sql`
|
||||
and `tests/read_only.sql`, with synthetic data only: each rule with an accepted and a refused case,
|
||||
two customers, a vehicle as asset and holder, licences, and what `fuel_ro` may read.
|
||||
- **Loader:** fuel-monitoring's `make selftest` and `bin/fmdb.py dbtest`.
|
||||
- **Row comparison:** before this rewrite, all 18 views were dumped from a database loaded from the
|
||||
corpus and a recorded review manifest; after it, the same corpus gave identical rows, leaving out
|
||||
`operator_uid`. That one-time comparison covers only data the corpus has; rules with no corpus data
|
||||
rest on `rules.sql`.
|
||||
Reference in new issue
Block a user