Compare commits

..
3 Commits
Author SHA1 Message Date
Nasim SadatandClaude Opus 5.5 78a498f1ce Sign-in by customer: realm petrolyst, row rules, guest role (PD-21)
Keycloak import for realm petrolyst: two clients, petrolyst with MFA for
Superset and Traefik, petrolyst-app password only. It lists only
app.petrolyst.com and traefik.petrolyst.com, and no customer names, as this
repository is public. public.v_operator shows each customer's group. Dashboards
overview and site, role Petrolyst Guest, filtered by row. Superset refuses a
user with no mapped role. The realm travels with the release: each release's
compose files name their realm and secret file, and a rollback needs only its
target's secret. Not deployed.

Checks earlier on this content: tests/run.sh 109 of 109 and read-only, dashboards
import twice, column drift 0, compose config on old and new trees, the secret
check on a scratch folder, a rehearsal on a throwaway Keycloak 26.4. Never a real
deploy or rollback; shellcheck not installed today, clean when last run. Codex
inspected once; its fixes are not re-inspected, on the owner's word.

Bands: persistence, Superset. Rules: 1, 2, 3, 6, 14.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 18:31:14 +01:00
Nasim SadatandClaude Opus 5.5 ab553526f6 fuel-db: customer on every row, place and holder for site, licence tables
Draft migrations rewritten in place (PD-16, PD-18), with 0004 for the licence
tables (PD-20) and rule tests in fuel-db/tests. Dashboards' column list
regenerated; coverage_detail selects operator_uid.

Checks: tests/run.sh 106 of 106; 18 views, 686 rows, 0 differences against a
reference from the same corpus. Codex's fixes to its inspection are not
re-inspected.

Bands: persistence, Superset. Rules: 6, 8.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 14:40:16 +01:00
Nasim SadatandClaude Opus 5.5 e4806f5799 Trino out of the code; commit messages name bands and rules
The rename step in init.sh and the guard in deploy/lib.sh are gone. README's
procedure is now "Removing the old project"; it keeps the old names until it
has run on the server. Migrations stay here (open point 8), and their README
says so. AGENTS.md adds the commit-message rule. Not deployed: live still runs
Trino.

Bands: 4 (Superset), persistence (docs). Rules: 2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 13:50:57 +01:00
25 changed files with 1520 additions and 532 deletions

No files matched your search

+5 -2
View File
@@ -12,9 +12,11 @@ keep the rules there; a change against a rule or an open conflict goes to the ow
- `stacks/`, `traefik/`, `deploy/`, `host/`: how and where things run.
- `petrolyst-insights/`: what Superset looks like and does (config, security manager, the guest
role, branding, patches, dashboards as code). Its README maps each file to the image.
- `keycloak/`: the `petrolyst` client import.
- `keycloak/`: the import of realm `petrolyst`'s two clients (`petrolyst`, `petrolyst-app`).
## Rules
- **Commit messages** end their body with the bands and rules touched, for example
`Bands: 4 (server). Rules: 2, 3.`, or `Bands: none (docs).` for text only (architecture rule 1).
- **Deploys:** only by README "Manual deploy", only when the owner asks. There is no CI. Commit on
`main` first; never push or deploy unprompted.
- **No secrets** in the repository; `deploy/deploy.sh` refuses if one appears.
@@ -24,7 +26,8 @@ keep the rules there; a change against a rule or an open conflict goes to the ow
export. `import_bundle.py` never deletes; old objects go by hand (`dashboards/README.md`).
- **Embedding** stays off in production (`superset_config.py`) until PD-12's points are decided;
the local stack turns it on in its own override.
- **ATOMA's users** have no Superset role (PD-13): `atoma-reader` opens the Petrolyst app only.
- **Customers' users** have no Superset account (PD-13, PD-21): role `reader` of client `petrolyst-app`
opens the Petrolyst app only, for the customer of their Keycloak group; Superset refuses them.
- fuel-db migrations (`stacks/fuel-monitoring/fuel-db/migrations/`) are applied by hand (P3).
## Checks
+81 -47
View File
@@ -6,7 +6,7 @@ Everything that runs on the petrolyst server (109.199.127.84, Ubuntu 24.04):
https://traefik.petrolyst.com, behind Keycloak.
- **fuel-monitoring** (ATOMA): Superset at https://atoma.petrolyst.com, reading the operational
store `fuel-db` (Postgres 17 with TimescaleDB) directly, as the read-only role `fuel_ro`
(Trino was removed on 29 Oct 2026). Sign-in goes through Keycloak
(no Trino since 29 Oct 2026). Sign-in goes through Keycloak
client `petrolyst`, and the UI carries the Petrolyst brand.
**Deploys are manual.** A person deploys a clean commit of `main` from the Mac ("Manual deploy"
@@ -21,7 +21,7 @@ Customization and deployment are separate, as with media-insights and media-inte
| `stacks/fuel-monitoring/` | Compose stack, Superset Dockerfile and init, fuel-db roles and migrations |
| `deploy/` | `deploy.sh` (run on the server by a manual deploy), `rollback.sh` (manual), `lib.sh` (shared) |
| `host/` | one-time server setup: `bootstrap.sh`, `secrets-init.sh` |
| `keycloak/` | the `petrolyst` client for Keycloak's partial import |
| `keycloak/` | the clients `petrolyst` and `petrolyst-app` for Keycloak's partial import |
No credential is in this repository. Secrets exist only on the server, under
`/opt/petrolyst/secrets/`. A deploy needs only root SSH access to the server (`ssh petrolyst`).
@@ -88,16 +88,23 @@ Create the private repository `Petrolyst/petrolyst-infra` and push this reposito
it. Gitea only keeps the history: deploys are manual, so it needs no runner, no Actions and no
deploy secrets.
### 2. Keycloak (realm `thedigitalbridge` on iam.thedigitalbridge.io)
### 2. Keycloak (realm `petrolyst` on iam.thedigitalbridge.io)
1. Realm settings → Action → **Partial import**: choose `keycloak/petrolyst-partial-import.json`,
1. Realm settings → Action → **Partial import**: choose `keycloak/petrolyst-realm-import.json`,
and "Skip" if a resource exists. This creates:
- the confidential client `petrolyst`: standard flow only, PKCE S256, redirect URIs for both
hosts;
- its client roles `superset-admin`, `superset-editor`, `superset-viewer`, `atoma-reader`
and `traefik-admin`;
- a mapper that puts those roles into the claim `petrolyst_roles`.
- the confidential client `petrolyst`, for Superset and Traefik: standard flow only, PKCE S256,
redirect URIs for both hosts;
- its client roles `superset-admin`, `superset-editor`, `superset-viewer` and `traefik-admin`;
- a mapper that puts those roles into the claim `petrolyst_roles`;
- the confidential client `petrolyst-app`, for the Petrolyst app, with its role `reader` and two
mappers: `petrolyst_roles`, and the person's groups with their full path in `petrolyst_groups`,
both in the ID token. Both clients' addresses are on app.petrolyst.com, where everything moves
(PD-16, PD-21); atoma.petrolyst.com is going away and is not in the import.
2. Clients → petrolyst → Credentials: copy the client secret (step 3 stores it on the server).
**Realm settings:** registration off; brute-force detection on; a password policy of at least 12
characters, because a password is the only lock for customers' users; SSO session idle 8 hours,
max 10 hours, because the app's session is renewed through Keycloak about hourly and the default
idle limit of 30 minutes would ask for a password each time.
3. **MFA**, which Exhibit B requires. A copy of the built-in `browser` flow is not enough, for two
reasons: its Cookie step accepts an existing password-only session from another client of the
realm, and its conditional OTP subflow skips users who have not set up OTP. Instead:
@@ -107,14 +114,17 @@ deploy secrets.
- Clients → petrolyst → Advanced → Authentication flow overrides → Browser flow:
`petrolyst-mfa`.
Result: every sign-in to a Petrolyst app asks for password and OTP, and a user without OTP
enrols first. Other clients of the realm are unaffected. There is no silent SSO between
Superset and the dashboard.
4. Give people roles: Users → (user) → Role mapping → Assign role → filter by clients →
`petrolyst`. Group mappings work too. ATOMA's users get `atoma-reader` and nothing else:
it opens the Petrolyst app (petrolyst-platform) and maps to no Superset role, so in
Superset they see nothing. A realm that already has the client gets the role by hand
(Clients → petrolyst → Roles → Create role) or by importing again with "Skip".
Result: every sign-in to Superset or Traefik asks for password and OTP, and a user without OTP
enrols first. There is no silent SSO between them. Client `petrolyst-app` keeps the realm's own
flow: customers' users sign in with a password only, the owner's choice "for now" (PD-21).
4. **A customer is a top-level group.** Groups → Create group, named as the customer's
`keycloak_group` in fuel-db (without the slash), then Role mapping → Assign role → filter by
clients → `petrolyst-app` `reader`. Adding a person to the group is then the only step, and they
see that customer in the app and no other. Exactly one customer per person: two give "No access".
They need no role on client `petrolyst`, and Superset refuses them. A person in no customer's
group sees "No access"; that includes our own staff, who join a customer's group to see it in
the app.
**Administrators:** Users → (user) → Role mapping → Assign role → filter by clients → `petrolyst`.
5. Users of the Traefik dashboard need **Email verified** switched on, because oauth2-proxy
refuses unverified addresses.
@@ -158,7 +168,7 @@ becomes the first Superset admin; there is no local admin account.
| Task | How |
|---|---|
| Deploy | "Manual deploy" below, from a clean `main` |
| Re-deploy a kept release | `sudo -u deploy /opt/petrolyst/infra/deploy/deploy.sh <sha>` |
| Re-deploy a kept release | `sudo -u deploy /opt/petrolyst/infra/deploy/deploy.sh <sha>`. Not across a change of Keycloak realm: see "Moving to realm petrolyst" |
| Roll back | see below |
| Traefik dashboard | https://traefik.petrolyst.com (role `traefik-admin`) |
| fuel-db shell | `docker exec -it petrolyst-fuel-db-1 psql -U postgres -d fuel` |
@@ -245,18 +255,14 @@ Tables it creates, hypertables included, then belong to `fuel_owner`, and `fuel_
After `SET ROLE NONE`, it cannot create anything in `public`. To repair a table that another
role created, run `ALTER TABLE <t> OWNER TO fuel_owner`, then `GRANT SELECT ON <t> TO fuel_ro`.
**One-time steps after 29 Oct 2026**, at the first deploy that includes them. Two changes
meet there:
- Trino is removed: Superset reads fuel-db directly. The role `trino_ro` becomes `fuel_ro` and
keeps its password (a SCRAM password survives a rename); its secret file moves with it.
- The Compose project `fuel-monitoring` is renamed `petrolyst`. Containers are then called
`petrolyst-<service>-1`. The data volumes keep their names (`fuel-monitoring_superset_db_data`,
`fuel-monitoring_fuel_db_data`; pinned in `compose.yaml`), so no data moves. The old project's
containers must be gone first: two PostgreSQL servers on one data volume corrupt it.
Preflight refuses to deploy while any of them exists.
**Removing the old project**, once, right before the next deploy; delete this paragraph when it
is done. The server still runs the Compose project `fuel-monitoring`, with Trino. Afterwards it runs
`petrolyst`, without it: containers are `petrolyst-<service>-1`, and the data volumes keep their
names (pinned in `compose.yaml`), so no data moves. The old containers must be gone first: two
PostgreSQL servers on one data volume corrupt it, and preflight refuses to deploy while any exists.
As root on the server, right before the deploy (Superset is down from step 3 until the deploy
finishes):
As root on the server (Superset is down from step 3 until the deploy finishes). The role keeps its
password, because a SCRAM password survives a rename:
```bash
S=/opt/petrolyst/secrets/fuel-monitoring
@@ -267,20 +273,52 @@ docker exec -i fuel-monitoring-fuel-db-1 psql -U postgres -d fuel -v ON_ERROR_ST
docker rm -f $(docker ps -aq --filter label=com.docker.compose.project=fuel-monitoring) # 3
```
4. Deploy as usual ("Manual deploy"). Compose starts the `petrolyst` project on the same volumes
and no longer starts Trino. Compose warns once that the volumes belong to another project;
that is expected. `superset-init` renames the connection "Fuel (Trino)" to "Fuel" and points
it at fuel-db; the smoke test checks that it reads as `fuel_ro`, read-only.
5. Clean up: `rm /opt/petrolyst/env/fuel-monitoring.env`, `rmdir /opt/petrolyst/env`,
4. Deploy as usual ("Manual deploy"). Compose warns once that the volumes belong to another project;
that is expected. `superset-init` registers the connection "Fuel"; the smoke test checks that it
reads as `fuel_ro`, read-only.
5. Delete the old connection. It refuses if a dataset still uses it; then stop and look:
```bash
docker exec -i petrolyst-superset-1 /app/.venv/bin/python - <<'PY'
from superset.app import create_app
with create_app().app_context():
from superset import db
from superset.connectors.sqla.models import SqlaTable
from superset.models.core import Database
old = db.session.query(Database).filter_by(database_name="Fuel (Trino)").one_or_none()
if old:
used = db.session.query(SqlaTable).filter_by(database_id=old.id).count()
if used:
raise SystemExit(f"{used} datasets still use the old connection")
db.session.delete(old)
db.session.commit()
print("old connection deleted")
PY
```
6. Clean up: `rm /opt/petrolyst/env/fuel-monitoring.env`, `rmdir /opt/petrolyst/env`,
`docker image rm trinodb/trino:484`,
`docker network rm fuel-monitoring_backend fuel-monitoring_fuel-db-tunnel`.
7. Releases from before this deploy can no longer run, as their role is gone, and `rollback.sh`
stops on them. Remove them from `/opt/petrolyst/releases/`.
**Rollback across the Trino removal** (not tested). `rollback.sh` stops for a release from before
it. To go back anyway: remove the `petrolyst` project's containers, reverse steps 1 and 2 above (rename `fuel_ro` back to `trino_ro` and
`ALTER ROLE trino_ro RESET default_transaction_read_only`, move the secret file back, write
`/opt/petrolyst/env/fuel-monitoring.env` with `FUEL_DB_TRINO_PASSWORD=<that secret>`), rename
the Superset connection "Fuel" back to "Fuel (Trino)" with `trino://superset@trino:8080/fuel`,
then run that release's own script, `/opt/petrolyst/releases/<sha>/deploy/rollback.sh <sha>`.
**Moving to realm `petrolyst`, one time.** A deploy of its own, after the deploy that removes Trino:
releases from before that one can no longer run, so a combined deploy leaves nothing to roll back to.
1. The realm is ready (Keycloak, above) and an admin has signed in on client `petrolyst` there. While
Superset still answers at atoma.petrolyst.com, add that address's redirect URI to the client by
hand, or nobody can sign in after step 3; remove it when the domain goes.
2. As root: `host/secrets-init.sh keycloak-secret`. The value goes into the new file
`shared/keycloak_petrolyst_realm_client_secret`, so nothing changes for anyone yet. Leave the old
file `shared/keycloak_petrolyst_client_secret` alone.
3. Deploy by "Manual deploy" only. That one step moves Superset and Traefik together.
4. **Prove it by hand:** an admin signs in to Superset and to the Traefik dashboard. The smoke tests
check redirects only, and pass with a wrong secret.
5. **To go back:** `rollback.sh <previous sha>` returns to the old realm with the old secret. Afterwards go
forward again by "Manual deploy" only: "Re-deploy a kept release" runs the old `lib.sh`, whose realm
address would override the new release's.
6. Not before 7 days of working sign-ins, and on the owner's word: retire client `petrolyst` in realm
`thedigitalbridge` and delete the old secret file. From then on there is no way back.
**Adding a project**: create `stacks/<name>/compose.yaml`, attach the public service to the
external network `proxy`, and set `traefik.*` labels with a unique router name and
@@ -299,8 +337,6 @@ its own gate in its stack:
## Verified locally (2026-10-23)
Before Trino's removal (29 Oct 2026): its lines below are history.
An end-to-end run ran the real `deploy.sh` and `rollback.sh` in a Linux container against
Docker Desktop, with a scratch `/opt/petrolyst`. Production differences in that run:
- a test CA certificate stood in for Let's Encrypt;
@@ -310,7 +346,6 @@ Docker Desktop, with a scratch `/opt/petrolyst`. Production differences in that
Results:
- The fresh install took the init path.
- An identical re-run recreated nothing and left `ARTIFACTS` byte-identical.
- A Trino-only change recreated only `trino`.
- A Superset change stopped, backed up, migrated and recreated only `superset`.
- A change to `superset/init.sh` alone did the same, and the changed script ran.
- A same-SHA upload with different content was refused.
@@ -327,13 +362,12 @@ Results:
- Retention pruned read-only releases.
- Retention removed a queued upload that a plain `rsync -a` had given an old checkout's
directory times. It kept one uploaded with `--omit-dir-times`.
- In `fuel-db`, a loader that created a table as its own role left it unreadable for `trino_ro`.
With the "Data loaders" recipe, tables (hypertables too) belonged to `fuel_owner` and Trino
could read them. After `SET ROLE NONE`, the loader could not create anything.
- In `fuel-db`, a loader that created a table as its own role left it unreadable for the read-only
role. With the "Data loaders" recipe, tables (hypertables too) belonged to `fuel_owner` and that
role could read them. After `SET ROLE NONE`, the loader could not create anything.
- Smoke tests passed, including the redirects to the real Keycloak with `client_id=petrolyst`
and PKCE S256.
- TLS 1.1 and unknown SNI were refused, and HTTP redirected to HTTPS.
- Trino and `trino_ro` both refused writes.
Not verified locally:
- Let's Encrypt issuance;
+29 -16
View File
@@ -17,9 +17,11 @@ ROLLBACK_MARK=$STATE/rollback-in-progress
ATOMA_HOST=${ATOMA_HOST:-atoma.petrolyst.com}
TRAEFIK_HOST=${TRAEFIK_HOST:-traefik.petrolyst.com}
KEYCLOAK_REALM_URL=${KEYCLOAK_REALM_URL:-https://iam.thedigitalbridge.io/realms/thedigitalbridge}
PETROLYST_ROOT=$ROOT
export ATOMA_HOST TRAEFIK_HOST KEYCLOAK_REALM_URL PETROLYST_ROOT
# The Keycloak realm is not set here: each release's compose files name their own, with the secret
# file that belongs to it. A rollback, which runs this file from the release it leaves, therefore
# returns to the old release's realm and secret. Exporting a value here would override them.
export ATOMA_HOST TRAEFIK_HOST PETROLYST_ROOT
# --host pins the daemon, as in petrolyst-web/deploy/deploy.sh: a stale
# `docker context use` must not send this deploy to another daemon.
@@ -39,8 +41,8 @@ SUPERSET_INPUTS=(
)
# Every file the stacks need from /opt/petrolyst (host/secrets-init.sh).
# The Keycloak client secret is not listed: it depends on the release (check_keycloak_secrets).
REQUIRED_FILES=(
secrets/shared/keycloak_petrolyst_client_secret
secrets/traefik/oauth2_proxy_cookie_secret
secrets/fuel-monitoring/superset_secret_key
secrets/fuel-monitoring/superset_db_password
@@ -94,11 +96,6 @@ load_artifacts() {
while IFS= read -r line; do
key=${line%%=*}
value=${line#*=}
# A release from before 29 Oct 2026 still runs Trino, which logs in as
# `trino_ro`, since renamed `fuel_ro`: it cannot start (README "Rollback
# across the Trino removal").
[ "$key" != TRINO_CONFIG_HASH ] ||
die "$file is from before Trino was removed; see README \"Rollback across the Trino removal\""
[[ $key =~ ^(SUPERSET_IMAGE_TAG|TRAEFIK_CONFIG_HASH)$ && $value =~ ^[0-9a-f]{16}$ ]] ||
die "unexpected line in $file: $line"
printf -v "$key" '%s' "$value"
@@ -131,10 +128,7 @@ preflight() {
for f in "${REQUIRED_FILES[@]}"; do
[ -s "$ROOT/$f" ] || die "missing or empty: $ROOT/$f (run host/secrets-init.sh as root)"
done
# oauth2-proxy uses the client secret file byte for byte.
f=$ROOT/secrets/shared/keycloak_petrolyst_client_secret
[ "$(tail -c1 "$f" | wc -l)" -eq 0 ] ||
die "$f ends with a newline; store it again with: host/secrets-init.sh keycloak-secret"
check_keycloak_secrets "$release"
bad=$(find "$release/petrolyst-insights" \( -name 'Dockerfile*' -o -name '*compose*.y*ml' \
-o -name '*.env' -o -name '.env' -o -path '*/.gitea/*' -o -path '*/.github/*' \) -print)
[ -z "$bad" ] || die "petrolyst-insights/ holds customization only; found: $bad"
@@ -143,7 +137,22 @@ preflight() {
# The stack was the Compose project "fuel-monitoring" until 29 Oct 2026; "petrolyst" uses the
# same data volumes, so both must never run at once (README "One-time steps after 29 Oct 2026").
[ -z "$("${DOCKER[@]}" ps -aq --filter label=com.docker.compose.project=fuel-monitoring)" ] ||
die "containers of the old project fuel-monitoring exist; remove them first (README \"One-time steps after 29 Oct 2026\")"
die "containers of the old project fuel-monitoring exist; remove them first (README \"Removing the old project\")"
}
# check_keycloak_secrets <release>: the client secret that release's compose files name, which
# differs per realm. So a rollback needs only its target's secret, not the one of the release it leaves.
check_keycloak_secrets() {
local release=$1 names name f
names=$(cat "$release/stacks/fuel-monitoring/compose.yaml" "$release/traefik/compose.yaml" |
grep -oE 'secrets/shared/keycloak_[a-z_]+' | sort -u) || die "$release names no Keycloak client secret"
for name in $names; do
f=$ROOT/$name
[ -s "$f" ] || die "missing or empty: $f (run host/secrets-init.sh as root)"
# oauth2-proxy uses the client secret file byte for byte.
[ "$(tail -c1 "$f" | wc -l)" -eq 0 ] ||
die "$f ends with a newline; store it again"
done
}
# promote <release> <sha>: make the release the live tree. --chmod applies to
@@ -239,7 +248,7 @@ probe() {
}
smoke_tests() {
local i body code location
local i body code location realm
log "smoke tests"
# First deploy: Let's Encrypt may need a minute to issue the certificate.
for i in $(seq 1 24); do
@@ -249,17 +258,21 @@ smoke_tests() {
sleep 5
done
# The realm this release runs on. These tests check redirects only, not a real sign-in: they
# pass with a wrong client secret, so a person signs in after a change of realm or secret.
realm=$(fm exec -T superset printenv KEYCLOAK_REALM_URL) || die "Superset has no KEYCLOAK_REALM_URL"
read -r code location < <(probe "$ATOMA_HOST" /login/)
[ "$code" = 302 ] && [[ $location == "https://$ATOMA_HOST/login/keycloak"* ]] ||
die "/login/ should redirect to /login/keycloak, got $code $location"
read -r code location < <(probe "$ATOMA_HOST" /login/keycloak)
[ "$code" = 302 ] && [[ $location == "$KEYCLOAK_REALM_URL/protocol/openid-connect/auth?"* ]] &&
[ "$code" = 302 ] && [[ $location == "$realm/protocol/openid-connect/auth?"* ]] &&
[[ $location == *client_id=petrolyst* && $location == *code_challenge_method=S256* ]] ||
die "/login/keycloak should redirect to Keycloak with client_id=petrolyst and PKCE S256, got $code $location"
read -r code location < <(probe "$TRAEFIK_HOST" /)
[ "$code" = 302 ] && [[ $location == "$KEYCLOAK_REALM_URL/"* || $location == *"/oauth2/start"* ]] ||
[ "$code" = 302 ] && [[ $location == "$realm/"* || $location == *"/oauth2/start"* ]] ||
die "https://$TRAEFIK_HOST/ should redirect to Keycloak, got $code $location"
# Superset's own connection "Fuel" (fuel_ro, password from /run/secrets) reads fuel-db and
+6 -5
View File
@@ -2,7 +2,8 @@
# Server secrets, run as root after bootstrap.sh.
# secrets-init.sh generate every missing secret; never overwrites one
# secrets-init.sh keycloak-secret store the client secret of Keycloak client
# "petrolyst" (Clients > petrolyst > Credentials),
# "petrolyst" of realm "petrolyst"
# (Clients > petrolyst > Credentials),
# read from the terminal or stdin
#
# Layout: /opt/petrolyst/secrets/{shared,traefik,fuel-monitoring}/ are root:deploy
@@ -50,8 +51,8 @@ case ${1:-all} in
fi
value=$(printf '%s' "$value" | tr -d '[:space:]')
[ -n "$value" ] || die "empty secret"
write_secret "$SECRETS/shared/keycloak_petrolyst_client_secret" "$value"
echo "stored $SECRETS/shared/keycloak_petrolyst_client_secret"
write_secret "$SECRETS/shared/keycloak_petrolyst_realm_client_secret" "$value"
echo "stored $SECRETS/shared/keycloak_petrolyst_realm_client_secret"
exit 0
;;
all) ;;
@@ -66,10 +67,10 @@ generate "$SECRETS/fuel-monitoring/fuel_db_ro_password" 32
# 32 hex characters = a 32-byte cookie key, as oauth2-proxy requires.
generate "$SECRETS/traefik/oauth2_proxy_cookie_secret" 16
if [ ! -s "$SECRETS/shared/keycloak_petrolyst_client_secret" ]; then
if [ ! -s "$SECRETS/shared/keycloak_petrolyst_realm_client_secret" ]; then
echo
echo "Still missing: the Keycloak client secret. Copy it from Keycloak"
echo "(realm thedigitalbridge > Clients > petrolyst > Credentials), then run:"
echo "(realm petrolyst > Clients > petrolyst > Credentials), then run:"
echo " $0 keycloak-secret"
exit 1
fi
+114
View File
@@ -0,0 +1,114 @@
{
"clients": [
{
"clientId": "petrolyst",
"name": "Petrolyst administration",
"description": "Superset and the Traefik dashboard (oauth2-proxy), for our own people, with MFA (browser flow override petrolyst-mfa, set by hand). Managed from petrolyst-infra/keycloak/.",
"enabled": true,
"protocol": "openid-connect",
"publicClient": false,
"clientAuthenticatorType": "client-secret",
"standardFlowEnabled": true,
"implicitFlowEnabled": false,
"directAccessGrantsEnabled": false,
"serviceAccountsEnabled": false,
"frontchannelLogout": true,
"baseUrl": "https://app.petrolyst.com/",
"redirectUris": [
"https://app.petrolyst.com/oauth-authorized/keycloak",
"https://traefik.petrolyst.com/oauth2/callback"
],
"webOrigins": ["+"],
"attributes": {
"pkce.code.challenge.method": "S256",
"post.logout.redirect.uris": "https://app.petrolyst.com/*##https://traefik.petrolyst.com/*"
},
"protocolMappers": [
{
"name": "petrolyst client roles",
"protocol": "openid-connect",
"protocolMapper": "oidc-usermodel-client-role-mapper",
"consentRequired": false,
"config": {
"usermodel.clientRoleMapping.clientId": "petrolyst",
"multivalued": "true",
"claim.name": "petrolyst_roles",
"jsonType.label": "String",
"id.token.claim": "true",
"access.token.claim": "true",
"userinfo.token.claim": "true",
"introspection.token.claim": "true"
}
}
]
},
{
"clientId": "petrolyst-app",
"name": "Petrolyst",
"description": "The Petrolyst app (petrolyst-platform server/), for customers' users. Password only (PD-21). The customer comes from the person's top-level group. Managed from petrolyst-infra/keycloak/.",
"enabled": true,
"protocol": "openid-connect",
"publicClient": false,
"clientAuthenticatorType": "client-secret",
"standardFlowEnabled": true,
"implicitFlowEnabled": false,
"directAccessGrantsEnabled": false,
"serviceAccountsEnabled": false,
"frontchannelLogout": true,
"baseUrl": "https://app.petrolyst.com/app/",
"redirectUris": [
"https://app.petrolyst.com/app/auth/callback"
],
"webOrigins": [],
"attributes": {
"pkce.code.challenge.method": "S256",
"post.logout.redirect.uris": "https://app.petrolyst.com/app/*"
},
"protocolMappers": [
{
"name": "petrolyst client roles",
"protocol": "openid-connect",
"protocolMapper": "oidc-usermodel-client-role-mapper",
"consentRequired": false,
"config": {
"usermodel.clientRoleMapping.clientId": "petrolyst-app",
"multivalued": "true",
"claim.name": "petrolyst_roles",
"jsonType.label": "String",
"id.token.claim": "true",
"access.token.claim": "false",
"userinfo.token.claim": "false",
"introspection.token.claim": "false"
}
},
{
"name": "petrolyst groups",
"protocol": "openid-connect",
"protocolMapper": "oidc-group-membership-mapper",
"consentRequired": false,
"config": {
"full.path": "true",
"claim.name": "petrolyst_groups",
"id.token.claim": "true",
"access.token.claim": "false",
"userinfo.token.claim": "false",
"introspection.token.claim": "false"
}
}
]
}
],
"roles": {
"client": {
"petrolyst": [
{ "name": "superset-admin", "description": "Superset Admin" },
{ "name": "superset-editor", "description": "Superset Alpha: build charts and dashboards" },
{ "name": "superset-viewer", "description": "Superset Gamma: view dashboards shared with the role" },
{ "name": "traefik-admin", "description": "Traefik dashboard at traefik.petrolyst.com" }
],
"petrolyst-app": [
{ "name": "reader", "description": "Reads their customer's data in the Petrolyst app; no role in Superset. Mapped onto each customer's group." }
]
}
}
}
+7 -7
View File
@@ -15,8 +15,8 @@ and where it runs lives in `stacks/`, `traefik/`, `deploy/` and `host/`.
|---|---|---|
| `superset_config.py` | Keycloak sign-in, role mapping, proxy and cookie settings, Redis caches, theme and chart palettes | `/app/pythonpath/` |
| `petrolyst_security.py` | `PetrolystSecurityManager`: user info and roles from the Keycloak ID token, `/login/` straight to Keycloak, `/logout/` through Keycloak | `/app/pythonpath/` |
| `roles/` | `atoma_guest.py`: the role "ATOMA Guest" for the Petrolyst app's embedded dashboards, with exactly its permissions; run by hand (see "Embedding") | `/app/petrolyst-roles/` |
| `assets/branding/` | logos (full colour and white): the wordmark alone for the navbar, and the stacked logo; favicons, app icons; ATOMA's logo, which the Petrolyst app shows in its map's corner (`PETROLYST_OPERATOR_LOGO`) | `/app/superset/static/assets/branding/`, served at `/static/assets/branding/…` |
| `roles/` | `petrolyst_guest.py`: the role "Petrolyst Guest" for the Petrolyst app's embedded dashboards, with exactly its permissions; run by hand (see "Embedding") | `/app/petrolyst-roles/` |
| `assets/branding/` | logos (full colour and white): the wordmark alone for the navbar, and the stacked logo; favicons, app icons; ATOMA's logo, which the Petrolyst app shows in its map's corner (the server reads this file itself: `PETROLYST_OPERATOR_LOGO_FILE`) | `/app/superset/static/assets/branding/`, served at `/static/assets/branding/…` |
| `patches/` | runtime patches, `fix_*.py` with `--verify` (see its README); none at present | `/app/patches/petrolyst/`, applied during the build |
| `dashboards/` | exported dashboards and import scripts (see its README) | not in the image |
@@ -35,13 +35,13 @@ value.
|---|---|
| `superset_secret_key` | Flask `SECRET_KEY` |
| `superset_db_password` | password of user `superset` on the metadata database |
| `keycloak_petrolyst_client_secret` | client secret of Keycloak client `petrolyst` |
| `keycloak_petrolyst_client_secret` | client secret of Keycloak client `petrolyst`, realm `petrolyst`; on the server in `shared/keycloak_petrolyst_realm_client_secret` |
On the server they come from `/opt/petrolyst/secrets/` (see the deployment README).
## Sign-in and roles
- Provider: Keycloak realm `thedigitalbridge` at `iam.thedigitalbridge.io`, client `petrolyst`,
- Provider: Keycloak realm `petrolyst` at `iam.thedigitalbridge.io`, client `petrolyst`,
authorization code flow with PKCE S256. The realm URL can be overridden with
`KEYCLOAK_REALM_URL`.
- Roles come from the client roles of `petrolyst`, which a protocol mapper puts into the claim
@@ -68,14 +68,14 @@ stack** (`.local/fmdb-scratch`); `superset_config.py` here leaves it off, and lo
guest secret.
- The scratch stack's `superset_config_local.py` turns on `EMBEDDED_SUPERSET`, names the guest
role `GUEST_ROLE_NAME = "ATOMA Guest"`, reads the signing secret with
role `GUEST_ROLE_NAME = "Petrolyst Guest"`, reads the signing secret with
`_read_secret("superset_guest_token_secret")` and expects the audience `petrolyst-app`. The
Petrolyst server signs the guest passes itself with the same secret.
- `roles/atoma_guest.py` gives "ATOMA Guest" exactly what a dashboard needs to be read. Which
- `roles/petrolyst_guest.py` gives "Petrolyst Guest" exactly what a dashboard needs to be read. Which
dashboard a guest sees comes from the pass, not the role. It is not in `init.sh`; run it once,
and again whenever its permissions change:
docker exec <superset> /app/.venv/bin/python /app/petrolyst-roles/atoma_guest.py
docker exec <superset> /app/.venv/bin/python /app/petrolyst-roles/petrolyst_guest.py
- The app's dashboards and their embedding ids: `dashboards/README.md`.
+1 -1
View File
@@ -4,7 +4,7 @@ Superset dashboards as code and the operator scripts that import them. Not baked
image, so changes here do not rebuild or restart Superset.
- `atoma/`: ATOMA's two dashboards for the Petrolyst app (PD-12, PD-13; SPEC.md §10):
`atoma-overview` (fleet totals) and `atoma-site` (one site, its four sections as tabs). The
`overview` (fleet totals) and `site` (one site, its four sections as tabs), named `atoma-overview` and `atoma-site` until 30 Oct. The
app embeds them; admins open them in Superset, with the filter column. `SPEC.md` is the
design; `build_bundle.py` generates the import bundle and `import_bundle.py` imports it
(overwriting the objects the bundle defines).
+6 -6
View File
@@ -1,6 +1,7 @@
# ATOMA dashboards: Overview and Site, for the Petrolyst app
**Status (29 Oct 2026, PD-13): two dashboards, `atoma-overview` and `atoma-site` (§10), built and
**Status (30 Oct 2026, PD-21: every customer sees these two, limited by row; slugs `overview` and `site`, formerly
`atoma-overview` and `atoma-site`, which the rest of this file still uses). Two dashboards (§10), built and
tested on a local Superset; not imported live.** The Petrolyst app embeds them, and admins open
them in Superset with the filter column.
- `build_bundle.py` generates the dashboards, every dataset and every chart from this
@@ -16,8 +17,7 @@ them in Superset with the filter column.
**Datasets.** The `public.v_*` views of
`stacks/fuel-monitoring/fuel-db/migrations/0003_bi_views.sql`, read from fuel-db directly (database
`fuel`, schema `public`) as the read-only role `fuel_ro`. *Until 29 Oct 2026 Superset read them
through Trino as `fuel.public`; Trino was removed then.*
`fuel`, schema `public`) as the read-only role `fuel_ro`.
**Design and rationale** live in the fuel-monitoring corpus, in
`.context/plans/2026-10-24-fuel-db-baseline-dashboard.md`:
@@ -231,7 +231,7 @@ Physical datasets, one per `public.v_*` view in fuel-db's `public` schema, plus
so the opening refill is included) or a bill's AFN amount.
Both use only `v_coverage`, `v_delivery` and `v_electricity_cost`. Every dataset has the same
four site columns from `v_site`: `site_uid`, `site_code`, `site_name`, `site_label`.
five columns from `v_site`: `site_uid`, `site_code`, `site_name`, `site_label`, `operator_uid` (the customer; the row filter uses it, PD-16).
| Dataset | Grain | Used by | Native filters that apply |
|---|---|---|---|
@@ -259,7 +259,7 @@ across methods or generator sets. The site's L/h figure itself comes only from a
(petrolyst-platform `views/src/shared/deepLink.ts`, retired) from its iframe address: the
dashboard path, the Site filter's id and the tab. The Petrolyst app uses the same rison shape for
the Site filter's preset (`app/src/app/core/site-filter.ts`). The rison shape is the one the sites
list used until 28 Oct, confirmed in P2, and the view's tests compared it with that list's Trino
list used until 28 Oct, confirmed in P2, and the view's tests compared it with that list's
output:
```
@@ -409,7 +409,7 @@ Run in P2 against a local Superset, and repeated as a smoke test in P3.
## 8. P2 results (local, 24 Oct 2026)
Local Superset 6.1.0 image built from this repository, Trino 484, fuel-db with migrations
Local Superset 6.1.0 image built from this repository, fuel-db with migrations
0001–0003 and the corpus data loaded; browser checks in headless Chrome.
| Check | Result |
@@ -1,9 +1,10 @@
#!/usr/bin/env python3
"""Generate ATOMA's dashboards for the Petrolyst app as one Superset 6.1 import bundle.
"""Generate the dashboards of the Petrolyst app as one Superset 6.1 import bundle.
Implements SPEC.md in this folder: the two dashboards the Petrolyst app embeds, "ATOMA · Overview"
(atoma-overview) and "ATOMA · Site" (atoma-site, the four detail tabs), one dataset per
public.v_* view and every chart. Nothing in the bundle names a site.
Implements SPEC.md in this folder: the two dashboards the app embeds, "Overview" (overview) and
"Site" (site, the four detail tabs), one dataset per public.v_* view and every chart. Every
customer sees the same dashboards, limited by row: the app's guest pass names the customer (PD-16).
Nothing in the bundle names a site or a customer.
build_bundle.py --database-uuid <uuid of "Fuel"> [--out atoma-dashboards.zip]
@@ -15,7 +16,8 @@ dashboards (the stock dashboard import leaves existing datasets and charts
unchanged).
Before writing, the bundle is checked: every column a chart or filter uses
exists in its dataset, every chart in a Site filter's scope has site_label, the
exists in its dataset, every dataset has operator_uid (the guest pass's row rule needs
it), every chart in a Site filter's scope has site_label, the
Overview sums recorded values only and money only per currency, and no row is
wider than 12. Requires PyYAML.
"""
@@ -109,7 +111,7 @@ CALCULATED = {
# opening visit's delivery is included and the closing visit's excluded.
VIRTUAL = {
"coverage_detail": ("v_coverage", [("lane_name", "character varying(60)"), ("amount", "character varying(120)")], """\
SELECT c.site_uid, c.site_code, c.site_name, c.site_label, c.lane, c.status,
SELECT c.site_uid, c.site_code, c.site_name, c.site_label, c.operator_uid, c.lane, c.status,
c.start_on, c.end_on, c.start_ts, c.end_ts,
regexp_replace(c.lane, '^[0-9]+ ', '') AS lane_name,
CASE WHEN c.lane = '1 Stock intervals' THEN
@@ -166,6 +168,7 @@ METRICS = {
LABELS = {
"site_code": "Code", "site_name": "Site name", "site_label": "Site", "site_uid": "Site uid",
"operator_uid": "Customer",
"pilot_status": "Pilot status", "baseline_status_label": "Baseline status",
"baseline_status": "Baseline status code", "baseline_validity_note": "Validity note",
"lph_estimate": "L/h", "lph_method": "L/h method", "lph_window_start": "L/h window start",
@@ -495,8 +498,11 @@ def timeseries_common(x: str, metrics: list[Any], groupby: list[str], grain: str
# --- Overview (PD-12, embedded in the Petrolyst app) ---------------------------------------
OVERVIEW_SLUG = "atoma-overview"
SITE_SLUG = "atoma-site" # one embedded dashboard with the DETAIL_TABS as its tabs
OVERVIEW_SLUG = "overview"
SITE_SLUG = "site" # one embedded dashboard with the DETAIL_TABS as its tabs
# A dashboard's identity, apart from its slug: the names it was first built under, so a re-import
# updates the existing two (and keeps their embedding ids) rather than adding two new ones.
DASHBOARD_KEYS = {OVERVIEW_SLUG: "atoma-overview", SITE_SLUG: "atoma-site"}
# SPEC §10: only sums and counts of recorded values, never an average of per-site rates; money
# only grouped by currency. The value says whether the metric is money.
@@ -934,7 +940,7 @@ def dashboard_yaml(key: str, title: str, slug: str, layout: Layout, metadata: di
"certified_by": None,
"certification_details": None,
"published": True,
"uuid": uid("dashboard", key),
"uuid": uid("dashboard", DASHBOARD_KEYS[key]),
"position": layout.nodes,
"metadata": metadata,
"version": VERSION,
@@ -999,7 +1005,7 @@ def build(database_uuid: str, columns_path: Path) -> dict[str, Any]:
all_charts = {**{c.key: c for c in detail_charts()}, **overview}
dashboards: list[tuple[Layout, dict[str, Chart], dict[str, Any]]] = []
for key, title in [(OVERVIEW_SLUG, "Overview"), (SITE_SLUG, "Site")]:
page = Layout(f"ATOMA · {title}", all_charts)
page = Layout(title, all_charts)
page.markdown.update({f"notes-{k}": v for k, v in TAB_NOTES.items()})
page.markdown["notes-overview"] = OVERVIEW_NOTES
if key == OVERVIEW_SLUG:
@@ -1012,7 +1018,7 @@ def build(database_uuid: str, columns_path: Path) -> dict[str, Any]:
placed = {k: all_charts[k] for k in page.ids}
page_metadata = base_metadata(page, filters, cross_filters=False)
page_metadata["label_colors"].update(STOCK_STATUS_COLOURS)
files[f"dashboards/{key}.yaml"] = dashboard_yaml(key, f"ATOMA · {title}", key, page,
files[f"dashboards/{key}.yaml"] = dashboard_yaml(key, title, key, page,
page_metadata)
dashboards.append((page, placed, page_metadata))
@@ -1030,6 +1036,12 @@ def check(files: dict[str, Any], views: dict[str, list[tuple[str, str]]],
problems = []
columns = {v: {c for c, _ in cols} | {c[0] for c in CALCULATED.get(v, [])}
for v, cols in views.items()}
for view, cols in columns.items():
if "operator_uid" not in cols:
problems.append(f"{view}: no operator_uid, so the guest pass's row rule cannot hold")
for name, (_, _, sql) in VIRTUAL.items():
if "operator_uid" not in sql:
problems.append(f"{name}: its SQL does not select operator_uid")
for view in list(MAIN_DTTM) + list(CALCULATED) + list(METRICS):
if view not in columns:
problems.append(f"{view}: configured but not in the column snapshot")
@@ -9,6 +9,7 @@ v_assumption site_uid character varying(12)
v_assumption site_code character varying(20)
v_assumption site_name character varying(120)
v_assumption site_label character varying(170)
v_assumption operator_uid character varying(12)
v_assumption code character varying(60)
v_assumption topic character varying(30)
v_assumption asset_uid character varying(40)
@@ -30,6 +31,7 @@ v_baseline_input site_uid character varying(12)
v_baseline_input site_code character varying(20)
v_baseline_input site_name character varying(120)
v_baseline_input site_label character varying(170)
v_baseline_input operator_uid character varying(12)
v_baseline_input baseline_code character varying(60)
v_baseline_input baseline_status character varying(12)
v_baseline_input interval_label character varying(30)
@@ -47,6 +49,7 @@ v_baseline_version site_uid character varying(12)
v_baseline_version site_code character varying(20)
v_baseline_version site_name character varying(120)
v_baseline_version site_label character varying(170)
v_baseline_version operator_uid character varying(12)
v_baseline_version code character varying(60)
v_baseline_version metric character varying(40)
v_baseline_version status character varying(12)
@@ -75,6 +78,7 @@ v_coverage site_uid character varying(12)
v_coverage site_code character varying(20)
v_coverage site_name character varying(120)
v_coverage site_label character varying(170)
v_coverage operator_uid character varying(12)
v_coverage lane character varying(60)
v_coverage status character varying(40)
v_coverage start_on date
@@ -85,6 +89,7 @@ v_data_quality site_uid character varying(12)
v_data_quality site_code character varying(20)
v_data_quality site_name character varying(120)
v_data_quality site_label character varying(170)
v_data_quality operator_uid character varying(12)
v_data_quality check_order integer
v_data_quality check_code character varying(40)
v_data_quality label character varying(80)
@@ -94,6 +99,7 @@ v_delivery site_uid character varying(12)
v_delivery site_code character varying(20)
v_delivery site_name character varying(120)
v_delivery site_label character varying(170)
v_delivery operator_uid character varying(12)
v_delivery event_key character varying(80)
v_delivery delivery_date date
v_delivery seq_in_day integer
@@ -108,6 +114,7 @@ v_electricity_cost site_uid character varying(12)
v_electricity_cost site_code character varying(20)
v_electricity_cost site_name character varying(120)
v_electricity_cost site_label character varying(170)
v_electricity_cost operator_uid character varying(12)
v_electricity_cost bill_month date
v_electricity_cost period_start date
v_electricity_cost period_end date
@@ -122,6 +129,7 @@ v_evidence site_uid character varying(12)
v_evidence site_code character varying(20)
v_evidence site_name character varying(120)
v_evidence site_label character varying(170)
v_evidence operator_uid character varying(12)
v_evidence document_key character varying(16)
v_evidence title character varying(200)
v_evidence kind character varying(20)
@@ -133,6 +141,7 @@ v_hour_meter_evidence site_uid character varying(12)
v_hour_meter_evidence site_code character varying(20)
v_hour_meter_evidence site_name character varying(120)
v_hour_meter_evidence site_label character varying(170)
v_hour_meter_evidence operator_uid character varying(12)
v_hour_meter_evidence meter_label character varying(40)
v_hour_meter_evidence mapped_set character varying(60)
v_hour_meter_evidence reading_date date
@@ -146,6 +155,7 @@ v_imported_calculation site_uid character varying(12)
v_imported_calculation site_code character varying(20)
v_imported_calculation site_name character varying(120)
v_imported_calculation site_label character varying(170)
v_imported_calculation operator_uid character varying(12)
v_imported_calculation cell_ref character varying(40)
v_imported_calculation source_label character varying(80)
v_imported_calculation formula character varying(200)
@@ -160,6 +170,7 @@ v_interval site_uid character varying(12)
v_interval site_code character varying(20)
v_interval site_name character varying(120)
v_interval site_label character varying(170)
v_interval operator_uid character varying(12)
v_interval interval_label character varying(30)
v_interval start_on date
v_interval end_on date
@@ -193,6 +204,7 @@ v_kpi_measures site_uid character varying(12)
v_kpi_measures site_code character varying(20)
v_kpi_measures site_name character varying(120)
v_kpi_measures site_label character varying(170)
v_kpi_measures operator_uid character varying(12)
v_kpi_measures measure_order integer
v_kpi_measures measure character varying(60)
v_kpi_measures value numeric(12,2)
@@ -205,6 +217,7 @@ v_modelled_lph site_uid character varying(12)
v_modelled_lph site_code character varying(20)
v_modelled_lph site_name character varying(120)
v_modelled_lph site_label character varying(170)
v_modelled_lph operator_uid character varying(12)
v_modelled_lph assumption_code character varying(60)
v_modelled_lph status character varying(12)
v_modelled_lph basis character varying(20)
@@ -223,6 +236,7 @@ v_reconciliation site_uid character varying(12)
v_reconciliation site_code character varying(20)
v_reconciliation site_name character varying(120)
v_reconciliation site_label character varying(170)
v_reconciliation operator_uid character varying(12)
v_reconciliation interval_label character varying(30)
v_reconciliation start_on date
v_reconciliation end_on date
@@ -242,6 +256,7 @@ v_runtime site_uid character varying(12)
v_runtime site_code character varying(20)
v_runtime site_name character varying(120)
v_runtime site_label character varying(170)
v_runtime operator_uid character varying(12)
v_runtime meter_label character varying(40)
v_runtime mapped_set character varying(60)
v_runtime interval_label character varying(80)
@@ -259,6 +274,7 @@ v_site site_uid character varying(12)
v_site site_code character varying(20)
v_site site_name character varying(120)
v_site site_label character varying(170)
v_site operator_uid character varying(12)
v_site zone character varying(40)
v_site province character varying(40)
v_site district character varying(60)
@@ -283,9 +299,10 @@ v_spec_reference site_uid character varying(12)
v_spec_reference site_code character varying(20)
v_spec_reference site_name character varying(120)
v_spec_reference site_label character varying(170)
v_spec_reference operator_uid character varying(12)
v_spec_reference slot character varying(20)
v_spec_reference asset_uid character varying(40)
v_spec_reference asset_kind character varying(10)
v_spec_reference asset_kind character varying(20)
v_spec_reference make_model character varying(120)
v_spec_reference attribute character varying(40)
v_spec_reference load_pct numeric(5,1)
@@ -301,6 +318,7 @@ v_stock_reading site_uid character varying(12)
v_stock_reading site_code character varying(20)
v_stock_reading site_name character varying(120)
v_stock_reading site_label character varying(170)
v_stock_reading operator_uid character varying(12)
v_stock_reading observed_on date
v_stock_reading seq_in_day integer
v_stock_reading reading_type character varying(20)
Can't render this file because it contains an unexpected character in line 3 and column 80.
@@ -17,7 +17,7 @@ live view and any drift is reported (exit status 2).
The database is never created or changed: the bundle's database uuid must
already exist under the name "Fuel", otherwise nothing is imported.
The dashboards (atoma-overview, atoma-site) get no role: admins see them in Superset, and
The dashboards (overview, site) get no role: admins see them in Superset, and
the Petrolyst app shows them with guest passes (PD-12). With --embedded-out FILE
--embed-domain URL, each of them is made embeddable from that origin (its embedding id stays
the same across imports), and FILE receives {slug: embedding id} for the Petrolyst server:
@@ -36,7 +36,7 @@ import zipfile
import yaml
DATABASE_NAME = "Fuel"
EMBEDDED_PREFIX = "atoma-"
EMBEDDED_SLUGS = {"overview", "site"}
STEPS = [("datasets", "SqlaTable"), ("charts", "Slice"), ("dashboards", "Dashboard")]
@@ -104,12 +104,17 @@ def main() -> int:
if path.startswith("dashboards/"):
dashboard = db.session.query(Dashboard).filter_by(
uuid=yaml.safe_load(text)["uuid"]).one()
if dashboard.slug.startswith(EMBEDDED_PREFIX) and args.embedded_out:
if dashboard.slug in EMBEDDED_SLUGS and args.embedded_out:
from superset.daos.dashboard import EmbeddedDashboardDAO
config = EmbeddedDashboardDAO.upsert(dashboard, [args.embed_domain])
db.session.flush()
embedded[dashboard.slug] = str(config.uuid)
# An empty or short list would stop the app's dashboards from opening.
if args.embedded_out and set(embedded) != EMBEDDED_SLUGS:
db.session.rollback()
print(f"embedded: expected {sorted(EMBEDDED_SLUGS)}, got {sorted(embedded)}", file=sys.stderr)
return 3
db.session.commit()
if args.embedded_out:
with open(args.embedded_out, "w") as out:
+11 -1
View File
@@ -1,7 +1,7 @@
"""Keycloak sign-in for Superset at atoma.petrolyst.com.
Set as CUSTOM_SECURITY_MANAGER in superset_config.py. Checked against
Flask-AppBuilder 5.0.2 as bundled in apache/superset:6.1.0. Three changes to
Flask-AppBuilder 5.0.2 as bundled in apache/superset:6.1.0. Four changes to
the stock behaviour:
- User info and roles come from the ID token that Keycloak issues for client
@@ -11,6 +11,8 @@ the stock behaviour:
login page on /login/, which only shows a "Sign in with keycloak" button.
- /logout/ also ends the Keycloak session, so the next visit asks for the
password and OTP again.
- A Keycloak user with no mapped Superset role is refused, and no account is
made. Customers' users belong in the Petrolyst app, never here (PD-21).
"""
import logging
@@ -70,6 +72,14 @@ class PetrolystSecurityManager(SupersetSecurityManager):
# FAB's OAuth view, and that page wins the route.
register_superset_auth_view = False
def auth_user_oauth(self, userinfo: dict[str, Any]) -> Any:
# Only the mapped roles count: FAB's own calculation always adds the
# registration role, so it is never empty.
if not self.get_roles_from_keys(userinfo.get("role_keys", [])):
logger.warning("Keycloak sign-in refused: no mapped Superset role")
return None
return super().auth_user_oauth(userinfo)
def oauth_user_info(
self, provider: str, response: Optional[dict[str, Any]] = None
) -> dict[str, Any]:
@@ -1,14 +1,15 @@
"""The role "ATOMA Guest": what the Petrolyst app's embedded dashboards may do (PD-12).
"""The role "Petrolyst Guest": what the Petrolyst app's embedded dashboards may do (PD-12).
Superset gives every embedded viewer (a guest pass signed for one dashboard) the role named by
GUEST_ROLE_NAME. Which dashboard a guest sees comes from the pass, not from the role. Superset
GUEST_ROLE_NAME. Which dashboard and which customer's rows a guest sees come from the pass, not from
the role. Superset
leaves the SQL out of chart-data responses for guests.
Embedding is switched on in the local stack only (.local/fmdb-scratch, superset_config_local.py),
so this script is not part of init.sh; run it by hand inside the Superset container, once, and
again whenever PERMISSIONS change:
docker exec <superset> /app/.venv/bin/python /app/petrolyst-roles/atoma_guest.py
docker exec <superset> /app/.venv/bin/python /app/petrolyst-roles/petrolyst_guest.py
Creates the role if needed and gives it exactly PERMISSIONS: missing ones are added, any other
is removed (also one added by hand in the UI). A second run changes nothing.
@@ -22,7 +23,7 @@ from __future__ import annotations
import sys
ROLE = "ATOMA Guest"
ROLE = "Petrolyst Guest"
PERMISSIONS = [
("can_read", "Dashboard"), # the dashboard and its charts' definitions
("can_dashboard", "Superset"), # the dashboard page
+5 -5
View File
@@ -46,7 +46,7 @@ SQLALCHEMY_DATABASE_URI = URL.create(
).render_as_string(hide_password=False)
# --- The data: fuel-db ----------------------------------------------------------
# Superset reads fuel-db directly as the read-only role fuel_ro (no Trino since 29 Oct 2026);
# Superset reads fuel-db directly as the read-only role fuel_ro;
# superset-init registers that connection, "Fuel", without a password. Passwords of data
# connections come from secret files at connect time, so none is kept in Superset's own
# database. A connection added in the UI gets no password this way.
@@ -64,7 +64,7 @@ SQLALCHEMY_CUSTOM_PASSWORD_STORE = _database_password
# Users and roles are managed in Keycloak; there is no local admin account.
# Break-glass access: `superset fab ...` inside the container on the server.
KEYCLOAK_REALM_URL = os.environ.get(
"KEYCLOAK_REALM_URL", "https://iam.thedigitalbridge.io/realms/thedigitalbridge"
"KEYCLOAK_REALM_URL", "https://iam.thedigitalbridge.io/realms/petrolyst"
).rstrip("/")
AUTH_TYPE = AUTH_OAUTH
@@ -94,12 +94,12 @@ AUTH_ROLES_MAPPING = {
"superset-admin": ["Admin"],
"superset-editor": ["Alpha"],
"superset-viewer": ["Gamma"],
# "atoma-reader" opens the Petrolyst app only; here it maps to nothing (Public).
# Customers' users have a role of client "petrolyst-app", not of this one: refused here.
}
AUTH_ROLES_SYNC_AT_LOGIN = True
AUTH_USER_REGISTRATION = True
# Public has no permissions: a Keycloak user without a Superset role signs in
# and sees nothing.
# Public has no permissions. A Keycloak user without a mapped role never gets
# this far: PetrolystSecurityManager refuses the sign-in and makes no account.
AUTH_USER_REGISTRATION_ROLE = "Public"
# Viewers (Gamma) see a dashboard once it lists their role.
+3 -4
View File
@@ -1,12 +1,11 @@
# Fuel-monitoring analytics: Superset on atoma.petrolyst.com, reading the
# operational store (fuel-db) directly as the read-only role fuel_ro (Trino was
# removed on 29 Oct 2026). Run only through deploy/deploy.sh or
# operational store (fuel-db) directly as the read-only role fuel_ro. Run only through deploy/deploy.sh or
# deploy/rollback.sh: they export SUPERSET_IMAGE_TAG from the release's
# ARTIFACTS file, and the :? guard below stops Compose from running without it.
name: petrolyst
x-superset-env: &superset-env
KEYCLOAK_REALM_URL: ${KEYCLOAK_REALM_URL:-https://iam.thedigitalbridge.io/realms/thedigitalbridge}
KEYCLOAK_REALM_URL: ${KEYCLOAK_REALM_URL:-https://iam.thedigitalbridge.io/realms/petrolyst}
x-superset: &superset
image: fuel-superset:${SUPERSET_IMAGE_TAG:?SUPERSET_IMAGE_TAG unset - run through deploy/deploy.sh}
@@ -159,4 +158,4 @@ secrets:
fuel_db_ro_password:
file: ${PETROLYST_ROOT:-/opt/petrolyst}/secrets/fuel-monitoring/fuel_db_ro_password
keycloak_petrolyst_client_secret:
file: ${PETROLYST_ROOT:-/opt/petrolyst}/secrets/shared/keycloak_petrolyst_client_secret
file: ${PETROLYST_ROOT:-/opt/petrolyst}/secrets/shared/keycloak_petrolyst_realm_client_secret
@@ -2,7 +2,9 @@
-- DRAFT, not applied. Applied by hand by a loader acting as fuel_owner (README in this folder);
-- deploy.sh never touches fuel-db. Design and rationale: fuel-monitoring
-- .context/plans/2026-10-24-fuel-db-baseline-dashboard.md (sections cited as "design §n").
-- Order: 0001, 0002, 0003, each in its own transaction.
-- Order: 0001 to 0004, each in its own transaction.
-- Every table but fm.asset_kind carries operator_uid, the customer (PD-16). A child names its
-- parent and the customer together, so it cannot point at another customer's parent.
BEGIN;
CREATE SCHEMA fm AUTHORIZATION fuel_owner;
CREATE EXTENSION IF NOT EXISTS btree_gist WITH SCHEMA fm; -- trusted extension; for EXCLUDE on text
@@ -10,9 +12,21 @@ CREATE EXTENSION IF NOT EXISTS btree_gist WITH SCHEMA fm; -- trusted extension
-- anyway, and the local test applies this file in a session whose search_path is the loader's default.
SET LOCAL search_path = fm, pg_catalog, public;
-- Customers -------------------------------------------------------------------------------
CREATE TABLE fm.operator (
operator_uid text PRIMARY KEY CHECK (operator_uid ~ '^OPR-[0-9]{4}$'),
name text NOT NULL,
label_issuer text NOT NULL, -- whose labels this customer's people see (holder_alias.issuer)
programme text NOT NULL, -- the programme v_site reports (participation.programme)
timezone text NOT NULL, -- for a holder with no place, or a place without its own
keycloak_group text NOT NULL UNIQUE -- the top-level Keycloak group of this customer's people (PD-21)
CHECK (keycloak_group ~ '^/[^/]+$')
);
-- Provenance ------------------------------------------------------------------------------
CREATE TABLE fm.source_document (
sha256 text PRIMARY KEY CHECK (sha256 ~ '^[0-9a-f]{64}$'),
operator_uid text NOT NULL REFERENCES fm.operator,
source_family text NOT NULL, -- stable across versions: '<operator>-<workbook>', 'site-yaml:SITE-0001'
legacy_key16 text UNIQUE CHECK (legacy_key16 ~ '^[0-9a-f]{16}$'), -- catalog.yaml key
kind text NOT NULL CHECK (kind IN ('delivery_form','site_logbook','operator_workbook',
@@ -23,12 +37,15 @@ CREATE TABLE fm.source_document (
corpus_path text, -- NULL when held outside the repository
transcription_path text, -- the corpus note that transcribes it
contains_personal_data boolean NOT NULL,
supersedes_sha256 text UNIQUE REFERENCES fm.source_document,
supersedes_sha256 text UNIQUE,
null_reasons jsonb NOT NULL DEFAULT '{}',
CHECK (corpus_path IS NOT NULL OR transcription_path IS NOT NULL)
CHECK (corpus_path IS NOT NULL OR transcription_path IS NOT NULL),
UNIQUE (operator_uid, sha256),
FOREIGN KEY (operator_uid, supersedes_sha256) REFERENCES fm.source_document (operator_uid, sha256)
);
CREATE TABLE fm.import_batch (
import_batch_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
operator_uid text NOT NULL REFERENCES fm.operator,
source_kind text NOT NULL CHECK (source_kind IN ('site_yaml','operator_workbook','catalog','review_manifest','baseline_run')),
source_path text NOT NULL,
source_sha256 text NOT NULL CHECK (source_sha256 ~ '^[0-9a-f]{64}$'),
@@ -37,33 +54,86 @@ CREATE TABLE fm.import_batch (
started_at timestamptz NOT NULL DEFAULT now(),
finished_at timestamptz,
status text NOT NULL DEFAULT 'running' CHECK (status IN ('running','loaded','failed')),
report jsonb NOT NULL DEFAULT '{}' -- counts, unmapped paths, conflicts
report jsonb NOT NULL DEFAULT '{}', -- counts, unmapped paths, conflicts
UNIQUE (operator_uid, import_batch_id)
);
CREATE TABLE fm.import_record (
import_record_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
operator_uid text NOT NULL REFERENCES fm.operator,
logical_key text NOT NULL, -- = source_key: '<family>:delivery:SITE-0001:2026-01-31:1'
source_locator text NOT NULL, -- YAML path or sheet!range as read
payload jsonb NOT NULL, -- original values, strings as written, 'UNKNOWN' kept
payload_sha256 text NOT NULL,
first_batch_id bigint NOT NULL REFERENCES fm.import_batch,
last_seen_batch_id bigint NOT NULL REFERENCES fm.import_batch,
UNIQUE (logical_key, payload_sha256)
first_batch_id bigint NOT NULL,
last_seen_batch_id bigint NOT NULL,
UNIQUE (operator_uid, logical_key, payload_sha256),
UNIQUE (operator_uid, import_record_id),
FOREIGN KEY (operator_uid, first_batch_id) REFERENCES fm.import_batch (operator_uid, import_batch_id),
FOREIGN KEY (operator_uid, last_seen_batch_id) REFERENCES fm.import_batch (operator_uid, import_batch_id)
);
-- Identity --------------------------------------------------------------------------------
CREATE TABLE fm.site (
site_uid text PRIMARY KEY CHECK (site_uid ~ '^SITE-[0-9]{4}$'),
name text, name_on_plate text,
zone text NOT NULL, province text NOT NULL, district text,
timezone text NOT NULL DEFAULT 'Asia/Kabul',
import_record_id bigint NOT NULL REFERENCES fm.import_record, -- full site.yaml payload
null_reasons jsonb NOT NULL DEFAULT '{}'
-- Identity: asset kinds and assets, places and holders (PD-18) --------------------------------
CREATE TABLE fm.asset_kind ( -- what an asset does with fuel (PD-18); global, not per customer
kind text PRIMARY KEY,
fuel_role text CHECK (fuel_role IN ('container','consumer','transfer','sensor'))
-- NULL: carries other assets and holds no fuel itself
);
CREATE TABLE fm.site_location ( -- coordinates are claims; the map uses only an accepted one
INSERT INTO fm.asset_kind (kind, fuel_role) VALUES
('generator','consumer'), ('engine','consumer'), ('tank','container'), ('compartment','container'),
('vehicle_tank','container'), ('pump','transfer'), ('vehicle',NULL);
CREATE TABLE fm.asset (
asset_uid text PRIMARY KEY CHECK (asset_uid ~ '^AST-[A-Z0-9-]+$'),
operator_uid text NOT NULL REFERENCES fm.operator,
kind text NOT NULL REFERENCES fm.asset_kind,
make text, model text, serial text, year_of_manufacture smallint,
import_record_id bigint NOT NULL,
null_reasons jsonb NOT NULL DEFAULT '{}',
UNIQUE (operator_uid, asset_uid),
FOREIGN KEY (operator_uid, import_record_id) REFERENCES fm.import_record (operator_uid, import_record_id)
);
CREATE TABLE fm.place ( -- fixed geography; its coordinates are claims (place_location)
place_uid text PRIMARY KEY CHECK (place_uid ~ '^PLC-[0-9]{4,8}$'),
operator_uid text NOT NULL REFERENCES fm.operator,
role text NOT NULL CHECK (role IN ('tower_site','depot','yard','fuel_station')),
zone text NOT NULL, province text NOT NULL, district text,
timezone text, -- NULL: the operator's
import_record_id bigint NOT NULL,
null_reasons jsonb NOT NULL DEFAULT '{}',
UNIQUE (operator_uid, place_uid),
FOREIGN KEY (operator_uid, import_record_id) REFERENCES fm.import_record (operator_uid, import_record_id)
);
CREATE TABLE fm.holder ( -- what assets are in or on (PD-18). For a telecom site: one
-- fixed installation at one place. Ids are opaque and permanent
holder_uid text PRIMARY KEY CHECK (holder_uid ~ '^[A-Z]{2,4}-[0-9]{4,7}$'),
operator_uid text NOT NULL REFERENCES fm.operator,
kind text NOT NULL CHECK (kind IN ('fixed_installation','vehicle','vessel','trailer')),
name text, name_on_plate text,
asset_uid text, -- a vehicle is both an asset and a holder
import_record_id bigint NOT NULL,
null_reasons jsonb NOT NULL DEFAULT '{}',
CHECK ((kind = 'vehicle') = (asset_uid IS NOT NULL)),
UNIQUE (operator_uid, holder_uid),
FOREIGN KEY (operator_uid, asset_uid) REFERENCES fm.asset (operator_uid, asset_uid),
FOREIGN KEY (operator_uid, import_record_id) REFERENCES fm.import_record (operator_uid, import_record_id)
);
CREATE TABLE fm.holder_place ( -- where a fixed holder stands; a vehicle's position is telemetry
holder_uid text NOT NULL,
operator_uid text NOT NULL REFERENCES fm.operator,
place_uid text NOT NULL,
valid_from date, valid_to date, -- NULL from = unknown start; NULL to = still there
null_reasons jsonb NOT NULL DEFAULT '{}',
CHECK (valid_from IS NOT NULL OR null_reasons ? 'valid_from'),
EXCLUDE USING gist (holder_uid WITH =,
daterange(coalesce(valid_from,'-infinity'::date), valid_to, '[)') WITH &&),
FOREIGN KEY (operator_uid, holder_uid) REFERENCES fm.holder (operator_uid, holder_uid),
FOREIGN KEY (operator_uid, place_uid) REFERENCES fm.place (operator_uid, place_uid)
);
CREATE TABLE fm.place_location ( -- coordinates are claims; the map uses only an accepted one
location_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
operator_uid text NOT NULL REFERENCES fm.operator,
source_key text NOT NULL, -- 'site-yaml:SITE-0001:coordinates'
assertion_sha256 text NOT NULL CHECK (assertion_sha256 ~ '^[0-9a-f]{64}$'), -- site, lat, lon, precision, basis
site_uid text NOT NULL REFERENCES fm.site,
assertion_sha256 text NOT NULL CHECK (assertion_sha256 ~ '^[0-9a-f]{64}$'), -- operator, place, lat, lon, precision, basis
place_uid text NOT NULL,
latitude numeric(9,6) NOT NULL CHECK (latitude BETWEEN -90 AND 90),
longitude numeric(9,6) NOT NULL CHECK (longitude BETWEEN -180 AND 180),
precision_m integer, -- e.g. 110 where the source rounds to 0.001 deg
@@ -71,91 +141,111 @@ CREATE TABLE fm.site_location ( -- coordinates are claims; the
source_note text NOT NULL, -- 'operator site plate photo <catalog key>'
acceptance text NOT NULL DEFAULT 'pending' CHECK (acceptance IN ('pending','accepted','rejected','superseded','retracted')),
accepted_on date, accepted_by_role text, acceptance_ref text, -- copied from the matching fm.review row
superseded_by bigint REFERENCES fm.site_location,
import_record_id bigint NOT NULL REFERENCES fm.import_record,
superseded_by bigint,
import_record_id bigint NOT NULL,
CHECK (acceptance <> 'accepted' OR (accepted_on IS NOT NULL AND accepted_by_role IS NOT NULL)),
CHECK (acceptance <> 'superseded' OR superseded_by IS NOT NULL)
CHECK (acceptance <> 'superseded' OR superseded_by IS NOT NULL),
UNIQUE (operator_uid, location_id),
FOREIGN KEY (operator_uid, place_uid) REFERENCES fm.place (operator_uid, place_uid),
FOREIGN KEY (operator_uid, superseded_by) REFERENCES fm.place_location (operator_uid, location_id),
FOREIGN KEY (operator_uid, import_record_id) REFERENCES fm.import_record (operator_uid, import_record_id)
);
CREATE UNIQUE INDEX site_location_live_key ON fm.site_location (source_key)
CREATE UNIQUE INDEX place_location_live_key ON fm.place_location (operator_uid, source_key)
WHERE acceptance IN ('pending','accepted','rejected');
CREATE UNIQUE INDEX site_location_one_accepted ON fm.site_location (site_uid) WHERE acceptance = 'accepted';
CREATE TABLE fm.site_alias (
site_alias_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
site_uid text NOT NULL REFERENCES fm.site,
CREATE UNIQUE INDEX place_location_one_accepted ON fm.place_location (place_uid) WHERE acceptance = 'accepted';
CREATE TABLE fm.holder_alias (
holder_alias_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
operator_uid text NOT NULL REFERENCES fm.operator,
holder_uid text NOT NULL,
label text NOT NULL, issuer text NOT NULL,
valid_from date, valid_to date, -- NULL from = unknown start; NULL to = still valid
is_primary boolean NOT NULL DEFAULT false,
null_reasons jsonb NOT NULL DEFAULT '{}',
CHECK (valid_from IS NOT NULL OR null_reasons ? 'valid_from'),
EXCLUDE USING gist (issuer WITH =, label WITH =,
EXCLUDE USING gist (operator_uid WITH =, issuer WITH =, label WITH =,
daterange(coalesce(valid_from,'-infinity'::date), valid_to, '[)') WITH &&),
EXCLUDE USING gist (site_uid WITH =, issuer WITH =,
daterange(coalesce(valid_from,'-infinity'::date), valid_to, '[)') WITH &&) WHERE (is_primary)
EXCLUDE USING gist (holder_uid WITH =, issuer WITH =,
daterange(coalesce(valid_from,'-infinity'::date), valid_to, '[)') WITH &&) WHERE (is_primary),
FOREIGN KEY (operator_uid, holder_uid) REFERENCES fm.holder (operator_uid, holder_uid)
);
CREATE TABLE fm.pilot_participation (
site_uid text NOT NULL REFERENCES fm.site,
programme text NOT NULL DEFAULT 'ATOMA OFMS pilot',
CREATE TABLE fm.participation (
holder_uid text NOT NULL,
operator_uid text NOT NULL REFERENCES fm.operator,
programme text NOT NULL,
status text NOT NULL CHECK (status IN ('active','reference','candidate','withdrawn')),
valid_from date NOT NULL, valid_to date,
decision_ref text, note text,
EXCLUDE USING gist (site_uid WITH =, programme WITH =, daterange(valid_from, valid_to, '[)') WITH &&)
EXCLUDE USING gist (holder_uid WITH =, programme WITH =, daterange(valid_from, valid_to, '[)') WITH &&),
FOREIGN KEY (operator_uid, holder_uid) REFERENCES fm.holder (operator_uid, holder_uid)
);
CREATE TABLE fm.asset (
asset_uid text PRIMARY KEY CHECK (asset_uid ~ '^AST-[A-Z0-9-]+$'),
kind text NOT NULL CHECK (kind IN ('generator','tank')),
make text, model text, serial text, year_of_manufacture smallint,
import_record_id bigint NOT NULL REFERENCES fm.import_record,
null_reasons jsonb NOT NULL DEFAULT '{}'
);
CREATE TABLE fm.asset_placement (
CREATE TABLE fm.asset_placement ( -- in or on a holder, or at a place: exactly one (PD-18)
placement_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
asset_uid text NOT NULL REFERENCES fm.asset,
site_uid text NOT NULL REFERENCES fm.site,
operator_uid text NOT NULL REFERENCES fm.operator,
asset_uid text NOT NULL,
holder_uid text,
place_uid text,
slot_label text NOT NULL, -- 'GEN-01': a position label, not an identity
occupied_from date, occupied_to date, -- NULL from = before any record (conservative)
earliest_evidence_on date,
installed_on date,
null_reasons jsonb NOT NULL DEFAULT '{}',
CHECK (num_nonnulls(holder_uid, place_uid) = 1),
CHECK (installed_on IS NOT NULL OR null_reasons ? 'installed_on'),
EXCLUDE USING gist (asset_uid WITH =,
daterange(coalesce(occupied_from,'-infinity'::date), occupied_to, '[)') WITH &&),
EXCLUDE USING gist (site_uid WITH =, slot_label WITH =,
daterange(coalesce(occupied_from,'-infinity'::date), occupied_to, '[)') WITH &&)
EXCLUDE USING gist (holder_uid WITH =, slot_label WITH =,
daterange(coalesce(occupied_from,'-infinity'::date), occupied_to, '[)') WITH &&) WHERE (holder_uid IS NOT NULL),
EXCLUDE USING gist (place_uid WITH =, slot_label WITH =,
daterange(coalesce(occupied_from,'-infinity'::date), occupied_to, '[)') WITH &&) WHERE (place_uid IS NOT NULL),
FOREIGN KEY (operator_uid, asset_uid) REFERENCES fm.asset (operator_uid, asset_uid),
FOREIGN KEY (operator_uid, holder_uid) REFERENCES fm.holder (operator_uid, holder_uid),
FOREIGN KEY (operator_uid, place_uid) REFERENCES fm.place (operator_uid, place_uid)
);
CREATE TABLE fm.runtime_meter (
meter_id text PRIMARY KEY, -- ours, stable: 'MTR-SITE-0001-GENSET1'
site_uid text NOT NULL REFERENCES fm.site,
operator_uid text NOT NULL REFERENCES fm.operator,
holder_uid text NOT NULL,
source_label text NOT NULL, -- as written: 'Genset 1', 'R.H'
asset_uid text REFERENCES fm.asset,
asset_uid text,
mapping_status text NOT NULL CHECK (mapping_status IN ('confirmed','proposed','unmapped')),
mapping_basis text,
UNIQUE (site_uid, source_label),
CHECK ((mapping_status = 'unmapped') = (asset_uid IS NULL))
UNIQUE (holder_uid, source_label),
CHECK ((mapping_status = 'unmapped') = (asset_uid IS NULL)),
UNIQUE (operator_uid, meter_id),
FOREIGN KEY (operator_uid, holder_uid) REFERENCES fm.holder (operator_uid, holder_uid),
FOREIGN KEY (operator_uid, asset_uid) REFERENCES fm.asset (operator_uid, asset_uid)
);
CREATE TABLE fm.meter_epoch (
epoch_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
meter_id text NOT NULL REFERENCES fm.runtime_meter,
operator_uid text NOT NULL REFERENCES fm.operator,
meter_id text NOT NULL,
epoch_no smallint NOT NULL,
starts_on date, -- NULL: began before the first reading held
start_reason text NOT NULL CHECK (start_reason IN ('first_seen','display_replaced','controller_replaced','reset','set_replaced')),
continuity text NOT NULL CHECK (continuity IN ('confirmed','assumed','uncertain')),
rollover_at numeric(9,1),
source_sha256 text REFERENCES fm.source_document, note text,
UNIQUE (meter_id, epoch_no)
source_sha256 text, note text,
UNIQUE (meter_id, epoch_no),
UNIQUE (operator_uid, epoch_id),
FOREIGN KEY (operator_uid, meter_id) REFERENCES fm.runtime_meter (operator_uid, meter_id),
FOREIGN KEY (operator_uid, source_sha256) REFERENCES fm.source_document (operator_uid, sha256)
);
-- Operations (claims) ---------------------------------------------------------------------
CREATE TABLE fm.visit (
visit_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
operator_uid text NOT NULL REFERENCES fm.operator,
visit_key text NOT NULL UNIQUE, -- 'SITE-0001:2026-01-31:1'
site_uid text NOT NULL REFERENCES fm.site,
visit_on date NOT NULL, -- site-local calendar date
holder_uid text NOT NULL,
visit_on date NOT NULL, -- holder-local calendar date
seq_in_day smallint NOT NULL DEFAULT 1,
time_in_local time, time_out_local time, -- as written; NULL when the source has a date only
kind text NOT NULL CHECK (kind IN ('delivery','stock_check','maintenance','survey')),
-- descriptive only: views use fm.v_visit_live and the live claims
UNIQUE (site_uid, visit_on, seq_in_day),
UNIQUE (visit_id, site_uid) -- target of site-checked composite foreign keys
UNIQUE (holder_uid, visit_on, seq_in_day),
UNIQUE (operator_uid, visit_id, holder_uid), -- target of holder-checked composite foreign keys
UNIQUE (operator_uid, visit_id),
FOREIGN KEY (operator_uid, holder_uid) REFERENCES fm.holder (operator_uid, holder_uid)
);
-- A visit is never deleted. Once all its claims are superseded or retracted it stays for provenance
-- only, and drops out of every analytical view (fm.v_visit_live).
@@ -164,42 +254,52 @@ CREATE TABLE fm.visit (
-- superseded_by, review_note, null_reasons
CREATE TABLE fm.fuel_movement (
movement_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
operator_uid text NOT NULL REFERENCES fm.operator,
source_key text NOT NULL, -- '<family>:delivery:SITE-0001:2026-01-31:1'
assertion_sha256 text NOT NULL CHECK (assertion_sha256 ~ '^[0-9a-f]{64}$'),
event_key text NOT NULL, -- the physical movement: 'SITE-0001:2026-01-31:1:delivery:1';
-- claims sharing it are alternative reports of ONE movement
visit_id bigint NOT NULL REFERENCES fm.visit,
site_uid text NOT NULL REFERENCES fm.site,
visit_id bigint NOT NULL,
holder_uid text NOT NULL,
kind text NOT NULL CHECK (kind IN ('delivery','external_in','internal_transfer',
'withdrawal','drain','spill','loss_reported')),
from_tank_uid text REFERENCES fm.asset, to_tank_uid text REFERENCES fm.asset,
from_tank_uid text, to_tank_uid text,
source_tank_label text,
litres numeric(8,1) CHECK (litres >= 0),
litres_state text NOT NULL CHECK (litres_state IN ('stated','converted_from_level','converted_from_mass','not_recorded','illegible')),
density_kg_per_l numeric(5,3),
unit_price numeric(8,2), price_currency text,
source_sha256 text NOT NULL REFERENCES fm.source_document,
source_sha256 text NOT NULL,
record_ref text NOT NULL, -- 'form <no>' | '<sheet>!J20' | 'logbook p2 r1'
import_record_id bigint NOT NULL REFERENCES fm.import_record,
import_record_id bigint NOT NULL,
claim_status text NOT NULL DEFAULT 'asserted' CHECK (claim_status IN ('asserted','reviewed','disputed','superseded','retracted')),
superseded_by bigint REFERENCES fm.fuel_movement,
superseded_by bigint,
review_note text,
null_reasons jsonb NOT NULL DEFAULT '{}',
CHECK ((litres IS NULL) = (litres_state IN ('not_recorded','illegible'))),
CHECK (claim_status <> 'superseded' OR superseded_by IS NOT NULL),
CHECK (kind <> 'internal_transfer' OR (from_tank_uid IS NOT NULL AND to_tank_uid IS NOT NULL))
CHECK (kind <> 'internal_transfer' OR (from_tank_uid IS NOT NULL AND to_tank_uid IS NOT NULL)),
UNIQUE (operator_uid, movement_id),
FOREIGN KEY (operator_uid, visit_id) REFERENCES fm.visit (operator_uid, visit_id),
FOREIGN KEY (operator_uid, holder_uid) REFERENCES fm.holder (operator_uid, holder_uid),
FOREIGN KEY (operator_uid, from_tank_uid) REFERENCES fm.asset (operator_uid, asset_uid),
FOREIGN KEY (operator_uid, to_tank_uid) REFERENCES fm.asset (operator_uid, asset_uid),
FOREIGN KEY (operator_uid, source_sha256) REFERENCES fm.source_document (operator_uid, sha256),
FOREIGN KEY (operator_uid, import_record_id) REFERENCES fm.import_record (operator_uid, import_record_id),
FOREIGN KEY (operator_uid, superseded_by) REFERENCES fm.fuel_movement (operator_uid, movement_id)
);
CREATE UNIQUE INDEX fuel_movement_live_key ON fm.fuel_movement (source_key)
CREATE UNIQUE INDEX fuel_movement_live_key ON fm.fuel_movement (operator_uid, source_key)
WHERE claim_status IN ('asserted','reviewed','disputed');
CREATE INDEX ON fm.fuel_movement (visit_id, event_key);
CREATE TABLE fm.stock_observation (
stock_obs_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
operator_uid text NOT NULL REFERENCES fm.operator,
source_key text NOT NULL,
assertion_sha256 text NOT NULL CHECK (assertion_sha256 ~ '^[0-9a-f]{64}$'),
visit_id bigint NOT NULL REFERENCES fm.visit,
site_uid text NOT NULL REFERENCES fm.site,
visit_id bigint NOT NULL,
holder_uid text NOT NULL,
scope text NOT NULL CHECK (scope IN ('site_total','tank')),
tank_uid text REFERENCES fm.asset, -- only when the source identifies the tank
tank_uid text, -- only when the source identifies the tank
source_tank_label text, -- 'T1' as written, even when unmapped
phase text NOT NULL CHECK (phase IN ('before_delivery','after_delivery','standalone')),
level_cm numeric(6,1),
@@ -208,27 +308,35 @@ CREATE TABLE fm.stock_observation (
'stated_total','level_only','not_recorded','illegible','placeholder_zero')),
resolution_l numeric(6,1), -- ±1 cm of dip at this level, from tank geometry
resolution_basis text,
source_sha256 text NOT NULL REFERENCES fm.source_document,
source_sha256 text NOT NULL,
record_ref text NOT NULL,
import_record_id bigint NOT NULL REFERENCES fm.import_record,
import_record_id bigint NOT NULL,
claim_status text NOT NULL DEFAULT 'asserted' CHECK (claim_status IN ('asserted','reviewed','disputed','superseded','retracted')),
superseded_by bigint REFERENCES fm.stock_observation,
superseded_by bigint,
review_note text,
null_reasons jsonb NOT NULL DEFAULT '{}',
CHECK (scope = 'site_total' OR tank_uid IS NOT NULL OR source_tank_label IS NOT NULL),
CHECK ((litres IS NULL) = (litres_basis IN ('level_only','not_recorded','illegible','placeholder_zero'))),
CHECK (claim_status <> 'superseded' OR superseded_by IS NOT NULL)
CHECK (claim_status <> 'superseded' OR superseded_by IS NOT NULL),
UNIQUE (operator_uid, stock_obs_id),
FOREIGN KEY (operator_uid, visit_id) REFERENCES fm.visit (operator_uid, visit_id),
FOREIGN KEY (operator_uid, holder_uid) REFERENCES fm.holder (operator_uid, holder_uid),
FOREIGN KEY (operator_uid, tank_uid) REFERENCES fm.asset (operator_uid, asset_uid),
FOREIGN KEY (operator_uid, source_sha256) REFERENCES fm.source_document (operator_uid, sha256),
FOREIGN KEY (operator_uid, import_record_id) REFERENCES fm.import_record (operator_uid, import_record_id),
FOREIGN KEY (operator_uid, superseded_by) REFERENCES fm.stock_observation (operator_uid, stock_obs_id)
);
CREATE UNIQUE INDEX stock_obs_live_key ON fm.stock_observation (source_key)
CREATE UNIQUE INDEX stock_obs_live_key ON fm.stock_observation (operator_uid, source_key)
WHERE claim_status IN ('asserted','reviewed','disputed');
CREATE INDEX ON fm.stock_observation (visit_id, phase);
CREATE TABLE fm.hour_meter_reading (
reading_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
operator_uid text NOT NULL REFERENCES fm.operator,
source_key text NOT NULL,
assertion_sha256 text NOT NULL CHECK (assertion_sha256 ~ '^[0-9a-f]{64}$'),
visit_id bigint NOT NULL REFERENCES fm.visit,
meter_id text NOT NULL REFERENCES fm.runtime_meter,
epoch_id bigint REFERENCES fm.meter_epoch, -- NULL = not yet judged; excluded from runtime
visit_id bigint NOT NULL,
meter_id text NOT NULL,
epoch_id bigint, -- NULL = not yet judged; excluded from runtime
source_field text NOT NULL CHECK (source_field IN ('current','previous','logbook_rh','display_photo')),
date_basis text NOT NULL CHECK (date_basis IN ('stated','inferred_previous_visit','capture_date_unknown')),
hours numeric(9,1),
@@ -236,84 +344,109 @@ CREATE TABLE fm.hour_meter_reading (
reading_state text NOT NULL CHECK (reading_state IN ('clear','best_reading','ambiguous','illegible','crossed_out','not_recorded')),
alternatives text, -- competing readings of one illegible value, '|'-separated
transcribed_by text NOT NULL CHECK (transcribed_by IN ('source_operator','our_reading')),
source_sha256 text NOT NULL REFERENCES fm.source_document,
source_sha256 text NOT NULL,
record_ref text NOT NULL,
import_record_id bigint NOT NULL REFERENCES fm.import_record,
import_record_id bigint NOT NULL,
claim_status text NOT NULL DEFAULT 'asserted' CHECK (claim_status IN ('asserted','reviewed','disputed','superseded','retracted')),
superseded_by bigint REFERENCES fm.hour_meter_reading,
superseded_by bigint,
review_note text,
null_reasons jsonb NOT NULL DEFAULT '{}',
CHECK ((hours IS NULL) = (reading_state IN ('ambiguous','illegible','crossed_out','not_recorded'))),
CHECK (claim_status <> 'superseded' OR superseded_by IS NOT NULL)
CHECK (claim_status <> 'superseded' OR superseded_by IS NOT NULL),
UNIQUE (operator_uid, reading_id),
FOREIGN KEY (operator_uid, visit_id) REFERENCES fm.visit (operator_uid, visit_id),
FOREIGN KEY (operator_uid, meter_id) REFERENCES fm.runtime_meter (operator_uid, meter_id),
FOREIGN KEY (operator_uid, epoch_id) REFERENCES fm.meter_epoch (operator_uid, epoch_id),
FOREIGN KEY (operator_uid, source_sha256) REFERENCES fm.source_document (operator_uid, sha256),
FOREIGN KEY (operator_uid, import_record_id) REFERENCES fm.import_record (operator_uid, import_record_id),
FOREIGN KEY (operator_uid, superseded_by) REFERENCES fm.hour_meter_reading (operator_uid, reading_id)
);
CREATE UNIQUE INDEX hour_reading_live_key ON fm.hour_meter_reading (source_key)
CREATE UNIQUE INDEX hour_reading_live_key ON fm.hour_meter_reading (operator_uid, source_key)
WHERE claim_status IN ('asserted','reviewed','disputed');
CREATE INDEX ON fm.hour_meter_reading (visit_id, meter_id);
CREATE TABLE fm.runtime_interval ( -- future: controller / ATS logs, telemetry
runtime_interval_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
operator_uid text NOT NULL REFERENCES fm.operator,
source_key text NOT NULL, -- live-unique, as for every claim
assertion_sha256 text NOT NULL CHECK (assertion_sha256 ~ '^[0-9a-f]{64}$'),
site_uid text NOT NULL REFERENCES fm.site,
asset_uid text REFERENCES fm.asset,
holder_uid text NOT NULL,
asset_uid text,
scope text NOT NULL CHECK (scope IN ('generator','site_any_generator')),
started_at timestamptz NOT NULL, ended_at timestamptz NOT NULL CHECK (ended_at > started_at),
source_kind text NOT NULL CHECK (source_kind IN ('controller_event_log','ats_log','gateway_telemetry','manual_log')),
source_sha256 text REFERENCES fm.source_document,
source_sha256 text,
claim_status text NOT NULL DEFAULT 'asserted' CHECK (claim_status IN ('asserted','reviewed','disputed','superseded','retracted')),
superseded_by bigint REFERENCES fm.runtime_interval,
superseded_by bigint,
CHECK ((scope = 'generator') = (asset_uid IS NOT NULL)),
CHECK (claim_status <> 'superseded' OR superseded_by IS NOT NULL),
EXCLUDE USING gist (site_uid WITH =, scope WITH =, (coalesce(asset_uid,'')) WITH =, source_kind WITH =,
tstzrange(started_at, ended_at) WITH &&) WHERE (claim_status IN ('asserted','reviewed'))
EXCLUDE USING gist (holder_uid WITH =, scope WITH =, (coalesce(asset_uid,'')) WITH =, source_kind WITH =,
tstzrange(started_at, ended_at) WITH &&) WHERE (claim_status IN ('asserted','reviewed')),
UNIQUE (operator_uid, runtime_interval_id),
FOREIGN KEY (operator_uid, holder_uid) REFERENCES fm.holder (operator_uid, holder_uid),
FOREIGN KEY (operator_uid, asset_uid) REFERENCES fm.asset (operator_uid, asset_uid),
FOREIGN KEY (operator_uid, source_sha256) REFERENCES fm.source_document (operator_uid, sha256),
FOREIGN KEY (operator_uid, superseded_by) REFERENCES fm.runtime_interval (operator_uid, runtime_interval_id)
);
CREATE UNIQUE INDEX runtime_interval_live_key ON fm.runtime_interval (source_key)
CREATE UNIQUE INDEX runtime_interval_live_key ON fm.runtime_interval (operator_uid, source_key)
WHERE claim_status IN ('asserted','reviewed','disputed');
CREATE TABLE fm.engine_fuel_total ( -- future: flow-meter net fuel per period
total_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
operator_uid text NOT NULL REFERENCES fm.operator,
source_key text NOT NULL, -- live-unique, as for every claim
assertion_sha256 text NOT NULL CHECK (assertion_sha256 ~ '^[0-9a-f]{64}$'),
asset_uid text NOT NULL REFERENCES fm.asset,
asset_uid text NOT NULL,
started_at timestamptz NOT NULL, ended_at timestamptz NOT NULL CHECK (ended_at > started_at),
net_fuel_l numeric(10,2) NOT NULL, -- feed − return as the device reports
engine_hours numeric(8,2), -- same device, if reported
source_kind text NOT NULL CHECK (source_kind IN ('differential_flow_meter','single_flow_meter','controller_fuel_counter')),
source_sha256 text REFERENCES fm.source_document,
source_sha256 text,
claim_status text NOT NULL DEFAULT 'asserted' CHECK (claim_status IN ('asserted','reviewed','disputed','superseded','retracted')),
superseded_by bigint REFERENCES fm.engine_fuel_total,
superseded_by bigint,
CHECK (claim_status <> 'superseded' OR superseded_by IS NOT NULL),
EXCLUDE USING gist (asset_uid WITH =, tstzrange(started_at, ended_at) WITH &&)
WHERE (claim_status IN ('asserted','reviewed'))
WHERE (claim_status IN ('asserted','reviewed')),
UNIQUE (operator_uid, total_id),
FOREIGN KEY (operator_uid, asset_uid) REFERENCES fm.asset (operator_uid, asset_uid),
FOREIGN KEY (operator_uid, source_sha256) REFERENCES fm.source_document (operator_uid, sha256),
FOREIGN KEY (operator_uid, superseded_by) REFERENCES fm.engine_fuel_total (operator_uid, total_id)
);
CREATE UNIQUE INDEX engine_fuel_total_live_key ON fm.engine_fuel_total (source_key)
CREATE UNIQUE INDEX engine_fuel_total_live_key ON fm.engine_fuel_total (operator_uid, source_key)
WHERE claim_status IN ('asserted','reviewed','disputed');
CREATE TABLE fm.electricity_bill (
bill_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
operator_uid text NOT NULL REFERENCES fm.operator,
source_key text NOT NULL,
assertion_sha256 text NOT NULL CHECK (assertion_sha256 ~ '^[0-9a-f]{64}$'),
site_uid text NOT NULL REFERENCES fm.site,
holder_uid text NOT NULL,
bill_month date NOT NULL CHECK (extract(day FROM bill_month) = 1), -- month as the source assigns it
period_start date, period_end date,
amount numeric(12,2),
currency text, currency_basis text CHECK (currency_basis IN ('stated','inferred')),
kwh numeric(10,1),
source_sha256 text NOT NULL REFERENCES fm.source_document,
source_sha256 text NOT NULL,
record_ref text NOT NULL,
import_record_id bigint NOT NULL REFERENCES fm.import_record,
import_record_id bigint NOT NULL,
claim_status text NOT NULL DEFAULT 'asserted' CHECK (claim_status IN ('asserted','reviewed','disputed','superseded','retracted')),
superseded_by bigint REFERENCES fm.electricity_bill,
superseded_by bigint,
null_reasons jsonb NOT NULL DEFAULT '{}',
CHECK (kwh IS NOT NULL OR null_reasons ? 'kwh'),
CHECK (period_start IS NOT NULL OR null_reasons ? 'period_start')
CHECK (period_start IS NOT NULL OR null_reasons ? 'period_start'),
UNIQUE (operator_uid, bill_id),
FOREIGN KEY (operator_uid, holder_uid) REFERENCES fm.holder (operator_uid, holder_uid),
FOREIGN KEY (operator_uid, source_sha256) REFERENCES fm.source_document (operator_uid, sha256),
FOREIGN KEY (operator_uid, import_record_id) REFERENCES fm.import_record (operator_uid, import_record_id),
FOREIGN KEY (operator_uid, superseded_by) REFERENCES fm.electricity_bill (operator_uid, bill_id)
);
CREATE UNIQUE INDEX bill_live_key ON fm.electricity_bill (source_key)
CREATE UNIQUE INDEX bill_live_key ON fm.electricity_bill (operator_uid, source_key)
WHERE claim_status IN ('asserted','reviewed','disputed');
-- Specifications, assumptions, imported calculations -------------------------------------
CREATE TABLE fm.equipment_spec ( -- versioned like a claim: corrections supersede
spec_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
operator_uid text NOT NULL REFERENCES fm.operator,
source_key text NOT NULL, -- 'site-yaml:SITE-0001:AST-…:rated_prime_kw'
assertion_sha256 text NOT NULL CHECK (assertion_sha256 ~ '^[0-9a-f]{64}$'),
asset_uid text NOT NULL REFERENCES fm.asset,
asset_uid text NOT NULL,
attribute text NOT NULL, -- rated_prime_kw, capacity_l, fuel_rate_at_load, …
load_pct numeric(5,1),
value_num numeric(12,3), value_text text, unit text,
@@ -321,22 +454,28 @@ CREATE TABLE fm.equipment_spec ( -- versioned like a claim: corre
'operator_stated','our_inference','geometry')),
applicability text NOT NULL DEFAULT 'confirmed' CHECK (applicability IN ('confirmed','conditional','unconfirmed')),
applicability_note text,
source_sha256 text REFERENCES fm.source_document,
import_record_id bigint NOT NULL REFERENCES fm.import_record,
source_sha256 text,
import_record_id bigint NOT NULL,
claim_status text NOT NULL DEFAULT 'asserted' CHECK (claim_status IN ('asserted','reviewed','disputed','superseded','retracted')),
superseded_by bigint REFERENCES fm.equipment_spec,
superseded_by bigint,
CHECK (claim_status <> 'superseded' OR superseded_by IS NOT NULL),
CHECK ((attribute IN ('fuel_rate_at_load','specific_fuel_at_load')) = (load_pct IS NOT NULL))
CHECK ((attribute IN ('fuel_rate_at_load','specific_fuel_at_load')) = (load_pct IS NOT NULL)),
UNIQUE (operator_uid, spec_id),
FOREIGN KEY (operator_uid, asset_uid) REFERENCES fm.asset (operator_uid, asset_uid),
FOREIGN KEY (operator_uid, source_sha256) REFERENCES fm.source_document (operator_uid, sha256),
FOREIGN KEY (operator_uid, import_record_id) REFERENCES fm.import_record (operator_uid, import_record_id),
FOREIGN KEY (operator_uid, superseded_by) REFERENCES fm.equipment_spec (operator_uid, spec_id)
);
CREATE UNIQUE INDEX equipment_spec_live_key ON fm.equipment_spec (source_key)
CREATE UNIQUE INDEX equipment_spec_live_key ON fm.equipment_spec (operator_uid, source_key)
WHERE claim_status IN ('asserted','reviewed','disputed');
CREATE TABLE fm.assumption ( -- one row per version; a change is a new version
assumption_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
operator_uid text NOT NULL REFERENCES fm.operator,
code text NOT NULL UNIQUE, -- 'A-SITE-0001-GEN02-IDLE-v1'
fingerprint text NOT NULL CHECK (fingerprint ~ '^[0-9a-f]{64}$'), -- all fields below + sorted members
created_xact xid8 NOT NULL DEFAULT pg_current_xact_id(), -- rule 9: members only in this transaction
site_uid text NOT NULL REFERENCES fm.site,
asset_uid text REFERENCES fm.asset,
holder_uid text NOT NULL,
asset_uid text,
topic text NOT NULL CHECK (topic IN ('runtime_hours_per_day','generator_idle','meter_coverage',
'tank_linkage','no_unrecorded_movements','density','other')),
-- meter_coverage: its members name the exact counters,
@@ -347,77 +486,97 @@ CREATE TABLE fm.assumption ( -- one row per version; a chang
derived_from_fuel boolean NOT NULL, -- true: never a runtime input to an L/h figure
status text NOT NULL CHECK (status IN ('proposed','accepted','rejected','superseded')),
accepted_on date, accepted_by_role text, decision_ref text, -- copied from the matching fm.review row
supersedes_id bigint UNIQUE REFERENCES fm.assumption,
supersedes_id bigint UNIQUE,
rationale text NOT NULL, contradicted_by text,
source_sha256 text REFERENCES fm.source_document,
source_sha256 text,
CHECK (status <> 'accepted' OR (accepted_on IS NOT NULL AND accepted_by_role IS NOT NULL)),
CONSTRAINT assumption_no_runtime_from_fuel -- rule 5
CHECK (NOT (derived_from_fuel AND topic = 'runtime_hours_per_day' AND status = 'accepted')),
CHECK (topic <> 'generator_idle' OR asset_uid IS NOT NULL),
CHECK (topic <> 'meter_coverage' OR asset_uid IS NULL), -- its sets are in assumption_member
EXCLUDE USING gist (site_uid WITH =, daterange(applies_from, applies_to, '[]') WITH &&)
EXCLUDE USING gist (holder_uid WITH =, daterange(applies_from, applies_to, '[]') WITH &&)
WHERE (topic = 'meter_coverage' AND status IN ('proposed','accepted')),
EXCLUDE USING gist (asset_uid WITH =, daterange(applies_from, applies_to, '[]') WITH &&)
WHERE (topic = 'generator_idle' AND status IN ('proposed','accepted'))
WHERE (topic = 'generator_idle' AND status IN ('proposed','accepted')),
UNIQUE (operator_uid, assumption_id),
FOREIGN KEY (operator_uid, holder_uid) REFERENCES fm.holder (operator_uid, holder_uid),
FOREIGN KEY (operator_uid, asset_uid) REFERENCES fm.asset (operator_uid, asset_uid),
FOREIGN KEY (operator_uid, supersedes_id) REFERENCES fm.assumption (operator_uid, assumption_id),
FOREIGN KEY (operator_uid, source_sha256) REFERENCES fm.source_document (operator_uid, sha256)
);
CREATE TABLE fm.assumption_member ( -- exact membership of a meter_coverage assumption
assumption_id bigint NOT NULL REFERENCES fm.assumption,
assumption_id bigint NOT NULL,
operator_uid text NOT NULL REFERENCES fm.operator,
member_role text NOT NULL CHECK (member_role IN ('counter','covered_generator','idle_generator')),
meter_id text REFERENCES fm.runtime_meter,
asset_uid text REFERENCES fm.asset,
meter_id text,
asset_uid text,
CHECK ((member_role = 'counter') = (meter_id IS NOT NULL)),
CHECK ((member_role = 'counter') = (asset_uid IS NULL))
CHECK ((member_role = 'counter') = (asset_uid IS NULL)),
FOREIGN KEY (operator_uid, assumption_id) REFERENCES fm.assumption (operator_uid, assumption_id),
FOREIGN KEY (operator_uid, meter_id) REFERENCES fm.runtime_meter (operator_uid, meter_id),
FOREIGN KEY (operator_uid, asset_uid) REFERENCES fm.asset (operator_uid, asset_uid)
);
CREATE UNIQUE INDEX assumption_member_meter ON fm.assumption_member (assumption_id, meter_id) WHERE meter_id IS NOT NULL;
CREATE UNIQUE INDEX assumption_member_asset ON fm.assumption_member (assumption_id, asset_uid) WHERE asset_uid IS NOT NULL;
-- a set is either covered or idle within one assumption
CREATE TABLE fm.imported_calculation ( -- versioned like a claim: a revised formula supersedes
calc_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
operator_uid text NOT NULL REFERENCES fm.operator,
source_key text NOT NULL, -- '<family>:calc:SITE-0001:<label>'
assertion_sha256 text NOT NULL CHECK (assertion_sha256 ~ '^[0-9a-f]{64}$'),
source_sha256 text NOT NULL REFERENCES fm.source_document,
source_sha256 text NOT NULL,
cell_ref text NOT NULL, -- '<sheet>!K2' | '<survey item>'
site_uid text REFERENCES fm.site,
holder_uid text,
source_label text NOT NULL, -- as written in the source
formula text, cached_value numeric(14,6), unit_as_labelled text,
our_reading text NOT NULL, -- what it actually computes
baseline_use text NOT NULL CHECK (baseline_use IN ('excluded','context_only')),
exclusion_reason text,
import_record_id bigint NOT NULL REFERENCES fm.import_record,
import_record_id bigint NOT NULL,
claim_status text NOT NULL DEFAULT 'asserted' CHECK (claim_status IN ('asserted','reviewed','disputed','superseded','retracted')),
superseded_by bigint REFERENCES fm.imported_calculation,
superseded_by bigint,
CHECK (claim_status <> 'superseded' OR superseded_by IS NOT NULL),
CHECK (baseline_use <> 'excluded' OR exclusion_reason IS NOT NULL)
CHECK (baseline_use <> 'excluded' OR exclusion_reason IS NOT NULL),
UNIQUE (operator_uid, calc_id),
FOREIGN KEY (operator_uid, source_sha256) REFERENCES fm.source_document (operator_uid, sha256),
FOREIGN KEY (operator_uid, holder_uid) REFERENCES fm.holder (operator_uid, holder_uid),
FOREIGN KEY (operator_uid, import_record_id) REFERENCES fm.import_record (operator_uid, import_record_id),
FOREIGN KEY (operator_uid, superseded_by) REFERENCES fm.imported_calculation (operator_uid, calc_id)
);
CREATE UNIQUE INDEX imported_calc_live_key ON fm.imported_calculation (source_key)
CREATE UNIQUE INDEX imported_calc_live_key ON fm.imported_calculation (operator_uid, source_key)
WHERE claim_status IN ('asserted','reviewed','disputed');
-- Reviews ---------------------------------------------------------------------------------
CREATE TABLE fm.review ( -- one applied entry of the corpus review manifest
review_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
entry_key text NOT NULL UNIQUE, -- the manifest entry's own id
operator_uid text NOT NULL REFERENCES fm.operator,
entry_key text NOT NULL, -- the manifest entry's own id, per customer
subject_kind text NOT NULL CHECK (subject_kind IN ('claim','location','assumption')),
subject_key text NOT NULL, -- source_key, or the assumption code
assertion_sha256 text NOT NULL CHECK (assertion_sha256 ~ '^[0-9a-f]{64}$'), -- what was actually reviewed
checked_against_sha256 text REFERENCES fm.source_document, -- the file version compared with
checked_against_sha256 text, -- the file version compared with
verdict text NOT NULL CHECK (verdict IN ('reviewed','disputed','accepted','rejected')),
reviewed_on date NOT NULL, reviewer_role text NOT NULL, decision_ref text, note text,
import_batch_id bigint NOT NULL REFERENCES fm.import_batch,
CHECK ((subject_kind = 'claim') = (verdict IN ('reviewed','disputed')))
import_batch_id bigint NOT NULL,
CHECK ((subject_kind = 'claim') = (verdict IN ('reviewed','disputed'))),
UNIQUE (operator_uid, entry_key),
FOREIGN KEY (operator_uid, checked_against_sha256) REFERENCES fm.source_document (operator_uid, sha256),
FOREIGN KEY (operator_uid, import_batch_id) REFERENCES fm.import_batch (operator_uid, import_batch_id)
);
CREATE INDEX ON fm.review (subject_key, assertion_sha256);
CREATE INDEX ON fm.review (operator_uid, subject_key, assertion_sha256);
-- Baseline --------------------------------------------------------------------------------
CREATE TABLE fm.baseline_version (
baseline_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
operator_uid text NOT NULL REFERENCES fm.operator,
code text NOT NULL UNIQUE, -- 'BL-SITE-0001-APGH-v1'
site_uid text NOT NULL REFERENCES fm.site,
holder_uid text NOT NULL,
metric text NOT NULL CHECK (metric IN ('measured_generator_l_per_h','apparent_l_per_generator_h',
'apparent_l_per_set_h','apparent_l_per_site_dg_h','modelled_l_per_h','fuel_l_per_calendar_day')),
population text NOT NULL, -- human-readable: 'GEN-01 (AST-XXX-GEN-0001)'
population_key text NOT NULL, -- sorted asset uids, 'meter:<id>' for counters not tied
-- to a set plus 'covers:<asset uid>' for the sets they
-- cover, or 'stock:<site_uid>' for fuel_l_per_calendar_day
-- cover, or 'stock:<holder_uid>' for fuel_l_per_calendar_day
window_start date NOT NULL, window_end date NOT NULL CHECK (window_end > window_start),
policy_version text NOT NULL, -- 'BP-1'
meets_policy boolean NOT NULL DEFAULT false,-- set by `baseline propose` from the inputs (BP-1 test)
@@ -426,19 +585,23 @@ CREATE TABLE fm.baseline_version (
fuel_l numeric(10,1), runtime_h numeric(10,1), matched_days integer,
conditions text, -- assumption codes the value depends on
evidence_grade text CHECK (evidence_grade IN ('G1','G2')),
atoma_tier text CHECK (atoma_tier IN ('A','B','C')), -- descriptive only (Baseline Verification Plan)
operator_tier text CHECK (operator_tier IN ('A','B','C')), -- descriptive only (Baseline Verification Plan)
season_note text,
status text NOT NULL CHECK (status IN ('draft','proposed','approved','superseded','rejected')),
computed_at timestamptz NOT NULL, computed_by_role text NOT NULL,
approved_on date, approved_by_role text, decision_ref text,
supersedes_id bigint REFERENCES fm.baseline_version,
UNIQUE (baseline_id, site_uid),
CHECK (status <> 'approved' OR (approved_on IS NOT NULL AND decision_ref IS NOT NULL))
supersedes_id bigint,
UNIQUE (operator_uid, baseline_id, holder_uid),
CHECK (status <> 'approved' OR (approved_on IS NOT NULL AND decision_ref IS NOT NULL)),
UNIQUE (operator_uid, baseline_id),
FOREIGN KEY (operator_uid, holder_uid) REFERENCES fm.holder (operator_uid, holder_uid),
FOREIGN KEY (operator_uid, supersedes_id) REFERENCES fm.baseline_version (operator_uid, baseline_id)
);
CREATE UNIQUE INDEX baseline_one_approved ON fm.baseline_version (site_uid, metric) WHERE status = 'approved';
CREATE UNIQUE INDEX baseline_one_approved ON fm.baseline_version (holder_uid, metric) WHERE status = 'approved';
CREATE TABLE fm.baseline_input ( -- an interval is identified by its two endpoint visits
baseline_id bigint NOT NULL,
site_uid text NOT NULL,
operator_uid text NOT NULL REFERENCES fm.operator,
holder_uid text NOT NULL,
open_visit_id bigint NOT NULL,
close_visit_id bigint NOT NULL,
start_on date NOT NULL, end_on date NOT NULL CHECK (end_on >= start_on),
@@ -448,9 +611,9 @@ CREATE TABLE fm.baseline_input ( -- an interval is identified by
dependencies jsonb NOT NULL, -- {"stock": [...], "runtime": [...]} from fm.v_interval_dependency
stock_dep_sha256 text NOT NULL, runtime_dep_sha256 text, -- their hashes; runtime NULL for L/day versions
PRIMARY KEY (baseline_id, open_visit_id, close_visit_id),
FOREIGN KEY (baseline_id, site_uid) REFERENCES fm.baseline_version (baseline_id, site_uid),
FOREIGN KEY (open_visit_id, site_uid) REFERENCES fm.visit (visit_id, site_uid), -- both endpoints belong
FOREIGN KEY (close_visit_id, site_uid) REFERENCES fm.visit (visit_id, site_uid), -- to the baseline's site
FOREIGN KEY (operator_uid, baseline_id, holder_uid) REFERENCES fm.baseline_version (operator_uid, baseline_id, holder_uid),
FOREIGN KEY (operator_uid, open_visit_id, holder_uid) REFERENCES fm.visit (operator_uid, visit_id, holder_uid), -- both endpoints belong
FOREIGN KEY (operator_uid, close_visit_id, holder_uid) REFERENCES fm.visit (operator_uid, visit_id, holder_uid), -- to the baseline's holder
CHECK (included OR exclusion_reason IS NOT NULL),
CHECK (NOT included OR end_on > start_on), -- a same-day interval can only be an excluded input
EXCLUDE USING gist (baseline_id WITH =, daterange(start_on, end_on, '[)') WITH &&) WHERE (included)
@@ -463,7 +626,7 @@ CREATE TABLE fm.baseline_input ( -- an interval is identified by
-- Claims are corrected by supersession, never in place (design §2.1). Only these columns may change.
CREATE FUNCTION fm.mutable_columns(tbl text) RETURNS text[] LANGUAGE sql IMMUTABLE AS $$
SELECT CASE tbl
WHEN 'site_location' THEN ARRAY['acceptance','accepted_on','accepted_by_role','acceptance_ref','superseded_by']
WHEN 'place_location' THEN ARRAY['acceptance','accepted_on','accepted_by_role','acceptance_ref','superseded_by']
WHEN 'assumption' THEN ARRAY['status','accepted_on','accepted_by_role','decision_ref','contradicted_by']
ELSE ARRAY['claim_status','superseded_by','review_note']
END $$;
@@ -479,7 +642,7 @@ BEGIN
RAISE EXCEPTION '%: only the status columns change in place; a corrected value is a new row that supersedes this one',
TG_TABLE_NAME;
END IF;
IF TG_TABLE_NAME = 'site_location' THEN
IF TG_TABLE_NAME = 'place_location' THEN
v_kind := 'location'; v_subject := j->>'source_key'; v_fp := j->>'assertion_sha256'; v_verdict := j->>'acceptance';
IF v_verdict NOT IN ('accepted','rejected') THEN RETURN NEW; END IF;
ELSIF TG_TABLE_NAME = 'assumption' THEN
@@ -490,7 +653,7 @@ BEGIN
IF v_verdict NOT IN ('reviewed','disputed') THEN RETURN NEW; END IF;
END IF;
SELECT * INTO r FROM fm.review x
WHERE x.subject_kind = v_kind AND x.subject_key = v_subject AND x.assertion_sha256 = v_fp AND x.verdict = v_verdict
WHERE x.operator_uid = j->>'operator_uid' AND x.subject_kind = v_kind AND x.subject_key = v_subject AND x.assertion_sha256 = v_fp AND x.verdict = v_verdict
ORDER BY x.reviewed_on DESC, x.review_id DESC LIMIT 1;
IF NOT FOUND THEN
RAISE EXCEPTION '%: status % for % needs a review of this exact assertion (%)',
@@ -508,31 +671,31 @@ DECLARE t text;
BEGIN
FOREACH t IN ARRAY ARRAY['fuel_movement','stock_observation','hour_meter_reading','electricity_bill',
'equipment_spec','imported_calculation','runtime_interval','engine_fuel_total',
'site_location','assumption'] LOOP
'place_location','assumption'] LOOP
EXECUTE format('CREATE TRIGGER status_from_review BEFORE INSERT OR UPDATE ON fm.%I
FOR EACH ROW EXECUTE FUNCTION fm.tg_status_from_review()', t);
END LOOP;
END $$;
-- 2, 3, 10. A movement claim: same site as its visit; an internal transfer's two tanks placed at
-- that site on the visit date; the live claims of one event_key share visit, kind and tanks. A
-- 2, 3, 10. A movement claim: same holder as its visit; an internal transfer's two tanks placed on
-- that holder on the visit date; the live claims of one event_key share visit, kind and tanks. A
-- superseded or retracted claim keeps its own, so a correction can change them.
CREATE FUNCTION fm.tg_fuel_movement_rules() RETURNS trigger LANGUAGE plpgsql AS $$
DECLARE v fm.visit;
BEGIN
SELECT * INTO v FROM fm.visit WHERE visit_id = NEW.visit_id;
IF v.site_uid <> NEW.site_uid THEN
RAISE EXCEPTION 'fuel_movement %: site % differs from its visit''s site %', NEW.source_key, NEW.site_uid, v.site_uid;
IF v.holder_uid <> NEW.holder_uid THEN
RAISE EXCEPTION 'fuel_movement %: site % differs from its visit''s site %', NEW.source_key, NEW.holder_uid, v.holder_uid;
END IF;
IF NEW.kind = 'internal_transfer' AND EXISTS (
SELECT 1 FROM unnest(ARRAY[NEW.from_tank_uid, NEW.to_tank_uid]) t(tank)
WHERE NOT EXISTS (SELECT 1 FROM fm.asset_placement p
WHERE p.asset_uid = t.tank AND p.site_uid = NEW.site_uid
WHERE p.asset_uid = t.tank AND p.holder_uid = NEW.holder_uid
AND daterange(coalesce(p.occupied_from,'-infinity'::date), p.occupied_to, '[)') @> v.visit_on)) THEN
RAISE EXCEPTION 'fuel_movement %: both tanks of an internal transfer must be placed at % on %', NEW.source_key, NEW.site_uid, v.visit_on;
RAISE EXCEPTION 'fuel_movement %: both tanks of an internal transfer must be placed at % on %', NEW.source_key, NEW.holder_uid, v.visit_on;
END IF;
IF NEW.claim_status IN ('asserted','reviewed','disputed') AND EXISTS (SELECT 1 FROM fm.fuel_movement m
WHERE m.event_key = NEW.event_key AND m.movement_id <> NEW.movement_id
WHERE m.operator_uid = NEW.operator_uid AND m.event_key = NEW.event_key AND m.movement_id <> NEW.movement_id
AND m.claim_status IN ('asserted','reviewed','disputed')
AND (m.visit_id, m.kind, m.from_tank_uid, m.to_tank_uid)
IS DISTINCT FROM (NEW.visit_id, NEW.kind, NEW.from_tank_uid, NEW.to_tank_uid)) THEN
@@ -543,21 +706,21 @@ END $$;
CREATE TRIGGER fuel_movement_rules BEFORE INSERT OR UPDATE ON fm.fuel_movement
FOR EACH ROW EXECUTE FUNCTION fm.tg_fuel_movement_rules();
-- 3. A stock claim belongs to its visit's site.
-- 3. A stock claim belongs to its visit's holder.
CREATE FUNCTION fm.tg_stock_observation_rules() RETURNS trigger LANGUAGE plpgsql AS $$
BEGIN
IF NOT EXISTS (SELECT 1 FROM fm.visit v WHERE v.visit_id = NEW.visit_id AND v.site_uid = NEW.site_uid) THEN
RAISE EXCEPTION 'stock_observation %: site % differs from its visit''s site', NEW.source_key, NEW.site_uid;
IF NOT EXISTS (SELECT 1 FROM fm.visit v WHERE v.visit_id = NEW.visit_id AND v.holder_uid = NEW.holder_uid) THEN
RAISE EXCEPTION 'stock_observation %: site % differs from its visit''s site', NEW.source_key, NEW.holder_uid;
END IF;
RETURN NEW;
END $$;
CREATE TRIGGER stock_observation_rules BEFORE INSERT OR UPDATE ON fm.stock_observation
FOR EACH ROW EXECUTE FUNCTION fm.tg_stock_observation_rules();
-- 3, 4. A reading's meter belongs to its visit's site; its epoch belongs to its meter.
-- 3, 4. A reading's meter belongs to its visit's holder; its epoch belongs to its meter.
CREATE FUNCTION fm.tg_hour_meter_reading_rules() RETURNS trigger LANGUAGE plpgsql AS $$
BEGIN
IF NOT EXISTS (SELECT 1 FROM fm.visit v JOIN fm.runtime_meter m ON m.site_uid = v.site_uid
IF NOT EXISTS (SELECT 1 FROM fm.visit v JOIN fm.runtime_meter m ON m.holder_uid = v.holder_uid
WHERE v.visit_id = NEW.visit_id AND m.meter_id = NEW.meter_id) THEN
RAISE EXCEPTION 'hour_meter_reading %: meter % is not a meter of the visit''s site', NEW.source_key, NEW.meter_id;
END IF;
@@ -570,9 +733,13 @@ END $$;
CREATE TRIGGER hour_meter_reading_rules BEFORE INSERT OR UPDATE ON fm.hour_meter_reading
FOR EACH ROW EXECUTE FUNCTION fm.tg_hour_meter_reading_rules();
-- 7. Supersession stays within one source family.
-- 7. Supersession stays within one source family. A document belongs to one customer until
-- document sharing exists.
CREATE FUNCTION fm.tg_source_document_rules() RETURNS trigger LANGUAGE plpgsql AS $$
BEGIN
IF EXISTS (SELECT 1 FROM fm.source_document d WHERE d.sha256 = NEW.sha256 AND d.operator_uid <> NEW.operator_uid) THEN
RAISE EXCEPTION 'source_document %: already held for another customer; a document cannot be shared yet', left(NEW.sha256, 16);
END IF;
IF NEW.supersedes_sha256 IS NOT NULL AND NOT EXISTS (
SELECT 1 FROM fm.source_document d WHERE d.sha256 = NEW.supersedes_sha256 AND d.source_family = NEW.source_family) THEN
RAISE EXCEPTION 'source_document %: it may supersede only a document of family %', left(NEW.sha256, 16), NEW.source_family;
@@ -582,30 +749,32 @@ END $$;
CREATE TRIGGER source_document_rules BEFORE INSERT OR UPDATE ON fm.source_document
FOR EACH ROW EXECUTE FUNCTION fm.tg_source_document_rules();
-- 8. A confirmed or proposed meter mapping names a generator placed at the meter's site at some
-- 8. A confirmed or proposed meter mapping names a generator placed on the meter's holder at some
-- time. Whether the placement spans a given interval is fm.v_lph_interval's check.
CREATE FUNCTION fm.tg_runtime_meter_rules() RETURNS trigger LANGUAGE plpgsql AS $$
BEGIN
IF NEW.mapping_status IN ('confirmed','proposed') AND NOT EXISTS (
SELECT 1 FROM fm.asset_placement p JOIN fm.asset a ON a.asset_uid = p.asset_uid
WHERE p.asset_uid = NEW.asset_uid AND p.site_uid = NEW.site_uid AND a.kind = 'generator') THEN
RAISE EXCEPTION 'runtime_meter %: % is not a generator placed at %', NEW.meter_id, NEW.asset_uid, NEW.site_uid;
WHERE p.asset_uid = NEW.asset_uid AND p.holder_uid = NEW.holder_uid AND a.kind = 'generator') THEN
RAISE EXCEPTION 'runtime_meter %: % is not a generator placed at %', NEW.meter_id, NEW.asset_uid, NEW.holder_uid;
END IF;
RETURN NEW;
END $$;
CREATE TRIGGER runtime_meter_rules BEFORE INSERT OR UPDATE ON fm.runtime_meter
FOR EACH ROW EXECUTE FUNCTION fm.tg_runtime_meter_rules();
-- 12. A generator_idle assumption names a generator placed at its own site during its dates.
-- 12. A generator_idle assumption names a generator placed on its own holder during its dates.
-- 8, 9 and 12 stay generator-only: the runtime and L/h calculations have not been checked on
-- an engine, so a meter or idle assumption on one is refused rather than silently unused.
CREATE FUNCTION fm.tg_assumption_rules() RETURNS trigger LANGUAGE plpgsql AS $$
BEGIN
IF NEW.topic = 'generator_idle' AND NOT EXISTS (
SELECT 1 FROM fm.asset_placement p JOIN fm.asset a ON a.asset_uid = p.asset_uid
WHERE p.asset_uid = NEW.asset_uid AND p.site_uid = NEW.site_uid AND a.kind = 'generator'
WHERE p.asset_uid = NEW.asset_uid AND p.holder_uid = NEW.holder_uid AND a.kind = 'generator'
AND daterange(coalesce(p.occupied_from,'-infinity'::date), p.occupied_to, '[)')
&& daterange(NEW.applies_from, NEW.applies_to, '[]')) THEN
RAISE EXCEPTION 'assumption %: % is not a generator placed at % during the assumption''s dates',
NEW.code, NEW.asset_uid, NEW.site_uid;
NEW.code, NEW.asset_uid, NEW.holder_uid;
END IF;
RETURN NEW;
END $$;
@@ -629,12 +798,12 @@ BEGIN
END IF;
IF NEW.member_role = 'counter' THEN
IF NOT EXISTS (SELECT 1 FROM fm.runtime_meter m
WHERE m.meter_id = NEW.meter_id AND m.site_uid = a.site_uid AND m.mapping_status <> 'confirmed') THEN
RAISE EXCEPTION 'assumption_member: % is not an unconfirmed counter of %', NEW.meter_id, a.site_uid;
WHERE m.meter_id = NEW.meter_id AND m.holder_uid = a.holder_uid AND m.mapping_status <> 'confirmed') THEN
RAISE EXCEPTION 'assumption_member: % is not an unconfirmed counter of %', NEW.meter_id, a.holder_uid;
END IF;
ELSIF NOT EXISTS (SELECT 1 FROM fm.asset_placement p JOIN fm.asset s ON s.asset_uid = p.asset_uid
WHERE p.asset_uid = NEW.asset_uid AND p.site_uid = a.site_uid AND s.kind = 'generator') THEN
RAISE EXCEPTION 'assumption_member: % is not a generator placed at %', NEW.asset_uid, a.site_uid;
WHERE p.asset_uid = NEW.asset_uid AND p.holder_uid = a.holder_uid AND s.kind = 'generator') THEN
RAISE EXCEPTION 'assumption_member: % is not a generator placed at %', NEW.asset_uid, a.holder_uid;
END IF;
RETURN NEW;
END $$;
@@ -710,7 +879,7 @@ BEGIN
RAISE EXCEPTION 'baseline_input: the inputs of approved version % are immutable', v.code;
END IF;
IF TG_OP = 'DELETE' THEN RETURN OLD; END IF;
IF TG_OP = 'UPDATE' AND (NEW.baseline_id, NEW.site_uid) IS DISTINCT FROM (OLD.baseline_id, OLD.site_uid) THEN
IF TG_OP = 'UPDATE' AND (NEW.baseline_id, NEW.holder_uid) IS DISTINCT FROM (OLD.baseline_id, OLD.holder_uid) THEN
RAISE EXCEPTION 'baseline_input: an input of % never moves to another version', v.code;
END IF;
IF v.status NOT IN ('draft','proposed') THEN
@@ -736,6 +905,9 @@ CREATE FUNCTION fm.tg_fixed_columns() RETURNS trigger LANGUAGE plpgsql AS $$
DECLARE c text;
BEGIN
FOREACH c IN ARRAY TG_ARGV LOOP
IF NOT to_jsonb(NEW) ? c THEN
RAISE EXCEPTION '%: tg_fixed_columns names a column it does not have: %', TG_TABLE_NAME, c;
END IF;
IF to_jsonb(OLD)->c IS DISTINCT FROM to_jsonb(NEW)->c THEN
RAISE EXCEPTION '%: % never changes in place', TG_TABLE_NAME, c;
END IF;
@@ -743,14 +915,62 @@ BEGIN
RETURN NEW;
END $$;
CREATE TRIGGER visit_fixed BEFORE UPDATE ON fm.visit
FOR EACH ROW EXECUTE FUNCTION fm.tg_fixed_columns('visit_key','site_uid','visit_on','seq_in_day');
FOR EACH ROW EXECUTE FUNCTION fm.tg_fixed_columns('visit_key','holder_uid','visit_on','seq_in_day');
CREATE TRIGGER source_document_fixed BEFORE UPDATE ON fm.source_document
FOR EACH ROW EXECUTE FUNCTION fm.tg_fixed_columns('sha256','source_family','kind');
CREATE TRIGGER asset_fixed BEFORE UPDATE ON fm.asset
FOR EACH ROW EXECUTE FUNCTION fm.tg_fixed_columns('asset_uid','kind');
CREATE TRIGGER runtime_meter_fixed BEFORE UPDATE ON fm.runtime_meter -- readings follow their meter's site
FOR EACH ROW EXECUTE FUNCTION fm.tg_fixed_columns('meter_id','site_uid');
FOR EACH ROW EXECUTE FUNCTION fm.tg_fixed_columns('meter_id','holder_uid');
CREATE TRIGGER meter_epoch_fixed BEFORE UPDATE ON fm.meter_epoch -- and their epoch's meter
FOR EACH ROW EXECUTE FUNCTION fm.tg_fixed_columns('epoch_id','meter_id');
CREATE TRIGGER place_fixed BEFORE UPDATE ON fm.place
FOR EACH ROW EXECUTE FUNCTION fm.tg_fixed_columns('place_uid');
CREATE TRIGGER holder_fixed BEFORE UPDATE ON fm.holder
FOR EACH ROW EXECUTE FUNCTION fm.tg_fixed_columns('holder_uid','kind','asset_uid');
-- A row's customer never changes in place, on any table.
DO $$
DECLARE t text;
BEGIN
FOR t IN SELECT c.table_name FROM information_schema.columns c
WHERE c.table_schema = 'fm' AND c.column_name = 'operator_uid' LOOP
EXECUTE format('CREATE TRIGGER operator_fixed BEFORE UPDATE ON fm.%I
FOR EACH ROW EXECUTE FUNCTION fm.tg_fixed_columns(%L)', t, 'operator_uid');
END LOOP;
END $$;
-- 14. A time zone is one PostgreSQL knows: a wrong one would stop every view that uses it.
CREATE FUNCTION fm.tg_timezone_known() RETURNS trigger LANGUAGE plpgsql AS $$
BEGIN
IF NEW.timezone IS NOT NULL AND NOT EXISTS (SELECT 1 FROM pg_timezone_names n WHERE n.name = NEW.timezone) THEN
RAISE EXCEPTION '%: unknown time zone %', TG_TABLE_NAME, NEW.timezone;
END IF;
RETURN NEW;
END $$;
CREATE TRIGGER operator_timezone BEFORE INSERT OR UPDATE ON fm.operator
FOR EACH ROW EXECUTE FUNCTION fm.tg_timezone_known();
CREATE TRIGGER place_timezone BEFORE INSERT OR UPDATE ON fm.place
FOR EACH ROW EXECUTE FUNCTION fm.tg_timezone_known();
-- 15. A vehicle holder's asset is a vehicle; only a fixed holder stands at a place.
CREATE FUNCTION fm.tg_holder_rules() RETURNS trigger LANGUAGE plpgsql AS $$
BEGIN
IF NEW.asset_uid IS NOT NULL AND NOT EXISTS (
SELECT 1 FROM fm.asset a WHERE a.asset_uid = NEW.asset_uid AND a.kind = 'vehicle') THEN
RAISE EXCEPTION 'holder %: % is not a vehicle', NEW.holder_uid, NEW.asset_uid;
END IF;
RETURN NEW;
END $$;
CREATE TRIGGER holder_rules BEFORE INSERT OR UPDATE ON fm.holder
FOR EACH ROW EXECUTE FUNCTION fm.tg_holder_rules();
CREATE FUNCTION fm.tg_holder_place_rules() RETURNS trigger LANGUAGE plpgsql AS $$
BEGIN
IF NOT EXISTS (SELECT 1 FROM fm.holder h WHERE h.holder_uid = NEW.holder_uid AND h.kind = 'fixed_installation') THEN
RAISE EXCEPTION 'holder_place: % is not a fixed installation', NEW.holder_uid;
END IF;
RETURN NEW;
END $$;
CREATE TRIGGER holder_place_rules BEFORE INSERT OR UPDATE ON fm.holder_place
FOR EACH ROW EXECUTE FUNCTION fm.tg_holder_place_rules();
COMMIT;
@@ -4,6 +4,18 @@
BEGIN;
SET LOCAL search_path = fm, pg_catalog, public;
-- Each holder's customer, current place and local day. The current place is found by the customer's
-- local day; a holder without a place, or a place without its own zone, uses the customer's zone.
CREATE VIEW fm.v_holder_current AS
SELECT h.holder_uid, h.operator_uid, hp.place_uid,
coalesce(pl.timezone, o.timezone) AS timezone,
(now() AT TIME ZONE coalesce(pl.timezone, o.timezone))::date AS today
FROM fm.holder h
JOIN fm.operator o ON o.operator_uid = h.operator_uid
LEFT JOIN fm.holder_place hp ON hp.holder_uid = h.holder_uid
AND daterange(coalesce(hp.valid_from, '-infinity'::date), hp.valid_to, '[)') @> (now() AT TIME ZONE o.timezone)::date
LEFT JOIN fm.place pl ON pl.place_uid = hp.place_uid;
CREATE VIEW fm.v_source_rank AS
SELECT sha256, kind, CASE kind WHEN 'delivery_form' THEN 1 WHEN 'site_logbook' THEN 2
WHEN 'operator_record' THEN 3 WHEN 'operator_workbook' THEN 3 ELSE 9 END AS rank
@@ -74,7 +86,7 @@ HAVING count(*) FILTER (WHERE s.claim_status IN ('asserted','reviewed') AND s.li
-- v1: tank-level rows are summed only in a later version, and only when every placed tank has one.
CREATE VIEW fm.v_stock_point AS -- stock on arrival and on departure of each live visit
SELECT v.visit_id, v.visit_key, v.site_uid, v.visit_on, v.seq_in_day, v.kind,
SELECT v.visit_id, v.visit_key, v.holder_uid, v.visit_on, v.seq_in_day, v.kind,
d.delivered_l, coalesce(o.n_external, 0) AS n_other_movements,
coalesce(b.litres, st.litres, a.litres - d.delivered_l) AS arrive_l,
CASE WHEN coalesce(b.litres, st.litres) IS NOT NULL THEN 'stated'
@@ -104,10 +116,10 @@ LEFT JOIN fm.v_visit_stock st ON st.visit_id = v.visit_id AND st.phase = 'standa
CREATE VIEW fm.v_stock_interval AS -- consecutive stock points per site; no overlap by construction
WITH pts AS (
SELECT p.*, row_number() OVER (PARTITION BY p.site_uid ORDER BY p.visit_on, p.seq_in_day) AS n
SELECT p.*, row_number() OVER (PARTITION BY p.holder_uid ORDER BY p.visit_on, p.seq_in_day) AS n
FROM fm.v_stock_point p WHERE p.arrive_l IS NOT NULL OR p.depart_l IS NOT NULL),
pr AS (
SELECT o.site_uid, o.visit_key || '>' || c.visit_key AS interval_key,
SELECT o.holder_uid, o.visit_key || '>' || c.visit_key AS interval_key,
o.visit_id AS open_visit_id, c.visit_id AS close_visit_id,
o.visit_on AS start_on, c.visit_on AS end_on, c.visit_on - o.visit_on AS days,
o.depart_l AS opening_l, o.depart_basis AS opening_basis,
@@ -117,8 +129,8 @@ pr AS (
coalesce(o.depart_reviewed AND c.arrive_reviewed, false) AS endpoints_reviewed,
o.n_other_movements + c.n_other_movements AS n_endpoint_movements,
o.visit_on AS o_on, o.seq_in_day AS o_seq, c.visit_on AS c_on, c.seq_in_day AS c_seq
FROM pts o JOIN pts c ON c.site_uid = o.site_uid AND c.n = o.n + 1)
SELECT p.site_uid, p.interval_key, p.open_visit_id, p.close_visit_id, p.start_on, p.end_on, p.days,
FROM pts o JOIN pts c ON c.holder_uid = o.holder_uid AND c.n = o.n + 1)
SELECT p.holder_uid, p.interval_key, p.open_visit_id, p.close_visit_id, p.start_on, p.end_on, p.days,
p.opening_l, p.opening_basis, p.closing_l, p.closing_basis,
coalesce(mid.delivered_l, 0) AS deliveries_between_l, coalesce(mid.out_l, 0) AS out_l,
p.opening_l + coalesce(mid.delivered_l,0) - coalesce(mid.out_l,0) - p.closing_l AS consumed_l,
@@ -144,7 +156,7 @@ CROSS JOIN LATERAL (
FROM fm.v_visit_live v
LEFT JOIN fm.v_visit_delivery d ON d.visit_id = v.visit_id
LEFT JOIN fm.v_visit_other_movement x ON x.visit_id = v.visit_id
WHERE v.site_uid = p.site_uid
WHERE v.holder_uid = p.holder_uid
AND (v.visit_on, v.seq_in_day) > (p.o_on, p.o_seq)
AND (v.visit_on, v.seq_in_day) < (p.c_on, p.c_seq)) mid;
@@ -180,7 +192,7 @@ WITH gen AS ( -- generators placed at the site at any time in the
bool_or(daterange(coalesce(p.occupied_from,'-infinity'::date), p.occupied_to, '[)')
@> daterange(i.start_on, i.end_on, '[]')) AS full_cover
FROM fm.v_stock_interval i
JOIN fm.asset_placement p ON p.site_uid = i.site_uid
JOIN fm.asset_placement p ON p.holder_uid = i.holder_uid
JOIN fm.asset a ON a.asset_uid = p.asset_uid AND a.kind = 'generator'
WHERE daterange(coalesce(p.occupied_from,'-infinity'::date), p.occupied_to, '[)')
&& daterange(i.start_on, i.end_on, '[]')
@@ -201,13 +213,13 @@ pair AS ( -- every counter read at both endpoints, classified
o.all_reviewed AND c.all_reviewed AS reviewed,
(o.hours_max > o.hours_min OR c.hours_max > c.hours_min) AS conflict
FROM fm.v_stock_interval i
JOIN fm.runtime_meter m ON m.site_uid = i.site_uid
JOIN fm.runtime_meter m ON m.holder_uid = i.holder_uid
JOIN fm.v_visit_reading o ON o.visit_id = i.open_visit_id AND o.meter_id = m.meter_id
JOIN fm.v_visit_reading c ON c.visit_id = i.close_visit_id AND c.meter_id = m.meter_id),
idle AS ( -- at most one live generator_idle per asset and date (EXCLUDE)
SELECT g.interval_key, g.asset_uid, x.code, x.status
FROM gen g JOIN fm.v_stock_interval i USING (interval_key)
JOIN fm.assumption x ON x.asset_uid = g.asset_uid AND x.site_uid = i.site_uid -- this site's only
JOIN fm.assumption x ON x.asset_uid = g.asset_uid AND x.holder_uid = i.holder_uid -- this site's only
AND x.topic = 'generator_idle' AND x.status IN ('proposed','accepted')
AND daterange(x.applies_from, x.applies_to, '[]') @> daterange(i.start_on, i.end_on, '[]')),
per_gen AS (
@@ -234,7 +246,7 @@ loose AS ( -- counters read at both ends but not tied to a set
SELECT interval_key, array_agg(meter_id ORDER BY meter_id) AS meters
FROM pair WHERE tie = 'not_tied' GROUP BY interval_key),
cova AS ( -- live meter_coverage assumptions with their exact members
SELECT x.assumption_id, x.code, x.site_uid, x.status, x.applies_from, x.applies_to,
SELECT x.assumption_id, x.code, x.holder_uid, x.status, x.applies_from, x.applies_to,
coalesce(array_agg(m.meter_id ORDER BY m.meter_id)
FILTER (WHERE m.member_role = 'counter'), '{}') AS meters,
coalesce(array_agg(m.asset_uid ORDER BY m.asset_uid)
@@ -254,7 +266,7 @@ unm AS ( -- at most one live coverage assumption per site an
FROM loose l
JOIN fm.v_stock_interval i USING (interval_key)
LEFT JOIN cov cv USING (interval_key)
LEFT JOIN cova c ON c.site_uid = i.site_uid
LEFT JOIN cova c ON c.holder_uid = i.holder_uid
AND daterange(c.applies_from, c.applies_to, '[]') @> daterange(i.start_on, i.end_on, '[]')),
sel AS ( -- the validated population; nothing else is summed
SELECT p.* FROM pair p
@@ -343,7 +355,7 @@ FROM x;
-- M4 (L per calendar day) per stock interval: no runtime, so no generator population. The
-- population is the site's whole stock. Input to the L/day proposal path only; never a KPI.
CREATE VIEW fm.v_m4_interval AS
SELECT i.*, 'stock:' || i.site_uid AS population_key,
SELECT i.*, 'stock:' || i.holder_uid AS population_key,
CASE WHEN i.days > 0 THEN i.consumed_l / i.days END AS l_per_day,
CASE WHEN i.stock_status = 'ok' THEN 'eligible' ELSE 'excluded' END AS eligibility,
CASE WHEN i.stock_status <> 'ok' THEN 'stock interval: ' || i.stock_status ELSE '' END AS exclusion_reasons,
@@ -359,7 +371,7 @@ WITH span AS ( -- the endpoint visits and every visit between them
FROM fm.v_stock_interval i
JOIN fm.visit o ON o.visit_id = i.open_visit_id
JOIN fm.visit c ON c.visit_id = i.close_visit_id
JOIN fm.visit v ON v.site_uid = i.site_uid
JOIN fm.visit v ON v.holder_uid = i.holder_uid
AND (v.visit_on, v.seq_in_day) >= (o.visit_on, o.seq_in_day)
AND (v.visit_on, v.seq_in_day) <= (c.visit_on, c.seq_in_day)),
stock_deps AS (
@@ -379,27 +391,27 @@ runtime_deps AS (
FROM fm.hour_meter_reading x WHERE x.visit_id IN (i.open_visit_id, i.close_visit_id)
UNION ALL
SELECT 'meter:' || m.meter_id || ':' || m.mapping_status || ':' || coalesce(m.asset_uid, '-')
FROM fm.runtime_meter m WHERE m.site_uid = i.site_uid
FROM fm.runtime_meter m WHERE m.holder_uid = i.holder_uid
UNION ALL
SELECT 'epoch:' || e.meter_id || ':' || e.epoch_id || ':' || coalesce(e.starts_on::text, '-')
|| ':' || e.start_reason || ':' || e.continuity || ':' || coalesce(e.rollover_at::text, '-')
FROM fm.meter_epoch e JOIN fm.runtime_meter m ON m.meter_id = e.meter_id
WHERE m.site_uid = i.site_uid
WHERE m.holder_uid = i.holder_uid
UNION ALL -- placements at this site, and anywhere of the assets its meters name
SELECT 'placement:' || p.asset_uid || ':' || p.site_uid || ':' || coalesce(p.occupied_from::text, '-')
SELECT 'placement:' || p.asset_uid || ':' || coalesce(p.holder_uid, p.place_uid) || ':' || coalesce(p.occupied_from::text, '-')
|| ':' || coalesce(p.occupied_to::text, '-')
FROM fm.asset_placement p
WHERE (p.site_uid = i.site_uid
OR p.asset_uid IN (SELECT m.asset_uid FROM fm.runtime_meter m WHERE m.site_uid = i.site_uid))
WHERE (p.holder_uid = i.holder_uid
OR p.asset_uid IN (SELECT m.asset_uid FROM fm.runtime_meter m WHERE m.holder_uid = i.holder_uid))
AND daterange(coalesce(p.occupied_from, '-infinity'::date), p.occupied_to, '[)')
&& daterange(i.start_on, i.end_on, '[]')
UNION ALL
SELECT 'assumption:' || x.code || ':' || x.fingerprint || ':' || x.status
FROM fm.assumption x
WHERE x.site_uid = i.site_uid AND x.topic IN ('generator_idle','meter_coverage')
WHERE x.holder_uid = i.holder_uid AND x.topic IN ('generator_idle','meter_coverage')
AND daterange(x.applies_from, x.applies_to, '[]') && daterange(i.start_on, i.end_on, '[]')) d
GROUP BY i.interval_key)
SELECT i.interval_key, i.site_uid, i.open_visit_id, i.close_visit_id,
SELECT i.interval_key, i.holder_uid, i.open_visit_id, i.close_visit_id,
coalesce(s.deps, '[]'::jsonb) AS stock_deps, coalesce(r.deps, '[]'::jsonb) AS runtime_deps,
encode(sha256(convert_to(coalesce(s.deps, '[]'::jsonb)::text, 'UTF8')), 'hex') AS stock_dep_sha256,
encode(sha256(convert_to(coalesce(r.deps, '[]'::jsonb)::text, 'UTF8')), 'hex') AS runtime_dep_sha256
@@ -424,11 +436,11 @@ GROUP BY b.baseline_id, b.metric;
-- Consecutive readings per meter (live visits), giving hours per day: R1 / public.v_runtime.
CREATE VIEW fm.v_runtime_pair AS
WITH r AS (
SELECT v.site_uid, v.visit_id, v.visit_on, v.seq_in_day, x.meter_id, x.epoch_id, x.n_epochs,
SELECT v.holder_uid, v.visit_id, v.visit_on, v.seq_in_day, x.meter_id, x.epoch_id, x.n_epochs,
x.n_unassigned_epoch, x.hours, x.all_reviewed,
row_number() OVER (PARTITION BY x.meter_id ORDER BY v.visit_on, v.seq_in_day) AS n
FROM fm.v_visit_reading x JOIN fm.v_visit_live v ON v.visit_id = x.visit_id)
SELECT o.site_uid, o.meter_id, o.visit_id AS open_visit_id, c.visit_id AS close_visit_id,
SELECT o.holder_uid, o.meter_id, o.visit_id AS open_visit_id, c.visit_id AS close_visit_id,
o.visit_on AS start_on, c.visit_on AS end_on, c.visit_on - o.visit_on AS days,
c.hours - o.hours AS runtime_h,
CASE WHEN c.visit_on > o.visit_on THEN (c.hours - o.hours) / (c.visit_on - o.visit_on) END AS h_per_day,
@@ -443,7 +455,7 @@ FROM r o JOIN r c ON c.meter_id = o.meter_id AND c.n = o.n + 1;
-- assumption's dates, divided by that independently sourced runtime assumption (never one derived
-- from fuel). The window is the matched one. Reference tab only; never a KPI (design §4.1).
CREATE VIEW fm.v_modelled_lph AS
SELECT x.site_uid, x.code, x.status, x.basis,
SELECT x.holder_uid, x.code, x.status, x.basis,
x.value_num AS h_per_day, x.low_num AS h_per_day_low, x.high_num AS h_per_day_high,
m4.window_start, m4.window_end, m4.days AS matched_days, m4.fuel_l / NULLIF(m4.days, 0) AS l_per_day,
m4.fuel_l / NULLIF(m4.days, 0) / NULLIF(x.value_num, 0) AS modelled_l_per_h,
@@ -454,46 +466,48 @@ CROSS JOIN LATERAL (
SELECT sum(i.consumed_l) AS fuel_l, sum(i.days) AS days,
min(i.start_on) AS window_start, max(i.end_on) AS window_end
FROM fm.v_m4_interval i
WHERE i.site_uid = x.site_uid AND i.eligibility = 'eligible'
WHERE i.holder_uid = x.holder_uid AND i.eligibility = 'eligible'
AND daterange(x.applies_from, x.applies_to, '[]') @> daterange(i.start_on, i.end_on, '[]')) m4
WHERE x.topic = 'runtime_hours_per_day' AND NOT x.derived_from_fuel AND x.status IN ('proposed','accepted')
AND m4.days > 0;
-- Review entries whose fingerprint matches no live row: the value changed after it was reviewed,
-- or the subject was superseded or retracted. The site comes from every row the subject key has
-- had, in any status; for an asset's specification or engine total, from every site the asset was
-- placed at. So one entry can give one row per site, and none is lost when its subject leaves.
-- or the subject was superseded or retracted. The holder comes from every row the subject key has
-- had, in any status; for an asset's specification or engine total, from every holder the asset was
-- placed on; for a location, from every holder that has stood at the place. So one entry can give
-- one row per holder, and none is lost when its subject leaves.
CREATE VIEW fm.v_review_stale AS
WITH subj AS (
SELECT 'claim' AS kind, source_key AS subject_key, assertion_sha256, site_uid,
SELECT operator_uid, 'claim' AS kind, source_key AS subject_key, assertion_sha256, holder_uid,
claim_status IN ('asserted','reviewed','disputed') AS live
FROM fm.fuel_movement
UNION ALL SELECT 'claim', source_key, assertion_sha256, site_uid, claim_status IN ('asserted','reviewed','disputed')
UNION ALL SELECT operator_uid, 'claim', source_key, assertion_sha256, holder_uid, claim_status IN ('asserted','reviewed','disputed')
FROM fm.stock_observation
UNION ALL SELECT 'claim', h.source_key, h.assertion_sha256, m.site_uid, h.claim_status IN ('asserted','reviewed','disputed')
UNION ALL SELECT h.operator_uid, 'claim', h.source_key, h.assertion_sha256, m.holder_uid, h.claim_status IN ('asserted','reviewed','disputed')
FROM fm.hour_meter_reading h JOIN fm.runtime_meter m ON m.meter_id = h.meter_id
UNION ALL SELECT 'claim', source_key, assertion_sha256, site_uid, claim_status IN ('asserted','reviewed','disputed')
UNION ALL SELECT operator_uid, 'claim', source_key, assertion_sha256, holder_uid, claim_status IN ('asserted','reviewed','disputed')
FROM fm.electricity_bill
UNION ALL SELECT 'claim', source_key, assertion_sha256, site_uid, claim_status IN ('asserted','reviewed','disputed')
UNION ALL SELECT operator_uid, 'claim', source_key, assertion_sha256, holder_uid, claim_status IN ('asserted','reviewed','disputed')
FROM fm.imported_calculation
UNION ALL SELECT 'claim', e.source_key, e.assertion_sha256, p.site_uid, e.claim_status IN ('asserted','reviewed','disputed')
UNION ALL SELECT e.operator_uid, 'claim', e.source_key, e.assertion_sha256, p.holder_uid, e.claim_status IN ('asserted','reviewed','disputed')
FROM fm.equipment_spec e LEFT JOIN fm.asset_placement p ON p.asset_uid = e.asset_uid
UNION ALL SELECT 'claim', source_key, assertion_sha256, site_uid, claim_status IN ('asserted','reviewed','disputed')
UNION ALL SELECT operator_uid, 'claim', source_key, assertion_sha256, holder_uid, claim_status IN ('asserted','reviewed','disputed')
FROM fm.runtime_interval
UNION ALL SELECT 'claim', t.source_key, t.assertion_sha256, p.site_uid, t.claim_status IN ('asserted','reviewed','disputed')
UNION ALL SELECT t.operator_uid, 'claim', t.source_key, t.assertion_sha256, p.holder_uid, t.claim_status IN ('asserted','reviewed','disputed')
FROM fm.engine_fuel_total t LEFT JOIN fm.asset_placement p ON p.asset_uid = t.asset_uid
UNION ALL SELECT 'location', source_key, assertion_sha256, site_uid, acceptance IN ('pending','accepted','rejected')
FROM fm.site_location
UNION ALL SELECT 'assumption', code, fingerprint, site_uid, status <> 'superseded'
UNION ALL SELECT l.operator_uid, 'location', l.source_key, l.assertion_sha256, hp.holder_uid, l.acceptance IN ('pending','accepted','rejected')
FROM fm.place_location l LEFT JOIN fm.holder_place hp ON hp.place_uid = l.place_uid
UNION ALL SELECT operator_uid, 'assumption', code, fingerprint, holder_uid, status <> 'superseded'
FROM fm.assumption)
SELECT r.review_id, r.entry_key, r.subject_kind, r.subject_key, r.assertion_sha256, r.verdict,
r.reviewed_on, r.reviewer_role, st.site_uid
SELECT r.operator_uid, r.review_id, r.entry_key, r.subject_kind, r.subject_key, r.assertion_sha256, r.verdict,
r.reviewed_on, r.reviewer_role, st.holder_uid
FROM fm.review r
LEFT JOIN LATERAL (SELECT DISTINCT s.site_uid FROM subj s
WHERE s.kind = r.subject_kind AND s.subject_key = r.subject_key AND s.site_uid IS NOT NULL) st ON true
LEFT JOIN LATERAL (SELECT DISTINCT s.holder_uid FROM subj s
WHERE s.operator_uid = r.operator_uid AND s.kind = r.subject_kind
AND s.subject_key = r.subject_key AND s.holder_uid IS NOT NULL) st ON true
WHERE NOT EXISTS (SELECT 1 FROM subj l
WHERE l.live AND l.kind = r.subject_kind AND l.subject_key = r.subject_key
AND l.assertion_sha256 = r.assertion_sha256);
WHERE l.live AND l.operator_uid = r.operator_uid AND l.kind = r.subject_kind
AND l.subject_key = r.subject_key AND l.assertion_sha256 = r.assertion_sha256);
-- Visits kept for provenance only: every claim superseded or retracted.
CREATE VIEW fm.v_visit_provenance_only AS
@@ -18,16 +18,16 @@ SET LOCAL search_path = fm, pg_catalog, public;
-- insufficient_evidence: none of the above (today: every site, since there is no runtime).
CREATE VIEW fm.v_site_baseline_status AS
WITH cur AS (
SELECT DISTINCT ON (b.site_uid) b.*
SELECT DISTINCT ON (b.holder_uid) b.*
FROM fm.baseline_version b
WHERE b.status IN ('approved','proposed')
AND b.metric IN ('measured_generator_l_per_h','apparent_l_per_generator_h',
'apparent_l_per_set_h','apparent_l_per_site_dg_h') -- never modelled, never per-day
ORDER BY b.site_uid, (b.status = 'approved') DESC, b.approved_on DESC NULLS LAST, b.computed_at DESC),
ORDER BY b.holder_uid, (b.status = 'approved') DESC, b.approved_on DESC NULLS LAST, b.computed_at DESC),
ev AS (
SELECT site_uid, count(*) FILTER (WHERE eligibility <> 'excluded') AS n_usable
FROM fm.v_lph_interval GROUP BY site_uid)
SELECT s.site_uid,
SELECT holder_uid, count(*) FILTER (WHERE eligibility <> 'excluded') AS n_usable
FROM fm.v_lph_interval GROUP BY holder_uid)
SELECT s.holder_uid,
CASE WHEN cur.status = 'approved' THEN 'approved'
WHEN cur.status = 'proposed' AND cur.meets_policy AND val.inputs_unchanged THEN 'eligible'
WHEN cur.status = 'proposed' OR coalesce(ev.n_usable, 0) > 0 THEN 'provisional'
@@ -42,37 +42,45 @@ SELECT s.site_uid,
cur.metric AS lph_metric, cur.population, cur.window_start AS lph_window_start,
cur.window_end AS lph_window_end, cur.conditions, cur.decision_ref,
coalesce(ev.n_usable, 0) AS n_usable_intervals
FROM fm.site s LEFT JOIN cur USING (site_uid) LEFT JOIN ev USING (site_uid)
FROM fm.holder s LEFT JOIN cur USING (holder_uid) LEFT JOIN ev USING (holder_uid)
LEFT JOIN fm.v_baseline_validity val ON val.baseline_id = cur.baseline_id;
-- One row per site, every status: the map dataset and the single source of site-level fields.
-- "Current" means the period that contains today's date in Kabul, not "no end date".
-- One row per holder, every status: the map dataset and the single source of site-level fields.
-- Public column names stay site_* (a telecom site is a place with one fixed holder); internal fm uses
-- holder and place. "Current" means the period that contains today in the holder's local time.
CREATE VIEW public.v_site AS
WITH today AS (SELECT (now() AT TIME ZONE 'Asia/Kabul')::date AS d),
alias AS ( -- v1: the operator is ATOMA; later: the site's current operator
SELECT a.site_uid, a.label FROM fm.site_alias a, today
WHERE a.issuer = 'ATOMA' AND a.is_primary
AND daterange(coalesce(a.valid_from, '-infinity'::date), a.valid_to, '[)') @> today.d),
pp AS ( -- one programme, chosen explicitly
SELECT p.site_uid, p.status FROM fm.pilot_participation p, today
WHERE p.programme = 'ATOMA OFMS pilot' AND daterange(p.valid_from, p.valid_to, '[)') @> today.d),
-- alias and pp are at most one row per site: the exclusion constraints forbid overlapping primary
-- labels per (site, issuer) and overlapping periods per (site, programme).
loc AS (SELECT site_uid, latitude, longitude, basis, precision_m FROM fm.site_location WHERE acceptance = 'accepted'),
loc_any AS (SELECT site_uid, bool_or(acceptance = 'pending') AS has_pending FROM fm.site_location GROUP BY site_uid),
cal AS (SELECT site_uid, sum(consumed_l) AS fuel_l, sum(days) AS days
FROM fm.v_stock_interval WHERE stock_status = 'ok' GROUP BY site_uid),
latest AS (SELECT site_uid, max(visit_on) AS latest_record_on -- live records only: a fully
FROM fm.v_visit_live GROUP BY site_uid) -- retracted visit does not count
SELECT s.site_uid::varchar(12) AS site_uid,
coalesce(alias.label, s.site_uid)::varchar(20) AS site_code,
WITH alias AS ( -- the label the customer's people use
SELECT a.holder_uid, a.label FROM fm.holder_alias a
JOIN fm.v_holder_current z ON z.holder_uid = a.holder_uid
JOIN fm.operator o ON o.operator_uid = z.operator_uid
WHERE a.issuer = o.label_issuer AND a.is_primary
AND daterange(coalesce(a.valid_from, '-infinity'::date), a.valid_to, '[)') @> z.today),
pp AS ( -- the customer's one programme
SELECT p.holder_uid, p.status FROM fm.participation p
JOIN fm.v_holder_current z ON z.holder_uid = p.holder_uid
JOIN fm.operator o ON o.operator_uid = z.operator_uid
WHERE p.programme = o.programme AND daterange(p.valid_from, p.valid_to, '[)') @> z.today),
-- alias and pp are at most one row per holder: the exclusion constraints forbid overlapping primary
-- labels per (holder, issuer) and overlapping periods per (holder, programme).
loc AS (SELECT z.holder_uid, l.latitude, l.longitude, l.basis, l.precision_m
FROM fm.place_location l JOIN fm.v_holder_current z ON z.place_uid = l.place_uid
WHERE l.acceptance = 'accepted'),
loc_any AS (SELECT z.holder_uid, bool_or(l.acceptance = 'pending') AS has_pending
FROM fm.place_location l JOIN fm.v_holder_current z ON z.place_uid = l.place_uid GROUP BY z.holder_uid),
cal AS (SELECT holder_uid, sum(consumed_l) AS fuel_l, sum(days) AS days
FROM fm.v_stock_interval WHERE stock_status = 'ok' GROUP BY holder_uid),
latest AS (SELECT holder_uid, max(visit_on) AS latest_record_on -- live records only: a fully
FROM fm.v_visit_live GROUP BY holder_uid) -- retracted visit does not count
SELECT s.holder_uid::varchar(12) AS site_uid,
coalesce(alias.label, s.holder_uid)::varchar(20) AS site_code,
coalesce(s.name, 'name not recorded')::varchar(120) AS site_name,
(coalesce(alias.label, s.site_uid) || ' - ' || coalesce(s.name, 'name not recorded')
|| ' - ' || s.site_uid)::varchar(170) AS site_label, -- the one filter value
s.zone::varchar(40) AS zone, s.province::varchar(40) AS province, s.district::varchar(60) AS district,
(coalesce(alias.label, s.holder_uid) || ' - ' || coalesce(s.name, 'name not recorded')
|| ' - ' || s.holder_uid)::varchar(170) AS site_label, -- the one filter value
s.operator_uid::varchar(12) AS operator_uid,
pl.zone::varchar(40) AS zone, pl.province::varchar(40) AS province, pl.district::varchar(60) AS district,
coalesce(pp.status, 'none')::varchar(12) AS pilot_status,
loc.latitude::numeric(9,6) AS latitude, loc.longitude::numeric(9,6) AS longitude, -- NULL unless accepted
(CASE WHEN loc.site_uid IS NOT NULL THEN 'accepted'
(CASE WHEN loc.holder_uid IS NOT NULL THEN 'accepted'
WHEN loc_any.has_pending THEN 'awaiting acceptance' ELSE 'missing' END)::varchar(20) AS location_status,
loc.basis::varchar(20) AS location_basis, loc.precision_m::integer AS location_precision_m,
bs.baseline_status::varchar(24) AS baseline_status,
@@ -91,19 +99,22 @@ SELECT s.site_uid::varchar(12) AS site_ui
bs.decision_ref::varchar(12) AS baseline_decision_ref,
round(cal.fuel_l / NULLIF(cal.days, 0), 1)::numeric(7,1) AS fuel_l_per_calendar_day,
latest.latest_record_on
FROM fm.site s
LEFT JOIN alias USING (site_uid) LEFT JOIN loc USING (site_uid) LEFT JOIN loc_any USING (site_uid)
LEFT JOIN pp USING (site_uid) LEFT JOIN fm.v_site_baseline_status bs USING (site_uid)
LEFT JOIN cal USING (site_uid) LEFT JOIN latest USING (site_uid);
FROM fm.holder s
JOIN fm.v_holder_current z ON z.holder_uid = s.holder_uid
LEFT JOIN fm.place pl ON pl.place_uid = z.place_uid
LEFT JOIN alias ON alias.holder_uid = s.holder_uid LEFT JOIN loc ON loc.holder_uid = s.holder_uid
LEFT JOIN loc_any ON loc_any.holder_uid = s.holder_uid LEFT JOIN pp ON pp.holder_uid = s.holder_uid
LEFT JOIN fm.v_site_baseline_status bs ON bs.holder_uid = s.holder_uid
LEFT JOIN cal ON cal.holder_uid = s.holder_uid LEFT JOIN latest ON latest.holder_uid = s.holder_uid;
-- Every view below joins public.v_site for site_uid, site_code, site_name and site_label, so one
-- Every view below joins public.v_site for site_uid, site_code, site_name, site_label and operator_uid, so one
-- Site filter drives every dataset, and casts to plain types (varchar(n), integer,
-- numeric(p,s), date, timestamp(0), boolean). Live = asserted, reviewed or disputed.
-- S2, R3: one row per site and measure; the rows stay when a value is not established.
CREATE VIEW public.v_kpi_measures AS
WITH cur AS ( -- the site's L/h version; its snapshot only while approved or its inputs unchanged
SELECT bs.site_uid, b.baseline_id, b.code, b.fuel_l, b.runtime_h, b.matched_days, b.window_start, b.window_end,
SELECT bs.holder_uid, b.baseline_id, b.code, b.fuel_l, b.runtime_h, b.matched_days, b.window_start, b.window_end,
(b.status = 'approved' OR coalesce(val.inputs_unchanged, false)) AS shown,
' (' || b.status || ')'
|| CASE WHEN coalesce(val.inputs_unchanged, false) THEN ''
@@ -112,35 +123,35 @@ WITH cur AS ( -- the site's L/h version; its snapshot only while
FROM fm.v_site_baseline_status bs JOIN fm.baseline_version b ON b.code = bs.baseline_code
LEFT JOIN fm.v_baseline_validity val ON val.baseline_id = b.baseline_id),
ninp AS (SELECT baseline_id, count(*) FILTER (WHERE included) AS n FROM fm.baseline_input GROUP BY baseline_id),
m4 AS (SELECT site_uid, sum(consumed_l) AS fuel_l, sum(days) AS days, min(start_on) AS first_on, max(end_on) AS last_on
FROM fm.v_m4_interval WHERE eligibility = 'eligible' GROUP BY site_uid),
span AS (SELECT site_uid, min(visit_on) AS first_on, max(visit_on) AS last_on FROM fm.v_visit_live GROUP BY site_uid),
del AS (SELECT v.site_uid, count(*) AS n FROM fm.v_movement_event e JOIN fm.v_visit_live v ON v.visit_id = e.visit_id
WHERE e.kind = 'delivery' GROUP BY v.site_uid),
rdg AS (SELECT m.site_uid, count(*) AS n FROM fm.hour_meter_reading h JOIN fm.runtime_meter m ON m.meter_id = h.meter_id
WHERE h.claim_status IN ('asserted','reviewed','disputed') GROUP BY m.site_uid),
kwh AS (SELECT site_uid, sum(kwh) AS kwh, count(kwh) AS n_kwh, min(bill_month) AS first_m, max(bill_month) AS last_m
FROM fm.electricity_bill WHERE claim_status IN ('asserted','reviewed') GROUP BY site_uid), -- values never from disputed
m4 AS (SELECT holder_uid, sum(consumed_l) AS fuel_l, sum(days) AS days, min(start_on) AS first_on, max(end_on) AS last_on
FROM fm.v_m4_interval WHERE eligibility = 'eligible' GROUP BY holder_uid),
span AS (SELECT holder_uid, min(visit_on) AS first_on, max(visit_on) AS last_on FROM fm.v_visit_live GROUP BY holder_uid),
del AS (SELECT v.holder_uid, count(*) AS n FROM fm.v_movement_event e JOIN fm.v_visit_live v ON v.visit_id = e.visit_id
WHERE e.kind = 'delivery' GROUP BY v.holder_uid),
rdg AS (SELECT m.holder_uid, count(*) AS n FROM fm.hour_meter_reading h JOIN fm.runtime_meter m ON m.meter_id = h.meter_id
WHERE h.claim_status IN ('asserted','reviewed','disputed') GROUP BY m.holder_uid),
kwh AS (SELECT holder_uid, sum(kwh) AS kwh, count(kwh) AS n_kwh, min(bill_month) AS first_m, max(bill_month) AS last_m
FROM fm.electricity_bill WHERE claim_status IN ('asserted','reviewed') GROUP BY holder_uid), -- values never from disputed
lday AS ( -- the L/day reference: the latest approved, else proposed, L/day version
SELECT DISTINCT ON (b.site_uid) b.site_uid, b.code, b.status, b.value, b.window_start, b.window_end,
SELECT DISTINCT ON (b.holder_uid) b.holder_uid, b.code, b.status, b.value, b.window_start, b.window_end,
coalesce(val.inputs_unchanged, false) AS inputs_unchanged
FROM fm.baseline_version b LEFT JOIN fm.v_baseline_validity val ON val.baseline_id = b.baseline_id
WHERE b.metric = 'fuel_l_per_calendar_day' AND b.status IN ('approved','proposed')
ORDER BY b.site_uid, (b.status = 'approved') DESC, b.approved_on DESC NULLS LAST, b.computed_at DESC)
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
ORDER BY b.holder_uid, (b.status = 'approved') DESC, b.approved_on DESC NULLS LAST, b.computed_at DESC)
SELECT s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
x.ord::integer AS measure_order, x.measure::varchar(60) AS measure,
round(x.value, 2)::numeric(12,2) AS value, x.unit::varchar(12) AS unit,
(CASE WHEN x.value IS NULL THEN 'not established' ELSE x.kind END)::varchar(16) AS value_kind,
x.window_start, x.window_end, x.note::varchar(200) AS note
FROM public.v_site s
LEFT JOIN cur ON cur.site_uid = s.site_uid
LEFT JOIN cur ON cur.holder_uid = s.site_uid
LEFT JOIN ninp ON ninp.baseline_id = cur.baseline_id
LEFT JOIN m4 ON m4.site_uid = s.site_uid
LEFT JOIN span ON span.site_uid = s.site_uid
LEFT JOIN del ON del.site_uid = s.site_uid
LEFT JOIN rdg ON rdg.site_uid = s.site_uid
LEFT JOIN kwh ON kwh.site_uid = s.site_uid
LEFT JOIN lday ON lday.site_uid = s.site_uid
LEFT JOIN m4 ON m4.holder_uid = s.site_uid
LEFT JOIN span ON span.holder_uid = s.site_uid
LEFT JOIN del ON del.holder_uid = s.site_uid
LEFT JOIN rdg ON rdg.holder_uid = s.site_uid
LEFT JOIN kwh ON kwh.holder_uid = s.site_uid
LEFT JOIN lday ON lday.holder_uid = s.site_uid
CROSS JOIN LATERAL (VALUES
(1, 'Matched fuel', CASE WHEN cur.shown THEN cur.fuel_l END, 'L', 'calculated', cur.window_start, cur.window_end,
'fuel used in the intervals of L/h baseline version ' || cur.code || cur.state),
@@ -175,7 +186,7 @@ CROSS JOIN LATERAL (VALUES
-- R2: timer readings as recorded (no rows in v1).
CREATE VIEW public.v_hour_meter_evidence AS
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
SELECT s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
m.source_label::varchar(40) AS meter_label,
(CASE WHEN m.asset_uid IS NULL THEN 'unmapped'
ELSE m.asset_uid || ' (' || m.mapping_status || ')' END)::varchar(60) AS mapped_set,
@@ -186,13 +197,13 @@ SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
FROM fm.hour_meter_reading h
JOIN fm.runtime_meter m ON m.meter_id = h.meter_id
JOIN fm.visit v ON v.visit_id = h.visit_id
JOIN public.v_site s ON s.site_uid = m.site_uid
JOIN public.v_site s ON s.site_uid = m.holder_uid
LEFT JOIN fm.meter_epoch e ON e.epoch_id = h.epoch_id
WHERE h.claim_status IN ('asserted','reviewed','disputed');
-- B5: assumption versions in force or proposed; a coverage assumption lists its members.
CREATE VIEW public.v_assumption AS
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
SELECT s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
x.code::varchar(60) AS code, x.topic::varchar(30) AS topic, x.asset_uid::varchar(40) AS asset_uid,
x.value_num::numeric(10,3) AS value, x.low_num::numeric(10,3) AS value_low,
x.high_num::numeric(10,3) AS value_high, x.unit::varchar(20) AS unit,
@@ -201,7 +212,7 @@ SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
x.rationale::varchar(400) AS rationale, x.contradicted_by::varchar(200) AS contradicted_by,
mem.members::varchar(400) AS members
FROM fm.assumption x
JOIN public.v_site s ON s.site_uid = x.site_uid
JOIN public.v_site s ON s.site_uid = x.holder_uid
LEFT JOIN LATERAL (
SELECT string_agg(m.member_role || ': ' || coalesce(m.meter_id, m.asset_uid), '; '
ORDER BY m.member_role, coalesce(m.meter_id, m.asset_uid)) AS members
@@ -210,7 +221,7 @@ WHERE x.status <> 'superseded';
-- B1: baseline versions, with whether their inputs are unchanged today.
CREATE VIEW public.v_baseline_version AS
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
SELECT s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
b.code::varchar(60) AS code, b.metric::varchar(40) AS metric, b.status::varchar(12) AS status,
b.population::varchar(200) AS population,
round(b.value, 3)::numeric(8,3) AS value, round(b.range_low, 3)::numeric(8,3) AS range_low,
@@ -222,14 +233,15 @@ SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
b.evidence_grade::varchar(2) AS evidence_grade, b.conditions::varchar(200) AS conditions,
b.season_note::varchar(200) AS season_note, b.approved_on, b.approved_by_role::varchar(40) AS approved_by_role,
b.decision_ref::varchar(12) AS decision_ref, coalesce(val.inputs_unchanged, false) AS inputs_unchanged,
(b.computed_at AT TIME ZONE 'Asia/Kabul')::timestamp(0) AS computed_at
(b.computed_at AT TIME ZONE z.timezone)::timestamp(0) AS computed_at
FROM fm.baseline_version b
JOIN public.v_site s ON s.site_uid = b.site_uid
JOIN public.v_site s ON s.site_uid = b.holder_uid
JOIN fm.v_holder_current z ON z.holder_uid = b.holder_uid
LEFT JOIN fm.v_baseline_validity val ON val.baseline_id = b.baseline_id;
-- B2: the inputs of each version, with dependency drift since computation.
CREATE VIEW public.v_baseline_input AS
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
SELECT s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
b.code::varchar(60) AS baseline_code, b.status::varchar(12) AS baseline_status,
(to_char(bi.start_on, 'YYYY-MM-DD') || ' to ' || to_char(bi.end_on, 'YYYY-MM-DD'))::varchar(30) AS interval_label,
bi.start_on, bi.end_on, bi.included, bi.exclusion_reason::varchar(300) AS exclusion_reason,
@@ -241,42 +253,42 @@ SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
AND d.runtime_dep_sha256 IS DISTINCT FROM bi.runtime_dep_sha256)) AS dependencies_changed
FROM fm.baseline_input bi
JOIN fm.baseline_version b ON b.baseline_id = bi.baseline_id
JOIN public.v_site s ON s.site_uid = bi.site_uid
JOIN public.v_site s ON s.site_uid = bi.holder_uid
LEFT JOIN fm.v_interval_dependency d ON d.open_visit_id = bi.open_visit_id AND d.close_visit_id = bi.close_visit_id;
-- B7: documents the site's figures rest on (the document itself is never served).
CREATE VIEW public.v_evidence AS
WITH cites AS (
SELECT site_uid, source_sha256, 'fuel movement' AS record_type
SELECT holder_uid, source_sha256, 'fuel movement' AS record_type
FROM fm.fuel_movement WHERE claim_status IN ('asserted','reviewed','disputed')
UNION SELECT site_uid, source_sha256, 'stock reading'
UNION SELECT holder_uid, source_sha256, 'stock reading'
FROM fm.stock_observation WHERE claim_status IN ('asserted','reviewed','disputed')
UNION SELECT m.site_uid, h.source_sha256, 'timer reading'
UNION SELECT m.holder_uid, h.source_sha256, 'timer reading'
FROM fm.hour_meter_reading h JOIN fm.runtime_meter m ON m.meter_id = h.meter_id
WHERE h.claim_status IN ('asserted','reviewed','disputed')
UNION SELECT site_uid, source_sha256, 'electricity bill'
UNION SELECT holder_uid, source_sha256, 'electricity bill'
FROM fm.electricity_bill WHERE claim_status IN ('asserted','reviewed','disputed')
UNION SELECT site_uid, source_sha256, 'imported calculation'
FROM fm.imported_calculation WHERE site_uid IS NOT NULL AND claim_status IN ('asserted','reviewed','disputed')
UNION SELECT p.site_uid, e.source_sha256, 'equipment specification'
UNION SELECT holder_uid, source_sha256, 'imported calculation'
FROM fm.imported_calculation WHERE holder_uid IS NOT NULL AND claim_status IN ('asserted','reviewed','disputed')
UNION SELECT p.holder_uid, e.source_sha256, 'equipment specification'
FROM fm.equipment_spec e JOIN fm.asset_placement p ON p.asset_uid = e.asset_uid
WHERE e.source_sha256 IS NOT NULL AND e.claim_status IN ('asserted','reviewed','disputed')
UNION SELECT site_uid, source_sha256, 'assumption'
UNION SELECT holder_uid, source_sha256, 'assumption'
FROM fm.assumption WHERE source_sha256 IS NOT NULL AND status <> 'superseded')
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
SELECT s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
left(d.sha256, 16)::varchar(16) AS document_key, d.title::varchar(200) AS title,
d.kind::varchar(20) AS kind, d.issuer::varchar(80) AS issuer, d.document_date,
string_agg(c.record_type, ', ' ORDER BY c.record_type)::varchar(200) AS used_for,
d.contains_personal_data
FROM cites c
JOIN fm.source_document d ON d.sha256 = c.source_sha256
JOIN public.v_site s ON s.site_uid = c.site_uid
GROUP BY s.site_uid, s.site_code, s.site_name, s.site_label,
JOIN public.v_site s ON s.site_uid = c.holder_uid
GROUP BY s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
d.sha256, d.title, d.kind, d.issuer, d.document_date, d.contains_personal_data;
-- F1, F4: one row per delivery event; quantity NULL = not recorded. No monthly sums.
CREATE VIEW public.v_delivery AS
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
SELECT s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
e.event_key::varchar(80) AS event_key, v.visit_on AS delivery_date, v.seq_in_day::integer AS seq_in_day,
round(e.litres, 1)::numeric(8,1) AS delivered_l,
b.unit_price::numeric(8,2) AS unit_price, b.price_currency::varchar(8) AS currency,
@@ -284,19 +296,19 @@ SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
e.n_claims::integer AS n_sources, (e.spread_l > 0) AS sources_disagree, e.all_reviewed AS reviewed
FROM fm.v_movement_event e
JOIN fm.v_visit_live v ON v.visit_id = e.visit_id
JOIN public.v_site s ON s.site_uid = v.site_uid
JOIN public.v_site s ON s.site_uid = v.holder_uid
CROSS JOIN LATERAL ( -- price and reference of the best-ranked claim;
-- a disputed claim gives the reference only, never a price
SELECT CASE WHEN m.claim_status <> 'disputed' THEN m.unit_price END AS unit_price,
CASE WHEN m.claim_status <> 'disputed' THEN m.price_currency END AS price_currency, m.record_ref
FROM fm.fuel_movement m JOIN fm.v_source_rank r ON r.sha256 = m.source_sha256
WHERE m.event_key = e.event_key AND m.claim_status IN ('asserted','reviewed','disputed')
WHERE m.visit_id = e.visit_id AND m.event_key = e.event_key AND m.claim_status IN ('asserted','reviewed','disputed')
ORDER BY (m.claim_status = 'disputed'), r.rank, m.movement_id LIMIT 1) b
WHERE e.kind = 'delivery';
-- F2: stock readings as recorded, plus after-delivery points derived as before + delivered.
CREATE VIEW public.v_stock_reading AS
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
SELECT s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
v.visit_on AS observed_on, v.seq_in_day::integer AS seq_in_day,
(CASE st.phase WHEN 'before_delivery' THEN 'before delivery' WHEN 'after_delivery' THEN 'after delivery'
ELSE 'stock check' END)::varchar(20) AS reading_type,
@@ -304,20 +316,20 @@ SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
round(st.spread_l, 1)::numeric(8,1) AS spread_l, st.all_reviewed AS reviewed
FROM fm.v_visit_stock st
JOIN fm.v_visit_live v ON v.visit_id = st.visit_id
JOIN public.v_site s ON s.site_uid = v.site_uid
JOIN public.v_site s ON s.site_uid = v.holder_uid
UNION ALL
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
SELECT s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
p.visit_on, p.seq_in_day::integer, 'after delivery'::varchar(20),
round(p.depart_l, 1)::numeric(8,1), 'calculated'::varchar(12),
round(p.depart_spread_l, 1)::numeric(8,1), p.depart_reviewed
FROM fm.v_stock_point p
JOIN public.v_site s ON s.site_uid = p.site_uid
JOIN public.v_site s ON s.site_uid = p.holder_uid
WHERE p.depart_basis = 'derived';
-- F3, F4, B3, B4: one row per stock interval, with its L/h evaluation. The approved baseline is set
-- beside an interval only when both have the same metric and population (B4 compares like with like).
CREATE VIEW public.v_interval AS
WITH cur AS (SELECT bs.site_uid, bs.baseline_code, bs.baseline_status, bs.lph_estimate, b.metric, b.population_key
WITH cur AS (SELECT bs.holder_uid, bs.baseline_code, bs.baseline_status, bs.lph_estimate, b.metric, b.population_key
FROM fm.v_site_baseline_status bs LEFT JOIN fm.baseline_version b ON b.code = bs.baseline_code),
inb AS ( -- the inputs of each site's KPI version
SELECT bi.open_visit_id, bi.close_visit_id, bi.included
@@ -327,12 +339,12 @@ inb AS ( -- the inputs of each site's KPI vers
cond AS ( -- assumptions a valid interval depends on
SELECT l.interval_key, string_agg(DISTINCT x.code, ', ') AS codes
FROM fm.v_lph_interval l
JOIN fm.assumption x ON x.site_uid = l.site_uid AND x.status IN ('proposed','accepted')
JOIN fm.assumption x ON x.holder_uid = l.holder_uid AND x.status IN ('proposed','accepted')
AND x.topic IN ('generator_idle','meter_coverage')
AND daterange(x.applies_from, x.applies_to, '[]') @> daterange(l.start_on, l.end_on, '[]')
WHERE l.population_valid
GROUP BY l.interval_key)
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
SELECT s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
(to_char(l.start_on, 'YYYY-MM-DD') || ' to ' || to_char(l.end_on, 'YYYY-MM-DD'))::varchar(30) AS interval_label,
l.start_on, l.end_on, l.days::integer AS days,
round(l.opening_l, 1)::numeric(9,1) AS opening_l, l.opening_basis::varchar(8) AS opening_basis,
@@ -357,15 +369,15 @@ SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
AND l.population_key = cur.population_key
THEN cur.lph_estimate END)::numeric(8,3) AS current_baseline_l_per_h
FROM fm.v_lph_interval l
JOIN public.v_site s ON s.site_uid = l.site_uid
JOIN cur ON cur.site_uid = l.site_uid
JOIN public.v_site s ON s.site_uid = l.holder_uid
JOIN cur ON cur.holder_uid = l.holder_uid
LEFT JOIN inb ON inb.open_visit_id = l.open_visit_id AND inb.close_visit_id = l.close_visit_id
LEFT JOIN cond ON cond.interval_key = l.interval_key;
-- R1: hours per day by meter interval (no rows in v1). Every pair is listed with its flags; runtime
-- and h/day only for a valid pair (one epoch, not decreasing, at most 24 h per day).
CREATE VIEW public.v_runtime AS
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
SELECT s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
m.source_label::varchar(40) AS meter_label,
(CASE WHEN m.asset_uid IS NULL THEN 'unmapped'
ELSE m.asset_uid || ' (' || m.mapping_status || ')' END)::varchar(60) AS mapped_set,
@@ -378,55 +390,56 @@ SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
FROM fm.v_runtime_pair r
CROSS JOIN LATERAL (SELECT coalesce(r.same_epoch AND r.monotonic AND NOT r.over_24h_per_day, false) AS pair_valid) v
JOIN fm.runtime_meter m ON m.meter_id = r.meter_id
JOIN public.v_site s ON s.site_uid = r.site_uid;
JOIN public.v_site s ON s.site_uid = r.holder_uid;
-- S3 (Gantt): which periods each kind of evidence covers; Kabul midnight as timestamp(0).
-- S3 (Gantt): which periods each kind of evidence covers; local midnight as timestamp(0).
CREATE VIEW public.v_coverage AS
WITH spans AS (
SELECT site_uid, '1 Stock intervals' AS lane, stock_status AS status, start_on, end_on
SELECT holder_uid, '1 Stock intervals' AS lane, stock_status AS status, start_on, end_on
FROM fm.v_stock_interval
UNION ALL
SELECT r.site_uid, '2 Runtime: ' || m.source_label,
SELECT r.holder_uid, '2 Runtime: ' || m.source_label,
CASE WHEN r.same_epoch AND r.monotonic AND NOT r.over_24h_per_day THEN 'ok' ELSE 'flagged' END,
r.start_on, r.end_on
FROM fm.v_runtime_pair r JOIN fm.runtime_meter m ON m.meter_id = r.meter_id
UNION ALL
SELECT site_uid, '3 Baseline window', baseline_status, lph_window_start, lph_window_end
SELECT holder_uid, '3 Baseline window', baseline_status, lph_window_start, lph_window_end
FROM fm.v_site_baseline_status WHERE lph_window_start IS NOT NULL
UNION ALL
SELECT site_uid, '4 Electricity bills',
SELECT holder_uid, '4 Electricity bills',
CASE WHEN claim_status = 'disputed' THEN 'disputed' WHEN kwh IS NULL THEN 'cost only' ELSE 'kWh' END,
coalesce(period_start, bill_month), coalesce(period_end, (bill_month + interval '1 month')::date)
FROM fm.electricity_bill WHERE claim_status IN ('asserted','reviewed','disputed'))
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
SELECT s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
x.lane::varchar(60) AS lane, x.status::varchar(40) AS status, x.start_on, x.end_on,
x.start_on::timestamp(0) AS start_ts, x.end_on::timestamp(0) AS end_ts
FROM spans x JOIN public.v_site s ON s.site_uid = x.site_uid;
FROM spans x JOIN public.v_site s ON s.site_uid = x.holder_uid;
-- B9: stock-balance fuel against engine-measured fuel inside the interval. No rows until flow-meter
-- data exist. The window runs from the opening visit's departure to the closing visit's arrival
-- (Kabul time) when both times are recorded, else from midnight to midnight. covered_h = the hours
-- (the holder's local time) when both times are recorded, else from midnight to midnight. covered_h = the hours
-- of that window the generator's records cover (live records never overlap per asset). The
-- difference is shown only for an `ok` stock interval with both times recorded, in which every
-- generator is placed at the site throughout (one placement spans the interval, as in
-- fm.v_lph_interval) and covered for the whole window.
CREATE VIEW public.v_reconciliation AS
WITH gen AS ( -- one row per interval and generator placed at the site at any time in it
SELECT l.interval_key, l.site_uid, l.start_on, l.end_on, l.consumed_l, l.stock_status, p.asset_uid,
SELECT l.interval_key, l.holder_uid, l.start_on, l.end_on, l.consumed_l, l.stock_status, p.asset_uid,
bool_or(daterange(coalesce(p.occupied_from, '-infinity'::date), p.occupied_to, '[)')
@> daterange(l.start_on, l.end_on, '[]')) AS placed_throughout,
(o.time_out_local IS NOT NULL AND c.time_in_local IS NOT NULL) AS times_known,
(l.start_on + coalesce(o.time_out_local, '00:00'::time)) AT TIME ZONE 'Asia/Kabul' AS t0,
(l.end_on + coalesce(c.time_in_local, '00:00'::time)) AT TIME ZONE 'Asia/Kabul' AS t1
(l.start_on + coalesce(o.time_out_local, '00:00'::time)) AT TIME ZONE z.timezone AS t0,
(l.end_on + coalesce(c.time_in_local, '00:00'::time)) AT TIME ZONE z.timezone AS t1
FROM fm.v_stock_interval l
JOIN fm.visit o ON o.visit_id = l.open_visit_id
JOIN fm.visit c ON c.visit_id = l.close_visit_id
JOIN fm.asset_placement p ON p.site_uid = l.site_uid
JOIN fm.v_holder_current z ON z.holder_uid = l.holder_uid
JOIN fm.asset_placement p ON p.holder_uid = l.holder_uid
JOIN fm.asset a ON a.asset_uid = p.asset_uid AND a.kind = 'generator'
WHERE daterange(coalesce(p.occupied_from, '-infinity'::date), p.occupied_to, '[)')
&& daterange(l.start_on, l.end_on, '[]')
GROUP BY l.interval_key, l.site_uid, l.start_on, l.end_on, l.consumed_l, l.stock_status, p.asset_uid,
o.time_out_local, c.time_in_local),
GROUP BY l.interval_key, l.holder_uid, l.start_on, l.end_on, l.consumed_l, l.stock_status, p.asset_uid,
o.time_out_local, c.time_in_local, z.timezone),
eng AS (
SELECT g.interval_key, g.asset_uid, sum(t.net_fuel_l) AS engine_l,
sum(extract(epoch FROM t.ended_at - t.started_at)) / 3600 AS covered_h,
@@ -440,7 +453,7 @@ tot AS (
count(*) FILTER (WHERE g.placed_throughout AND e.full_cover) AS n_complete, sum(e.engine_l) AS engine_all_l
FROM gen g LEFT JOIN eng e ON e.interval_key = g.interval_key AND e.asset_uid = g.asset_uid
GROUP BY g.interval_key)
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
SELECT s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
(to_char(g.start_on, 'YYYY-MM-DD') || ' to ' || to_char(g.end_on, 'YYYY-MM-DD'))::varchar(30) AS interval_label,
g.start_on, g.end_on, g.stock_status::varchar(40) AS stock_status, g.times_known AS endpoint_times_known,
g.asset_uid::varchar(40) AS asset_uid, g.placed_throughout,
@@ -456,60 +469,60 @@ SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
FROM gen g
JOIN eng e ON e.interval_key = g.interval_key AND e.asset_uid = g.asset_uid
JOIN tot t ON t.interval_key = g.interval_key
JOIN public.v_site s ON s.site_uid = g.site_uid;
JOIN public.v_site s ON s.site_uid = g.holder_uid;
-- B6: one row per site and check; rows stay when ok.
CREATE VIEW public.v_data_quality AS
WITH c AS (
SELECT s.site_uid,
(SELECT count(*) FROM fm.stock_observation o WHERE o.site_uid = s.site_uid
SELECT s.holder_uid,
(SELECT count(*) FROM fm.stock_observation o WHERE o.holder_uid = s.holder_uid
AND o.litres_basis = 'placeholder_zero' AND o.claim_status IN ('asserted','reviewed','disputed')) AS n_placeholder,
(SELECT count(*) FROM fm.v_movement_event e JOIN fm.visit v ON v.visit_id = e.visit_id
WHERE v.site_uid = s.site_uid AND e.spread_l > 0)
WHERE v.holder_uid = s.holder_uid AND e.spread_l > 0)
+ (SELECT count(*) FROM fm.v_visit_stock st JOIN fm.visit v ON v.visit_id = st.visit_id
WHERE v.site_uid = s.site_uid AND st.spread_l > 0) AS n_conflicts,
WHERE v.holder_uid = s.holder_uid AND st.spread_l > 0) AS n_conflicts,
(SELECT count(*) FROM fm.hour_meter_reading h JOIN fm.runtime_meter m ON m.meter_id = h.meter_id
WHERE m.site_uid = s.site_uid AND h.claim_status IN ('asserted','reviewed','disputed')) AS n_readings,
(SELECT count(*) FROM fm.runtime_meter m WHERE m.site_uid = s.site_uid AND m.mapping_status <> 'confirmed') AS n_unmapped,
WHERE m.holder_uid = s.holder_uid AND h.claim_status IN ('asserted','reviewed','disputed')) AS n_readings,
(SELECT count(*) FROM fm.runtime_meter m WHERE m.holder_uid = s.holder_uid AND m.mapping_status <> 'confirmed') AS n_unmapped,
(SELECT count(*) FROM fm.asset_placement p JOIN fm.asset a ON a.asset_uid = p.asset_uid
WHERE p.site_uid = s.site_uid AND a.kind = 'generator'
WHERE p.holder_uid = s.holder_uid AND a.kind = 'generator'
AND daterange(coalesce(p.occupied_from, '-infinity'::date), p.occupied_to, '[)')
@> (now() AT TIME ZONE 'Asia/Kabul')::date
@> (SELECT z.today FROM fm.v_holder_current z WHERE z.holder_uid = s.holder_uid)
AND NOT EXISTS (SELECT 1 FROM fm.runtime_meter m JOIN fm.hour_meter_reading h ON h.meter_id = m.meter_id
WHERE m.asset_uid = p.asset_uid AND m.mapping_status = 'confirmed'
AND h.claim_status IN ('asserted','reviewed'))) AS n_gen_no_readings,
(SELECT count(*) FROM fm.fuel_movement x WHERE x.site_uid = s.site_uid AND x.claim_status = 'asserted')
+ (SELECT count(*) FROM fm.stock_observation x WHERE x.site_uid = s.site_uid AND x.claim_status = 'asserted')
(SELECT count(*) FROM fm.fuel_movement x WHERE x.holder_uid = s.holder_uid AND x.claim_status = 'asserted')
+ (SELECT count(*) FROM fm.stock_observation x WHERE x.holder_uid = s.holder_uid AND x.claim_status = 'asserted')
+ (SELECT count(*) FROM fm.hour_meter_reading x JOIN fm.runtime_meter m ON m.meter_id = x.meter_id
WHERE m.site_uid = s.site_uid AND x.claim_status = 'asserted')
+ (SELECT count(*) FROM fm.electricity_bill x WHERE x.site_uid = s.site_uid AND x.claim_status = 'asserted') AS n_unreviewed,
(SELECT count(*) FROM fm.v_stock_point p WHERE p.site_uid = s.site_uid
WHERE m.holder_uid = s.holder_uid AND x.claim_status = 'asserted')
+ (SELECT count(*) FROM fm.electricity_bill x WHERE x.holder_uid = s.holder_uid AND x.claim_status = 'asserted') AS n_unreviewed,
(SELECT count(*) FROM fm.v_stock_point p WHERE p.holder_uid = s.holder_uid
AND abs(p.after_vs_before_plus_delivered_l) > 0.5) AS n_after_mismatch,
(SELECT count(*) FROM fm.v_stock_interval i WHERE i.site_uid = s.site_uid
(SELECT count(*) FROM fm.v_stock_interval i WHERE i.holder_uid = s.holder_uid
AND i.stock_status IN ('negative','same_day')) AS n_neg_same_day,
(SELECT count(*) FROM fm.v_stock_interval i WHERE i.site_uid = s.site_uid
(SELECT count(*) FROM fm.v_stock_interval i WHERE i.holder_uid = s.holder_uid
AND i.stock_status NOT IN ('ok','negative','same_day')) AS n_gaps,
(SELECT count(*) FROM fm.electricity_bill b WHERE b.site_uid = s.site_uid
(SELECT count(*) FROM fm.electricity_bill b WHERE b.holder_uid = s.holder_uid
AND b.claim_status IN ('asserted','reviewed','disputed')) AS n_bills,
(SELECT count(*) FROM fm.electricity_bill b WHERE b.site_uid = s.site_uid
(SELECT count(*) FROM fm.electricity_bill b WHERE b.holder_uid = s.holder_uid
AND (b.kwh IS NULL OR b.claim_status = 'disputed')
AND b.claim_status IN ('asserted','reviewed','disputed')) AS n_bills_no_kwh, -- no usable kWh
(SELECT count(*) FROM fm.fuel_movement m WHERE m.site_uid = s.site_uid
(SELECT count(*) FROM fm.fuel_movement m WHERE m.holder_uid = s.holder_uid
AND m.kind IN ('withdrawal','drain','spill','loss_reported','internal_transfer')
AND m.claim_status IN ('asserted','reviewed','disputed')) AS n_transfer_records,
(SELECT count(*) FROM fm.baseline_version b JOIN fm.v_baseline_validity v ON v.baseline_id = b.baseline_id
WHERE b.site_uid = s.site_uid AND b.status IN ('proposed','approved') AND NOT v.inputs_unchanged) AS n_drift,
(SELECT count(*) FROM fm.v_review_stale r WHERE r.site_uid = s.site_uid) AS n_stale,
(SELECT count(*) FROM fm.v_visit_provenance_only v WHERE v.site_uid = s.site_uid) AS n_provenance_only,
(SELECT count(*) FROM fm.v_lph_interval l WHERE l.site_uid = s.site_uid
WHERE b.holder_uid = s.holder_uid AND b.status IN ('proposed','approved') AND NOT v.inputs_unchanged) AS n_drift,
(SELECT count(*) FROM fm.v_review_stale r WHERE r.holder_uid = s.holder_uid) AS n_stale,
(SELECT count(*) FROM fm.v_visit_provenance_only v WHERE v.holder_uid = s.holder_uid) AS n_provenance_only,
(SELECT count(*) FROM fm.v_lph_interval l WHERE l.holder_uid = s.holder_uid
AND l.exclusion_reasons LIKE '%confirmed for a generator not placed here%') AS n_unplaced_counter,
(SELECT count(*) FROM fm.v_lph_interval l WHERE l.site_uid = s.site_uid
(SELECT count(*) FROM fm.v_lph_interval l WHERE l.holder_uid = s.holder_uid
AND l.exclusion_reasons LIKE '%meter-coverage assumption % does not name%') AS n_cov_mismatch
FROM fm.site s)
SELECT vs.site_uid, vs.site_code, vs.site_name, vs.site_label,
FROM fm.holder s)
SELECT vs.site_uid, vs.site_code, vs.site_name, vs.site_label, vs.operator_uid,
x.ord::integer AS check_order, x.code::varchar(40) AS check_code, x.label::varchar(80) AS label,
x.status::varchar(10) AS status, x.detail::varchar(300) AS detail
FROM c JOIN public.v_site vs ON vs.site_uid = c.site_uid
FROM c JOIN public.v_site vs ON vs.site_uid = c.holder_uid
CROSS JOIN LATERAL (VALUES
(1, 'placeholder_rows', 'Placeholder rows',
CASE WHEN c.n_placeholder > 0 THEN 'warning' ELSE 'ok' END,
@@ -564,8 +577,8 @@ CROSS JOIN LATERAL (VALUES
-- B10: specifications, labelled as such; live rows only. A disputed specification is listed with its
-- status and without its value.
CREATE VIEW public.v_spec_reference AS
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
p.slot_label::varchar(20) AS slot, e.asset_uid::varchar(40) AS asset_uid, a.kind::varchar(10) AS asset_kind,
SELECT s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
p.slot_label::varchar(20) AS slot, e.asset_uid::varchar(40) AS asset_uid, a.kind::varchar(20) AS asset_kind,
concat_ws(' ', a.make, a.model)::varchar(120) AS make_model,
e.attribute::varchar(40) AS attribute, e.load_pct::numeric(5,1) AS load_pct,
(CASE WHEN e.claim_status <> 'disputed' THEN e.value_num END)::numeric(12,3) AS value,
@@ -575,14 +588,14 @@ SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
'specification, not measured'::varchar(30) AS value_kind, e.claim_status::varchar(12) AS claim_status
FROM fm.equipment_spec e
JOIN fm.asset a ON a.asset_uid = e.asset_uid
JOIN (SELECT DISTINCT ON (asset_uid, site_uid) asset_uid, site_uid, slot_label
FROM fm.asset_placement ORDER BY asset_uid, site_uid, occupied_from DESC NULLS LAST) p ON p.asset_uid = e.asset_uid
JOIN public.v_site s ON s.site_uid = p.site_uid
JOIN (SELECT DISTINCT ON (asset_uid, holder_uid) asset_uid, holder_uid, slot_label
FROM fm.asset_placement ORDER BY asset_uid, holder_uid, occupied_from DESC NULLS LAST) p ON p.asset_uid = e.asset_uid
JOIN public.v_site s ON s.site_uid = p.holder_uid
WHERE e.claim_status IN ('asserted','reviewed','disputed');
-- M3 sensitivity table, reference tab only (shown only once Q9 approves a runtime assumption).
CREATE VIEW public.v_modelled_lph AS
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
SELECT s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
m.code::varchar(60) AS assumption_code, m.status::varchar(12) AS status, m.basis::varchar(20) AS basis,
m.h_per_day::numeric(5,2) AS h_per_day, m.h_per_day_low::numeric(5,2) AS h_per_day_low,
m.h_per_day_high::numeric(5,2) AS h_per_day_high, m.window_start, m.window_end,
@@ -591,31 +604,36 @@ SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
round(m.modelled_l_per_h_low, 2)::numeric(6,2) AS modelled_l_per_h_low,
round(m.modelled_l_per_h_high, 2)::numeric(6,2) AS modelled_l_per_h_high,
'modelled: L per calendar day / assumed runtime; not measured'::varchar(70) AS value_kind
FROM fm.v_modelled_lph m JOIN public.v_site s ON s.site_uid = m.site_uid;
FROM fm.v_modelled_lph m JOIN public.v_site s ON s.site_uid = m.holder_uid;
-- R4: electricity cost per bill month; cost context only, never joined into fuel metrics. A
-- disputed bill stays listed, without amount or kWh.
CREATE VIEW public.v_electricity_cost AS
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
SELECT s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
b.bill_month, b.period_start, b.period_end, (b.period_start IS NOT NULL) AS period_known,
(CASE WHEN b.claim_status <> 'disputed' THEN b.amount END)::numeric(12,2) AS amount, b.currency::varchar(8) AS currency,
b.currency_basis::varchar(10) AS currency_basis,
(CASE WHEN b.claim_status <> 'disputed' THEN b.kwh END)::numeric(10,1) AS kwh,
b.record_ref::varchar(60) AS source_ref, b.claim_status::varchar(12) AS claim_status
FROM fm.electricity_bill b JOIN public.v_site s ON s.site_uid = b.site_uid
FROM fm.electricity_bill b JOIN public.v_site s ON s.site_uid = b.holder_uid
WHERE b.claim_status IN ('asserted','reviewed','disputed');
-- B8: operator workbook calculations and our assessment; live rows only. A disputed transcription is
-- listed with its status and without its formula or value.
CREATE VIEW public.v_imported_calculation AS
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
SELECT s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
c.cell_ref::varchar(40) AS cell_ref, c.source_label::varchar(80) AS source_label,
(CASE WHEN c.claim_status <> 'disputed' THEN c.formula END)::varchar(200) AS formula,
(CASE WHEN c.claim_status <> 'disputed' THEN c.cached_value END)::numeric(14,6) AS cached_value,
c.unit_as_labelled::varchar(30) AS unit_as_labelled, c.our_reading::varchar(300) AS our_reading,
c.baseline_use::varchar(12) AS baseline_use, c.exclusion_reason::varchar(300) AS exclusion_reason,
left(c.source_sha256, 16)::varchar(16) AS document_key, c.claim_status::varchar(12) AS claim_status
FROM fm.imported_calculation c JOIN public.v_site s ON s.site_uid = c.site_uid
FROM fm.imported_calculation c JOIN public.v_site s ON s.site_uid = c.holder_uid
WHERE c.claim_status IN ('asserted','reviewed','disputed');
-- Which Keycloak group is which customer (PD-21). Only server/ reads this, at sign-in; it holds no
-- site data, and no dashboard has a dataset on it.
CREATE VIEW public.v_operator AS
SELECT operator_uid, name, keycloak_group FROM fm.operator;
COMMIT;
@@ -0,0 +1,91 @@
-- 0004_licence.sql: customer licences (PD-20) and plan prices (PD-19), in schema `licence`.
-- DRAFT, not applied. Applied after 0003 by a loader acting as fuel_owner (README in this folder).
-- fuel_ro has no access: its default privileges cover `public` only. The server gets its own role
-- when it starts checking licences. No customer's licence is seeded here; our admins write those.
BEGIN;
CREATE SCHEMA licence AUTHORIZATION fuel_owner;
SET LOCAL search_path = licence, fm, pg_catalog, public; -- fm holds btree_gist
CREATE TABLE licence.plan (
plan_code text PRIMARY KEY,
name text NOT NULL
);
CREATE TABLE licence.plan_price ( -- per sensor per month (PD-19); a change is a new dated row
plan_code text NOT NULL REFERENCES licence.plan,
payment_term text NOT NULL CHECK (payment_term IN ('annual','monthly')),
currency text NOT NULL CHECK (currency ~ '^[A-Z]{3}$'),
price numeric(8,2) NOT NULL CHECK (price >= 0),
valid_from date NOT NULL, valid_to date, -- NULL to = still valid
EXCLUDE USING gist (plan_code WITH =, payment_term WITH =, currency WITH =,
daterange(valid_from, valid_to, '[)') WITH &&)
);
CREATE TABLE licence.plan_entitlement (
plan_code text NOT NULL REFERENCES licence.plan,
feature text NOT NULL,
limit_value integer, -- NULL: no limit
PRIMARY KEY (plan_code, feature)
);
CREATE TABLE licence.licence ( -- a renewal is a new row
licence_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
operator_uid text NOT NULL REFERENCES fm.operator,
plan_code text NOT NULL REFERENCES licence.plan,
payment_term text NOT NULL CHECK (payment_term IN ('annual','monthly')),
valid_from date NOT NULL,
valid_until date NOT NULL, -- last valid day
grace_until date NOT NULL, -- last day of grace; alarms stop after it
status text NOT NULL CHECK (status IN ('active','suspended')),
contract_ref text NOT NULL, -- the signed agreement or amendment the dates come from
created_by text NOT NULL, -- a role, never a person's name
created_at timestamptz NOT NULL DEFAULT now(),
CHECK (valid_until >= valid_from),
CHECK (grace_until >= valid_until),
EXCLUDE USING gist (operator_uid WITH =, daterange(valid_from, valid_until, '[]') WITH &&)
);
CREATE TABLE licence.licence_override (
licence_id bigint NOT NULL REFERENCES licence.licence,
feature text NOT NULL,
limit_value integer, -- NULL: no limit
PRIMARY KEY (licence_id, feature)
);
-- Prices decided on 30 Oct 2026 (PD-19). The pilot plan costs nothing on either term.
INSERT INTO licence.plan (plan_code, name) VALUES ('pilot', 'Pilot'), ('standard', 'Standard');
INSERT INTO licence.plan_price (plan_code, payment_term, currency, price, valid_from) VALUES
('pilot', 'annual', 'USD', 0.00, '2026-01-01'),
('pilot', 'monthly', 'USD', 0.00, '2026-01-01'),
('standard', 'annual', 'USD', 15.00, '2026-10-30'),
('standard', 'monthly', 'USD', 17.00, '2026-10-30');
-- Every change is logged (PD-20), and the log is never changed.
CREATE TABLE licence.licence_event (
event_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
table_name text NOT NULL,
action text NOT NULL CHECK (action IN ('INSERT','UPDATE','DELETE')),
old_row jsonb, new_row jsonb,
changed_by text NOT NULL DEFAULT current_user,
changed_at timestamptz NOT NULL DEFAULT now()
);
CREATE FUNCTION licence.tg_log_change() RETURNS trigger LANGUAGE plpgsql AS $$
BEGIN
INSERT INTO licence.licence_event (table_name, action, old_row, new_row)
VALUES (TG_TABLE_NAME, TG_OP,
CASE WHEN TG_OP <> 'INSERT' THEN to_jsonb(OLD) END,
CASE WHEN TG_OP <> 'DELETE' THEN to_jsonb(NEW) END);
RETURN NULL;
END $$;
DO $$
DECLARE t text;
BEGIN
FOREACH t IN ARRAY ARRAY['plan','plan_price','plan_entitlement','licence','licence_override'] LOOP
EXECUTE format('CREATE TRIGGER log_change AFTER INSERT OR UPDATE OR DELETE ON licence.%I
FOR EACH ROW EXECUTE FUNCTION licence.tg_log_change()', t);
END LOOP;
END $$;
CREATE FUNCTION licence.tg_event_immutable() RETURNS trigger LANGUAGE plpgsql AS $$
BEGIN
RAISE EXCEPTION 'licence_event: the log is never changed';
END $$;
CREATE TRIGGER event_immutable BEFORE UPDATE OR DELETE ON licence.licence_event
FOR EACH ROW EXECUTE FUNCTION licence.tg_event_immutable();
COMMIT;
@@ -12,30 +12,48 @@ The design and its rationale live in the fuel-monitoring corpus:
| File | Creates | Readable by `fuel_ro` (Superset, Petrolyst server) |
|---|---|---|
| `0001_fm_core.sql` | schema `fm`, `btree_gist` in it, 25 tables, constraints, trigger-backed rules 1–13 | no |
| `0001_fm_core.sql` | schema `fm`, `btree_gist` in it, 29 tables (customers, assets, places and holders, claims, reviews, baselines), constraints, trigger-backed rules 1–15 | no |
| `0002_fm_derivations.sql` | internal views in `fm`: claims → visits → stock intervals → L/h and L/day intervals, dependency hashes, baseline validity | no |
| `0003_bi_views.sql` | `fm.v_site_baseline_status` and the 18 `public.v_*` views Superset reads | yes, read-only |
| `0003_bi_views.sql` | `fm.v_site_baseline_status` and the 19 `public.v_*` views Superset reads | yes, read-only |
| `0004_licence.sql` | schema `licence`: plans, prices, entitlements, customer licences, overrides and their change log (PD-19, PD-20). Holds no customer's licence | no |
Each file is one transaction. Apply them in order.
**The shape (PD-16, PD-18).** Every row in `fm` carries its customer, `operator_uid`, except
`asset_kind`. A child names its parent and the customer together in a composite foreign key, so it
cannot point at another customer's parent. A site is a `place` with one fixed `holder`; holders keep
their `SITE-0001` ids and places get `PLC-0001`. Internal `fm` speaks of holder and place. The public
views keep their `site_*` column names, because dashboards, the app and the server read them; each
gains `operator_uid`. They get renamed, or holder-named siblings, with the first fleet customer.
**Runtime is generator-only.** An engine can be a placed asset, but rules 8, 9 and 12 refuse a
meter or idle assumption on one: the runtime and L/h calculations have not been checked on engines.
**Who depends on the views.** Superset's dashboards list every column (`petrolyst-insights/dashboards/atoma/columns.tsv`;
the import reports drift). The Petrolyst server reads `public.v_site`; its integration test
(petrolyst-platform `server/tests/test_integration.py`) names the columns and types it needs. After
a change to `0003_bi_views.sql`, run both checks. The migrations stay in this repository (open
point 8, closed 30 Oct 2026).
## Applying (on the server)
1. Create a loader role as in the repository README, "Data loaders". Its sessions act as
`fuel_owner`, so every object belongs to `fuel_owner`. `fuel_ro` can then read the `public`
views through the existing default privileges, with no role or grant changes.
2. Open the tunnel and apply the three files as the loader:
2. Open the tunnel and apply the four files as the loader:
```sh
ssh -L 5433:127.0.0.1:5433 petrolyst
for f in 0001_fm_core 0002_fm_derivations 0003_bi_views; do
for f in 0001_fm_core 0002_fm_derivations 0003_bi_views 0004_licence; do
psql -h localhost -p 5433 -U <loader> -d fuel -v ON_ERROR_STOP=1 -f "$f.sql" || break
done
```
3. Check as `fuel_ro`
(`docker exec -it petrolyst-fuel-db-1 psql -U postgres -d fuel`, then `SET ROLE fuel_ro;`):
- `\dv public.*` lists the 18 `v_*` views;
- `SELECT count(*) FROM fm.site;` fails with "permission denied for schema fm".
- `\dv public.*` lists the 19 `v_*` views;
- `SELECT count(*) FROM fm.holder;` fails with "permission denied for schema fm";
- `SELECT count(*) FROM licence.plan;` fails with "permission denied for schema licence".
The `public` views are owned by `fuel_owner` and run with its rights, so `fuel_ro` never needs,
and never gets, read access to `fm`. PostgreSQL's catalog still shows every schema's table and
@@ -60,14 +78,15 @@ column names to any role, so Superset's SQL Lab can list fm's names; names only,
1. Baseline versions start as draft or proposed. Approval needs `meets_policy` and unchanged
inputs. Approved rows and their inputs are immutable; the only change is approved →
superseded. An input never moves to another version.
2. An internal transfer's tanks are placed at the movement's site on the visit date.
3. A claim's site is its visit's site, and a reading's meter belongs to that site.
2. An internal transfer's tanks are placed on the movement's holder on the visit date.
3. A claim's holder is its visit's holder, and a reading's meter belongs to that holder.
4. A reading's epoch belongs to its meter.
5. A runtime assumption derived from fuel cannot be accepted. This is a CHECK constraint.
6. A baseline input's dates equal its endpoint visits. Included inputs carry the version's
population.
7. A source document supersedes only one of its own family.
8. A confirmed or proposed meter mapping names a generator placed at the meter's site.
7. A source document supersedes only one of its own family. The same document under a second
customer is refused until document sharing exists.
8. A confirmed or proposed meter mapping names a generator placed on the meter's holder.
9. Meter-coverage members:
- valid when written, and written in the assumption's own transaction;
- immutable afterwards;
@@ -75,51 +94,34 @@ column names to any role, so Superset's SQL Lab can list fm's names; names only,
10. The live claims of one `event_key` share visit, kind and tanks. A superseded or retracted
claim keeps its own, so a correction can change them.
11. Status comes only from a matching review. Values are never corrected in place.
12. An idle assumption names a generator placed at its own site during its dates.
13. Identity and classification never change in place: a visit's key, site, date and sequence; a
source document's hash, family and kind (its rank); an asset's uid and kind; a meter's id and
site; an epoch's id and meter. A wrong visit date is corrected by a new visit, which the
superseding claims point to.
12. An idle assumption names a generator placed on its own holder during its dates.
13. Identity and classification never change in place: a visit's key, holder, date and sequence; a
source document's hash, family and kind (its rank); an asset's uid and kind; a place's uid; a
holder's uid, kind and asset; a meter's id and holder; an epoch's id and meter; any row's customer.
A wrong visit date is corrected by a new visit, which the superseding claims point to. Naming a
column a table does not have raises an error.
14. A time zone is one PostgreSQL knows.
15. A vehicle holder's asset is a vehicle; only a fixed installation stands at a place.
## Rollback
Before any data matters, run `DROP SCHEMA fm CASCADE` as the loader. This also drops every
`public.v_*` view, and leaves fuel-db as it was before 0001.
Before any data matters, run `DROP SCHEMA licence CASCADE; DROP SCHEMA fm CASCADE` as the loader. This
also drops every `public.v_*` view, and leaves fuel-db as it was before 0001.
Once data has been loaded, this deletes it. The corpus remains the source, so a re-load restores
it, except baseline versions and their approvals, which live only in fuel-db: export
`fm.baseline_version` and `fm.baseline_input` first.
## Verified locally (2026-10-24, scratch only)
## Verified locally (2026-10-30, scratch only)
With Trino, before its removal on 29 Oct 2026; `trino_ro` is now `fuel_ro`.
Not run against the live fuel-db. An earlier set of 150 checks was a scratch script, since removed;
these replace it:
Setup:
- `timescale/timescaledb:2.31.0-pg17`, with this repository's `01-roles.sh`;
- a loader created by the README recipe;
- `trinodb/trino:484`, with this repository's `trino/etc` connected as `trino_ro`.
Results:
- **Apply:** all three files applied as the loader, under the loader's default `search_path`.
- **Fixture:** the operator workbook of 21 Oct 2026, the five site records, and synthetic sites
for the runtime path.
- **Tests:** all 150 checks of the design's §9 pass: 98 written with the migrations, and 52
regressions for the implementation inspection's findings. These cover:
- figures and statuses;
- claims, reviews and revisions;
- movement events;
- coverage, placement and idle scope;
- baseline dependencies and approval;
- the L/day path;
- current periods;
- flow-meter reconciliation (coverage, placement, visit times), runtime validity, modelled-L/h
windows, disputed claims supplying no values, the L/day reference, S2's L/h rows following
the version's validity, corrections under one `event_key`, fixed meter and epoch owners, and
population keys that name the generators a coverage assumption covers;
- every refusal listed above.
- **Trino:**
- all 303 columns of the 18 views have identical names, order and types in PostgreSQL and
Trino;
- `fm` is unreadable: a query is refused, though its names are listed (see Applying);
- writes are refused.
- **Not run on the server:** none of this has been run against the live fuel-db.
- **Rules:** `../tests/run.sh` applies these files to a throw-away database and runs `tests/rules.sql`
and `tests/read_only.sql`, with synthetic data only: each rule with an accepted and a refused case,
two customers, a vehicle as asset and holder, licences, and what `fuel_ro` may read.
- **Loader:** fuel-monitoring's `make selftest` and `bin/fmdb.py dbtest`.
- **Row comparison:** before this rewrite, all 18 views were dumped from a database loaded from the
corpus and a recorded review manifest; after it, the same corpus gave identical rows, leaving out
`operator_uid`. That one-time comparison covers only data the corpus has; rules with no corpus data
rest on `rules.sql`.
@@ -0,0 +1,36 @@
-- read_only.sql: what fuel_ro may and may not do. Run by run.sh, logged in as fuel_ro.
\set ON_ERROR_STOP 1
DO $$
DECLARE
v text; n integer := 0; has boolean;
BEGIN
FOREACH v IN ARRAY ARRAY['v_site','v_kpi_measures','v_hour_meter_evidence','v_assumption','v_baseline_version',
'v_baseline_input','v_evidence','v_delivery','v_stock_reading','v_interval','v_runtime',
'v_coverage','v_reconciliation','v_data_quality','v_spec_reference','v_modelled_lph',
'v_electricity_cost','v_imported_calculation','v_operator'] LOOP
EXECUTE format('SELECT count(*) FROM public.%I', v);
SELECT EXISTS (SELECT 1 FROM information_schema.columns c
WHERE c.table_schema = 'public' AND c.table_name = v AND c.column_name = 'operator_uid') INTO has;
IF NOT has THEN RAISE EXCEPTION '% has no operator_uid', v; END IF;
n := n + 1;
END LOOP;
IF n <> 19 THEN RAISE EXCEPTION 'expected 19 views, checked %', n; END IF;
IF (SELECT count(*) FROM information_schema.views WHERE table_schema = 'public' AND table_name LIKE 'v\_%') <> 19 THEN
RAISE EXCEPTION 'public holds a view this list does not name';
END IF;
FOREACH v IN ARRAY ARRAY['fm.holder','fm.operator','licence.licence','licence.plan','licence.licence_event'] LOOP
BEGIN
EXECUTE format('SELECT count(*) FROM %s', v);
RAISE EXCEPTION USING ERRCODE = 'TT001', MESSAGE = v || ' is readable';
EXCEPTION WHEN insufficient_privilege THEN NULL;
END;
END LOOP;
BEGIN
EXECUTE 'CREATE TABLE public.written (a integer)';
RAISE EXCEPTION USING ERRCODE = 'TT001', MESSAGE = 'fuel_ro created a table';
EXCEPTION WHEN insufficient_privilege OR read_only_sql_transaction THEN NULL;
END;
RAISE NOTICE 'read-only role: 19 views readable, each with operator_uid; fm and licence refused; writes refused';
END $$;
@@ -0,0 +1,366 @@
-- rules.sql: every rule of the fuel-db migrations, each with an accepted and a refused case.
-- Synthetic data only. Run by run.sh on a throw-away database, as a loader acting as fuel_owner.
-- A refused case names the words its error must hold, so it fails if a row is refused for another
-- reason than the rule under test.
\set ON_ERROR_STOP 1
\o /dev/null
SET client_min_messages = warning;
SET search_path = fm, pg_catalog, public;
CREATE SCHEMA rules_test;
CREATE TABLE rules_test.result (n bigint GENERATED ALWAYS AS IDENTITY, label text NOT NULL, passed boolean NOT NULL, detail text);
CREATE FUNCTION rules_test.ok(label text, stmt text) RETURNS void LANGUAGE plpgsql AS $$
BEGIN
EXECUTE stmt;
INSERT INTO rules_test.result (label, passed) VALUES (label, true);
EXCEPTION WHEN OTHERS THEN
INSERT INTO rules_test.result (label, passed, detail) VALUES (label, false, 'refused: ' || SQLERRM);
END $$;
-- A refused case always rolls back, so a wrongly accepted row cannot disturb the later cases.
CREATE FUNCTION rules_test.no(label text, want text, stmt text) RETURNS void LANGUAGE plpgsql AS $$
BEGIN
BEGIN
EXECUTE stmt;
RAISE EXCEPTION USING ERRCODE = 'TT001';
EXCEPTION
WHEN SQLSTATE 'TT001' THEN
INSERT INTO rules_test.result (label, passed, detail) VALUES (label, false, 'accepted');
WHEN OTHERS THEN
INSERT INTO rules_test.result (label, passed, detail)
VALUES (label, position(want IN SQLERRM) > 0,
CASE WHEN position(want IN SQLERRM) = 0 THEN 'refused for another reason: ' || SQLERRM END);
END;
END $$;
CREATE FUNCTION rules_test.eq(label text, query text, want text) RETURNS void LANGUAGE plpgsql AS $$
DECLARE got text;
BEGIN
EXECUTE query INTO got;
INSERT INTO rules_test.result (label, passed, detail)
VALUES (label, got IS NOT DISTINCT FROM want, format('got %s, want %s', got, want));
END $$;
CREATE FUNCTION rules_test.rec(op text) RETURNS bigint LANGUAGE sql STABLE AS
$$ SELECT min(import_record_id) FROM fm.import_record WHERE operator_uid = op $$;
CREATE FUNCTION rules_test.batch(op text) RETURNS bigint LANGUAGE sql STABLE AS
$$ SELECT min(import_batch_id) FROM fm.import_batch WHERE operator_uid = op $$;
CREATE FUNCTION rules_test.visit(k text) RETURNS bigint LANGUAGE sql STABLE AS
$$ SELECT visit_id FROM fm.visit WHERE visit_key = k $$;
CREATE FUNCTION rules_test.sha(c text) RETURNS text LANGUAGE sql IMMUTABLE AS
$$ SELECT repeat(c, 64) $$;
-- Two customers ------------------------------------------------------------------------------
INSERT INTO fm.operator (operator_uid, name, label_issuer, programme, timezone, keycloak_group) VALUES
('OPR-0001', 'Customer A', 'Issuer A', 'Programme A', 'Asia/Kabul', '/customer-a'),
('OPR-0002', 'Customer B', 'Issuer B', 'Programme B', 'Europe/Berlin', '/customer-b');
INSERT INTO fm.source_document (sha256, operator_uid, source_family, kind, issuer, title, corpus_path, contains_personal_data) VALUES
(rules_test.sha('a'), 'OPR-0001', 'family-a', 'operator_workbook', 'Issuer A', 'workbook A', 'a.xlsx', false),
(rules_test.sha('b'), 'OPR-0002', 'family-b', 'operator_workbook', 'Issuer B', 'workbook B', 'b.xlsx', false);
INSERT INTO fm.import_batch (operator_uid, source_kind, source_path, source_sha256, importer_version) VALUES
('OPR-0001', 'operator_workbook', 'a.xlsx', rules_test.sha('a'), 'test'),
('OPR-0002', 'operator_workbook', 'b.xlsx', rules_test.sha('b'), 'test');
INSERT INTO fm.import_record (operator_uid, logical_key, source_locator, payload, payload_sha256, first_batch_id, last_seen_batch_id)
SELECT o, 'record-' || o, 'test', '{}', rules_test.sha('c'), rules_test.batch(o), rules_test.batch(o)
FROM unnest(ARRAY['OPR-0001','OPR-0002']) o;
INSERT INTO fm.place (place_uid, operator_uid, role, zone, province, timezone, import_record_id) VALUES
('PLC-0001', 'OPR-0001', 'tower_site', 'z', 'p', NULL, rules_test.rec('OPR-0001')),
('PLC-0002', 'OPR-0002', 'depot', 'z', 'p', 'Asia/Tokyo', rules_test.rec('OPR-0002'));
INSERT INTO fm.holder (holder_uid, operator_uid, kind, name, import_record_id) VALUES
('SITE-0001', 'OPR-0001', 'fixed_installation', 'Holder A', rules_test.rec('OPR-0001')),
('SITE-0002', 'OPR-0002', 'fixed_installation', 'Holder B', rules_test.rec('OPR-0002'));
INSERT INTO fm.holder_place (holder_uid, operator_uid, place_uid, null_reasons) VALUES
('SITE-0001', 'OPR-0001', 'PLC-0001', '{"valid_from": {}}'),
('SITE-0002', 'OPR-0002', 'PLC-0002', '{"valid_from": {}}');
INSERT INTO fm.holder_alias (holder_uid, operator_uid, label, issuer, is_primary, null_reasons) VALUES
('SITE-0001', 'OPR-0001', 'A1', 'Issuer A', true, '{"valid_from": {}}'),
('SITE-0002', 'OPR-0002', 'B1', 'Issuer B', true, '{"valid_from": {}}');
INSERT INTO fm.participation (holder_uid, operator_uid, programme, status, valid_from) VALUES
('SITE-0001', 'OPR-0001', 'Programme A', 'active', '2026-01-01'),
('SITE-0002', 'OPR-0002', 'Programme B', 'active', '2026-01-01');
INSERT INTO fm.asset (asset_uid, operator_uid, kind, import_record_id) VALUES
('AST-A-GEN', 'OPR-0001', 'generator', rules_test.rec('OPR-0001')),
('AST-A-GEN2', 'OPR-0001', 'generator', rules_test.rec('OPR-0001')),
('AST-A-ENG', 'OPR-0001', 'engine', rules_test.rec('OPR-0001')),
('AST-A-T1', 'OPR-0001', 'tank', rules_test.rec('OPR-0001')),
('AST-A-T2', 'OPR-0001', 'tank', rules_test.rec('OPR-0001')),
('AST-B-GEN', 'OPR-0002', 'generator', rules_test.rec('OPR-0002'));
INSERT INTO fm.asset_placement (asset_uid, operator_uid, holder_uid, slot_label, occupied_from, installed_on) VALUES
('AST-A-GEN', 'OPR-0001', 'SITE-0001', 'GEN-01', '2025-01-01', '2025-01-01'),
('AST-A-GEN2', 'OPR-0001', 'SITE-0001', 'GEN-02', '2025-01-01', '2025-01-01'),
('AST-A-ENG', 'OPR-0001', 'SITE-0001', 'ENG-01', '2025-01-01', '2025-01-01'),
('AST-A-T1', 'OPR-0001', 'SITE-0001', 'T1', '2025-01-01', '2025-01-01'),
('AST-B-GEN', 'OPR-0002', 'SITE-0002', 'GEN-01', '2025-01-01', '2025-01-01');
INSERT INTO fm.visit (visit_key, operator_uid, holder_uid, visit_on, kind) VALUES
('SITE-0001:2026-01-01:1', 'OPR-0001', 'SITE-0001', '2026-01-01', 'stock_check'),
('SITE-0001:2026-01-11:1', 'OPR-0001', 'SITE-0001', '2026-01-11', 'stock_check'),
('SITE-0002:2026-01-01:1', 'OPR-0002', 'SITE-0002', '2026-01-01', 'stock_check');
INSERT INTO fm.stock_observation (source_key, assertion_sha256, operator_uid, visit_id, holder_uid, scope, phase, litres, litres_basis, source_sha256, record_ref, import_record_id)
SELECT 'stock:' || k, rules_test.sha('d'), 'OPR-0001', rules_test.visit(k), 'SITE-0001', 'site_total', 'standalone', l, 'stated_without_level',
rules_test.sha('a'), 'test', rules_test.rec('OPR-0001')
FROM (VALUES ('SITE-0001:2026-01-01:1', 100.0), ('SITE-0001:2026-01-11:1', 50.0)) x(k, l);
SELECT rules_test.ok('fixture holds two customers', 'SELECT 1');
SELECT rules_test.no('customers: one Keycloak group is one customer', 'duplicate key',
'UPDATE fm.operator SET keycloak_group = ''/customer-a'' WHERE operator_uid = ''OPR-0002''');
SELECT rules_test.no('customers: the group is a top-level one', 'keycloak_group_check',
'UPDATE fm.operator SET keycloak_group = ''/customer-a/team'' WHERE operator_uid = ''OPR-0001''');
SELECT rules_test.eq('customers: v_operator gives the group', 'SELECT operator_uid FROM public.v_operator WHERE keycloak_group = ''/customer-b''', 'OPR-0002');
-- Rule 1: baseline versions --------------------------------------------------------------------
\set bv 'INSERT INTO fm.baseline_version (code, operator_uid, holder_uid, metric, population, population_key, window_start, window_end, policy_version, meets_policy, computed_at, computed_by_role'
SELECT rules_test.no('rule 1: a version does not start approved', 'starts as draft or proposed',
:'bv' || ', status, approved_on, decision_ref) VALUES (''BL-X'', ''OPR-0001'', ''SITE-0001'', ''fuel_l_per_calendar_day'', ''x'', ''stock:SITE-0001'', ''2026-01-01'', ''2026-01-11'', ''BP-1'', true, now(), ''r'', ''approved'', ''2026-01-12'', ''D-1'')');
SELECT rules_test.ok('rule 1: a version starts proposed',
:'bv' || ', status) VALUES (''BL-1'', ''OPR-0001'', ''SITE-0001'', ''fuel_l_per_calendar_day'', ''x'', ''stock:SITE-0001'', ''2026-01-01'', ''2026-01-11'', ''BP-1'', true, now(), ''r'', ''proposed'')');
SELECT rules_test.ok('rule 1: a version below policy starts proposed',
:'bv' || ', status) VALUES (''BL-2'', ''OPR-0001'', ''SITE-0001'', ''fuel_l_per_calendar_day'', ''x'', ''stock:SITE-0001'', ''2026-01-01'', ''2026-01-11'', ''BP-1'', false, now(), ''r'', ''proposed'')');
SELECT rules_test.no('rule 1: below policy cannot be approved', 'does not meet',
'UPDATE fm.baseline_version SET status = ''approved'', approved_on = ''2026-01-12'', decision_ref = ''D-1'' WHERE code = ''BL-2''');
SELECT rules_test.no('rule 1: no inputs cannot be approved', 'its inputs changed',
'UPDATE fm.baseline_version SET status = ''approved'', approved_on = ''2026-01-12'', decision_ref = ''D-1'' WHERE code = ''BL-1''');
-- Rule 6: inputs (needed before an approval) ------------------------------------------------------
\set bi 'INSERT INTO fm.baseline_input (baseline_id, operator_uid, holder_uid, open_visit_id, close_visit_id, start_on, end_on, included, days, population_key, eligibility_at_computation, dependencies, stock_dep_sha256) SELECT b.baseline_id, ''OPR-0001'', ''SITE-0001'', rules_test.visit(''SITE-0001:2026-01-01:1''), rules_test.visit(''SITE-0001:2026-01-11:1''), '
SELECT rules_test.no('rule 6: dates must equal the endpoint visits', 'differ from the endpoint visits',
:'bi' || '''2026-01-02'', ''2026-01-11'', true, 10, ''stock:SITE-0001'', ''eligible'', ''{}'', d.stock_dep_sha256 FROM fm.baseline_version b, fm.v_interval_dependency d WHERE b.code = ''BL-1'' AND d.holder_uid = ''SITE-0001''');
SELECT rules_test.no('rule 6: an included input carries the population', 'differs from version',
:'bi' || '''2026-01-01'', ''2026-01-11'', true, 10, ''stock:OTHER'', ''eligible'', ''{}'', d.stock_dep_sha256 FROM fm.baseline_version b, fm.v_interval_dependency d WHERE b.code = ''BL-1'' AND d.holder_uid = ''SITE-0001''');
SELECT rules_test.ok('rule 6: a correct input is accepted',
:'bi' || '''2026-01-01'', ''2026-01-11'', true, 10, ''stock:SITE-0001'', ''eligible'', ''{}'', d.stock_dep_sha256 FROM fm.baseline_version b, fm.v_interval_dependency d WHERE b.code = ''BL-1'' AND d.holder_uid = ''SITE-0001''');
SELECT rules_test.ok('rule 1: unchanged inputs and policy met can be approved',
'UPDATE fm.baseline_version SET status = ''approved'', approved_on = ''2026-01-12'', approved_by_role = ''r'', decision_ref = ''D-1'' WHERE code = ''BL-1''');
SELECT rules_test.no('rule 1: an approved version is immutable', 'approved rows are immutable',
'UPDATE fm.baseline_version SET value = 1 WHERE code = ''BL-1''');
SELECT rules_test.no('rule 1: an approved version is never deleted', 'never deleted',
'DELETE FROM fm.baseline_version WHERE code = ''BL-1''');
SELECT rules_test.no('rule 1: an approved version''s inputs are immutable', 'are immutable',
'UPDATE fm.baseline_input SET days = 9 WHERE included');
-- Rules 2, 3, 10: claims -----------------------------------------------------------------------
\set mv 'INSERT INTO fm.fuel_movement (source_key, assertion_sha256, operator_uid, event_key, visit_id, holder_uid, kind, from_tank_uid, to_tank_uid, litres, litres_state, source_sha256, record_ref, import_record_id) VALUES '
SELECT rules_test.no('rule 3: a movement''s holder is its visit''s holder', 'differs from its visit',
:'mv' || '(''m0'', rules_test.sha(''e''), ''OPR-0001'', ''e0'', rules_test.visit(''SITE-0001:2026-01-01:1''), ''SITE-0003'', ''delivery'', NULL, NULL, 10, ''stated'', rules_test.sha(''a''), ''r'', rules_test.rec(''OPR-0001''))');
SELECT rules_test.no('rule 2: an internal transfer''s tanks are placed', 'must be placed',
:'mv' || '(''m1'', rules_test.sha(''e''), ''OPR-0001'', ''e1'', rules_test.visit(''SITE-0001:2026-01-01:1''), ''SITE-0001'', ''internal_transfer'', ''AST-A-T1'', ''AST-A-T2'', 10, ''stated'', rules_test.sha(''a''), ''r'', rules_test.rec(''OPR-0001''))');
INSERT INTO fm.asset_placement (asset_uid, operator_uid, holder_uid, slot_label, occupied_from, installed_on)
VALUES ('AST-A-T2', 'OPR-0001', 'SITE-0001', 'T2', '2025-01-01', '2025-01-01');
SELECT rules_test.ok('rule 2: placed tanks are accepted',
:'mv' || '(''m1'', rules_test.sha(''e''), ''OPR-0001'', ''e1'', rules_test.visit(''SITE-0001:2026-01-01:1''), ''SITE-0001'', ''internal_transfer'', ''AST-A-T1'', ''AST-A-T2'', 10, ''stated'', rules_test.sha(''a''), ''r'', rules_test.rec(''OPR-0001''))');
SELECT rules_test.no('rule 10: claims of one event share kind and tanks', 'must share visit',
:'mv' || '(''m2'', rules_test.sha(''e''), ''OPR-0001'', ''e1'', rules_test.visit(''SITE-0001:2026-01-01:1''), ''SITE-0001'', ''delivery'', NULL, NULL, 10, ''stated'', rules_test.sha(''a''), ''r'', rules_test.rec(''OPR-0001''))');
SELECT rules_test.ok('rule 10: a second claim of one event is accepted',
:'mv' || '(''m2'', rules_test.sha(''e''), ''OPR-0001'', ''e1'', rules_test.visit(''SITE-0001:2026-01-01:1''), ''SITE-0001'', ''internal_transfer'', ''AST-A-T1'', ''AST-A-T2'', 11, ''stated'', rules_test.sha(''a''), ''r'', rules_test.rec(''OPR-0001''))');
SELECT rules_test.no('rule 3: a stock claim''s holder is its visit''s holder', 'differs from its visit',
'INSERT INTO fm.stock_observation (source_key, assertion_sha256, operator_uid, visit_id, holder_uid, scope, phase, litres, litres_basis, source_sha256, record_ref, import_record_id) VALUES (''s0'', rules_test.sha(''d''), ''OPR-0001'', rules_test.visit(''SITE-0001:2026-01-01:1''), ''SITE-0003'', ''site_total'', ''standalone'', 1, ''stated_without_level'', rules_test.sha(''a''), ''r'', rules_test.rec(''OPR-0001''))');
-- Rules 8, 12, 9: meters and assumptions, generator-only (decision 6) ---------------------------
\set mt 'INSERT INTO fm.runtime_meter (meter_id, operator_uid, holder_uid, source_label, asset_uid, mapping_status) VALUES '
SELECT rules_test.ok('rule 8: a confirmed meter on a placed generator',
:'mt' || '(''MTR-A-1'', ''OPR-0001'', ''SITE-0001'', ''Genset 1'', ''AST-A-GEN'', ''confirmed'')');
SELECT rules_test.no('rule 8: a meter on an engine is refused', 'is not a generator placed',
:'mt' || '(''MTR-A-E'', ''OPR-0001'', ''SITE-0001'', ''Engine'', ''AST-A-ENG'', ''confirmed'')');
SELECT rules_test.no('rule 8: a meter on a tank is refused', 'is not a generator placed',
:'mt' || '(''MTR-A-T'', ''OPR-0001'', ''SITE-0001'', ''Tank'', ''AST-A-T1'', ''proposed'')');
SELECT rules_test.ok('rule 8: an unmapped counter names no asset',
:'mt' || '(''MTR-A-2'', ''OPR-0001'', ''SITE-0001'', ''R.H'', NULL, ''unmapped'')');
INSERT INTO fm.meter_epoch (meter_id, operator_uid, epoch_no, start_reason, continuity) VALUES
('MTR-A-1', 'OPR-0001', 1, 'first_seen', 'assumed'),
('MTR-A-2', 'OPR-0001', 1, 'first_seen', 'assumed');
\set rd 'INSERT INTO fm.hour_meter_reading (source_key, assertion_sha256, operator_uid, visit_id, meter_id, epoch_id, source_field, date_basis, hours, reading_state, transcribed_by, source_sha256, record_ref, import_record_id) VALUES '
SELECT rules_test.ok('rules 3, 4: a reading of the visit''s own meter and epoch',
:'rd' || '(''h1'', rules_test.sha(''f''), ''OPR-0001'', rules_test.visit(''SITE-0001:2026-01-01:1''), ''MTR-A-1'', (SELECT epoch_id FROM fm.meter_epoch WHERE meter_id = ''MTR-A-1''), ''current'', ''stated'', 10, ''clear'', ''source_operator'', rules_test.sha(''a''), ''r'', rules_test.rec(''OPR-0001''))');
SELECT rules_test.no('rule 4: a reading''s epoch belongs to its meter', 'does not belong to meter',
:'rd' || '(''h2'', rules_test.sha(''f''), ''OPR-0001'', rules_test.visit(''SITE-0001:2026-01-01:1''), ''MTR-A-1'', (SELECT epoch_id FROM fm.meter_epoch WHERE meter_id = ''MTR-A-2''), ''current'', ''stated'', 10, ''clear'', ''source_operator'', rules_test.sha(''a''), ''r'', rules_test.rec(''OPR-0001''))');
\set ax 'INSERT INTO fm.assumption (code, fingerprint, operator_uid, holder_uid, asset_uid, topic, basis, derived_from_fuel, status, rationale'
SELECT rules_test.ok('rule 12: an idle assumption on a placed generator',
:'ax' || ') VALUES (''A-IDLE'', rules_test.sha(''0''), ''OPR-0001'', ''SITE-0001'', ''AST-A-GEN2'', ''generator_idle'', ''our_assumption'', false, ''proposed'', ''r'')');
SELECT rules_test.no('rule 12: an idle assumption on an engine is refused', 'is not a generator placed',
:'ax' || ') VALUES (''A-IDLE-E'', rules_test.sha(''0''), ''OPR-0001'', ''SITE-0001'', ''AST-A-ENG'', ''generator_idle'', ''our_assumption'', false, ''proposed'', ''r'')');
-- Rule 5: needs a matching review first, or rule 11 refuses before the CHECK is reached
INSERT INTO fm.review (entry_key, operator_uid, subject_kind, subject_key, assertion_sha256, verdict, reviewed_on, reviewer_role, import_batch_id)
VALUES ('R-5', 'OPR-0001', 'assumption', 'A-R5', rules_test.sha('5'), 'accepted', '2026-01-02', 'r', rules_test.batch('OPR-0001'));
SELECT rules_test.no('rule 5: a runtime derived from fuel cannot be accepted', 'assumption_no_runtime_from_fuel',
:'ax' || ', accepted_on, accepted_by_role) VALUES (''A-R5'', rules_test.sha(''5''), ''OPR-0001'', ''SITE-0001'', NULL, ''runtime_hours_per_day'', ''our_assumption'', true, ''accepted'', ''r'', ''2026-01-02'', ''r'')');
SELECT rules_test.ok('rule 5: one not derived from fuel can be accepted',
:'ax' || ', accepted_on, accepted_by_role) VALUES (''A-R5'', rules_test.sha(''5''), ''OPR-0001'', ''SITE-0001'', NULL, ''runtime_hours_per_day'', ''our_assumption'', false, ''accepted'', ''r'', ''2026-01-02'', ''r'')');
SELECT rules_test.no('rule 9: members complete at commit', 'as many counters as covered',
:'ax' || ') VALUES (''A-COV0'', rules_test.sha(''6''), ''OPR-0001'', ''SITE-0001'', NULL, ''meter_coverage'', ''our_assumption'', false, ''proposed'', ''r''); '
'INSERT INTO fm.assumption_member (assumption_id, operator_uid, member_role, meter_id) SELECT assumption_id, ''OPR-0001'', ''counter'', ''MTR-A-2'' FROM fm.assumption WHERE code = ''A-COV0''; '
'SET CONSTRAINTS ALL IMMEDIATE');
SELECT rules_test.ok('rule 9: valid and complete members in the assumption''s own transaction',
:'ax' || ') VALUES (''A-COV'', rules_test.sha(''6''), ''OPR-0001'', ''SITE-0001'', NULL, ''meter_coverage'', ''our_assumption'', false, ''proposed'', ''r''); '
'INSERT INTO fm.assumption_member (assumption_id, operator_uid, member_role, meter_id) SELECT assumption_id, ''OPR-0001'', ''counter'', ''MTR-A-2'' FROM fm.assumption WHERE code = ''A-COV''; '
'INSERT INTO fm.assumption_member (assumption_id, operator_uid, member_role, asset_uid) SELECT assumption_id, ''OPR-0001'', ''covered_generator'', ''AST-A-GEN'' FROM fm.assumption WHERE code = ''A-COV''; '
'SET CONSTRAINTS ALL IMMEDIATE');
SELECT rules_test.no('rule 9: members are written in the same transaction', 'written in the transaction',
'INSERT INTO fm.assumption_member (assumption_id, operator_uid, member_role, asset_uid) SELECT assumption_id, ''OPR-0001'', ''idle_generator'', ''AST-A-GEN2'' FROM fm.assumption WHERE code = ''A-COV''');
SELECT rules_test.no('rule 9: members are immutable', 'members are immutable',
'DELETE FROM fm.assumption_member');
SELECT rules_test.no('rule 9: an engine cannot be a covered member', 'is not a generator placed',
'INSERT INTO fm.assumption (code, fingerprint, operator_uid, holder_uid, topic, basis, derived_from_fuel, status, rationale) '
'SELECT ''A-COV2'', rules_test.sha(''7''), ''OPR-0002'', ''SITE-0002'', ''meter_coverage'', ''our_assumption'', false, ''proposed'', ''r''; '
'INSERT INTO fm.asset (asset_uid, operator_uid, kind, import_record_id) VALUES (''AST-B-ENG'', ''OPR-0002'', ''engine'', rules_test.rec(''OPR-0002'')); '
'INSERT INTO fm.asset_placement (asset_uid, operator_uid, holder_uid, slot_label, occupied_from, installed_on) VALUES (''AST-B-ENG'', ''OPR-0002'', ''SITE-0002'', ''E1'', ''2025-01-01'', ''2025-01-01''); '
'INSERT INTO fm.assumption_member (assumption_id, operator_uid, member_role, asset_uid) SELECT assumption_id, ''OPR-0002'', ''covered_generator'', ''AST-B-ENG'' FROM fm.assumption WHERE code = ''A-COV2''');
-- Rule 7: source documents ---------------------------------------------------------------------
\set sd 'INSERT INTO fm.source_document (sha256, operator_uid, source_family, kind, issuer, title, corpus_path, contains_personal_data, supersedes_sha256) VALUES '
SELECT rules_test.ok('rule 7: a later version in its own family',
:'sd' || '(rules_test.sha(''1''), ''OPR-0001'', ''family-a'', ''operator_workbook'', ''Issuer A'', ''later'', ''a2.xlsx'', false, rules_test.sha(''a''))');
SELECT rules_test.no('rule 7: no supersession across families', 'may supersede only',
:'sd' || '(rules_test.sha(''2''), ''OPR-0001'', ''family-other'', ''operator_workbook'', ''Issuer A'', ''x'', ''x.xlsx'', false, rules_test.sha(''1''))');
SELECT rules_test.no('rule 7: a document cannot be shared with a second customer', 'another customer',
:'sd' || '(rules_test.sha(''a''), ''OPR-0002'', ''family-a'', ''operator_workbook'', ''Issuer A'', ''x'', ''x.xlsx'', false, NULL)');
-- Rule 11: status from a review ------------------------------------------------------------------
SELECT rules_test.no('rule 11: no status without a review', 'needs a review',
'UPDATE fm.stock_observation SET claim_status = ''reviewed'' WHERE source_key = ''stock:SITE-0001:2026-01-01:1''');
SELECT rules_test.no('rule 11: a value is never corrected in place', 'only the status columns',
'UPDATE fm.stock_observation SET litres = 101 WHERE source_key = ''stock:SITE-0001:2026-01-01:1''');
INSERT INTO fm.review (entry_key, operator_uid, subject_kind, subject_key, assertion_sha256, verdict, reviewed_on, reviewer_role, import_batch_id) VALUES
('R-OTHER', 'OPR-0002', 'claim', 'stock:SITE-0001:2026-01-01:1', rules_test.sha('d'), 'reviewed', '2026-01-02', 'r', rules_test.batch('OPR-0002'));
SELECT rules_test.no('rule 11: another customer''s review does not count', 'needs a review',
'UPDATE fm.stock_observation SET claim_status = ''reviewed'' WHERE source_key = ''stock:SITE-0001:2026-01-01:1''');
INSERT INTO fm.review (entry_key, operator_uid, subject_kind, subject_key, assertion_sha256, verdict, reviewed_on, reviewer_role, import_batch_id) VALUES
('R-OTHER', 'OPR-0001', 'claim', 'stock:SITE-0001:2026-01-01:1', rules_test.sha('d'), 'reviewed', '2026-01-02', 'r', rules_test.batch('OPR-0001'));
SELECT rules_test.ok('rule 11: a matching review gives the status; entry ids are per customer',
'UPDATE fm.stock_observation SET claim_status = ''reviewed'' WHERE source_key = ''stock:SITE-0001:2026-01-01:1''');
SELECT rules_test.no('review: an entry id is unique within one customer', 'duplicate key',
'INSERT INTO fm.review (entry_key, operator_uid, subject_kind, subject_key, assertion_sha256, verdict, reviewed_on, reviewer_role, import_batch_id) VALUES (''R-OTHER'', ''OPR-0001'', ''claim'', ''elsewhere'', rules_test.sha(''d''), ''disputed'', ''2026-01-03'', ''r'', rules_test.batch(''OPR-0001''))');
-- Rule 13: identity never changes in place -------------------------------------------------------
SELECT rules_test.no('rule 13: a visit''s date', 'never changes in place', 'UPDATE fm.visit SET visit_on = ''2026-01-02'' WHERE visit_key = ''SITE-0001:2026-01-01:1''');
SELECT rules_test.no('rule 13: a document''s family', 'never changes in place', 'UPDATE fm.source_document SET source_family = ''x'' WHERE sha256 = rules_test.sha(''a'')');
SELECT rules_test.no('rule 13: an asset''s kind', 'never changes in place', 'UPDATE fm.asset SET kind = ''engine'' WHERE asset_uid = ''AST-A-GEN''');
SELECT rules_test.no('rule 13: a meter''s holder', 'never changes in place', 'UPDATE fm.runtime_meter SET holder_uid = ''SITE-0002'' WHERE meter_id = ''MTR-A-1''');
SELECT rules_test.no('rule 13: an epoch''s meter', 'never changes in place', 'UPDATE fm.meter_epoch SET meter_id = ''MTR-A-2'' WHERE meter_id = ''MTR-A-1''');
SELECT rules_test.no('rule 13: a holder''s kind', 'never changes in place', 'UPDATE fm.holder SET kind = ''trailer'' WHERE holder_uid = ''SITE-0001''');
SELECT rules_test.ok('rule 13: a descriptive field may change', 'UPDATE fm.holder SET name = ''Renamed'' WHERE holder_uid = ''SITE-0001''');
CREATE TABLE rules_test.wrong_name (a integer);
CREATE TRIGGER wrong_name BEFORE UPDATE ON rules_test.wrong_name FOR EACH ROW EXECUTE FUNCTION fm.tg_fixed_columns('nope');
INSERT INTO rules_test.wrong_name VALUES (1);
SELECT rules_test.no('rule 13: a wrong column name raises, not a silent pass', 'does not have', 'UPDATE rules_test.wrong_name SET a = 2');
-- Rules 14, 15 --------------------------------------------------------------------------------
SELECT rules_test.no('rule 14: an unknown time zone', 'unknown time zone', 'UPDATE fm.operator SET timezone = ''Asia/Nowhere'' WHERE operator_uid = ''OPR-0001''');
SELECT rules_test.no('rule 14: an unknown time zone on a place', 'unknown time zone', 'UPDATE fm.place SET timezone = ''Asia/Nowhere'' WHERE place_uid = ''PLC-0001''');
SELECT rules_test.ok('rule 14: a known time zone', 'UPDATE fm.place SET timezone = ''Asia/Kabul'' WHERE place_uid = ''PLC-0001''');
-- Two customers: a child cannot name another customer's parent -----------------------------------
SELECT rules_test.no('customers: a visit on another''s holder', 'foreign key',
'INSERT INTO fm.visit (visit_key, operator_uid, holder_uid, visit_on, kind) VALUES (''x'', ''OPR-0002'', ''SITE-0001'', ''2026-02-01'', ''survey'')');
SELECT rules_test.no('customers: a claim on another''s visit', 'foreign key',
:'mv' || '(''mx'', rules_test.sha(''e''), ''OPR-0002'', ''ex'', rules_test.visit(''SITE-0001:2026-01-01:1''), ''SITE-0001'', ''delivery'', NULL, NULL, 10, ''stated'', rules_test.sha(''b''), ''r'', rules_test.rec(''OPR-0002''))');
SELECT rules_test.no('customers: a placement of another''s asset', 'foreign key',
'INSERT INTO fm.asset (asset_uid, operator_uid, kind, import_record_id) VALUES (''AST-A-SPARE'', ''OPR-0001'', ''generator'', rules_test.rec(''OPR-0001'')); '
'INSERT INTO fm.asset_placement (asset_uid, operator_uid, holder_uid, slot_label, null_reasons) VALUES (''AST-A-SPARE'', ''OPR-0002'', ''SITE-0002'', ''X'', ''{"installed_on": {}}'')');
SELECT rules_test.no('customers: a meter on another''s holder', 'foreign key',
:'mt' || '(''MTR-X'', ''OPR-0002'', ''SITE-0001'', ''X'', NULL, ''unmapped'')');
SELECT rules_test.no('customers: a reading of another''s meter', 'foreign key',
:'rd' || '(''hx'', rules_test.sha(''f''), ''OPR-0002'', rules_test.visit(''SITE-0001:2026-01-01:1''), ''MTR-A-1'', NULL, ''current'', ''stated'', 10, ''clear'', ''source_operator'', rules_test.sha(''b''), ''r'', rules_test.rec(''OPR-0002''))');
SELECT rules_test.no('customers: a baseline on another''s holder', 'foreign key',
:'bv' || ', status) VALUES (''BL-X2'', ''OPR-0002'', ''SITE-0001'', ''fuel_l_per_calendar_day'', ''x'', ''k'', ''2026-01-01'', ''2026-01-11'', ''BP-1'', true, now(), ''r'', ''proposed'')');
SELECT rules_test.no('customers: an input of another''s baseline', 'foreign key',
'INSERT INTO fm.baseline_input (baseline_id, operator_uid, holder_uid, open_visit_id, close_visit_id, start_on, end_on, included, exclusion_reason, eligibility_at_computation, dependencies, stock_dep_sha256) SELECT b.baseline_id, ''OPR-0002'', ''SITE-0002'', rules_test.visit(''SITE-0002:2026-01-01:1''), rules_test.visit(''SITE-0002:2026-01-01:1''), ''2026-01-01'', ''2026-01-01'', false, ''x'', ''excluded'', ''{}'', ''x'' FROM fm.baseline_version b WHERE b.code = ''BL-2''');
SELECT rules_test.no('customers: a claim citing another''s document', 'foreign key',
:'mv' || '(''my'', rules_test.sha(''e''), ''OPR-0002'', ''ey'', rules_test.visit(''SITE-0002:2026-01-01:1''), ''SITE-0002'', ''delivery'', NULL, NULL, 10, ''stated'', rules_test.sha(''a''), ''r'', rules_test.rec(''OPR-0002''))');
SELECT rules_test.no('customers: a holder standing at another''s place', 'foreign key',
'INSERT INTO fm.holder (holder_uid, operator_uid, kind, import_record_id) VALUES (''SITE-0004'', ''OPR-0002'', ''fixed_installation'', rules_test.rec(''OPR-0002'')); '
'INSERT INTO fm.holder_place (holder_uid, operator_uid, place_uid, valid_from) VALUES (''SITE-0004'', ''OPR-0002'', ''PLC-0001'', ''2030-01-01'')');
\set me 'INSERT INTO fm.fuel_movement (source_key, assertion_sha256, operator_uid, event_key, visit_id, holder_uid, kind, litres, litres_state, unit_price, price_currency, source_sha256, record_ref, import_record_id) VALUES '
SELECT rules_test.ok('customers: one event key under the first customer',
:'me' || '(''shared-a'', rules_test.sha(''e''), ''OPR-0001'', ''shared-event'', rules_test.visit(''SITE-0001:2026-01-11:1''), ''SITE-0001'', ''delivery'', 10, ''stated'', 70, ''AAA'', rules_test.sha(''a''), ''ref-a'', rules_test.rec(''OPR-0001''))');
SELECT rules_test.ok('customers: the same event key under the second, with another kind (rule 10 is per customer)',
:'me' || '(''shared-b'', rules_test.sha(''e''), ''OPR-0002'', ''shared-event'', rules_test.visit(''SITE-0002:2026-01-01:1''), ''SITE-0002'', ''withdrawal'', 20, ''stated'', 90, ''BBB'', rules_test.sha(''b''), ''ref-b'', rules_test.rec(''OPR-0002''))');
SELECT rules_test.ok('customers: and as a delivery',
:'me' || '(''shared-c'', rules_test.sha(''e''), ''OPR-0002'', ''shared-event-2'', rules_test.visit(''SITE-0002:2026-01-01:1''), ''SITE-0002'', ''delivery'', 20, ''stated'', 90, ''BBB'', rules_test.sha(''b''), ''ref-b'', rules_test.rec(''OPR-0002''))');
SELECT rules_test.ok('customers: the first again, under the second''s delivery key',
:'me' || '(''shared-d'', rules_test.sha(''e''), ''OPR-0001'', ''shared-event-2'', rules_test.visit(''SITE-0001:2026-01-11:1''), ''SITE-0001'', ''delivery'', 10, ''stated'', 70, ''AAA'', rules_test.sha(''a''), ''ref-a'', rules_test.rec(''OPR-0001''))');
SELECT rules_test.eq('customers: v_delivery gives each its own price and reference under one event key',
'SELECT string_agg(operator_uid || '':'' || unit_price || '':'' || currency || '':'' || source_ref, '','' ORDER BY operator_uid) FROM public.v_delivery WHERE event_key = ''shared-event-2''',
'OPR-0001:70.00:AAA:ref-a,OPR-0002:90.00:BBB:ref-b');
SELECT rules_test.no('customers: a row''s customer cannot change: holder', 'never changes in place', 'UPDATE fm.holder SET operator_uid = ''OPR-0002'' WHERE holder_uid = ''SITE-0001''');
SELECT rules_test.no('customers: a row''s customer cannot change: review', 'never changes in place', 'UPDATE fm.review SET operator_uid = ''OPR-0002'' WHERE entry_key = ''R-5''');
SELECT rules_test.no('customers: a row''s customer cannot change: batch', 'never changes in place', 'UPDATE fm.import_batch SET operator_uid = ''OPR-0002'' WHERE import_batch_id = rules_test.batch(''OPR-0001'')');
SELECT rules_test.eq('customers: every table but asset_kind has the fixed-customer trigger',
'SELECT string_agg(c.relname, '','' ORDER BY c.relname) FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace WHERE n.nspname = ''fm'' AND c.relkind = ''r'' AND NOT EXISTS (SELECT 1 FROM pg_trigger t WHERE t.tgrelid = c.oid AND t.tgname = ''operator_fixed'')',
'asset_kind');
SELECT rules_test.eq('customers: v_site gives each holder its customer',
'SELECT string_agg(site_uid || ''='' || operator_uid, '','' ORDER BY site_uid) FROM public.v_site', 'SITE-0001=OPR-0001,SITE-0002=OPR-0002');
SELECT rules_test.eq('customers: labels and programme come from each customer''s record',
'SELECT string_agg(site_code || '':'' || pilot_status, '','' ORDER BY site_uid) FROM public.v_site', 'A1:active,B1:active');
SELECT rules_test.eq('customers: local day uses the place''s zone, else the customer''s',
'SELECT string_agg(holder_uid || '':'' || timezone, '','' ORDER BY holder_uid) FROM fm.v_holder_current', 'SITE-0001:Asia/Kabul,SITE-0002:Asia/Tokyo');
-- A vehicle is both an asset and a holder (PD-18) -------------------------------------------------
INSERT INTO fm.asset (asset_uid, operator_uid, kind, import_record_id) VALUES
('AST-VEH-1', 'OPR-0002', 'vehicle', rules_test.rec('OPR-0002')),
('AST-VEH-T', 'OPR-0002', 'vehicle_tank', rules_test.rec('OPR-0002')),
('AST-VEH-E', 'OPR-0002', 'engine', rules_test.rec('OPR-0002'));
\set hd 'INSERT INTO fm.holder (holder_uid, operator_uid, kind, asset_uid, import_record_id) VALUES '
SELECT rules_test.no('vehicle: a vehicle holder needs its asset', 'holder_check', :'hd' || '(''VEH-0001'', ''OPR-0002'', ''vehicle'', NULL, rules_test.rec(''OPR-0002''))');
SELECT rules_test.no('vehicle: a fixed holder has no asset', 'holder_check', :'hd' || '(''SITE-0009'', ''OPR-0002'', ''fixed_installation'', ''AST-VEH-1'', rules_test.rec(''OPR-0002''))');
SELECT rules_test.no('rule 15: a vehicle holder''s asset is a vehicle', 'is not a vehicle', :'hd' || '(''VEH-0001'', ''OPR-0002'', ''vehicle'', ''AST-VEH-T'', rules_test.rec(''OPR-0002''))');
SELECT rules_test.ok('vehicle: asset and holder', :'hd' || '(''VEH-0001'', ''OPR-0002'', ''vehicle'', ''AST-VEH-1'', rules_test.rec(''OPR-0002''))');
SELECT rules_test.no('rule 15: a vehicle stands at no place', 'not a fixed installation',
'INSERT INTO fm.holder_place (holder_uid, operator_uid, place_uid, valid_from) VALUES (''VEH-0001'', ''OPR-0002'', ''PLC-0002'', ''2026-01-01'')');
SELECT rules_test.ok('vehicle: a tank placed on it',
'INSERT INTO fm.asset_placement (asset_uid, operator_uid, holder_uid, slot_label, occupied_from, installed_on) VALUES (''AST-VEH-T'', ''OPR-0002'', ''VEH-0001'', ''C1'', ''2026-01-01'', ''2026-01-01'')');
SELECT rules_test.ok('vehicle: an engine placed on it',
'INSERT INTO fm.asset_placement (asset_uid, operator_uid, holder_uid, slot_label, occupied_from, installed_on) VALUES (''AST-VEH-E'', ''OPR-0002'', ''VEH-0001'', ''E1'', ''2026-01-01'', ''2026-01-01'')');
SELECT rules_test.no('rule 8: a meter on the vehicle''s engine is refused', 'is not a generator placed',
:'mt' || '(''MTR-V-E'', ''OPR-0002'', ''VEH-0001'', ''Engine'', ''AST-VEH-E'', ''confirmed'')');
SELECT rules_test.ok('vehicle: a visit', 'INSERT INTO fm.visit (visit_key, operator_uid, holder_uid, visit_on, kind) VALUES (''VEH-0001:2026-02-01:1'', ''OPR-0002'', ''VEH-0001'', ''2026-02-01'', ''delivery'')');
SELECT rules_test.ok('vehicle: a delivery on its visit',
:'mv' || '(''m-veh'', rules_test.sha(''e''), ''OPR-0002'', ''e-veh'', rules_test.visit(''VEH-0001:2026-02-01:1''), ''VEH-0001'', ''delivery'', NULL, NULL, 200, ''stated'', rules_test.sha(''b''), ''r'', rules_test.rec(''OPR-0002''))');
SELECT rules_test.eq('vehicle: the delivery reaches v_delivery with its customer',
'SELECT operator_uid || '':'' || delivered_l FROM public.v_delivery WHERE site_uid = ''VEH-0001''', 'OPR-0002:200.0');
SELECT rules_test.eq('vehicle: v_site lists it, with no place', 'SELECT coalesce(province, ''no place'') FROM public.v_site WHERE site_uid = ''VEH-0001''', 'no place');
-- Placement: in or on a holder, or at a place, exactly one -----------------------------------------
\set pl 'INSERT INTO fm.asset_placement (asset_uid, operator_uid, holder_uid, place_uid, slot_label, occupied_from, installed_on) VALUES '
INSERT INTO fm.asset (asset_uid, operator_uid, kind, import_record_id) VALUES
('AST-B-T1', 'OPR-0002', 'tank', rules_test.rec('OPR-0002')), ('AST-B-T2', 'OPR-0002', 'tank', rules_test.rec('OPR-0002'));
SELECT rules_test.no('placement: not both', 'asset_placement_check', :'pl' || '(''AST-B-T1'', ''OPR-0002'', ''SITE-0002'', ''PLC-0002'', ''P1'', ''2026-01-01'', ''2026-01-01'')');
SELECT rules_test.no('placement: not neither', 'asset_placement_check', :'pl' || '(''AST-B-T1'', ''OPR-0002'', NULL, NULL, ''P1'', ''2026-01-01'', ''2026-01-01'')');
SELECT rules_test.ok('placement: at a place', :'pl' || '(''AST-B-T1'', ''OPR-0002'', NULL, ''PLC-0002'', ''P1'', ''2026-01-01'', ''2026-01-01'')');
SELECT rules_test.no('placement: one slot at a place at a time', 'conflicting key', :'pl' || '(''AST-B-T2'', ''OPR-0002'', NULL, ''PLC-0002'', ''P1'', ''2026-06-01'', ''2026-06-01'')');
SELECT rules_test.no('placement: one slot on a holder at a time', 'conflicting key', :'pl' || '(''AST-B-T2'', ''OPR-0002'', ''SITE-0002'', NULL, ''GEN-01'', ''2026-06-01'', ''2026-06-01'')');
SELECT rules_test.no('placement: an asset is in one place at a time', 'conflicting key', :'pl' || '(''AST-B-T1'', ''OPR-0002'', ''SITE-0002'', NULL, ''P2'', ''2026-06-01'', ''2026-06-01'')');
SELECT rules_test.eq('placement: a place-held tank does not break the views',
'SELECT count(*)::text FROM public.v_spec_reference WHERE asset_uid = ''AST-B-T1''', '0');
-- Licences -------------------------------------------------------------------------------------
\set li 'INSERT INTO licence.licence (operator_uid, plan_code, payment_term, valid_from, valid_until, grace_until, status, contract_ref, created_by) VALUES '
SELECT rules_test.eq('licence: PD-19''s prices are seeded',
'SELECT string_agg(payment_term || ''='' || price, '','' ORDER BY payment_term) FROM licence.plan_price WHERE plan_code = ''standard''', 'annual=15.00,monthly=17.00');
SELECT rules_test.eq('licence: no customer''s licence is seeded', 'SELECT count(*)::text FROM licence.licence', '0');
SELECT rules_test.eq('licence: seeds are not in the log', 'SELECT count(*)::text FROM licence.licence_event', '0');
SELECT rules_test.ok('licence: a first licence', :'li' || '(''OPR-0001'', ''pilot'', ''annual'', ''2026-01-01'', ''2026-12-31'', ''2027-01-30'', ''active'', ''agreement'', ''our admin'')');
SELECT rules_test.no('licence: overlapping dates refused', 'conflicting key', :'li' || '(''OPR-0001'', ''standard'', ''annual'', ''2026-12-31'', ''2027-12-31'', ''2028-01-30'', ''active'', ''agreement'', ''our admin'')');
SELECT rules_test.ok('licence: a renewal may start during the old grace', :'li' || '(''OPR-0001'', ''standard'', ''annual'', ''2027-01-01'', ''2027-12-31'', ''2028-01-30'', ''active'', ''agreement'', ''our admin'')');
SELECT rules_test.ok('licence: another customer may share the dates', :'li' || '(''OPR-0002'', ''standard'', ''monthly'', ''2026-01-01'', ''2026-12-31'', ''2027-01-30'', ''active'', ''agreement'', ''our admin'')');
SELECT rules_test.no('licence: grace at least until expiry', 'licence_check', :'li' || '(''OPR-0002'', ''standard'', ''annual'', ''2028-01-01'', ''2028-12-31'', ''2028-12-30'', ''active'', ''agreement'', ''our admin'')');
SELECT rules_test.no('licence: an unknown customer', 'foreign key', :'li' || '(''OPR-0009'', ''standard'', ''annual'', ''2028-01-01'', ''2028-12-31'', ''2029-01-30'', ''active'', ''agreement'', ''our admin'')');
SELECT rules_test.eq('licence: every change writes an event', 'SELECT count(*)::text FROM licence.licence_event WHERE table_name = ''licence'' AND action = ''INSERT''', '3');
SELECT rules_test.ok('licence: a suspension', 'UPDATE licence.licence SET status = ''suspended'' WHERE operator_uid = ''OPR-0002''');
SELECT rules_test.eq('licence: the event holds the old and new row',
'SELECT old_row->>''status'' || ''>'' || (new_row->>''status'') FROM licence.licence_event WHERE action = ''UPDATE''', 'active>suspended');
SELECT rules_test.no('licence: the log is never changed', 'never changed', 'DELETE FROM licence.licence_event');
-- Result ------------------------------------------------------------------------------------------
\o
\pset format unaligned
\pset tuples_only on
SELECT CASE WHEN passed THEN 'ok ' ELSE 'FAIL ' END || label || CASE WHEN NOT passed THEN ' - ' || coalesce(detail, '') ELSE '' END
FROM rules_test.result ORDER BY n;
SELECT 'rules: ' || count(*) || ' checks, ' || count(*) FILTER (WHERE NOT passed) || ' failed' FROM rules_test.result;
DO $$ BEGIN
IF EXISTS (SELECT 1 FROM rules_test.result WHERE NOT passed) THEN RAISE EXCEPTION 'rule tests failed'; END IF;
END $$;
+48
View File
@@ -0,0 +1,48 @@
#!/usr/bin/env bash
# Applies the migrations to a throw-away database and runs the rule tests. Synthetic data only.
# Needs a running fuel-db container with this repository's roles (fuel_owner, fuel_ro), e.g. the
# local stack: FUEL_DB_CONTAINER=fmdb-scratch-db stacks/fuel-monitoring/fuel-db/tests/run.sh
# Never point it at the server: it creates a role and a database.
set -euo pipefail
here=$(cd "$(dirname "$0")" && pwd)
container=${FUEL_DB_CONTAINER:?set FUEL_DB_CONTAINER to a local fuel-db container}
db=fuel_rules_$$
loader=fuel_rules_loader
psql_as() {
local user=$1
shift
docker exec -i "$container" psql -X -q -v ON_ERROR_STOP=1 -U "$user" "$@"
}
cleanup() {
psql_as postgres -d postgres -c "DROP DATABASE IF EXISTS $db;" >/dev/null
}
trap cleanup EXIT
psql_as postgres -d postgres -c "CREATE DATABASE $db;" >/dev/null
psql_as postgres -d postgres >/dev/null <<SQL
DO \$\$ BEGIN
IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = '$loader') THEN
CREATE ROLE $loader LOGIN;
GRANT fuel_owner TO $loader WITH INHERIT FALSE, SET TRUE;
ALTER ROLE $loader SET role TO 'fuel_owner';
END IF;
END \$\$;
ALTER DATABASE $db OWNER TO fuel_owner;
GRANT CONNECT ON DATABASE $db TO fuel_ro;
SQL
psql_as postgres -d "$db" >/dev/null <<SQL
REVOKE CREATE ON SCHEMA public FROM PUBLIC;
GRANT USAGE ON SCHEMA public TO fuel_ro;
ALTER DEFAULT PRIVILEGES FOR ROLE fuel_owner IN SCHEMA public GRANT SELECT ON TABLES TO fuel_ro;
SQL
for file in "$here"/../migrations/*.sql; do
psql_as "$loader" -d "$db" <"$file" >/dev/null
done
echo "migrations applied: $(cd "$here/../migrations" && echo *.sql)"
psql_as "$loader" -d "$db" <"$here/rules.sql"
psql_as fuel_ro -d "$db" <"$here/read_only.sql" 2>&1 | sed 's/^psql:[^ ]* NOTICE: //'
-17
View File
@@ -10,23 +10,6 @@ set -euo pipefail
superset db upgrade
superset init
# Trino was removed on 29 Oct 2026. An instance still holding its connection
# "Fuel (Trino)" gets it renamed, so its datasets and dashboards keep their ids;
# the next step points it at fuel-db. Drop this once no instance has it.
/app/.venv/bin/python - <<'PY'
from superset.app import create_app
with create_app().app_context():
from superset import db
from superset.models.core import Database
old = db.session.query(Database).filter_by(database_name="Fuel (Trino)").one_or_none()
if old and not db.session.query(Database).filter_by(database_name="Fuel").count():
old.database_name = "Fuel"
db.session.commit()
print('renamed the connection "Fuel (Trino)" to "Fuel"')
PY
# No password here: superset_config.py reads it from /run/secrets at connect time.
superset set-database-uri \
--database_name "Fuel" \
+2 -2
View File
@@ -69,7 +69,7 @@ services:
image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.5
command:
- --provider=keycloak-oidc
- --oidc-issuer-url=${KEYCLOAK_REALM_URL:-https://iam.thedigitalbridge.io/realms/thedigitalbridge}
- --oidc-issuer-url=${KEYCLOAK_REALM_URL:-https://iam.thedigitalbridge.io/realms/petrolyst}
- --client-id=petrolyst
- --client-secret-file=/run/secrets/keycloak_petrolyst_client_secret
- --cookie-secret-file=/run/secrets/oauth2_proxy_cookie_secret
@@ -114,6 +114,6 @@ volumes:
secrets:
keycloak_petrolyst_client_secret:
file: ${PETROLYST_ROOT:-/opt/petrolyst}/secrets/shared/keycloak_petrolyst_client_secret
file: ${PETROLYST_ROOT:-/opt/petrolyst}/secrets/shared/keycloak_petrolyst_realm_client_secret
oauth2_proxy_cookie_secret:
file: ${PETROLYST_ROOT:-/opt/petrolyst}/secrets/traefik/oauth2_proxy_cookie_secret