Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
78a498f1ce | ||
|
|
ab553526f6 | ||
|
|
e4806f5799 |
No files matched your search
@@ -12,9 +12,11 @@ keep the rules there; a change against a rule or an open conflict goes to the ow
|
||||
- `stacks/`, `traefik/`, `deploy/`, `host/`: how and where things run.
|
||||
- `petrolyst-insights/`: what Superset looks like and does (config, security manager, the guest
|
||||
role, branding, patches, dashboards as code). Its README maps each file to the image.
|
||||
- `keycloak/`: the `petrolyst` client import.
|
||||
- `keycloak/`: the import of realm `petrolyst`'s two clients (`petrolyst`, `petrolyst-app`).
|
||||
|
||||
## Rules
|
||||
- **Commit messages** end their body with the bands and rules touched, for example
|
||||
`Bands: 4 (server). Rules: 2, 3.`, or `Bands: none (docs).` for text only (architecture rule 1).
|
||||
- **Deploys:** only by README "Manual deploy", only when the owner asks. There is no CI. Commit on
|
||||
`main` first; never push or deploy unprompted.
|
||||
- **No secrets** in the repository; `deploy/deploy.sh` refuses if one appears.
|
||||
@@ -24,7 +26,8 @@ keep the rules there; a change against a rule or an open conflict goes to the ow
|
||||
export. `import_bundle.py` never deletes; old objects go by hand (`dashboards/README.md`).
|
||||
- **Embedding** stays off in production (`superset_config.py`) until PD-12's points are decided;
|
||||
the local stack turns it on in its own override.
|
||||
- **ATOMA's users** have no Superset role (PD-13): `atoma-reader` opens the Petrolyst app only.
|
||||
- **Customers' users** have no Superset account (PD-13, PD-21): role `reader` of client `petrolyst-app`
|
||||
opens the Petrolyst app only, for the customer of their Keycloak group; Superset refuses them.
|
||||
- fuel-db migrations (`stacks/fuel-monitoring/fuel-db/migrations/`) are applied by hand (P3).
|
||||
|
||||
## Checks
|
||||
|
||||
@@ -6,7 +6,7 @@ Everything that runs on the petrolyst server (109.199.127.84, Ubuntu 24.04):
|
||||
https://traefik.petrolyst.com, behind Keycloak.
|
||||
- **fuel-monitoring** (ATOMA): Superset at https://atoma.petrolyst.com, reading the operational
|
||||
store `fuel-db` (Postgres 17 with TimescaleDB) directly, as the read-only role `fuel_ro`
|
||||
(Trino was removed on 29 Oct 2026). Sign-in goes through Keycloak
|
||||
(no Trino since 29 Oct 2026). Sign-in goes through Keycloak
|
||||
client `petrolyst`, and the UI carries the Petrolyst brand.
|
||||
|
||||
**Deploys are manual.** A person deploys a clean commit of `main` from the Mac ("Manual deploy"
|
||||
@@ -21,7 +21,7 @@ Customization and deployment are separate, as with media-insights and media-inte
|
||||
| `stacks/fuel-monitoring/` | Compose stack, Superset Dockerfile and init, fuel-db roles and migrations |
|
||||
| `deploy/` | `deploy.sh` (run on the server by a manual deploy), `rollback.sh` (manual), `lib.sh` (shared) |
|
||||
| `host/` | one-time server setup: `bootstrap.sh`, `secrets-init.sh` |
|
||||
| `keycloak/` | the `petrolyst` client for Keycloak's partial import |
|
||||
| `keycloak/` | the clients `petrolyst` and `petrolyst-app` for Keycloak's partial import |
|
||||
|
||||
No credential is in this repository. Secrets exist only on the server, under
|
||||
`/opt/petrolyst/secrets/`. A deploy needs only root SSH access to the server (`ssh petrolyst`).
|
||||
@@ -88,16 +88,23 @@ Create the private repository `Petrolyst/petrolyst-infra` and push this reposito
|
||||
it. Gitea only keeps the history: deploys are manual, so it needs no runner, no Actions and no
|
||||
deploy secrets.
|
||||
|
||||
### 2. Keycloak (realm `thedigitalbridge` on iam.thedigitalbridge.io)
|
||||
### 2. Keycloak (realm `petrolyst` on iam.thedigitalbridge.io)
|
||||
|
||||
1. Realm settings → Action → **Partial import**: choose `keycloak/petrolyst-partial-import.json`,
|
||||
1. Realm settings → Action → **Partial import**: choose `keycloak/petrolyst-realm-import.json`,
|
||||
and "Skip" if a resource exists. This creates:
|
||||
- the confidential client `petrolyst`: standard flow only, PKCE S256, redirect URIs for both
|
||||
hosts;
|
||||
- its client roles `superset-admin`, `superset-editor`, `superset-viewer`, `atoma-reader`
|
||||
and `traefik-admin`;
|
||||
- a mapper that puts those roles into the claim `petrolyst_roles`.
|
||||
- the confidential client `petrolyst`, for Superset and Traefik: standard flow only, PKCE S256,
|
||||
redirect URIs for both hosts;
|
||||
- its client roles `superset-admin`, `superset-editor`, `superset-viewer` and `traefik-admin`;
|
||||
- a mapper that puts those roles into the claim `petrolyst_roles`;
|
||||
- the confidential client `petrolyst-app`, for the Petrolyst app, with its role `reader` and two
|
||||
mappers: `petrolyst_roles`, and the person's groups with their full path in `petrolyst_groups`,
|
||||
both in the ID token. Both clients' addresses are on app.petrolyst.com, where everything moves
|
||||
(PD-16, PD-21); atoma.petrolyst.com is going away and is not in the import.
|
||||
2. Clients → petrolyst → Credentials: copy the client secret (step 3 stores it on the server).
|
||||
**Realm settings:** registration off; brute-force detection on; a password policy of at least 12
|
||||
characters, because a password is the only lock for customers' users; SSO session idle 8 hours,
|
||||
max 10 hours, because the app's session is renewed through Keycloak about hourly and the default
|
||||
idle limit of 30 minutes would ask for a password each time.
|
||||
3. **MFA**, which Exhibit B requires. A copy of the built-in `browser` flow is not enough, for two
|
||||
reasons: its Cookie step accepts an existing password-only session from another client of the
|
||||
realm, and its conditional OTP subflow skips users who have not set up OTP. Instead:
|
||||
@@ -107,14 +114,17 @@ deploy secrets.
|
||||
- Clients → petrolyst → Advanced → Authentication flow overrides → Browser flow:
|
||||
`petrolyst-mfa`.
|
||||
|
||||
Result: every sign-in to a Petrolyst app asks for password and OTP, and a user without OTP
|
||||
enrols first. Other clients of the realm are unaffected. There is no silent SSO between
|
||||
Superset and the dashboard.
|
||||
4. Give people roles: Users → (user) → Role mapping → Assign role → filter by clients →
|
||||
`petrolyst`. Group mappings work too. ATOMA's users get `atoma-reader` and nothing else:
|
||||
it opens the Petrolyst app (petrolyst-platform) and maps to no Superset role, so in
|
||||
Superset they see nothing. A realm that already has the client gets the role by hand
|
||||
(Clients → petrolyst → Roles → Create role) or by importing again with "Skip".
|
||||
Result: every sign-in to Superset or Traefik asks for password and OTP, and a user without OTP
|
||||
enrols first. There is no silent SSO between them. Client `petrolyst-app` keeps the realm's own
|
||||
flow: customers' users sign in with a password only, the owner's choice "for now" (PD-21).
|
||||
4. **A customer is a top-level group.** Groups → Create group, named as the customer's
|
||||
`keycloak_group` in fuel-db (without the slash), then Role mapping → Assign role → filter by
|
||||
clients → `petrolyst-app` `reader`. Adding a person to the group is then the only step, and they
|
||||
see that customer in the app and no other. Exactly one customer per person: two give "No access".
|
||||
They need no role on client `petrolyst`, and Superset refuses them. A person in no customer's
|
||||
group sees "No access"; that includes our own staff, who join a customer's group to see it in
|
||||
the app.
|
||||
**Administrators:** Users → (user) → Role mapping → Assign role → filter by clients → `petrolyst`.
|
||||
5. Users of the Traefik dashboard need **Email verified** switched on, because oauth2-proxy
|
||||
refuses unverified addresses.
|
||||
|
||||
@@ -158,7 +168,7 @@ becomes the first Superset admin; there is no local admin account.
|
||||
| Task | How |
|
||||
|---|---|
|
||||
| Deploy | "Manual deploy" below, from a clean `main` |
|
||||
| Re-deploy a kept release | `sudo -u deploy /opt/petrolyst/infra/deploy/deploy.sh <sha>` |
|
||||
| Re-deploy a kept release | `sudo -u deploy /opt/petrolyst/infra/deploy/deploy.sh <sha>`. Not across a change of Keycloak realm: see "Moving to realm petrolyst" |
|
||||
| Roll back | see below |
|
||||
| Traefik dashboard | https://traefik.petrolyst.com (role `traefik-admin`) |
|
||||
| fuel-db shell | `docker exec -it petrolyst-fuel-db-1 psql -U postgres -d fuel` |
|
||||
@@ -245,18 +255,14 @@ Tables it creates, hypertables included, then belong to `fuel_owner`, and `fuel_
|
||||
After `SET ROLE NONE`, it cannot create anything in `public`. To repair a table that another
|
||||
role created, run `ALTER TABLE <t> OWNER TO fuel_owner`, then `GRANT SELECT ON <t> TO fuel_ro`.
|
||||
|
||||
**One-time steps after 29 Oct 2026**, at the first deploy that includes them. Two changes
|
||||
meet there:
|
||||
- Trino is removed: Superset reads fuel-db directly. The role `trino_ro` becomes `fuel_ro` and
|
||||
keeps its password (a SCRAM password survives a rename); its secret file moves with it.
|
||||
- The Compose project `fuel-monitoring` is renamed `petrolyst`. Containers are then called
|
||||
`petrolyst-<service>-1`. The data volumes keep their names (`fuel-monitoring_superset_db_data`,
|
||||
`fuel-monitoring_fuel_db_data`; pinned in `compose.yaml`), so no data moves. The old project's
|
||||
containers must be gone first: two PostgreSQL servers on one data volume corrupt it.
|
||||
Preflight refuses to deploy while any of them exists.
|
||||
**Removing the old project**, once, right before the next deploy; delete this paragraph when it
|
||||
is done. The server still runs the Compose project `fuel-monitoring`, with Trino. Afterwards it runs
|
||||
`petrolyst`, without it: containers are `petrolyst-<service>-1`, and the data volumes keep their
|
||||
names (pinned in `compose.yaml`), so no data moves. The old containers must be gone first: two
|
||||
PostgreSQL servers on one data volume corrupt it, and preflight refuses to deploy while any exists.
|
||||
|
||||
As root on the server, right before the deploy (Superset is down from step 3 until the deploy
|
||||
finishes):
|
||||
As root on the server (Superset is down from step 3 until the deploy finishes). The role keeps its
|
||||
password, because a SCRAM password survives a rename:
|
||||
|
||||
```bash
|
||||
S=/opt/petrolyst/secrets/fuel-monitoring
|
||||
@@ -267,20 +273,52 @@ docker exec -i fuel-monitoring-fuel-db-1 psql -U postgres -d fuel -v ON_ERROR_ST
|
||||
docker rm -f $(docker ps -aq --filter label=com.docker.compose.project=fuel-monitoring) # 3
|
||||
```
|
||||
|
||||
4. Deploy as usual ("Manual deploy"). Compose starts the `petrolyst` project on the same volumes
|
||||
and no longer starts Trino. Compose warns once that the volumes belong to another project;
|
||||
that is expected. `superset-init` renames the connection "Fuel (Trino)" to "Fuel" and points
|
||||
it at fuel-db; the smoke test checks that it reads as `fuel_ro`, read-only.
|
||||
5. Clean up: `rm /opt/petrolyst/env/fuel-monitoring.env`, `rmdir /opt/petrolyst/env`,
|
||||
4. Deploy as usual ("Manual deploy"). Compose warns once that the volumes belong to another project;
|
||||
that is expected. `superset-init` registers the connection "Fuel"; the smoke test checks that it
|
||||
reads as `fuel_ro`, read-only.
|
||||
5. Delete the old connection. It refuses if a dataset still uses it; then stop and look:
|
||||
|
||||
```bash
|
||||
docker exec -i petrolyst-superset-1 /app/.venv/bin/python - <<'PY'
|
||||
from superset.app import create_app
|
||||
|
||||
with create_app().app_context():
|
||||
from superset import db
|
||||
from superset.connectors.sqla.models import SqlaTable
|
||||
from superset.models.core import Database
|
||||
|
||||
old = db.session.query(Database).filter_by(database_name="Fuel (Trino)").one_or_none()
|
||||
if old:
|
||||
used = db.session.query(SqlaTable).filter_by(database_id=old.id).count()
|
||||
if used:
|
||||
raise SystemExit(f"{used} datasets still use the old connection")
|
||||
db.session.delete(old)
|
||||
db.session.commit()
|
||||
print("old connection deleted")
|
||||
PY
|
||||
```
|
||||
6. Clean up: `rm /opt/petrolyst/env/fuel-monitoring.env`, `rmdir /opt/petrolyst/env`,
|
||||
`docker image rm trinodb/trino:484`,
|
||||
`docker network rm fuel-monitoring_backend fuel-monitoring_fuel-db-tunnel`.
|
||||
7. Releases from before this deploy can no longer run, as their role is gone, and `rollback.sh`
|
||||
stops on them. Remove them from `/opt/petrolyst/releases/`.
|
||||
|
||||
**Rollback across the Trino removal** (not tested). `rollback.sh` stops for a release from before
|
||||
it. To go back anyway: remove the `petrolyst` project's containers, reverse steps 1 and 2 above (rename `fuel_ro` back to `trino_ro` and
|
||||
`ALTER ROLE trino_ro RESET default_transaction_read_only`, move the secret file back, write
|
||||
`/opt/petrolyst/env/fuel-monitoring.env` with `FUEL_DB_TRINO_PASSWORD=<that secret>`), rename
|
||||
the Superset connection "Fuel" back to "Fuel (Trino)" with `trino://superset@trino:8080/fuel`,
|
||||
then run that release's own script, `/opt/petrolyst/releases/<sha>/deploy/rollback.sh <sha>`.
|
||||
**Moving to realm `petrolyst`, one time.** A deploy of its own, after the deploy that removes Trino:
|
||||
releases from before that one can no longer run, so a combined deploy leaves nothing to roll back to.
|
||||
1. The realm is ready (Keycloak, above) and an admin has signed in on client `petrolyst` there. While
|
||||
Superset still answers at atoma.petrolyst.com, add that address's redirect URI to the client by
|
||||
hand, or nobody can sign in after step 3; remove it when the domain goes.
|
||||
2. As root: `host/secrets-init.sh keycloak-secret`. The value goes into the new file
|
||||
`shared/keycloak_petrolyst_realm_client_secret`, so nothing changes for anyone yet. Leave the old
|
||||
file `shared/keycloak_petrolyst_client_secret` alone.
|
||||
3. Deploy by "Manual deploy" only. That one step moves Superset and Traefik together.
|
||||
4. **Prove it by hand:** an admin signs in to Superset and to the Traefik dashboard. The smoke tests
|
||||
check redirects only, and pass with a wrong secret.
|
||||
5. **To go back:** `rollback.sh <previous sha>` returns to the old realm with the old secret. Afterwards go
|
||||
forward again by "Manual deploy" only: "Re-deploy a kept release" runs the old `lib.sh`, whose realm
|
||||
address would override the new release's.
|
||||
6. Not before 7 days of working sign-ins, and on the owner's word: retire client `petrolyst` in realm
|
||||
`thedigitalbridge` and delete the old secret file. From then on there is no way back.
|
||||
|
||||
**Adding a project**: create `stacks/<name>/compose.yaml`, attach the public service to the
|
||||
external network `proxy`, and set `traefik.*` labels with a unique router name and
|
||||
@@ -299,8 +337,6 @@ its own gate in its stack:
|
||||
|
||||
## Verified locally (2026-10-23)
|
||||
|
||||
Before Trino's removal (29 Oct 2026): its lines below are history.
|
||||
|
||||
An end-to-end run ran the real `deploy.sh` and `rollback.sh` in a Linux container against
|
||||
Docker Desktop, with a scratch `/opt/petrolyst`. Production differences in that run:
|
||||
- a test CA certificate stood in for Let's Encrypt;
|
||||
@@ -310,7 +346,6 @@ Docker Desktop, with a scratch `/opt/petrolyst`. Production differences in that
|
||||
Results:
|
||||
- The fresh install took the init path.
|
||||
- An identical re-run recreated nothing and left `ARTIFACTS` byte-identical.
|
||||
- A Trino-only change recreated only `trino`.
|
||||
- A Superset change stopped, backed up, migrated and recreated only `superset`.
|
||||
- A change to `superset/init.sh` alone did the same, and the changed script ran.
|
||||
- A same-SHA upload with different content was refused.
|
||||
@@ -327,13 +362,12 @@ Results:
|
||||
- Retention pruned read-only releases.
|
||||
- Retention removed a queued upload that a plain `rsync -a` had given an old checkout's
|
||||
directory times. It kept one uploaded with `--omit-dir-times`.
|
||||
- In `fuel-db`, a loader that created a table as its own role left it unreadable for `trino_ro`.
|
||||
With the "Data loaders" recipe, tables (hypertables too) belonged to `fuel_owner` and Trino
|
||||
could read them. After `SET ROLE NONE`, the loader could not create anything.
|
||||
- In `fuel-db`, a loader that created a table as its own role left it unreadable for the read-only
|
||||
role. With the "Data loaders" recipe, tables (hypertables too) belonged to `fuel_owner` and that
|
||||
role could read them. After `SET ROLE NONE`, the loader could not create anything.
|
||||
- Smoke tests passed, including the redirects to the real Keycloak with `client_id=petrolyst`
|
||||
and PKCE S256.
|
||||
- TLS 1.1 and unknown SNI were refused, and HTTP redirected to HTTPS.
|
||||
- Trino and `trino_ro` both refused writes.
|
||||
|
||||
Not verified locally:
|
||||
- Let's Encrypt issuance;
|
||||
|
||||
+29
-16
@@ -17,9 +17,11 @@ ROLLBACK_MARK=$STATE/rollback-in-progress
|
||||
|
||||
ATOMA_HOST=${ATOMA_HOST:-atoma.petrolyst.com}
|
||||
TRAEFIK_HOST=${TRAEFIK_HOST:-traefik.petrolyst.com}
|
||||
KEYCLOAK_REALM_URL=${KEYCLOAK_REALM_URL:-https://iam.thedigitalbridge.io/realms/thedigitalbridge}
|
||||
PETROLYST_ROOT=$ROOT
|
||||
export ATOMA_HOST TRAEFIK_HOST KEYCLOAK_REALM_URL PETROLYST_ROOT
|
||||
# The Keycloak realm is not set here: each release's compose files name their own, with the secret
|
||||
# file that belongs to it. A rollback, which runs this file from the release it leaves, therefore
|
||||
# returns to the old release's realm and secret. Exporting a value here would override them.
|
||||
export ATOMA_HOST TRAEFIK_HOST PETROLYST_ROOT
|
||||
|
||||
# --host pins the daemon, as in petrolyst-web/deploy/deploy.sh: a stale
|
||||
# `docker context use` must not send this deploy to another daemon.
|
||||
@@ -39,8 +41,8 @@ SUPERSET_INPUTS=(
|
||||
)
|
||||
|
||||
# Every file the stacks need from /opt/petrolyst (host/secrets-init.sh).
|
||||
# The Keycloak client secret is not listed: it depends on the release (check_keycloak_secrets).
|
||||
REQUIRED_FILES=(
|
||||
secrets/shared/keycloak_petrolyst_client_secret
|
||||
secrets/traefik/oauth2_proxy_cookie_secret
|
||||
secrets/fuel-monitoring/superset_secret_key
|
||||
secrets/fuel-monitoring/superset_db_password
|
||||
@@ -94,11 +96,6 @@ load_artifacts() {
|
||||
while IFS= read -r line; do
|
||||
key=${line%%=*}
|
||||
value=${line#*=}
|
||||
# A release from before 29 Oct 2026 still runs Trino, which logs in as
|
||||
# `trino_ro`, since renamed `fuel_ro`: it cannot start (README "Rollback
|
||||
# across the Trino removal").
|
||||
[ "$key" != TRINO_CONFIG_HASH ] ||
|
||||
die "$file is from before Trino was removed; see README \"Rollback across the Trino removal\""
|
||||
[[ $key =~ ^(SUPERSET_IMAGE_TAG|TRAEFIK_CONFIG_HASH)$ && $value =~ ^[0-9a-f]{16}$ ]] ||
|
||||
die "unexpected line in $file: $line"
|
||||
printf -v "$key" '%s' "$value"
|
||||
@@ -131,10 +128,7 @@ preflight() {
|
||||
for f in "${REQUIRED_FILES[@]}"; do
|
||||
[ -s "$ROOT/$f" ] || die "missing or empty: $ROOT/$f (run host/secrets-init.sh as root)"
|
||||
done
|
||||
# oauth2-proxy uses the client secret file byte for byte.
|
||||
f=$ROOT/secrets/shared/keycloak_petrolyst_client_secret
|
||||
[ "$(tail -c1 "$f" | wc -l)" -eq 0 ] ||
|
||||
die "$f ends with a newline; store it again with: host/secrets-init.sh keycloak-secret"
|
||||
check_keycloak_secrets "$release"
|
||||
bad=$(find "$release/petrolyst-insights" \( -name 'Dockerfile*' -o -name '*compose*.y*ml' \
|
||||
-o -name '*.env' -o -name '.env' -o -path '*/.gitea/*' -o -path '*/.github/*' \) -print)
|
||||
[ -z "$bad" ] || die "petrolyst-insights/ holds customization only; found: $bad"
|
||||
@@ -143,7 +137,22 @@ preflight() {
|
||||
# The stack was the Compose project "fuel-monitoring" until 29 Oct 2026; "petrolyst" uses the
|
||||
# same data volumes, so both must never run at once (README "One-time steps after 29 Oct 2026").
|
||||
[ -z "$("${DOCKER[@]}" ps -aq --filter label=com.docker.compose.project=fuel-monitoring)" ] ||
|
||||
die "containers of the old project fuel-monitoring exist; remove them first (README \"One-time steps after 29 Oct 2026\")"
|
||||
die "containers of the old project fuel-monitoring exist; remove them first (README \"Removing the old project\")"
|
||||
}
|
||||
|
||||
# check_keycloak_secrets <release>: the client secret that release's compose files name, which
|
||||
# differs per realm. So a rollback needs only its target's secret, not the one of the release it leaves.
|
||||
check_keycloak_secrets() {
|
||||
local release=$1 names name f
|
||||
names=$(cat "$release/stacks/fuel-monitoring/compose.yaml" "$release/traefik/compose.yaml" |
|
||||
grep -oE 'secrets/shared/keycloak_[a-z_]+' | sort -u) || die "$release names no Keycloak client secret"
|
||||
for name in $names; do
|
||||
f=$ROOT/$name
|
||||
[ -s "$f" ] || die "missing or empty: $f (run host/secrets-init.sh as root)"
|
||||
# oauth2-proxy uses the client secret file byte for byte.
|
||||
[ "$(tail -c1 "$f" | wc -l)" -eq 0 ] ||
|
||||
die "$f ends with a newline; store it again"
|
||||
done
|
||||
}
|
||||
|
||||
# promote <release> <sha>: make the release the live tree. --chmod applies to
|
||||
@@ -239,7 +248,7 @@ probe() {
|
||||
}
|
||||
|
||||
smoke_tests() {
|
||||
local i body code location
|
||||
local i body code location realm
|
||||
log "smoke tests"
|
||||
# First deploy: Let's Encrypt may need a minute to issue the certificate.
|
||||
for i in $(seq 1 24); do
|
||||
@@ -249,17 +258,21 @@ smoke_tests() {
|
||||
sleep 5
|
||||
done
|
||||
|
||||
# The realm this release runs on. These tests check redirects only, not a real sign-in: they
|
||||
# pass with a wrong client secret, so a person signs in after a change of realm or secret.
|
||||
realm=$(fm exec -T superset printenv KEYCLOAK_REALM_URL) || die "Superset has no KEYCLOAK_REALM_URL"
|
||||
|
||||
read -r code location < <(probe "$ATOMA_HOST" /login/)
|
||||
[ "$code" = 302 ] && [[ $location == "https://$ATOMA_HOST/login/keycloak"* ]] ||
|
||||
die "/login/ should redirect to /login/keycloak, got $code $location"
|
||||
|
||||
read -r code location < <(probe "$ATOMA_HOST" /login/keycloak)
|
||||
[ "$code" = 302 ] && [[ $location == "$KEYCLOAK_REALM_URL/protocol/openid-connect/auth?"* ]] &&
|
||||
[ "$code" = 302 ] && [[ $location == "$realm/protocol/openid-connect/auth?"* ]] &&
|
||||
[[ $location == *client_id=petrolyst* && $location == *code_challenge_method=S256* ]] ||
|
||||
die "/login/keycloak should redirect to Keycloak with client_id=petrolyst and PKCE S256, got $code $location"
|
||||
|
||||
read -r code location < <(probe "$TRAEFIK_HOST" /)
|
||||
[ "$code" = 302 ] && [[ $location == "$KEYCLOAK_REALM_URL/"* || $location == *"/oauth2/start"* ]] ||
|
||||
[ "$code" = 302 ] && [[ $location == "$realm/"* || $location == *"/oauth2/start"* ]] ||
|
||||
die "https://$TRAEFIK_HOST/ should redirect to Keycloak, got $code $location"
|
||||
|
||||
# Superset's own connection "Fuel" (fuel_ro, password from /run/secrets) reads fuel-db and
|
||||
|
||||
@@ -2,7 +2,8 @@
|
||||
# Server secrets, run as root after bootstrap.sh.
|
||||
# secrets-init.sh generate every missing secret; never overwrites one
|
||||
# secrets-init.sh keycloak-secret store the client secret of Keycloak client
|
||||
# "petrolyst" (Clients > petrolyst > Credentials),
|
||||
# "petrolyst" of realm "petrolyst"
|
||||
# (Clients > petrolyst > Credentials),
|
||||
# read from the terminal or stdin
|
||||
#
|
||||
# Layout: /opt/petrolyst/secrets/{shared,traefik,fuel-monitoring}/ are root:deploy
|
||||
@@ -50,8 +51,8 @@ case ${1:-all} in
|
||||
fi
|
||||
value=$(printf '%s' "$value" | tr -d '[:space:]')
|
||||
[ -n "$value" ] || die "empty secret"
|
||||
write_secret "$SECRETS/shared/keycloak_petrolyst_client_secret" "$value"
|
||||
echo "stored $SECRETS/shared/keycloak_petrolyst_client_secret"
|
||||
write_secret "$SECRETS/shared/keycloak_petrolyst_realm_client_secret" "$value"
|
||||
echo "stored $SECRETS/shared/keycloak_petrolyst_realm_client_secret"
|
||||
exit 0
|
||||
;;
|
||||
all) ;;
|
||||
@@ -66,10 +67,10 @@ generate "$SECRETS/fuel-monitoring/fuel_db_ro_password" 32
|
||||
# 32 hex characters = a 32-byte cookie key, as oauth2-proxy requires.
|
||||
generate "$SECRETS/traefik/oauth2_proxy_cookie_secret" 16
|
||||
|
||||
if [ ! -s "$SECRETS/shared/keycloak_petrolyst_client_secret" ]; then
|
||||
if [ ! -s "$SECRETS/shared/keycloak_petrolyst_realm_client_secret" ]; then
|
||||
echo
|
||||
echo "Still missing: the Keycloak client secret. Copy it from Keycloak"
|
||||
echo "(realm thedigitalbridge > Clients > petrolyst > Credentials), then run:"
|
||||
echo "(realm petrolyst > Clients > petrolyst > Credentials), then run:"
|
||||
echo " $0 keycloak-secret"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
@@ -0,0 +1,114 @@
|
||||
{
|
||||
"clients": [
|
||||
{
|
||||
"clientId": "petrolyst",
|
||||
"name": "Petrolyst administration",
|
||||
"description": "Superset and the Traefik dashboard (oauth2-proxy), for our own people, with MFA (browser flow override petrolyst-mfa, set by hand). Managed from petrolyst-infra/keycloak/.",
|
||||
"enabled": true,
|
||||
"protocol": "openid-connect",
|
||||
"publicClient": false,
|
||||
"clientAuthenticatorType": "client-secret",
|
||||
"standardFlowEnabled": true,
|
||||
"implicitFlowEnabled": false,
|
||||
"directAccessGrantsEnabled": false,
|
||||
"serviceAccountsEnabled": false,
|
||||
"frontchannelLogout": true,
|
||||
"baseUrl": "https://app.petrolyst.com/",
|
||||
"redirectUris": [
|
||||
"https://app.petrolyst.com/oauth-authorized/keycloak",
|
||||
"https://traefik.petrolyst.com/oauth2/callback"
|
||||
],
|
||||
"webOrigins": ["+"],
|
||||
"attributes": {
|
||||
"pkce.code.challenge.method": "S256",
|
||||
"post.logout.redirect.uris": "https://app.petrolyst.com/*##https://traefik.petrolyst.com/*"
|
||||
},
|
||||
"protocolMappers": [
|
||||
{
|
||||
"name": "petrolyst client roles",
|
||||
"protocol": "openid-connect",
|
||||
"protocolMapper": "oidc-usermodel-client-role-mapper",
|
||||
"consentRequired": false,
|
||||
"config": {
|
||||
"usermodel.clientRoleMapping.clientId": "petrolyst",
|
||||
"multivalued": "true",
|
||||
"claim.name": "petrolyst_roles",
|
||||
"jsonType.label": "String",
|
||||
"id.token.claim": "true",
|
||||
"access.token.claim": "true",
|
||||
"userinfo.token.claim": "true",
|
||||
"introspection.token.claim": "true"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"clientId": "petrolyst-app",
|
||||
"name": "Petrolyst",
|
||||
"description": "The Petrolyst app (petrolyst-platform server/), for customers' users. Password only (PD-21). The customer comes from the person's top-level group. Managed from petrolyst-infra/keycloak/.",
|
||||
"enabled": true,
|
||||
"protocol": "openid-connect",
|
||||
"publicClient": false,
|
||||
"clientAuthenticatorType": "client-secret",
|
||||
"standardFlowEnabled": true,
|
||||
"implicitFlowEnabled": false,
|
||||
"directAccessGrantsEnabled": false,
|
||||
"serviceAccountsEnabled": false,
|
||||
"frontchannelLogout": true,
|
||||
"baseUrl": "https://app.petrolyst.com/app/",
|
||||
"redirectUris": [
|
||||
"https://app.petrolyst.com/app/auth/callback"
|
||||
],
|
||||
"webOrigins": [],
|
||||
"attributes": {
|
||||
"pkce.code.challenge.method": "S256",
|
||||
"post.logout.redirect.uris": "https://app.petrolyst.com/app/*"
|
||||
},
|
||||
"protocolMappers": [
|
||||
{
|
||||
"name": "petrolyst client roles",
|
||||
"protocol": "openid-connect",
|
||||
"protocolMapper": "oidc-usermodel-client-role-mapper",
|
||||
"consentRequired": false,
|
||||
"config": {
|
||||
"usermodel.clientRoleMapping.clientId": "petrolyst-app",
|
||||
"multivalued": "true",
|
||||
"claim.name": "petrolyst_roles",
|
||||
"jsonType.label": "String",
|
||||
"id.token.claim": "true",
|
||||
"access.token.claim": "false",
|
||||
"userinfo.token.claim": "false",
|
||||
"introspection.token.claim": "false"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "petrolyst groups",
|
||||
"protocol": "openid-connect",
|
||||
"protocolMapper": "oidc-group-membership-mapper",
|
||||
"consentRequired": false,
|
||||
"config": {
|
||||
"full.path": "true",
|
||||
"claim.name": "petrolyst_groups",
|
||||
"id.token.claim": "true",
|
||||
"access.token.claim": "false",
|
||||
"userinfo.token.claim": "false",
|
||||
"introspection.token.claim": "false"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"roles": {
|
||||
"client": {
|
||||
"petrolyst": [
|
||||
{ "name": "superset-admin", "description": "Superset Admin" },
|
||||
{ "name": "superset-editor", "description": "Superset Alpha: build charts and dashboards" },
|
||||
{ "name": "superset-viewer", "description": "Superset Gamma: view dashboards shared with the role" },
|
||||
{ "name": "traefik-admin", "description": "Traefik dashboard at traefik.petrolyst.com" }
|
||||
],
|
||||
"petrolyst-app": [
|
||||
{ "name": "reader", "description": "Reads their customer's data in the Petrolyst app; no role in Superset. Mapped onto each customer's group." }
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -15,8 +15,8 @@ and where it runs lives in `stacks/`, `traefik/`, `deploy/` and `host/`.
|
||||
|---|---|---|
|
||||
| `superset_config.py` | Keycloak sign-in, role mapping, proxy and cookie settings, Redis caches, theme and chart palettes | `/app/pythonpath/` |
|
||||
| `petrolyst_security.py` | `PetrolystSecurityManager`: user info and roles from the Keycloak ID token, `/login/` straight to Keycloak, `/logout/` through Keycloak | `/app/pythonpath/` |
|
||||
| `roles/` | `atoma_guest.py`: the role "ATOMA Guest" for the Petrolyst app's embedded dashboards, with exactly its permissions; run by hand (see "Embedding") | `/app/petrolyst-roles/` |
|
||||
| `assets/branding/` | logos (full colour and white): the wordmark alone for the navbar, and the stacked logo; favicons, app icons; ATOMA's logo, which the Petrolyst app shows in its map's corner (`PETROLYST_OPERATOR_LOGO`) | `/app/superset/static/assets/branding/`, served at `/static/assets/branding/…` |
|
||||
| `roles/` | `petrolyst_guest.py`: the role "Petrolyst Guest" for the Petrolyst app's embedded dashboards, with exactly its permissions; run by hand (see "Embedding") | `/app/petrolyst-roles/` |
|
||||
| `assets/branding/` | logos (full colour and white): the wordmark alone for the navbar, and the stacked logo; favicons, app icons; ATOMA's logo, which the Petrolyst app shows in its map's corner (the server reads this file itself: `PETROLYST_OPERATOR_LOGO_FILE`) | `/app/superset/static/assets/branding/`, served at `/static/assets/branding/…` |
|
||||
| `patches/` | runtime patches, `fix_*.py` with `--verify` (see its README); none at present | `/app/patches/petrolyst/`, applied during the build |
|
||||
| `dashboards/` | exported dashboards and import scripts (see its README) | not in the image |
|
||||
|
||||
@@ -35,13 +35,13 @@ value.
|
||||
|---|---|
|
||||
| `superset_secret_key` | Flask `SECRET_KEY` |
|
||||
| `superset_db_password` | password of user `superset` on the metadata database |
|
||||
| `keycloak_petrolyst_client_secret` | client secret of Keycloak client `petrolyst` |
|
||||
| `keycloak_petrolyst_client_secret` | client secret of Keycloak client `petrolyst`, realm `petrolyst`; on the server in `shared/keycloak_petrolyst_realm_client_secret` |
|
||||
|
||||
On the server they come from `/opt/petrolyst/secrets/` (see the deployment README).
|
||||
|
||||
## Sign-in and roles
|
||||
|
||||
- Provider: Keycloak realm `thedigitalbridge` at `iam.thedigitalbridge.io`, client `petrolyst`,
|
||||
- Provider: Keycloak realm `petrolyst` at `iam.thedigitalbridge.io`, client `petrolyst`,
|
||||
authorization code flow with PKCE S256. The realm URL can be overridden with
|
||||
`KEYCLOAK_REALM_URL`.
|
||||
- Roles come from the client roles of `petrolyst`, which a protocol mapper puts into the claim
|
||||
@@ -68,14 +68,14 @@ stack** (`.local/fmdb-scratch`); `superset_config.py` here leaves it off, and lo
|
||||
guest secret.
|
||||
|
||||
- The scratch stack's `superset_config_local.py` turns on `EMBEDDED_SUPERSET`, names the guest
|
||||
role `GUEST_ROLE_NAME = "ATOMA Guest"`, reads the signing secret with
|
||||
role `GUEST_ROLE_NAME = "Petrolyst Guest"`, reads the signing secret with
|
||||
`_read_secret("superset_guest_token_secret")` and expects the audience `petrolyst-app`. The
|
||||
Petrolyst server signs the guest passes itself with the same secret.
|
||||
- `roles/atoma_guest.py` gives "ATOMA Guest" exactly what a dashboard needs to be read. Which
|
||||
- `roles/petrolyst_guest.py` gives "Petrolyst Guest" exactly what a dashboard needs to be read. Which
|
||||
dashboard a guest sees comes from the pass, not the role. It is not in `init.sh`; run it once,
|
||||
and again whenever its permissions change:
|
||||
|
||||
docker exec <superset> /app/.venv/bin/python /app/petrolyst-roles/atoma_guest.py
|
||||
docker exec <superset> /app/.venv/bin/python /app/petrolyst-roles/petrolyst_guest.py
|
||||
|
||||
- The app's dashboards and their embedding ids: `dashboards/README.md`.
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ Superset dashboards as code and the operator scripts that import them. Not baked
|
||||
image, so changes here do not rebuild or restart Superset.
|
||||
|
||||
- `atoma/`: ATOMA's two dashboards for the Petrolyst app (PD-12, PD-13; SPEC.md §10):
|
||||
`atoma-overview` (fleet totals) and `atoma-site` (one site, its four sections as tabs). The
|
||||
`overview` (fleet totals) and `site` (one site, its four sections as tabs), named `atoma-overview` and `atoma-site` until 30 Oct. The
|
||||
app embeds them; admins open them in Superset, with the filter column. `SPEC.md` is the
|
||||
design; `build_bundle.py` generates the import bundle and `import_bundle.py` imports it
|
||||
(overwriting the objects the bundle defines).
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
# ATOMA dashboards: Overview and Site, for the Petrolyst app
|
||||
|
||||
**Status (29 Oct 2026, PD-13): two dashboards, `atoma-overview` and `atoma-site` (§10), built and
|
||||
**Status (30 Oct 2026, PD-21: every customer sees these two, limited by row; slugs `overview` and `site`, formerly
|
||||
`atoma-overview` and `atoma-site`, which the rest of this file still uses). Two dashboards (§10), built and
|
||||
tested on a local Superset; not imported live.** The Petrolyst app embeds them, and admins open
|
||||
them in Superset with the filter column.
|
||||
- `build_bundle.py` generates the dashboards, every dataset and every chart from this
|
||||
@@ -16,8 +17,7 @@ them in Superset with the filter column.
|
||||
|
||||
**Datasets.** The `public.v_*` views of
|
||||
`stacks/fuel-monitoring/fuel-db/migrations/0003_bi_views.sql`, read from fuel-db directly (database
|
||||
`fuel`, schema `public`) as the read-only role `fuel_ro`. *Until 29 Oct 2026 Superset read them
|
||||
through Trino as `fuel.public`; Trino was removed then.*
|
||||
`fuel`, schema `public`) as the read-only role `fuel_ro`.
|
||||
|
||||
**Design and rationale** live in the fuel-monitoring corpus, in
|
||||
`.context/plans/2026-10-24-fuel-db-baseline-dashboard.md`:
|
||||
@@ -231,7 +231,7 @@ Physical datasets, one per `public.v_*` view in fuel-db's `public` schema, plus
|
||||
so the opening refill is included) or a bill's AFN amount.
|
||||
|
||||
Both use only `v_coverage`, `v_delivery` and `v_electricity_cost`. Every dataset has the same
|
||||
four site columns from `v_site`: `site_uid`, `site_code`, `site_name`, `site_label`.
|
||||
five columns from `v_site`: `site_uid`, `site_code`, `site_name`, `site_label`, `operator_uid` (the customer; the row filter uses it, PD-16).
|
||||
|
||||
| Dataset | Grain | Used by | Native filters that apply |
|
||||
|---|---|---|---|
|
||||
@@ -259,7 +259,7 @@ across methods or generator sets. The site's L/h figure itself comes only from a
|
||||
(petrolyst-platform `views/src/shared/deepLink.ts`, retired) from its iframe address: the
|
||||
dashboard path, the Site filter's id and the tab. The Petrolyst app uses the same rison shape for
|
||||
the Site filter's preset (`app/src/app/core/site-filter.ts`). The rison shape is the one the sites
|
||||
list used until 28 Oct, confirmed in P2, and the view's tests compared it with that list's Trino
|
||||
list used until 28 Oct, confirmed in P2, and the view's tests compared it with that list's
|
||||
output:
|
||||
|
||||
```
|
||||
@@ -409,7 +409,7 @@ Run in P2 against a local Superset, and repeated as a smoke test in P3.
|
||||
|
||||
## 8. P2 results (local, 24 Oct 2026)
|
||||
|
||||
Local Superset 6.1.0 image built from this repository, Trino 484, fuel-db with migrations
|
||||
Local Superset 6.1.0 image built from this repository, fuel-db with migrations
|
||||
0001–0003 and the corpus data loaded; browser checks in headless Chrome.
|
||||
|
||||
| Check | Result |
|
||||
|
||||
@@ -1,9 +1,10 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Generate ATOMA's dashboards for the Petrolyst app as one Superset 6.1 import bundle.
|
||||
"""Generate the dashboards of the Petrolyst app as one Superset 6.1 import bundle.
|
||||
|
||||
Implements SPEC.md in this folder: the two dashboards the Petrolyst app embeds, "ATOMA · Overview"
|
||||
(atoma-overview) and "ATOMA · Site" (atoma-site, the four detail tabs), one dataset per
|
||||
public.v_* view and every chart. Nothing in the bundle names a site.
|
||||
Implements SPEC.md in this folder: the two dashboards the app embeds, "Overview" (overview) and
|
||||
"Site" (site, the four detail tabs), one dataset per public.v_* view and every chart. Every
|
||||
customer sees the same dashboards, limited by row: the app's guest pass names the customer (PD-16).
|
||||
Nothing in the bundle names a site or a customer.
|
||||
|
||||
build_bundle.py --database-uuid <uuid of "Fuel"> [--out atoma-dashboards.zip]
|
||||
|
||||
@@ -15,7 +16,8 @@ dashboards (the stock dashboard import leaves existing datasets and charts
|
||||
unchanged).
|
||||
|
||||
Before writing, the bundle is checked: every column a chart or filter uses
|
||||
exists in its dataset, every chart in a Site filter's scope has site_label, the
|
||||
exists in its dataset, every dataset has operator_uid (the guest pass's row rule needs
|
||||
it), every chart in a Site filter's scope has site_label, the
|
||||
Overview sums recorded values only and money only per currency, and no row is
|
||||
wider than 12. Requires PyYAML.
|
||||
"""
|
||||
@@ -109,7 +111,7 @@ CALCULATED = {
|
||||
# opening visit's delivery is included and the closing visit's excluded.
|
||||
VIRTUAL = {
|
||||
"coverage_detail": ("v_coverage", [("lane_name", "character varying(60)"), ("amount", "character varying(120)")], """\
|
||||
SELECT c.site_uid, c.site_code, c.site_name, c.site_label, c.lane, c.status,
|
||||
SELECT c.site_uid, c.site_code, c.site_name, c.site_label, c.operator_uid, c.lane, c.status,
|
||||
c.start_on, c.end_on, c.start_ts, c.end_ts,
|
||||
regexp_replace(c.lane, '^[0-9]+ ', '') AS lane_name,
|
||||
CASE WHEN c.lane = '1 Stock intervals' THEN
|
||||
@@ -166,6 +168,7 @@ METRICS = {
|
||||
|
||||
LABELS = {
|
||||
"site_code": "Code", "site_name": "Site name", "site_label": "Site", "site_uid": "Site uid",
|
||||
"operator_uid": "Customer",
|
||||
"pilot_status": "Pilot status", "baseline_status_label": "Baseline status",
|
||||
"baseline_status": "Baseline status code", "baseline_validity_note": "Validity note",
|
||||
"lph_estimate": "L/h", "lph_method": "L/h method", "lph_window_start": "L/h window start",
|
||||
@@ -495,8 +498,11 @@ def timeseries_common(x: str, metrics: list[Any], groupby: list[str], grain: str
|
||||
|
||||
# --- Overview (PD-12, embedded in the Petrolyst app) ---------------------------------------
|
||||
|
||||
OVERVIEW_SLUG = "atoma-overview"
|
||||
SITE_SLUG = "atoma-site" # one embedded dashboard with the DETAIL_TABS as its tabs
|
||||
OVERVIEW_SLUG = "overview"
|
||||
SITE_SLUG = "site" # one embedded dashboard with the DETAIL_TABS as its tabs
|
||||
# A dashboard's identity, apart from its slug: the names it was first built under, so a re-import
|
||||
# updates the existing two (and keeps their embedding ids) rather than adding two new ones.
|
||||
DASHBOARD_KEYS = {OVERVIEW_SLUG: "atoma-overview", SITE_SLUG: "atoma-site"}
|
||||
|
||||
# SPEC §10: only sums and counts of recorded values, never an average of per-site rates; money
|
||||
# only grouped by currency. The value says whether the metric is money.
|
||||
@@ -934,7 +940,7 @@ def dashboard_yaml(key: str, title: str, slug: str, layout: Layout, metadata: di
|
||||
"certified_by": None,
|
||||
"certification_details": None,
|
||||
"published": True,
|
||||
"uuid": uid("dashboard", key),
|
||||
"uuid": uid("dashboard", DASHBOARD_KEYS[key]),
|
||||
"position": layout.nodes,
|
||||
"metadata": metadata,
|
||||
"version": VERSION,
|
||||
@@ -999,7 +1005,7 @@ def build(database_uuid: str, columns_path: Path) -> dict[str, Any]:
|
||||
all_charts = {**{c.key: c for c in detail_charts()}, **overview}
|
||||
dashboards: list[tuple[Layout, dict[str, Chart], dict[str, Any]]] = []
|
||||
for key, title in [(OVERVIEW_SLUG, "Overview"), (SITE_SLUG, "Site")]:
|
||||
page = Layout(f"ATOMA · {title}", all_charts)
|
||||
page = Layout(title, all_charts)
|
||||
page.markdown.update({f"notes-{k}": v for k, v in TAB_NOTES.items()})
|
||||
page.markdown["notes-overview"] = OVERVIEW_NOTES
|
||||
if key == OVERVIEW_SLUG:
|
||||
@@ -1012,7 +1018,7 @@ def build(database_uuid: str, columns_path: Path) -> dict[str, Any]:
|
||||
placed = {k: all_charts[k] for k in page.ids}
|
||||
page_metadata = base_metadata(page, filters, cross_filters=False)
|
||||
page_metadata["label_colors"].update(STOCK_STATUS_COLOURS)
|
||||
files[f"dashboards/{key}.yaml"] = dashboard_yaml(key, f"ATOMA · {title}", key, page,
|
||||
files[f"dashboards/{key}.yaml"] = dashboard_yaml(key, title, key, page,
|
||||
page_metadata)
|
||||
dashboards.append((page, placed, page_metadata))
|
||||
|
||||
@@ -1030,6 +1036,12 @@ def check(files: dict[str, Any], views: dict[str, list[tuple[str, str]]],
|
||||
problems = []
|
||||
columns = {v: {c for c, _ in cols} | {c[0] for c in CALCULATED.get(v, [])}
|
||||
for v, cols in views.items()}
|
||||
for view, cols in columns.items():
|
||||
if "operator_uid" not in cols:
|
||||
problems.append(f"{view}: no operator_uid, so the guest pass's row rule cannot hold")
|
||||
for name, (_, _, sql) in VIRTUAL.items():
|
||||
if "operator_uid" not in sql:
|
||||
problems.append(f"{name}: its SQL does not select operator_uid")
|
||||
for view in list(MAIN_DTTM) + list(CALCULATED) + list(METRICS):
|
||||
if view not in columns:
|
||||
problems.append(f"{view}: configured but not in the column snapshot")
|
||||
|
||||
@@ -9,6 +9,7 @@ v_assumption site_uid character varying(12)
|
||||
v_assumption site_code character varying(20)
|
||||
v_assumption site_name character varying(120)
|
||||
v_assumption site_label character varying(170)
|
||||
v_assumption operator_uid character varying(12)
|
||||
v_assumption code character varying(60)
|
||||
v_assumption topic character varying(30)
|
||||
v_assumption asset_uid character varying(40)
|
||||
@@ -30,6 +31,7 @@ v_baseline_input site_uid character varying(12)
|
||||
v_baseline_input site_code character varying(20)
|
||||
v_baseline_input site_name character varying(120)
|
||||
v_baseline_input site_label character varying(170)
|
||||
v_baseline_input operator_uid character varying(12)
|
||||
v_baseline_input baseline_code character varying(60)
|
||||
v_baseline_input baseline_status character varying(12)
|
||||
v_baseline_input interval_label character varying(30)
|
||||
@@ -47,6 +49,7 @@ v_baseline_version site_uid character varying(12)
|
||||
v_baseline_version site_code character varying(20)
|
||||
v_baseline_version site_name character varying(120)
|
||||
v_baseline_version site_label character varying(170)
|
||||
v_baseline_version operator_uid character varying(12)
|
||||
v_baseline_version code character varying(60)
|
||||
v_baseline_version metric character varying(40)
|
||||
v_baseline_version status character varying(12)
|
||||
@@ -75,6 +78,7 @@ v_coverage site_uid character varying(12)
|
||||
v_coverage site_code character varying(20)
|
||||
v_coverage site_name character varying(120)
|
||||
v_coverage site_label character varying(170)
|
||||
v_coverage operator_uid character varying(12)
|
||||
v_coverage lane character varying(60)
|
||||
v_coverage status character varying(40)
|
||||
v_coverage start_on date
|
||||
@@ -85,6 +89,7 @@ v_data_quality site_uid character varying(12)
|
||||
v_data_quality site_code character varying(20)
|
||||
v_data_quality site_name character varying(120)
|
||||
v_data_quality site_label character varying(170)
|
||||
v_data_quality operator_uid character varying(12)
|
||||
v_data_quality check_order integer
|
||||
v_data_quality check_code character varying(40)
|
||||
v_data_quality label character varying(80)
|
||||
@@ -94,6 +99,7 @@ v_delivery site_uid character varying(12)
|
||||
v_delivery site_code character varying(20)
|
||||
v_delivery site_name character varying(120)
|
||||
v_delivery site_label character varying(170)
|
||||
v_delivery operator_uid character varying(12)
|
||||
v_delivery event_key character varying(80)
|
||||
v_delivery delivery_date date
|
||||
v_delivery seq_in_day integer
|
||||
@@ -108,6 +114,7 @@ v_electricity_cost site_uid character varying(12)
|
||||
v_electricity_cost site_code character varying(20)
|
||||
v_electricity_cost site_name character varying(120)
|
||||
v_electricity_cost site_label character varying(170)
|
||||
v_electricity_cost operator_uid character varying(12)
|
||||
v_electricity_cost bill_month date
|
||||
v_electricity_cost period_start date
|
||||
v_electricity_cost period_end date
|
||||
@@ -122,6 +129,7 @@ v_evidence site_uid character varying(12)
|
||||
v_evidence site_code character varying(20)
|
||||
v_evidence site_name character varying(120)
|
||||
v_evidence site_label character varying(170)
|
||||
v_evidence operator_uid character varying(12)
|
||||
v_evidence document_key character varying(16)
|
||||
v_evidence title character varying(200)
|
||||
v_evidence kind character varying(20)
|
||||
@@ -133,6 +141,7 @@ v_hour_meter_evidence site_uid character varying(12)
|
||||
v_hour_meter_evidence site_code character varying(20)
|
||||
v_hour_meter_evidence site_name character varying(120)
|
||||
v_hour_meter_evidence site_label character varying(170)
|
||||
v_hour_meter_evidence operator_uid character varying(12)
|
||||
v_hour_meter_evidence meter_label character varying(40)
|
||||
v_hour_meter_evidence mapped_set character varying(60)
|
||||
v_hour_meter_evidence reading_date date
|
||||
@@ -146,6 +155,7 @@ v_imported_calculation site_uid character varying(12)
|
||||
v_imported_calculation site_code character varying(20)
|
||||
v_imported_calculation site_name character varying(120)
|
||||
v_imported_calculation site_label character varying(170)
|
||||
v_imported_calculation operator_uid character varying(12)
|
||||
v_imported_calculation cell_ref character varying(40)
|
||||
v_imported_calculation source_label character varying(80)
|
||||
v_imported_calculation formula character varying(200)
|
||||
@@ -160,6 +170,7 @@ v_interval site_uid character varying(12)
|
||||
v_interval site_code character varying(20)
|
||||
v_interval site_name character varying(120)
|
||||
v_interval site_label character varying(170)
|
||||
v_interval operator_uid character varying(12)
|
||||
v_interval interval_label character varying(30)
|
||||
v_interval start_on date
|
||||
v_interval end_on date
|
||||
@@ -193,6 +204,7 @@ v_kpi_measures site_uid character varying(12)
|
||||
v_kpi_measures site_code character varying(20)
|
||||
v_kpi_measures site_name character varying(120)
|
||||
v_kpi_measures site_label character varying(170)
|
||||
v_kpi_measures operator_uid character varying(12)
|
||||
v_kpi_measures measure_order integer
|
||||
v_kpi_measures measure character varying(60)
|
||||
v_kpi_measures value numeric(12,2)
|
||||
@@ -205,6 +217,7 @@ v_modelled_lph site_uid character varying(12)
|
||||
v_modelled_lph site_code character varying(20)
|
||||
v_modelled_lph site_name character varying(120)
|
||||
v_modelled_lph site_label character varying(170)
|
||||
v_modelled_lph operator_uid character varying(12)
|
||||
v_modelled_lph assumption_code character varying(60)
|
||||
v_modelled_lph status character varying(12)
|
||||
v_modelled_lph basis character varying(20)
|
||||
@@ -223,6 +236,7 @@ v_reconciliation site_uid character varying(12)
|
||||
v_reconciliation site_code character varying(20)
|
||||
v_reconciliation site_name character varying(120)
|
||||
v_reconciliation site_label character varying(170)
|
||||
v_reconciliation operator_uid character varying(12)
|
||||
v_reconciliation interval_label character varying(30)
|
||||
v_reconciliation start_on date
|
||||
v_reconciliation end_on date
|
||||
@@ -242,6 +256,7 @@ v_runtime site_uid character varying(12)
|
||||
v_runtime site_code character varying(20)
|
||||
v_runtime site_name character varying(120)
|
||||
v_runtime site_label character varying(170)
|
||||
v_runtime operator_uid character varying(12)
|
||||
v_runtime meter_label character varying(40)
|
||||
v_runtime mapped_set character varying(60)
|
||||
v_runtime interval_label character varying(80)
|
||||
@@ -259,6 +274,7 @@ v_site site_uid character varying(12)
|
||||
v_site site_code character varying(20)
|
||||
v_site site_name character varying(120)
|
||||
v_site site_label character varying(170)
|
||||
v_site operator_uid character varying(12)
|
||||
v_site zone character varying(40)
|
||||
v_site province character varying(40)
|
||||
v_site district character varying(60)
|
||||
@@ -283,9 +299,10 @@ v_spec_reference site_uid character varying(12)
|
||||
v_spec_reference site_code character varying(20)
|
||||
v_spec_reference site_name character varying(120)
|
||||
v_spec_reference site_label character varying(170)
|
||||
v_spec_reference operator_uid character varying(12)
|
||||
v_spec_reference slot character varying(20)
|
||||
v_spec_reference asset_uid character varying(40)
|
||||
v_spec_reference asset_kind character varying(10)
|
||||
v_spec_reference asset_kind character varying(20)
|
||||
v_spec_reference make_model character varying(120)
|
||||
v_spec_reference attribute character varying(40)
|
||||
v_spec_reference load_pct numeric(5,1)
|
||||
@@ -301,6 +318,7 @@ v_stock_reading site_uid character varying(12)
|
||||
v_stock_reading site_code character varying(20)
|
||||
v_stock_reading site_name character varying(120)
|
||||
v_stock_reading site_label character varying(170)
|
||||
v_stock_reading operator_uid character varying(12)
|
||||
v_stock_reading observed_on date
|
||||
v_stock_reading seq_in_day integer
|
||||
v_stock_reading reading_type character varying(20)
|
||||
|
||||
|
Can't render this file because it contains an unexpected character in line 3 and column 80.
|
@@ -17,7 +17,7 @@ live view and any drift is reported (exit status 2).
|
||||
The database is never created or changed: the bundle's database uuid must
|
||||
already exist under the name "Fuel", otherwise nothing is imported.
|
||||
|
||||
The dashboards (atoma-overview, atoma-site) get no role: admins see them in Superset, and
|
||||
The dashboards (overview, site) get no role: admins see them in Superset, and
|
||||
the Petrolyst app shows them with guest passes (PD-12). With --embedded-out FILE
|
||||
--embed-domain URL, each of them is made embeddable from that origin (its embedding id stays
|
||||
the same across imports), and FILE receives {slug: embedding id} for the Petrolyst server:
|
||||
@@ -36,7 +36,7 @@ import zipfile
|
||||
import yaml
|
||||
|
||||
DATABASE_NAME = "Fuel"
|
||||
EMBEDDED_PREFIX = "atoma-"
|
||||
EMBEDDED_SLUGS = {"overview", "site"}
|
||||
STEPS = [("datasets", "SqlaTable"), ("charts", "Slice"), ("dashboards", "Dashboard")]
|
||||
|
||||
|
||||
@@ -104,12 +104,17 @@ def main() -> int:
|
||||
if path.startswith("dashboards/"):
|
||||
dashboard = db.session.query(Dashboard).filter_by(
|
||||
uuid=yaml.safe_load(text)["uuid"]).one()
|
||||
if dashboard.slug.startswith(EMBEDDED_PREFIX) and args.embedded_out:
|
||||
if dashboard.slug in EMBEDDED_SLUGS and args.embedded_out:
|
||||
from superset.daos.dashboard import EmbeddedDashboardDAO
|
||||
|
||||
config = EmbeddedDashboardDAO.upsert(dashboard, [args.embed_domain])
|
||||
db.session.flush()
|
||||
embedded[dashboard.slug] = str(config.uuid)
|
||||
# An empty or short list would stop the app's dashboards from opening.
|
||||
if args.embedded_out and set(embedded) != EMBEDDED_SLUGS:
|
||||
db.session.rollback()
|
||||
print(f"embedded: expected {sorted(EMBEDDED_SLUGS)}, got {sorted(embedded)}", file=sys.stderr)
|
||||
return 3
|
||||
db.session.commit()
|
||||
if args.embedded_out:
|
||||
with open(args.embedded_out, "w") as out:
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
"""Keycloak sign-in for Superset at atoma.petrolyst.com.
|
||||
|
||||
Set as CUSTOM_SECURITY_MANAGER in superset_config.py. Checked against
|
||||
Flask-AppBuilder 5.0.2 as bundled in apache/superset:6.1.0. Three changes to
|
||||
Flask-AppBuilder 5.0.2 as bundled in apache/superset:6.1.0. Four changes to
|
||||
the stock behaviour:
|
||||
|
||||
- User info and roles come from the ID token that Keycloak issues for client
|
||||
@@ -11,6 +11,8 @@ the stock behaviour:
|
||||
login page on /login/, which only shows a "Sign in with keycloak" button.
|
||||
- /logout/ also ends the Keycloak session, so the next visit asks for the
|
||||
password and OTP again.
|
||||
- A Keycloak user with no mapped Superset role is refused, and no account is
|
||||
made. Customers' users belong in the Petrolyst app, never here (PD-21).
|
||||
"""
|
||||
|
||||
import logging
|
||||
@@ -70,6 +72,14 @@ class PetrolystSecurityManager(SupersetSecurityManager):
|
||||
# FAB's OAuth view, and that page wins the route.
|
||||
register_superset_auth_view = False
|
||||
|
||||
def auth_user_oauth(self, userinfo: dict[str, Any]) -> Any:
|
||||
# Only the mapped roles count: FAB's own calculation always adds the
|
||||
# registration role, so it is never empty.
|
||||
if not self.get_roles_from_keys(userinfo.get("role_keys", [])):
|
||||
logger.warning("Keycloak sign-in refused: no mapped Superset role")
|
||||
return None
|
||||
return super().auth_user_oauth(userinfo)
|
||||
|
||||
def oauth_user_info(
|
||||
self, provider: str, response: Optional[dict[str, Any]] = None
|
||||
) -> dict[str, Any]:
|
||||
|
||||
+5
-4
@@ -1,14 +1,15 @@
|
||||
"""The role "ATOMA Guest": what the Petrolyst app's embedded dashboards may do (PD-12).
|
||||
"""The role "Petrolyst Guest": what the Petrolyst app's embedded dashboards may do (PD-12).
|
||||
|
||||
Superset gives every embedded viewer (a guest pass signed for one dashboard) the role named by
|
||||
GUEST_ROLE_NAME. Which dashboard a guest sees comes from the pass, not from the role. Superset
|
||||
GUEST_ROLE_NAME. Which dashboard and which customer's rows a guest sees come from the pass, not from
|
||||
the role. Superset
|
||||
leaves the SQL out of chart-data responses for guests.
|
||||
|
||||
Embedding is switched on in the local stack only (.local/fmdb-scratch, superset_config_local.py),
|
||||
so this script is not part of init.sh; run it by hand inside the Superset container, once, and
|
||||
again whenever PERMISSIONS change:
|
||||
|
||||
docker exec <superset> /app/.venv/bin/python /app/petrolyst-roles/atoma_guest.py
|
||||
docker exec <superset> /app/.venv/bin/python /app/petrolyst-roles/petrolyst_guest.py
|
||||
|
||||
Creates the role if needed and gives it exactly PERMISSIONS: missing ones are added, any other
|
||||
is removed (also one added by hand in the UI). A second run changes nothing.
|
||||
@@ -22,7 +23,7 @@ from __future__ import annotations
|
||||
|
||||
import sys
|
||||
|
||||
ROLE = "ATOMA Guest"
|
||||
ROLE = "Petrolyst Guest"
|
||||
PERMISSIONS = [
|
||||
("can_read", "Dashboard"), # the dashboard and its charts' definitions
|
||||
("can_dashboard", "Superset"), # the dashboard page
|
||||
@@ -46,7 +46,7 @@ SQLALCHEMY_DATABASE_URI = URL.create(
|
||||
).render_as_string(hide_password=False)
|
||||
|
||||
# --- The data: fuel-db ----------------------------------------------------------
|
||||
# Superset reads fuel-db directly as the read-only role fuel_ro (no Trino since 29 Oct 2026);
|
||||
# Superset reads fuel-db directly as the read-only role fuel_ro;
|
||||
# superset-init registers that connection, "Fuel", without a password. Passwords of data
|
||||
# connections come from secret files at connect time, so none is kept in Superset's own
|
||||
# database. A connection added in the UI gets no password this way.
|
||||
@@ -64,7 +64,7 @@ SQLALCHEMY_CUSTOM_PASSWORD_STORE = _database_password
|
||||
# Users and roles are managed in Keycloak; there is no local admin account.
|
||||
# Break-glass access: `superset fab ...` inside the container on the server.
|
||||
KEYCLOAK_REALM_URL = os.environ.get(
|
||||
"KEYCLOAK_REALM_URL", "https://iam.thedigitalbridge.io/realms/thedigitalbridge"
|
||||
"KEYCLOAK_REALM_URL", "https://iam.thedigitalbridge.io/realms/petrolyst"
|
||||
).rstrip("/")
|
||||
|
||||
AUTH_TYPE = AUTH_OAUTH
|
||||
@@ -94,12 +94,12 @@ AUTH_ROLES_MAPPING = {
|
||||
"superset-admin": ["Admin"],
|
||||
"superset-editor": ["Alpha"],
|
||||
"superset-viewer": ["Gamma"],
|
||||
# "atoma-reader" opens the Petrolyst app only; here it maps to nothing (Public).
|
||||
# Customers' users have a role of client "petrolyst-app", not of this one: refused here.
|
||||
}
|
||||
AUTH_ROLES_SYNC_AT_LOGIN = True
|
||||
AUTH_USER_REGISTRATION = True
|
||||
# Public has no permissions: a Keycloak user without a Superset role signs in
|
||||
# and sees nothing.
|
||||
# Public has no permissions. A Keycloak user without a mapped role never gets
|
||||
# this far: PetrolystSecurityManager refuses the sign-in and makes no account.
|
||||
AUTH_USER_REGISTRATION_ROLE = "Public"
|
||||
|
||||
# Viewers (Gamma) see a dashboard once it lists their role.
|
||||
|
||||
@@ -1,12 +1,11 @@
|
||||
# Fuel-monitoring analytics: Superset on atoma.petrolyst.com, reading the
|
||||
# operational store (fuel-db) directly as the read-only role fuel_ro (Trino was
|
||||
# removed on 29 Oct 2026). Run only through deploy/deploy.sh or
|
||||
# operational store (fuel-db) directly as the read-only role fuel_ro. Run only through deploy/deploy.sh or
|
||||
# deploy/rollback.sh: they export SUPERSET_IMAGE_TAG from the release's
|
||||
# ARTIFACTS file, and the :? guard below stops Compose from running without it.
|
||||
name: petrolyst
|
||||
|
||||
x-superset-env: &superset-env
|
||||
KEYCLOAK_REALM_URL: ${KEYCLOAK_REALM_URL:-https://iam.thedigitalbridge.io/realms/thedigitalbridge}
|
||||
KEYCLOAK_REALM_URL: ${KEYCLOAK_REALM_URL:-https://iam.thedigitalbridge.io/realms/petrolyst}
|
||||
|
||||
x-superset: &superset
|
||||
image: fuel-superset:${SUPERSET_IMAGE_TAG:?SUPERSET_IMAGE_TAG unset - run through deploy/deploy.sh}
|
||||
@@ -159,4 +158,4 @@ secrets:
|
||||
fuel_db_ro_password:
|
||||
file: ${PETROLYST_ROOT:-/opt/petrolyst}/secrets/fuel-monitoring/fuel_db_ro_password
|
||||
keycloak_petrolyst_client_secret:
|
||||
file: ${PETROLYST_ROOT:-/opt/petrolyst}/secrets/shared/keycloak_petrolyst_client_secret
|
||||
file: ${PETROLYST_ROOT:-/opt/petrolyst}/secrets/shared/keycloak_petrolyst_realm_client_secret
|
||||
@@ -2,7 +2,9 @@
|
||||
-- DRAFT, not applied. Applied by hand by a loader acting as fuel_owner (README in this folder);
|
||||
-- deploy.sh never touches fuel-db. Design and rationale: fuel-monitoring
|
||||
-- .context/plans/2026-10-24-fuel-db-baseline-dashboard.md (sections cited as "design §n").
|
||||
-- Order: 0001, 0002, 0003, each in its own transaction.
|
||||
-- Order: 0001 to 0004, each in its own transaction.
|
||||
-- Every table but fm.asset_kind carries operator_uid, the customer (PD-16). A child names its
|
||||
-- parent and the customer together, so it cannot point at another customer's parent.
|
||||
BEGIN;
|
||||
CREATE SCHEMA fm AUTHORIZATION fuel_owner;
|
||||
CREATE EXTENSION IF NOT EXISTS btree_gist WITH SCHEMA fm; -- trusted extension; for EXCLUDE on text
|
||||
@@ -10,9 +12,21 @@ CREATE EXTENSION IF NOT EXISTS btree_gist WITH SCHEMA fm; -- trusted extension
|
||||
-- anyway, and the local test applies this file in a session whose search_path is the loader's default.
|
||||
SET LOCAL search_path = fm, pg_catalog, public;
|
||||
|
||||
-- Customers -------------------------------------------------------------------------------
|
||||
CREATE TABLE fm.operator (
|
||||
operator_uid text PRIMARY KEY CHECK (operator_uid ~ '^OPR-[0-9]{4}$'),
|
||||
name text NOT NULL,
|
||||
label_issuer text NOT NULL, -- whose labels this customer's people see (holder_alias.issuer)
|
||||
programme text NOT NULL, -- the programme v_site reports (participation.programme)
|
||||
timezone text NOT NULL, -- for a holder with no place, or a place without its own
|
||||
keycloak_group text NOT NULL UNIQUE -- the top-level Keycloak group of this customer's people (PD-21)
|
||||
CHECK (keycloak_group ~ '^/[^/]+$')
|
||||
);
|
||||
|
||||
-- Provenance ------------------------------------------------------------------------------
|
||||
CREATE TABLE fm.source_document (
|
||||
sha256 text PRIMARY KEY CHECK (sha256 ~ '^[0-9a-f]{64}$'),
|
||||
operator_uid text NOT NULL REFERENCES fm.operator,
|
||||
source_family text NOT NULL, -- stable across versions: '<operator>-<workbook>', 'site-yaml:SITE-0001'
|
||||
legacy_key16 text UNIQUE CHECK (legacy_key16 ~ '^[0-9a-f]{16}$'), -- catalog.yaml key
|
||||
kind text NOT NULL CHECK (kind IN ('delivery_form','site_logbook','operator_workbook',
|
||||
@@ -23,12 +37,15 @@ CREATE TABLE fm.source_document (
|
||||
corpus_path text, -- NULL when held outside the repository
|
||||
transcription_path text, -- the corpus note that transcribes it
|
||||
contains_personal_data boolean NOT NULL,
|
||||
supersedes_sha256 text UNIQUE REFERENCES fm.source_document,
|
||||
supersedes_sha256 text UNIQUE,
|
||||
null_reasons jsonb NOT NULL DEFAULT '{}',
|
||||
CHECK (corpus_path IS NOT NULL OR transcription_path IS NOT NULL)
|
||||
CHECK (corpus_path IS NOT NULL OR transcription_path IS NOT NULL),
|
||||
UNIQUE (operator_uid, sha256),
|
||||
FOREIGN KEY (operator_uid, supersedes_sha256) REFERENCES fm.source_document (operator_uid, sha256)
|
||||
);
|
||||
CREATE TABLE fm.import_batch (
|
||||
import_batch_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
operator_uid text NOT NULL REFERENCES fm.operator,
|
||||
source_kind text NOT NULL CHECK (source_kind IN ('site_yaml','operator_workbook','catalog','review_manifest','baseline_run')),
|
||||
source_path text NOT NULL,
|
||||
source_sha256 text NOT NULL CHECK (source_sha256 ~ '^[0-9a-f]{64}$'),
|
||||
@@ -37,33 +54,86 @@ CREATE TABLE fm.import_batch (
|
||||
started_at timestamptz NOT NULL DEFAULT now(),
|
||||
finished_at timestamptz,
|
||||
status text NOT NULL DEFAULT 'running' CHECK (status IN ('running','loaded','failed')),
|
||||
report jsonb NOT NULL DEFAULT '{}' -- counts, unmapped paths, conflicts
|
||||
report jsonb NOT NULL DEFAULT '{}', -- counts, unmapped paths, conflicts
|
||||
UNIQUE (operator_uid, import_batch_id)
|
||||
);
|
||||
CREATE TABLE fm.import_record (
|
||||
import_record_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
operator_uid text NOT NULL REFERENCES fm.operator,
|
||||
logical_key text NOT NULL, -- = source_key: '<family>:delivery:SITE-0001:2026-01-31:1'
|
||||
source_locator text NOT NULL, -- YAML path or sheet!range as read
|
||||
payload jsonb NOT NULL, -- original values, strings as written, 'UNKNOWN' kept
|
||||
payload_sha256 text NOT NULL,
|
||||
first_batch_id bigint NOT NULL REFERENCES fm.import_batch,
|
||||
last_seen_batch_id bigint NOT NULL REFERENCES fm.import_batch,
|
||||
UNIQUE (logical_key, payload_sha256)
|
||||
first_batch_id bigint NOT NULL,
|
||||
last_seen_batch_id bigint NOT NULL,
|
||||
UNIQUE (operator_uid, logical_key, payload_sha256),
|
||||
UNIQUE (operator_uid, import_record_id),
|
||||
FOREIGN KEY (operator_uid, first_batch_id) REFERENCES fm.import_batch (operator_uid, import_batch_id),
|
||||
FOREIGN KEY (operator_uid, last_seen_batch_id) REFERENCES fm.import_batch (operator_uid, import_batch_id)
|
||||
);
|
||||
|
||||
-- Identity --------------------------------------------------------------------------------
|
||||
CREATE TABLE fm.site (
|
||||
site_uid text PRIMARY KEY CHECK (site_uid ~ '^SITE-[0-9]{4}$'),
|
||||
name text, name_on_plate text,
|
||||
zone text NOT NULL, province text NOT NULL, district text,
|
||||
timezone text NOT NULL DEFAULT 'Asia/Kabul',
|
||||
import_record_id bigint NOT NULL REFERENCES fm.import_record, -- full site.yaml payload
|
||||
null_reasons jsonb NOT NULL DEFAULT '{}'
|
||||
-- Identity: asset kinds and assets, places and holders (PD-18) --------------------------------
|
||||
CREATE TABLE fm.asset_kind ( -- what an asset does with fuel (PD-18); global, not per customer
|
||||
kind text PRIMARY KEY,
|
||||
fuel_role text CHECK (fuel_role IN ('container','consumer','transfer','sensor'))
|
||||
-- NULL: carries other assets and holds no fuel itself
|
||||
);
|
||||
CREATE TABLE fm.site_location ( -- coordinates are claims; the map uses only an accepted one
|
||||
INSERT INTO fm.asset_kind (kind, fuel_role) VALUES
|
||||
('generator','consumer'), ('engine','consumer'), ('tank','container'), ('compartment','container'),
|
||||
('vehicle_tank','container'), ('pump','transfer'), ('vehicle',NULL);
|
||||
CREATE TABLE fm.asset (
|
||||
asset_uid text PRIMARY KEY CHECK (asset_uid ~ '^AST-[A-Z0-9-]+$'),
|
||||
operator_uid text NOT NULL REFERENCES fm.operator,
|
||||
kind text NOT NULL REFERENCES fm.asset_kind,
|
||||
make text, model text, serial text, year_of_manufacture smallint,
|
||||
import_record_id bigint NOT NULL,
|
||||
null_reasons jsonb NOT NULL DEFAULT '{}',
|
||||
UNIQUE (operator_uid, asset_uid),
|
||||
FOREIGN KEY (operator_uid, import_record_id) REFERENCES fm.import_record (operator_uid, import_record_id)
|
||||
);
|
||||
CREATE TABLE fm.place ( -- fixed geography; its coordinates are claims (place_location)
|
||||
place_uid text PRIMARY KEY CHECK (place_uid ~ '^PLC-[0-9]{4,8}$'),
|
||||
operator_uid text NOT NULL REFERENCES fm.operator,
|
||||
role text NOT NULL CHECK (role IN ('tower_site','depot','yard','fuel_station')),
|
||||
zone text NOT NULL, province text NOT NULL, district text,
|
||||
timezone text, -- NULL: the operator's
|
||||
import_record_id bigint NOT NULL,
|
||||
null_reasons jsonb NOT NULL DEFAULT '{}',
|
||||
UNIQUE (operator_uid, place_uid),
|
||||
FOREIGN KEY (operator_uid, import_record_id) REFERENCES fm.import_record (operator_uid, import_record_id)
|
||||
);
|
||||
CREATE TABLE fm.holder ( -- what assets are in or on (PD-18). For a telecom site: one
|
||||
-- fixed installation at one place. Ids are opaque and permanent
|
||||
holder_uid text PRIMARY KEY CHECK (holder_uid ~ '^[A-Z]{2,4}-[0-9]{4,7}$'),
|
||||
operator_uid text NOT NULL REFERENCES fm.operator,
|
||||
kind text NOT NULL CHECK (kind IN ('fixed_installation','vehicle','vessel','trailer')),
|
||||
name text, name_on_plate text,
|
||||
asset_uid text, -- a vehicle is both an asset and a holder
|
||||
import_record_id bigint NOT NULL,
|
||||
null_reasons jsonb NOT NULL DEFAULT '{}',
|
||||
CHECK ((kind = 'vehicle') = (asset_uid IS NOT NULL)),
|
||||
UNIQUE (operator_uid, holder_uid),
|
||||
FOREIGN KEY (operator_uid, asset_uid) REFERENCES fm.asset (operator_uid, asset_uid),
|
||||
FOREIGN KEY (operator_uid, import_record_id) REFERENCES fm.import_record (operator_uid, import_record_id)
|
||||
);
|
||||
CREATE TABLE fm.holder_place ( -- where a fixed holder stands; a vehicle's position is telemetry
|
||||
holder_uid text NOT NULL,
|
||||
operator_uid text NOT NULL REFERENCES fm.operator,
|
||||
place_uid text NOT NULL,
|
||||
valid_from date, valid_to date, -- NULL from = unknown start; NULL to = still there
|
||||
null_reasons jsonb NOT NULL DEFAULT '{}',
|
||||
CHECK (valid_from IS NOT NULL OR null_reasons ? 'valid_from'),
|
||||
EXCLUDE USING gist (holder_uid WITH =,
|
||||
daterange(coalesce(valid_from,'-infinity'::date), valid_to, '[)') WITH &&),
|
||||
FOREIGN KEY (operator_uid, holder_uid) REFERENCES fm.holder (operator_uid, holder_uid),
|
||||
FOREIGN KEY (operator_uid, place_uid) REFERENCES fm.place (operator_uid, place_uid)
|
||||
);
|
||||
CREATE TABLE fm.place_location ( -- coordinates are claims; the map uses only an accepted one
|
||||
location_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
operator_uid text NOT NULL REFERENCES fm.operator,
|
||||
source_key text NOT NULL, -- 'site-yaml:SITE-0001:coordinates'
|
||||
assertion_sha256 text NOT NULL CHECK (assertion_sha256 ~ '^[0-9a-f]{64}$'), -- site, lat, lon, precision, basis
|
||||
site_uid text NOT NULL REFERENCES fm.site,
|
||||
assertion_sha256 text NOT NULL CHECK (assertion_sha256 ~ '^[0-9a-f]{64}$'), -- operator, place, lat, lon, precision, basis
|
||||
place_uid text NOT NULL,
|
||||
latitude numeric(9,6) NOT NULL CHECK (latitude BETWEEN -90 AND 90),
|
||||
longitude numeric(9,6) NOT NULL CHECK (longitude BETWEEN -180 AND 180),
|
||||
precision_m integer, -- e.g. 110 where the source rounds to 0.001 deg
|
||||
@@ -71,91 +141,111 @@ CREATE TABLE fm.site_location ( -- coordinates are claims; the
|
||||
source_note text NOT NULL, -- 'operator site plate photo <catalog key>'
|
||||
acceptance text NOT NULL DEFAULT 'pending' CHECK (acceptance IN ('pending','accepted','rejected','superseded','retracted')),
|
||||
accepted_on date, accepted_by_role text, acceptance_ref text, -- copied from the matching fm.review row
|
||||
superseded_by bigint REFERENCES fm.site_location,
|
||||
import_record_id bigint NOT NULL REFERENCES fm.import_record,
|
||||
superseded_by bigint,
|
||||
import_record_id bigint NOT NULL,
|
||||
CHECK (acceptance <> 'accepted' OR (accepted_on IS NOT NULL AND accepted_by_role IS NOT NULL)),
|
||||
CHECK (acceptance <> 'superseded' OR superseded_by IS NOT NULL)
|
||||
CHECK (acceptance <> 'superseded' OR superseded_by IS NOT NULL),
|
||||
UNIQUE (operator_uid, location_id),
|
||||
FOREIGN KEY (operator_uid, place_uid) REFERENCES fm.place (operator_uid, place_uid),
|
||||
FOREIGN KEY (operator_uid, superseded_by) REFERENCES fm.place_location (operator_uid, location_id),
|
||||
FOREIGN KEY (operator_uid, import_record_id) REFERENCES fm.import_record (operator_uid, import_record_id)
|
||||
);
|
||||
CREATE UNIQUE INDEX site_location_live_key ON fm.site_location (source_key)
|
||||
CREATE UNIQUE INDEX place_location_live_key ON fm.place_location (operator_uid, source_key)
|
||||
WHERE acceptance IN ('pending','accepted','rejected');
|
||||
CREATE UNIQUE INDEX site_location_one_accepted ON fm.site_location (site_uid) WHERE acceptance = 'accepted';
|
||||
CREATE TABLE fm.site_alias (
|
||||
site_alias_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
site_uid text NOT NULL REFERENCES fm.site,
|
||||
CREATE UNIQUE INDEX place_location_one_accepted ON fm.place_location (place_uid) WHERE acceptance = 'accepted';
|
||||
CREATE TABLE fm.holder_alias (
|
||||
holder_alias_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
operator_uid text NOT NULL REFERENCES fm.operator,
|
||||
holder_uid text NOT NULL,
|
||||
label text NOT NULL, issuer text NOT NULL,
|
||||
valid_from date, valid_to date, -- NULL from = unknown start; NULL to = still valid
|
||||
is_primary boolean NOT NULL DEFAULT false,
|
||||
null_reasons jsonb NOT NULL DEFAULT '{}',
|
||||
CHECK (valid_from IS NOT NULL OR null_reasons ? 'valid_from'),
|
||||
EXCLUDE USING gist (issuer WITH =, label WITH =,
|
||||
EXCLUDE USING gist (operator_uid WITH =, issuer WITH =, label WITH =,
|
||||
daterange(coalesce(valid_from,'-infinity'::date), valid_to, '[)') WITH &&),
|
||||
EXCLUDE USING gist (site_uid WITH =, issuer WITH =,
|
||||
daterange(coalesce(valid_from,'-infinity'::date), valid_to, '[)') WITH &&) WHERE (is_primary)
|
||||
EXCLUDE USING gist (holder_uid WITH =, issuer WITH =,
|
||||
daterange(coalesce(valid_from,'-infinity'::date), valid_to, '[)') WITH &&) WHERE (is_primary),
|
||||
FOREIGN KEY (operator_uid, holder_uid) REFERENCES fm.holder (operator_uid, holder_uid)
|
||||
);
|
||||
CREATE TABLE fm.pilot_participation (
|
||||
site_uid text NOT NULL REFERENCES fm.site,
|
||||
programme text NOT NULL DEFAULT 'ATOMA OFMS pilot',
|
||||
CREATE TABLE fm.participation (
|
||||
holder_uid text NOT NULL,
|
||||
operator_uid text NOT NULL REFERENCES fm.operator,
|
||||
programme text NOT NULL,
|
||||
status text NOT NULL CHECK (status IN ('active','reference','candidate','withdrawn')),
|
||||
valid_from date NOT NULL, valid_to date,
|
||||
decision_ref text, note text,
|
||||
EXCLUDE USING gist (site_uid WITH =, programme WITH =, daterange(valid_from, valid_to, '[)') WITH &&)
|
||||
EXCLUDE USING gist (holder_uid WITH =, programme WITH =, daterange(valid_from, valid_to, '[)') WITH &&),
|
||||
FOREIGN KEY (operator_uid, holder_uid) REFERENCES fm.holder (operator_uid, holder_uid)
|
||||
);
|
||||
CREATE TABLE fm.asset (
|
||||
asset_uid text PRIMARY KEY CHECK (asset_uid ~ '^AST-[A-Z0-9-]+$'),
|
||||
kind text NOT NULL CHECK (kind IN ('generator','tank')),
|
||||
make text, model text, serial text, year_of_manufacture smallint,
|
||||
import_record_id bigint NOT NULL REFERENCES fm.import_record,
|
||||
null_reasons jsonb NOT NULL DEFAULT '{}'
|
||||
);
|
||||
CREATE TABLE fm.asset_placement (
|
||||
CREATE TABLE fm.asset_placement ( -- in or on a holder, or at a place: exactly one (PD-18)
|
||||
placement_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
asset_uid text NOT NULL REFERENCES fm.asset,
|
||||
site_uid text NOT NULL REFERENCES fm.site,
|
||||
operator_uid text NOT NULL REFERENCES fm.operator,
|
||||
asset_uid text NOT NULL,
|
||||
holder_uid text,
|
||||
place_uid text,
|
||||
slot_label text NOT NULL, -- 'GEN-01': a position label, not an identity
|
||||
occupied_from date, occupied_to date, -- NULL from = before any record (conservative)
|
||||
earliest_evidence_on date,
|
||||
installed_on date,
|
||||
null_reasons jsonb NOT NULL DEFAULT '{}',
|
||||
CHECK (num_nonnulls(holder_uid, place_uid) = 1),
|
||||
CHECK (installed_on IS NOT NULL OR null_reasons ? 'installed_on'),
|
||||
EXCLUDE USING gist (asset_uid WITH =,
|
||||
daterange(coalesce(occupied_from,'-infinity'::date), occupied_to, '[)') WITH &&),
|
||||
EXCLUDE USING gist (site_uid WITH =, slot_label WITH =,
|
||||
daterange(coalesce(occupied_from,'-infinity'::date), occupied_to, '[)') WITH &&)
|
||||
EXCLUDE USING gist (holder_uid WITH =, slot_label WITH =,
|
||||
daterange(coalesce(occupied_from,'-infinity'::date), occupied_to, '[)') WITH &&) WHERE (holder_uid IS NOT NULL),
|
||||
EXCLUDE USING gist (place_uid WITH =, slot_label WITH =,
|
||||
daterange(coalesce(occupied_from,'-infinity'::date), occupied_to, '[)') WITH &&) WHERE (place_uid IS NOT NULL),
|
||||
FOREIGN KEY (operator_uid, asset_uid) REFERENCES fm.asset (operator_uid, asset_uid),
|
||||
FOREIGN KEY (operator_uid, holder_uid) REFERENCES fm.holder (operator_uid, holder_uid),
|
||||
FOREIGN KEY (operator_uid, place_uid) REFERENCES fm.place (operator_uid, place_uid)
|
||||
);
|
||||
CREATE TABLE fm.runtime_meter (
|
||||
meter_id text PRIMARY KEY, -- ours, stable: 'MTR-SITE-0001-GENSET1'
|
||||
site_uid text NOT NULL REFERENCES fm.site,
|
||||
operator_uid text NOT NULL REFERENCES fm.operator,
|
||||
holder_uid text NOT NULL,
|
||||
source_label text NOT NULL, -- as written: 'Genset 1', 'R.H'
|
||||
asset_uid text REFERENCES fm.asset,
|
||||
asset_uid text,
|
||||
mapping_status text NOT NULL CHECK (mapping_status IN ('confirmed','proposed','unmapped')),
|
||||
mapping_basis text,
|
||||
UNIQUE (site_uid, source_label),
|
||||
CHECK ((mapping_status = 'unmapped') = (asset_uid IS NULL))
|
||||
UNIQUE (holder_uid, source_label),
|
||||
CHECK ((mapping_status = 'unmapped') = (asset_uid IS NULL)),
|
||||
UNIQUE (operator_uid, meter_id),
|
||||
FOREIGN KEY (operator_uid, holder_uid) REFERENCES fm.holder (operator_uid, holder_uid),
|
||||
FOREIGN KEY (operator_uid, asset_uid) REFERENCES fm.asset (operator_uid, asset_uid)
|
||||
);
|
||||
CREATE TABLE fm.meter_epoch (
|
||||
epoch_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
meter_id text NOT NULL REFERENCES fm.runtime_meter,
|
||||
operator_uid text NOT NULL REFERENCES fm.operator,
|
||||
meter_id text NOT NULL,
|
||||
epoch_no smallint NOT NULL,
|
||||
starts_on date, -- NULL: began before the first reading held
|
||||
start_reason text NOT NULL CHECK (start_reason IN ('first_seen','display_replaced','controller_replaced','reset','set_replaced')),
|
||||
continuity text NOT NULL CHECK (continuity IN ('confirmed','assumed','uncertain')),
|
||||
rollover_at numeric(9,1),
|
||||
source_sha256 text REFERENCES fm.source_document, note text,
|
||||
UNIQUE (meter_id, epoch_no)
|
||||
source_sha256 text, note text,
|
||||
UNIQUE (meter_id, epoch_no),
|
||||
UNIQUE (operator_uid, epoch_id),
|
||||
FOREIGN KEY (operator_uid, meter_id) REFERENCES fm.runtime_meter (operator_uid, meter_id),
|
||||
FOREIGN KEY (operator_uid, source_sha256) REFERENCES fm.source_document (operator_uid, sha256)
|
||||
);
|
||||
|
||||
-- Operations (claims) ---------------------------------------------------------------------
|
||||
CREATE TABLE fm.visit (
|
||||
visit_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
operator_uid text NOT NULL REFERENCES fm.operator,
|
||||
visit_key text NOT NULL UNIQUE, -- 'SITE-0001:2026-01-31:1'
|
||||
site_uid text NOT NULL REFERENCES fm.site,
|
||||
visit_on date NOT NULL, -- site-local calendar date
|
||||
holder_uid text NOT NULL,
|
||||
visit_on date NOT NULL, -- holder-local calendar date
|
||||
seq_in_day smallint NOT NULL DEFAULT 1,
|
||||
time_in_local time, time_out_local time, -- as written; NULL when the source has a date only
|
||||
kind text NOT NULL CHECK (kind IN ('delivery','stock_check','maintenance','survey')),
|
||||
-- descriptive only: views use fm.v_visit_live and the live claims
|
||||
UNIQUE (site_uid, visit_on, seq_in_day),
|
||||
UNIQUE (visit_id, site_uid) -- target of site-checked composite foreign keys
|
||||
UNIQUE (holder_uid, visit_on, seq_in_day),
|
||||
UNIQUE (operator_uid, visit_id, holder_uid), -- target of holder-checked composite foreign keys
|
||||
UNIQUE (operator_uid, visit_id),
|
||||
FOREIGN KEY (operator_uid, holder_uid) REFERENCES fm.holder (operator_uid, holder_uid)
|
||||
);
|
||||
-- A visit is never deleted. Once all its claims are superseded or retracted it stays for provenance
|
||||
-- only, and drops out of every analytical view (fm.v_visit_live).
|
||||
@@ -164,42 +254,52 @@ CREATE TABLE fm.visit (
|
||||
-- superseded_by, review_note, null_reasons
|
||||
CREATE TABLE fm.fuel_movement (
|
||||
movement_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
operator_uid text NOT NULL REFERENCES fm.operator,
|
||||
source_key text NOT NULL, -- '<family>:delivery:SITE-0001:2026-01-31:1'
|
||||
assertion_sha256 text NOT NULL CHECK (assertion_sha256 ~ '^[0-9a-f]{64}$'),
|
||||
event_key text NOT NULL, -- the physical movement: 'SITE-0001:2026-01-31:1:delivery:1';
|
||||
-- claims sharing it are alternative reports of ONE movement
|
||||
visit_id bigint NOT NULL REFERENCES fm.visit,
|
||||
site_uid text NOT NULL REFERENCES fm.site,
|
||||
visit_id bigint NOT NULL,
|
||||
holder_uid text NOT NULL,
|
||||
kind text NOT NULL CHECK (kind IN ('delivery','external_in','internal_transfer',
|
||||
'withdrawal','drain','spill','loss_reported')),
|
||||
from_tank_uid text REFERENCES fm.asset, to_tank_uid text REFERENCES fm.asset,
|
||||
from_tank_uid text, to_tank_uid text,
|
||||
source_tank_label text,
|
||||
litres numeric(8,1) CHECK (litres >= 0),
|
||||
litres_state text NOT NULL CHECK (litres_state IN ('stated','converted_from_level','converted_from_mass','not_recorded','illegible')),
|
||||
density_kg_per_l numeric(5,3),
|
||||
unit_price numeric(8,2), price_currency text,
|
||||
source_sha256 text NOT NULL REFERENCES fm.source_document,
|
||||
source_sha256 text NOT NULL,
|
||||
record_ref text NOT NULL, -- 'form <no>' | '<sheet>!J20' | 'logbook p2 r1'
|
||||
import_record_id bigint NOT NULL REFERENCES fm.import_record,
|
||||
import_record_id bigint NOT NULL,
|
||||
claim_status text NOT NULL DEFAULT 'asserted' CHECK (claim_status IN ('asserted','reviewed','disputed','superseded','retracted')),
|
||||
superseded_by bigint REFERENCES fm.fuel_movement,
|
||||
superseded_by bigint,
|
||||
review_note text,
|
||||
null_reasons jsonb NOT NULL DEFAULT '{}',
|
||||
CHECK ((litres IS NULL) = (litres_state IN ('not_recorded','illegible'))),
|
||||
CHECK (claim_status <> 'superseded' OR superseded_by IS NOT NULL),
|
||||
CHECK (kind <> 'internal_transfer' OR (from_tank_uid IS NOT NULL AND to_tank_uid IS NOT NULL))
|
||||
CHECK (kind <> 'internal_transfer' OR (from_tank_uid IS NOT NULL AND to_tank_uid IS NOT NULL)),
|
||||
UNIQUE (operator_uid, movement_id),
|
||||
FOREIGN KEY (operator_uid, visit_id) REFERENCES fm.visit (operator_uid, visit_id),
|
||||
FOREIGN KEY (operator_uid, holder_uid) REFERENCES fm.holder (operator_uid, holder_uid),
|
||||
FOREIGN KEY (operator_uid, from_tank_uid) REFERENCES fm.asset (operator_uid, asset_uid),
|
||||
FOREIGN KEY (operator_uid, to_tank_uid) REFERENCES fm.asset (operator_uid, asset_uid),
|
||||
FOREIGN KEY (operator_uid, source_sha256) REFERENCES fm.source_document (operator_uid, sha256),
|
||||
FOREIGN KEY (operator_uid, import_record_id) REFERENCES fm.import_record (operator_uid, import_record_id),
|
||||
FOREIGN KEY (operator_uid, superseded_by) REFERENCES fm.fuel_movement (operator_uid, movement_id)
|
||||
);
|
||||
CREATE UNIQUE INDEX fuel_movement_live_key ON fm.fuel_movement (source_key)
|
||||
CREATE UNIQUE INDEX fuel_movement_live_key ON fm.fuel_movement (operator_uid, source_key)
|
||||
WHERE claim_status IN ('asserted','reviewed','disputed');
|
||||
CREATE INDEX ON fm.fuel_movement (visit_id, event_key);
|
||||
CREATE TABLE fm.stock_observation (
|
||||
stock_obs_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
operator_uid text NOT NULL REFERENCES fm.operator,
|
||||
source_key text NOT NULL,
|
||||
assertion_sha256 text NOT NULL CHECK (assertion_sha256 ~ '^[0-9a-f]{64}$'),
|
||||
visit_id bigint NOT NULL REFERENCES fm.visit,
|
||||
site_uid text NOT NULL REFERENCES fm.site,
|
||||
visit_id bigint NOT NULL,
|
||||
holder_uid text NOT NULL,
|
||||
scope text NOT NULL CHECK (scope IN ('site_total','tank')),
|
||||
tank_uid text REFERENCES fm.asset, -- only when the source identifies the tank
|
||||
tank_uid text, -- only when the source identifies the tank
|
||||
source_tank_label text, -- 'T1' as written, even when unmapped
|
||||
phase text NOT NULL CHECK (phase IN ('before_delivery','after_delivery','standalone')),
|
||||
level_cm numeric(6,1),
|
||||
@@ -208,27 +308,35 @@ CREATE TABLE fm.stock_observation (
|
||||
'stated_total','level_only','not_recorded','illegible','placeholder_zero')),
|
||||
resolution_l numeric(6,1), -- ±1 cm of dip at this level, from tank geometry
|
||||
resolution_basis text,
|
||||
source_sha256 text NOT NULL REFERENCES fm.source_document,
|
||||
source_sha256 text NOT NULL,
|
||||
record_ref text NOT NULL,
|
||||
import_record_id bigint NOT NULL REFERENCES fm.import_record,
|
||||
import_record_id bigint NOT NULL,
|
||||
claim_status text NOT NULL DEFAULT 'asserted' CHECK (claim_status IN ('asserted','reviewed','disputed','superseded','retracted')),
|
||||
superseded_by bigint REFERENCES fm.stock_observation,
|
||||
superseded_by bigint,
|
||||
review_note text,
|
||||
null_reasons jsonb NOT NULL DEFAULT '{}',
|
||||
CHECK (scope = 'site_total' OR tank_uid IS NOT NULL OR source_tank_label IS NOT NULL),
|
||||
CHECK ((litres IS NULL) = (litres_basis IN ('level_only','not_recorded','illegible','placeholder_zero'))),
|
||||
CHECK (claim_status <> 'superseded' OR superseded_by IS NOT NULL)
|
||||
CHECK (claim_status <> 'superseded' OR superseded_by IS NOT NULL),
|
||||
UNIQUE (operator_uid, stock_obs_id),
|
||||
FOREIGN KEY (operator_uid, visit_id) REFERENCES fm.visit (operator_uid, visit_id),
|
||||
FOREIGN KEY (operator_uid, holder_uid) REFERENCES fm.holder (operator_uid, holder_uid),
|
||||
FOREIGN KEY (operator_uid, tank_uid) REFERENCES fm.asset (operator_uid, asset_uid),
|
||||
FOREIGN KEY (operator_uid, source_sha256) REFERENCES fm.source_document (operator_uid, sha256),
|
||||
FOREIGN KEY (operator_uid, import_record_id) REFERENCES fm.import_record (operator_uid, import_record_id),
|
||||
FOREIGN KEY (operator_uid, superseded_by) REFERENCES fm.stock_observation (operator_uid, stock_obs_id)
|
||||
);
|
||||
CREATE UNIQUE INDEX stock_obs_live_key ON fm.stock_observation (source_key)
|
||||
CREATE UNIQUE INDEX stock_obs_live_key ON fm.stock_observation (operator_uid, source_key)
|
||||
WHERE claim_status IN ('asserted','reviewed','disputed');
|
||||
CREATE INDEX ON fm.stock_observation (visit_id, phase);
|
||||
CREATE TABLE fm.hour_meter_reading (
|
||||
reading_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
operator_uid text NOT NULL REFERENCES fm.operator,
|
||||
source_key text NOT NULL,
|
||||
assertion_sha256 text NOT NULL CHECK (assertion_sha256 ~ '^[0-9a-f]{64}$'),
|
||||
visit_id bigint NOT NULL REFERENCES fm.visit,
|
||||
meter_id text NOT NULL REFERENCES fm.runtime_meter,
|
||||
epoch_id bigint REFERENCES fm.meter_epoch, -- NULL = not yet judged; excluded from runtime
|
||||
visit_id bigint NOT NULL,
|
||||
meter_id text NOT NULL,
|
||||
epoch_id bigint, -- NULL = not yet judged; excluded from runtime
|
||||
source_field text NOT NULL CHECK (source_field IN ('current','previous','logbook_rh','display_photo')),
|
||||
date_basis text NOT NULL CHECK (date_basis IN ('stated','inferred_previous_visit','capture_date_unknown')),
|
||||
hours numeric(9,1),
|
||||
@@ -236,84 +344,109 @@ CREATE TABLE fm.hour_meter_reading (
|
||||
reading_state text NOT NULL CHECK (reading_state IN ('clear','best_reading','ambiguous','illegible','crossed_out','not_recorded')),
|
||||
alternatives text, -- competing readings of one illegible value, '|'-separated
|
||||
transcribed_by text NOT NULL CHECK (transcribed_by IN ('source_operator','our_reading')),
|
||||
source_sha256 text NOT NULL REFERENCES fm.source_document,
|
||||
source_sha256 text NOT NULL,
|
||||
record_ref text NOT NULL,
|
||||
import_record_id bigint NOT NULL REFERENCES fm.import_record,
|
||||
import_record_id bigint NOT NULL,
|
||||
claim_status text NOT NULL DEFAULT 'asserted' CHECK (claim_status IN ('asserted','reviewed','disputed','superseded','retracted')),
|
||||
superseded_by bigint REFERENCES fm.hour_meter_reading,
|
||||
superseded_by bigint,
|
||||
review_note text,
|
||||
null_reasons jsonb NOT NULL DEFAULT '{}',
|
||||
CHECK ((hours IS NULL) = (reading_state IN ('ambiguous','illegible','crossed_out','not_recorded'))),
|
||||
CHECK (claim_status <> 'superseded' OR superseded_by IS NOT NULL)
|
||||
CHECK (claim_status <> 'superseded' OR superseded_by IS NOT NULL),
|
||||
UNIQUE (operator_uid, reading_id),
|
||||
FOREIGN KEY (operator_uid, visit_id) REFERENCES fm.visit (operator_uid, visit_id),
|
||||
FOREIGN KEY (operator_uid, meter_id) REFERENCES fm.runtime_meter (operator_uid, meter_id),
|
||||
FOREIGN KEY (operator_uid, epoch_id) REFERENCES fm.meter_epoch (operator_uid, epoch_id),
|
||||
FOREIGN KEY (operator_uid, source_sha256) REFERENCES fm.source_document (operator_uid, sha256),
|
||||
FOREIGN KEY (operator_uid, import_record_id) REFERENCES fm.import_record (operator_uid, import_record_id),
|
||||
FOREIGN KEY (operator_uid, superseded_by) REFERENCES fm.hour_meter_reading (operator_uid, reading_id)
|
||||
);
|
||||
CREATE UNIQUE INDEX hour_reading_live_key ON fm.hour_meter_reading (source_key)
|
||||
CREATE UNIQUE INDEX hour_reading_live_key ON fm.hour_meter_reading (operator_uid, source_key)
|
||||
WHERE claim_status IN ('asserted','reviewed','disputed');
|
||||
CREATE INDEX ON fm.hour_meter_reading (visit_id, meter_id);
|
||||
CREATE TABLE fm.runtime_interval ( -- future: controller / ATS logs, telemetry
|
||||
runtime_interval_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
operator_uid text NOT NULL REFERENCES fm.operator,
|
||||
source_key text NOT NULL, -- live-unique, as for every claim
|
||||
assertion_sha256 text NOT NULL CHECK (assertion_sha256 ~ '^[0-9a-f]{64}$'),
|
||||
site_uid text NOT NULL REFERENCES fm.site,
|
||||
asset_uid text REFERENCES fm.asset,
|
||||
holder_uid text NOT NULL,
|
||||
asset_uid text,
|
||||
scope text NOT NULL CHECK (scope IN ('generator','site_any_generator')),
|
||||
started_at timestamptz NOT NULL, ended_at timestamptz NOT NULL CHECK (ended_at > started_at),
|
||||
source_kind text NOT NULL CHECK (source_kind IN ('controller_event_log','ats_log','gateway_telemetry','manual_log')),
|
||||
source_sha256 text REFERENCES fm.source_document,
|
||||
source_sha256 text,
|
||||
claim_status text NOT NULL DEFAULT 'asserted' CHECK (claim_status IN ('asserted','reviewed','disputed','superseded','retracted')),
|
||||
superseded_by bigint REFERENCES fm.runtime_interval,
|
||||
superseded_by bigint,
|
||||
CHECK ((scope = 'generator') = (asset_uid IS NOT NULL)),
|
||||
CHECK (claim_status <> 'superseded' OR superseded_by IS NOT NULL),
|
||||
EXCLUDE USING gist (site_uid WITH =, scope WITH =, (coalesce(asset_uid,'')) WITH =, source_kind WITH =,
|
||||
tstzrange(started_at, ended_at) WITH &&) WHERE (claim_status IN ('asserted','reviewed'))
|
||||
EXCLUDE USING gist (holder_uid WITH =, scope WITH =, (coalesce(asset_uid,'')) WITH =, source_kind WITH =,
|
||||
tstzrange(started_at, ended_at) WITH &&) WHERE (claim_status IN ('asserted','reviewed')),
|
||||
UNIQUE (operator_uid, runtime_interval_id),
|
||||
FOREIGN KEY (operator_uid, holder_uid) REFERENCES fm.holder (operator_uid, holder_uid),
|
||||
FOREIGN KEY (operator_uid, asset_uid) REFERENCES fm.asset (operator_uid, asset_uid),
|
||||
FOREIGN KEY (operator_uid, source_sha256) REFERENCES fm.source_document (operator_uid, sha256),
|
||||
FOREIGN KEY (operator_uid, superseded_by) REFERENCES fm.runtime_interval (operator_uid, runtime_interval_id)
|
||||
);
|
||||
CREATE UNIQUE INDEX runtime_interval_live_key ON fm.runtime_interval (source_key)
|
||||
CREATE UNIQUE INDEX runtime_interval_live_key ON fm.runtime_interval (operator_uid, source_key)
|
||||
WHERE claim_status IN ('asserted','reviewed','disputed');
|
||||
CREATE TABLE fm.engine_fuel_total ( -- future: flow-meter net fuel per period
|
||||
total_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
operator_uid text NOT NULL REFERENCES fm.operator,
|
||||
source_key text NOT NULL, -- live-unique, as for every claim
|
||||
assertion_sha256 text NOT NULL CHECK (assertion_sha256 ~ '^[0-9a-f]{64}$'),
|
||||
asset_uid text NOT NULL REFERENCES fm.asset,
|
||||
asset_uid text NOT NULL,
|
||||
started_at timestamptz NOT NULL, ended_at timestamptz NOT NULL CHECK (ended_at > started_at),
|
||||
net_fuel_l numeric(10,2) NOT NULL, -- feed − return as the device reports
|
||||
engine_hours numeric(8,2), -- same device, if reported
|
||||
source_kind text NOT NULL CHECK (source_kind IN ('differential_flow_meter','single_flow_meter','controller_fuel_counter')),
|
||||
source_sha256 text REFERENCES fm.source_document,
|
||||
source_sha256 text,
|
||||
claim_status text NOT NULL DEFAULT 'asserted' CHECK (claim_status IN ('asserted','reviewed','disputed','superseded','retracted')),
|
||||
superseded_by bigint REFERENCES fm.engine_fuel_total,
|
||||
superseded_by bigint,
|
||||
CHECK (claim_status <> 'superseded' OR superseded_by IS NOT NULL),
|
||||
EXCLUDE USING gist (asset_uid WITH =, tstzrange(started_at, ended_at) WITH &&)
|
||||
WHERE (claim_status IN ('asserted','reviewed'))
|
||||
WHERE (claim_status IN ('asserted','reviewed')),
|
||||
UNIQUE (operator_uid, total_id),
|
||||
FOREIGN KEY (operator_uid, asset_uid) REFERENCES fm.asset (operator_uid, asset_uid),
|
||||
FOREIGN KEY (operator_uid, source_sha256) REFERENCES fm.source_document (operator_uid, sha256),
|
||||
FOREIGN KEY (operator_uid, superseded_by) REFERENCES fm.engine_fuel_total (operator_uid, total_id)
|
||||
);
|
||||
CREATE UNIQUE INDEX engine_fuel_total_live_key ON fm.engine_fuel_total (source_key)
|
||||
CREATE UNIQUE INDEX engine_fuel_total_live_key ON fm.engine_fuel_total (operator_uid, source_key)
|
||||
WHERE claim_status IN ('asserted','reviewed','disputed');
|
||||
CREATE TABLE fm.electricity_bill (
|
||||
bill_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
operator_uid text NOT NULL REFERENCES fm.operator,
|
||||
source_key text NOT NULL,
|
||||
assertion_sha256 text NOT NULL CHECK (assertion_sha256 ~ '^[0-9a-f]{64}$'),
|
||||
site_uid text NOT NULL REFERENCES fm.site,
|
||||
holder_uid text NOT NULL,
|
||||
bill_month date NOT NULL CHECK (extract(day FROM bill_month) = 1), -- month as the source assigns it
|
||||
period_start date, period_end date,
|
||||
amount numeric(12,2),
|
||||
currency text, currency_basis text CHECK (currency_basis IN ('stated','inferred')),
|
||||
kwh numeric(10,1),
|
||||
source_sha256 text NOT NULL REFERENCES fm.source_document,
|
||||
source_sha256 text NOT NULL,
|
||||
record_ref text NOT NULL,
|
||||
import_record_id bigint NOT NULL REFERENCES fm.import_record,
|
||||
import_record_id bigint NOT NULL,
|
||||
claim_status text NOT NULL DEFAULT 'asserted' CHECK (claim_status IN ('asserted','reviewed','disputed','superseded','retracted')),
|
||||
superseded_by bigint REFERENCES fm.electricity_bill,
|
||||
superseded_by bigint,
|
||||
null_reasons jsonb NOT NULL DEFAULT '{}',
|
||||
CHECK (kwh IS NOT NULL OR null_reasons ? 'kwh'),
|
||||
CHECK (period_start IS NOT NULL OR null_reasons ? 'period_start')
|
||||
CHECK (period_start IS NOT NULL OR null_reasons ? 'period_start'),
|
||||
UNIQUE (operator_uid, bill_id),
|
||||
FOREIGN KEY (operator_uid, holder_uid) REFERENCES fm.holder (operator_uid, holder_uid),
|
||||
FOREIGN KEY (operator_uid, source_sha256) REFERENCES fm.source_document (operator_uid, sha256),
|
||||
FOREIGN KEY (operator_uid, import_record_id) REFERENCES fm.import_record (operator_uid, import_record_id),
|
||||
FOREIGN KEY (operator_uid, superseded_by) REFERENCES fm.electricity_bill (operator_uid, bill_id)
|
||||
);
|
||||
CREATE UNIQUE INDEX bill_live_key ON fm.electricity_bill (source_key)
|
||||
CREATE UNIQUE INDEX bill_live_key ON fm.electricity_bill (operator_uid, source_key)
|
||||
WHERE claim_status IN ('asserted','reviewed','disputed');
|
||||
|
||||
-- Specifications, assumptions, imported calculations -------------------------------------
|
||||
CREATE TABLE fm.equipment_spec ( -- versioned like a claim: corrections supersede
|
||||
spec_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
operator_uid text NOT NULL REFERENCES fm.operator,
|
||||
source_key text NOT NULL, -- 'site-yaml:SITE-0001:AST-…:rated_prime_kw'
|
||||
assertion_sha256 text NOT NULL CHECK (assertion_sha256 ~ '^[0-9a-f]{64}$'),
|
||||
asset_uid text NOT NULL REFERENCES fm.asset,
|
||||
asset_uid text NOT NULL,
|
||||
attribute text NOT NULL, -- rated_prime_kw, capacity_l, fuel_rate_at_load, …
|
||||
load_pct numeric(5,1),
|
||||
value_num numeric(12,3), value_text text, unit text,
|
||||
@@ -321,22 +454,28 @@ CREATE TABLE fm.equipment_spec ( -- versioned like a claim: corre
|
||||
'operator_stated','our_inference','geometry')),
|
||||
applicability text NOT NULL DEFAULT 'confirmed' CHECK (applicability IN ('confirmed','conditional','unconfirmed')),
|
||||
applicability_note text,
|
||||
source_sha256 text REFERENCES fm.source_document,
|
||||
import_record_id bigint NOT NULL REFERENCES fm.import_record,
|
||||
source_sha256 text,
|
||||
import_record_id bigint NOT NULL,
|
||||
claim_status text NOT NULL DEFAULT 'asserted' CHECK (claim_status IN ('asserted','reviewed','disputed','superseded','retracted')),
|
||||
superseded_by bigint REFERENCES fm.equipment_spec,
|
||||
superseded_by bigint,
|
||||
CHECK (claim_status <> 'superseded' OR superseded_by IS NOT NULL),
|
||||
CHECK ((attribute IN ('fuel_rate_at_load','specific_fuel_at_load')) = (load_pct IS NOT NULL))
|
||||
CHECK ((attribute IN ('fuel_rate_at_load','specific_fuel_at_load')) = (load_pct IS NOT NULL)),
|
||||
UNIQUE (operator_uid, spec_id),
|
||||
FOREIGN KEY (operator_uid, asset_uid) REFERENCES fm.asset (operator_uid, asset_uid),
|
||||
FOREIGN KEY (operator_uid, source_sha256) REFERENCES fm.source_document (operator_uid, sha256),
|
||||
FOREIGN KEY (operator_uid, import_record_id) REFERENCES fm.import_record (operator_uid, import_record_id),
|
||||
FOREIGN KEY (operator_uid, superseded_by) REFERENCES fm.equipment_spec (operator_uid, spec_id)
|
||||
);
|
||||
CREATE UNIQUE INDEX equipment_spec_live_key ON fm.equipment_spec (source_key)
|
||||
CREATE UNIQUE INDEX equipment_spec_live_key ON fm.equipment_spec (operator_uid, source_key)
|
||||
WHERE claim_status IN ('asserted','reviewed','disputed');
|
||||
CREATE TABLE fm.assumption ( -- one row per version; a change is a new version
|
||||
assumption_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
operator_uid text NOT NULL REFERENCES fm.operator,
|
||||
code text NOT NULL UNIQUE, -- 'A-SITE-0001-GEN02-IDLE-v1'
|
||||
fingerprint text NOT NULL CHECK (fingerprint ~ '^[0-9a-f]{64}$'), -- all fields below + sorted members
|
||||
created_xact xid8 NOT NULL DEFAULT pg_current_xact_id(), -- rule 9: members only in this transaction
|
||||
site_uid text NOT NULL REFERENCES fm.site,
|
||||
asset_uid text REFERENCES fm.asset,
|
||||
holder_uid text NOT NULL,
|
||||
asset_uid text,
|
||||
topic text NOT NULL CHECK (topic IN ('runtime_hours_per_day','generator_idle','meter_coverage',
|
||||
'tank_linkage','no_unrecorded_movements','density','other')),
|
||||
-- meter_coverage: its members name the exact counters,
|
||||
@@ -347,77 +486,97 @@ CREATE TABLE fm.assumption ( -- one row per version; a chang
|
||||
derived_from_fuel boolean NOT NULL, -- true: never a runtime input to an L/h figure
|
||||
status text NOT NULL CHECK (status IN ('proposed','accepted','rejected','superseded')),
|
||||
accepted_on date, accepted_by_role text, decision_ref text, -- copied from the matching fm.review row
|
||||
supersedes_id bigint UNIQUE REFERENCES fm.assumption,
|
||||
supersedes_id bigint UNIQUE,
|
||||
rationale text NOT NULL, contradicted_by text,
|
||||
source_sha256 text REFERENCES fm.source_document,
|
||||
source_sha256 text,
|
||||
CHECK (status <> 'accepted' OR (accepted_on IS NOT NULL AND accepted_by_role IS NOT NULL)),
|
||||
CONSTRAINT assumption_no_runtime_from_fuel -- rule 5
|
||||
CHECK (NOT (derived_from_fuel AND topic = 'runtime_hours_per_day' AND status = 'accepted')),
|
||||
CHECK (topic <> 'generator_idle' OR asset_uid IS NOT NULL),
|
||||
CHECK (topic <> 'meter_coverage' OR asset_uid IS NULL), -- its sets are in assumption_member
|
||||
EXCLUDE USING gist (site_uid WITH =, daterange(applies_from, applies_to, '[]') WITH &&)
|
||||
EXCLUDE USING gist (holder_uid WITH =, daterange(applies_from, applies_to, '[]') WITH &&)
|
||||
WHERE (topic = 'meter_coverage' AND status IN ('proposed','accepted')),
|
||||
EXCLUDE USING gist (asset_uid WITH =, daterange(applies_from, applies_to, '[]') WITH &&)
|
||||
WHERE (topic = 'generator_idle' AND status IN ('proposed','accepted'))
|
||||
WHERE (topic = 'generator_idle' AND status IN ('proposed','accepted')),
|
||||
UNIQUE (operator_uid, assumption_id),
|
||||
FOREIGN KEY (operator_uid, holder_uid) REFERENCES fm.holder (operator_uid, holder_uid),
|
||||
FOREIGN KEY (operator_uid, asset_uid) REFERENCES fm.asset (operator_uid, asset_uid),
|
||||
FOREIGN KEY (operator_uid, supersedes_id) REFERENCES fm.assumption (operator_uid, assumption_id),
|
||||
FOREIGN KEY (operator_uid, source_sha256) REFERENCES fm.source_document (operator_uid, sha256)
|
||||
);
|
||||
CREATE TABLE fm.assumption_member ( -- exact membership of a meter_coverage assumption
|
||||
assumption_id bigint NOT NULL REFERENCES fm.assumption,
|
||||
assumption_id bigint NOT NULL,
|
||||
operator_uid text NOT NULL REFERENCES fm.operator,
|
||||
member_role text NOT NULL CHECK (member_role IN ('counter','covered_generator','idle_generator')),
|
||||
meter_id text REFERENCES fm.runtime_meter,
|
||||
asset_uid text REFERENCES fm.asset,
|
||||
meter_id text,
|
||||
asset_uid text,
|
||||
CHECK ((member_role = 'counter') = (meter_id IS NOT NULL)),
|
||||
CHECK ((member_role = 'counter') = (asset_uid IS NULL))
|
||||
CHECK ((member_role = 'counter') = (asset_uid IS NULL)),
|
||||
FOREIGN KEY (operator_uid, assumption_id) REFERENCES fm.assumption (operator_uid, assumption_id),
|
||||
FOREIGN KEY (operator_uid, meter_id) REFERENCES fm.runtime_meter (operator_uid, meter_id),
|
||||
FOREIGN KEY (operator_uid, asset_uid) REFERENCES fm.asset (operator_uid, asset_uid)
|
||||
);
|
||||
CREATE UNIQUE INDEX assumption_member_meter ON fm.assumption_member (assumption_id, meter_id) WHERE meter_id IS NOT NULL;
|
||||
CREATE UNIQUE INDEX assumption_member_asset ON fm.assumption_member (assumption_id, asset_uid) WHERE asset_uid IS NOT NULL;
|
||||
-- a set is either covered or idle within one assumption
|
||||
CREATE TABLE fm.imported_calculation ( -- versioned like a claim: a revised formula supersedes
|
||||
calc_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
operator_uid text NOT NULL REFERENCES fm.operator,
|
||||
source_key text NOT NULL, -- '<family>:calc:SITE-0001:<label>'
|
||||
assertion_sha256 text NOT NULL CHECK (assertion_sha256 ~ '^[0-9a-f]{64}$'),
|
||||
source_sha256 text NOT NULL REFERENCES fm.source_document,
|
||||
source_sha256 text NOT NULL,
|
||||
cell_ref text NOT NULL, -- '<sheet>!K2' | '<survey item>'
|
||||
site_uid text REFERENCES fm.site,
|
||||
holder_uid text,
|
||||
source_label text NOT NULL, -- as written in the source
|
||||
formula text, cached_value numeric(14,6), unit_as_labelled text,
|
||||
our_reading text NOT NULL, -- what it actually computes
|
||||
baseline_use text NOT NULL CHECK (baseline_use IN ('excluded','context_only')),
|
||||
exclusion_reason text,
|
||||
import_record_id bigint NOT NULL REFERENCES fm.import_record,
|
||||
import_record_id bigint NOT NULL,
|
||||
claim_status text NOT NULL DEFAULT 'asserted' CHECK (claim_status IN ('asserted','reviewed','disputed','superseded','retracted')),
|
||||
superseded_by bigint REFERENCES fm.imported_calculation,
|
||||
superseded_by bigint,
|
||||
CHECK (claim_status <> 'superseded' OR superseded_by IS NOT NULL),
|
||||
CHECK (baseline_use <> 'excluded' OR exclusion_reason IS NOT NULL)
|
||||
CHECK (baseline_use <> 'excluded' OR exclusion_reason IS NOT NULL),
|
||||
UNIQUE (operator_uid, calc_id),
|
||||
FOREIGN KEY (operator_uid, source_sha256) REFERENCES fm.source_document (operator_uid, sha256),
|
||||
FOREIGN KEY (operator_uid, holder_uid) REFERENCES fm.holder (operator_uid, holder_uid),
|
||||
FOREIGN KEY (operator_uid, import_record_id) REFERENCES fm.import_record (operator_uid, import_record_id),
|
||||
FOREIGN KEY (operator_uid, superseded_by) REFERENCES fm.imported_calculation (operator_uid, calc_id)
|
||||
);
|
||||
CREATE UNIQUE INDEX imported_calc_live_key ON fm.imported_calculation (source_key)
|
||||
CREATE UNIQUE INDEX imported_calc_live_key ON fm.imported_calculation (operator_uid, source_key)
|
||||
WHERE claim_status IN ('asserted','reviewed','disputed');
|
||||
|
||||
-- Reviews ---------------------------------------------------------------------------------
|
||||
CREATE TABLE fm.review ( -- one applied entry of the corpus review manifest
|
||||
review_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
entry_key text NOT NULL UNIQUE, -- the manifest entry's own id
|
||||
operator_uid text NOT NULL REFERENCES fm.operator,
|
||||
entry_key text NOT NULL, -- the manifest entry's own id, per customer
|
||||
subject_kind text NOT NULL CHECK (subject_kind IN ('claim','location','assumption')),
|
||||
subject_key text NOT NULL, -- source_key, or the assumption code
|
||||
assertion_sha256 text NOT NULL CHECK (assertion_sha256 ~ '^[0-9a-f]{64}$'), -- what was actually reviewed
|
||||
checked_against_sha256 text REFERENCES fm.source_document, -- the file version compared with
|
||||
checked_against_sha256 text, -- the file version compared with
|
||||
verdict text NOT NULL CHECK (verdict IN ('reviewed','disputed','accepted','rejected')),
|
||||
reviewed_on date NOT NULL, reviewer_role text NOT NULL, decision_ref text, note text,
|
||||
import_batch_id bigint NOT NULL REFERENCES fm.import_batch,
|
||||
CHECK ((subject_kind = 'claim') = (verdict IN ('reviewed','disputed')))
|
||||
import_batch_id bigint NOT NULL,
|
||||
CHECK ((subject_kind = 'claim') = (verdict IN ('reviewed','disputed'))),
|
||||
UNIQUE (operator_uid, entry_key),
|
||||
FOREIGN KEY (operator_uid, checked_against_sha256) REFERENCES fm.source_document (operator_uid, sha256),
|
||||
FOREIGN KEY (operator_uid, import_batch_id) REFERENCES fm.import_batch (operator_uid, import_batch_id)
|
||||
);
|
||||
CREATE INDEX ON fm.review (subject_key, assertion_sha256);
|
||||
CREATE INDEX ON fm.review (operator_uid, subject_key, assertion_sha256);
|
||||
|
||||
-- Baseline --------------------------------------------------------------------------------
|
||||
CREATE TABLE fm.baseline_version (
|
||||
baseline_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
operator_uid text NOT NULL REFERENCES fm.operator,
|
||||
code text NOT NULL UNIQUE, -- 'BL-SITE-0001-APGH-v1'
|
||||
site_uid text NOT NULL REFERENCES fm.site,
|
||||
holder_uid text NOT NULL,
|
||||
metric text NOT NULL CHECK (metric IN ('measured_generator_l_per_h','apparent_l_per_generator_h',
|
||||
'apparent_l_per_set_h','apparent_l_per_site_dg_h','modelled_l_per_h','fuel_l_per_calendar_day')),
|
||||
population text NOT NULL, -- human-readable: 'GEN-01 (AST-XXX-GEN-0001)'
|
||||
population_key text NOT NULL, -- sorted asset uids, 'meter:<id>' for counters not tied
|
||||
-- to a set plus 'covers:<asset uid>' for the sets they
|
||||
-- cover, or 'stock:<site_uid>' for fuel_l_per_calendar_day
|
||||
-- cover, or 'stock:<holder_uid>' for fuel_l_per_calendar_day
|
||||
window_start date NOT NULL, window_end date NOT NULL CHECK (window_end > window_start),
|
||||
policy_version text NOT NULL, -- 'BP-1'
|
||||
meets_policy boolean NOT NULL DEFAULT false,-- set by `baseline propose` from the inputs (BP-1 test)
|
||||
@@ -426,19 +585,23 @@ CREATE TABLE fm.baseline_version (
|
||||
fuel_l numeric(10,1), runtime_h numeric(10,1), matched_days integer,
|
||||
conditions text, -- assumption codes the value depends on
|
||||
evidence_grade text CHECK (evidence_grade IN ('G1','G2')),
|
||||
atoma_tier text CHECK (atoma_tier IN ('A','B','C')), -- descriptive only (Baseline Verification Plan)
|
||||
operator_tier text CHECK (operator_tier IN ('A','B','C')), -- descriptive only (Baseline Verification Plan)
|
||||
season_note text,
|
||||
status text NOT NULL CHECK (status IN ('draft','proposed','approved','superseded','rejected')),
|
||||
computed_at timestamptz NOT NULL, computed_by_role text NOT NULL,
|
||||
approved_on date, approved_by_role text, decision_ref text,
|
||||
supersedes_id bigint REFERENCES fm.baseline_version,
|
||||
UNIQUE (baseline_id, site_uid),
|
||||
CHECK (status <> 'approved' OR (approved_on IS NOT NULL AND decision_ref IS NOT NULL))
|
||||
supersedes_id bigint,
|
||||
UNIQUE (operator_uid, baseline_id, holder_uid),
|
||||
CHECK (status <> 'approved' OR (approved_on IS NOT NULL AND decision_ref IS NOT NULL)),
|
||||
UNIQUE (operator_uid, baseline_id),
|
||||
FOREIGN KEY (operator_uid, holder_uid) REFERENCES fm.holder (operator_uid, holder_uid),
|
||||
FOREIGN KEY (operator_uid, supersedes_id) REFERENCES fm.baseline_version (operator_uid, baseline_id)
|
||||
);
|
||||
CREATE UNIQUE INDEX baseline_one_approved ON fm.baseline_version (site_uid, metric) WHERE status = 'approved';
|
||||
CREATE UNIQUE INDEX baseline_one_approved ON fm.baseline_version (holder_uid, metric) WHERE status = 'approved';
|
||||
CREATE TABLE fm.baseline_input ( -- an interval is identified by its two endpoint visits
|
||||
baseline_id bigint NOT NULL,
|
||||
site_uid text NOT NULL,
|
||||
operator_uid text NOT NULL REFERENCES fm.operator,
|
||||
holder_uid text NOT NULL,
|
||||
open_visit_id bigint NOT NULL,
|
||||
close_visit_id bigint NOT NULL,
|
||||
start_on date NOT NULL, end_on date NOT NULL CHECK (end_on >= start_on),
|
||||
@@ -448,9 +611,9 @@ CREATE TABLE fm.baseline_input ( -- an interval is identified by
|
||||
dependencies jsonb NOT NULL, -- {"stock": [...], "runtime": [...]} from fm.v_interval_dependency
|
||||
stock_dep_sha256 text NOT NULL, runtime_dep_sha256 text, -- their hashes; runtime NULL for L/day versions
|
||||
PRIMARY KEY (baseline_id, open_visit_id, close_visit_id),
|
||||
FOREIGN KEY (baseline_id, site_uid) REFERENCES fm.baseline_version (baseline_id, site_uid),
|
||||
FOREIGN KEY (open_visit_id, site_uid) REFERENCES fm.visit (visit_id, site_uid), -- both endpoints belong
|
||||
FOREIGN KEY (close_visit_id, site_uid) REFERENCES fm.visit (visit_id, site_uid), -- to the baseline's site
|
||||
FOREIGN KEY (operator_uid, baseline_id, holder_uid) REFERENCES fm.baseline_version (operator_uid, baseline_id, holder_uid),
|
||||
FOREIGN KEY (operator_uid, open_visit_id, holder_uid) REFERENCES fm.visit (operator_uid, visit_id, holder_uid), -- both endpoints belong
|
||||
FOREIGN KEY (operator_uid, close_visit_id, holder_uid) REFERENCES fm.visit (operator_uid, visit_id, holder_uid), -- to the baseline's holder
|
||||
CHECK (included OR exclusion_reason IS NOT NULL),
|
||||
CHECK (NOT included OR end_on > start_on), -- a same-day interval can only be an excluded input
|
||||
EXCLUDE USING gist (baseline_id WITH =, daterange(start_on, end_on, '[)') WITH &&) WHERE (included)
|
||||
@@ -463,7 +626,7 @@ CREATE TABLE fm.baseline_input ( -- an interval is identified by
|
||||
-- Claims are corrected by supersession, never in place (design §2.1). Only these columns may change.
|
||||
CREATE FUNCTION fm.mutable_columns(tbl text) RETURNS text[] LANGUAGE sql IMMUTABLE AS $$
|
||||
SELECT CASE tbl
|
||||
WHEN 'site_location' THEN ARRAY['acceptance','accepted_on','accepted_by_role','acceptance_ref','superseded_by']
|
||||
WHEN 'place_location' THEN ARRAY['acceptance','accepted_on','accepted_by_role','acceptance_ref','superseded_by']
|
||||
WHEN 'assumption' THEN ARRAY['status','accepted_on','accepted_by_role','decision_ref','contradicted_by']
|
||||
ELSE ARRAY['claim_status','superseded_by','review_note']
|
||||
END $$;
|
||||
@@ -479,7 +642,7 @@ BEGIN
|
||||
RAISE EXCEPTION '%: only the status columns change in place; a corrected value is a new row that supersedes this one',
|
||||
TG_TABLE_NAME;
|
||||
END IF;
|
||||
IF TG_TABLE_NAME = 'site_location' THEN
|
||||
IF TG_TABLE_NAME = 'place_location' THEN
|
||||
v_kind := 'location'; v_subject := j->>'source_key'; v_fp := j->>'assertion_sha256'; v_verdict := j->>'acceptance';
|
||||
IF v_verdict NOT IN ('accepted','rejected') THEN RETURN NEW; END IF;
|
||||
ELSIF TG_TABLE_NAME = 'assumption' THEN
|
||||
@@ -490,7 +653,7 @@ BEGIN
|
||||
IF v_verdict NOT IN ('reviewed','disputed') THEN RETURN NEW; END IF;
|
||||
END IF;
|
||||
SELECT * INTO r FROM fm.review x
|
||||
WHERE x.subject_kind = v_kind AND x.subject_key = v_subject AND x.assertion_sha256 = v_fp AND x.verdict = v_verdict
|
||||
WHERE x.operator_uid = j->>'operator_uid' AND x.subject_kind = v_kind AND x.subject_key = v_subject AND x.assertion_sha256 = v_fp AND x.verdict = v_verdict
|
||||
ORDER BY x.reviewed_on DESC, x.review_id DESC LIMIT 1;
|
||||
IF NOT FOUND THEN
|
||||
RAISE EXCEPTION '%: status % for % needs a review of this exact assertion (%)',
|
||||
@@ -508,31 +671,31 @@ DECLARE t text;
|
||||
BEGIN
|
||||
FOREACH t IN ARRAY ARRAY['fuel_movement','stock_observation','hour_meter_reading','electricity_bill',
|
||||
'equipment_spec','imported_calculation','runtime_interval','engine_fuel_total',
|
||||
'site_location','assumption'] LOOP
|
||||
'place_location','assumption'] LOOP
|
||||
EXECUTE format('CREATE TRIGGER status_from_review BEFORE INSERT OR UPDATE ON fm.%I
|
||||
FOR EACH ROW EXECUTE FUNCTION fm.tg_status_from_review()', t);
|
||||
END LOOP;
|
||||
END $$;
|
||||
|
||||
-- 2, 3, 10. A movement claim: same site as its visit; an internal transfer's two tanks placed at
|
||||
-- that site on the visit date; the live claims of one event_key share visit, kind and tanks. A
|
||||
-- 2, 3, 10. A movement claim: same holder as its visit; an internal transfer's two tanks placed on
|
||||
-- that holder on the visit date; the live claims of one event_key share visit, kind and tanks. A
|
||||
-- superseded or retracted claim keeps its own, so a correction can change them.
|
||||
CREATE FUNCTION fm.tg_fuel_movement_rules() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||
DECLARE v fm.visit;
|
||||
BEGIN
|
||||
SELECT * INTO v FROM fm.visit WHERE visit_id = NEW.visit_id;
|
||||
IF v.site_uid <> NEW.site_uid THEN
|
||||
RAISE EXCEPTION 'fuel_movement %: site % differs from its visit''s site %', NEW.source_key, NEW.site_uid, v.site_uid;
|
||||
IF v.holder_uid <> NEW.holder_uid THEN
|
||||
RAISE EXCEPTION 'fuel_movement %: site % differs from its visit''s site %', NEW.source_key, NEW.holder_uid, v.holder_uid;
|
||||
END IF;
|
||||
IF NEW.kind = 'internal_transfer' AND EXISTS (
|
||||
SELECT 1 FROM unnest(ARRAY[NEW.from_tank_uid, NEW.to_tank_uid]) t(tank)
|
||||
WHERE NOT EXISTS (SELECT 1 FROM fm.asset_placement p
|
||||
WHERE p.asset_uid = t.tank AND p.site_uid = NEW.site_uid
|
||||
WHERE p.asset_uid = t.tank AND p.holder_uid = NEW.holder_uid
|
||||
AND daterange(coalesce(p.occupied_from,'-infinity'::date), p.occupied_to, '[)') @> v.visit_on)) THEN
|
||||
RAISE EXCEPTION 'fuel_movement %: both tanks of an internal transfer must be placed at % on %', NEW.source_key, NEW.site_uid, v.visit_on;
|
||||
RAISE EXCEPTION 'fuel_movement %: both tanks of an internal transfer must be placed at % on %', NEW.source_key, NEW.holder_uid, v.visit_on;
|
||||
END IF;
|
||||
IF NEW.claim_status IN ('asserted','reviewed','disputed') AND EXISTS (SELECT 1 FROM fm.fuel_movement m
|
||||
WHERE m.event_key = NEW.event_key AND m.movement_id <> NEW.movement_id
|
||||
WHERE m.operator_uid = NEW.operator_uid AND m.event_key = NEW.event_key AND m.movement_id <> NEW.movement_id
|
||||
AND m.claim_status IN ('asserted','reviewed','disputed')
|
||||
AND (m.visit_id, m.kind, m.from_tank_uid, m.to_tank_uid)
|
||||
IS DISTINCT FROM (NEW.visit_id, NEW.kind, NEW.from_tank_uid, NEW.to_tank_uid)) THEN
|
||||
@@ -543,21 +706,21 @@ END $$;
|
||||
CREATE TRIGGER fuel_movement_rules BEFORE INSERT OR UPDATE ON fm.fuel_movement
|
||||
FOR EACH ROW EXECUTE FUNCTION fm.tg_fuel_movement_rules();
|
||||
|
||||
-- 3. A stock claim belongs to its visit's site.
|
||||
-- 3. A stock claim belongs to its visit's holder.
|
||||
CREATE FUNCTION fm.tg_stock_observation_rules() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||
BEGIN
|
||||
IF NOT EXISTS (SELECT 1 FROM fm.visit v WHERE v.visit_id = NEW.visit_id AND v.site_uid = NEW.site_uid) THEN
|
||||
RAISE EXCEPTION 'stock_observation %: site % differs from its visit''s site', NEW.source_key, NEW.site_uid;
|
||||
IF NOT EXISTS (SELECT 1 FROM fm.visit v WHERE v.visit_id = NEW.visit_id AND v.holder_uid = NEW.holder_uid) THEN
|
||||
RAISE EXCEPTION 'stock_observation %: site % differs from its visit''s site', NEW.source_key, NEW.holder_uid;
|
||||
END IF;
|
||||
RETURN NEW;
|
||||
END $$;
|
||||
CREATE TRIGGER stock_observation_rules BEFORE INSERT OR UPDATE ON fm.stock_observation
|
||||
FOR EACH ROW EXECUTE FUNCTION fm.tg_stock_observation_rules();
|
||||
|
||||
-- 3, 4. A reading's meter belongs to its visit's site; its epoch belongs to its meter.
|
||||
-- 3, 4. A reading's meter belongs to its visit's holder; its epoch belongs to its meter.
|
||||
CREATE FUNCTION fm.tg_hour_meter_reading_rules() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||
BEGIN
|
||||
IF NOT EXISTS (SELECT 1 FROM fm.visit v JOIN fm.runtime_meter m ON m.site_uid = v.site_uid
|
||||
IF NOT EXISTS (SELECT 1 FROM fm.visit v JOIN fm.runtime_meter m ON m.holder_uid = v.holder_uid
|
||||
WHERE v.visit_id = NEW.visit_id AND m.meter_id = NEW.meter_id) THEN
|
||||
RAISE EXCEPTION 'hour_meter_reading %: meter % is not a meter of the visit''s site', NEW.source_key, NEW.meter_id;
|
||||
END IF;
|
||||
@@ -570,9 +733,13 @@ END $$;
|
||||
CREATE TRIGGER hour_meter_reading_rules BEFORE INSERT OR UPDATE ON fm.hour_meter_reading
|
||||
FOR EACH ROW EXECUTE FUNCTION fm.tg_hour_meter_reading_rules();
|
||||
|
||||
-- 7. Supersession stays within one source family.
|
||||
-- 7. Supersession stays within one source family. A document belongs to one customer until
|
||||
-- document sharing exists.
|
||||
CREATE FUNCTION fm.tg_source_document_rules() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||
BEGIN
|
||||
IF EXISTS (SELECT 1 FROM fm.source_document d WHERE d.sha256 = NEW.sha256 AND d.operator_uid <> NEW.operator_uid) THEN
|
||||
RAISE EXCEPTION 'source_document %: already held for another customer; a document cannot be shared yet', left(NEW.sha256, 16);
|
||||
END IF;
|
||||
IF NEW.supersedes_sha256 IS NOT NULL AND NOT EXISTS (
|
||||
SELECT 1 FROM fm.source_document d WHERE d.sha256 = NEW.supersedes_sha256 AND d.source_family = NEW.source_family) THEN
|
||||
RAISE EXCEPTION 'source_document %: it may supersede only a document of family %', left(NEW.sha256, 16), NEW.source_family;
|
||||
@@ -582,30 +749,32 @@ END $$;
|
||||
CREATE TRIGGER source_document_rules BEFORE INSERT OR UPDATE ON fm.source_document
|
||||
FOR EACH ROW EXECUTE FUNCTION fm.tg_source_document_rules();
|
||||
|
||||
-- 8. A confirmed or proposed meter mapping names a generator placed at the meter's site at some
|
||||
-- 8. A confirmed or proposed meter mapping names a generator placed on the meter's holder at some
|
||||
-- time. Whether the placement spans a given interval is fm.v_lph_interval's check.
|
||||
CREATE FUNCTION fm.tg_runtime_meter_rules() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||
BEGIN
|
||||
IF NEW.mapping_status IN ('confirmed','proposed') AND NOT EXISTS (
|
||||
SELECT 1 FROM fm.asset_placement p JOIN fm.asset a ON a.asset_uid = p.asset_uid
|
||||
WHERE p.asset_uid = NEW.asset_uid AND p.site_uid = NEW.site_uid AND a.kind = 'generator') THEN
|
||||
RAISE EXCEPTION 'runtime_meter %: % is not a generator placed at %', NEW.meter_id, NEW.asset_uid, NEW.site_uid;
|
||||
WHERE p.asset_uid = NEW.asset_uid AND p.holder_uid = NEW.holder_uid AND a.kind = 'generator') THEN
|
||||
RAISE EXCEPTION 'runtime_meter %: % is not a generator placed at %', NEW.meter_id, NEW.asset_uid, NEW.holder_uid;
|
||||
END IF;
|
||||
RETURN NEW;
|
||||
END $$;
|
||||
CREATE TRIGGER runtime_meter_rules BEFORE INSERT OR UPDATE ON fm.runtime_meter
|
||||
FOR EACH ROW EXECUTE FUNCTION fm.tg_runtime_meter_rules();
|
||||
|
||||
-- 12. A generator_idle assumption names a generator placed at its own site during its dates.
|
||||
-- 12. A generator_idle assumption names a generator placed on its own holder during its dates.
|
||||
-- 8, 9 and 12 stay generator-only: the runtime and L/h calculations have not been checked on
|
||||
-- an engine, so a meter or idle assumption on one is refused rather than silently unused.
|
||||
CREATE FUNCTION fm.tg_assumption_rules() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||
BEGIN
|
||||
IF NEW.topic = 'generator_idle' AND NOT EXISTS (
|
||||
SELECT 1 FROM fm.asset_placement p JOIN fm.asset a ON a.asset_uid = p.asset_uid
|
||||
WHERE p.asset_uid = NEW.asset_uid AND p.site_uid = NEW.site_uid AND a.kind = 'generator'
|
||||
WHERE p.asset_uid = NEW.asset_uid AND p.holder_uid = NEW.holder_uid AND a.kind = 'generator'
|
||||
AND daterange(coalesce(p.occupied_from,'-infinity'::date), p.occupied_to, '[)')
|
||||
&& daterange(NEW.applies_from, NEW.applies_to, '[]')) THEN
|
||||
RAISE EXCEPTION 'assumption %: % is not a generator placed at % during the assumption''s dates',
|
||||
NEW.code, NEW.asset_uid, NEW.site_uid;
|
||||
NEW.code, NEW.asset_uid, NEW.holder_uid;
|
||||
END IF;
|
||||
RETURN NEW;
|
||||
END $$;
|
||||
@@ -629,12 +798,12 @@ BEGIN
|
||||
END IF;
|
||||
IF NEW.member_role = 'counter' THEN
|
||||
IF NOT EXISTS (SELECT 1 FROM fm.runtime_meter m
|
||||
WHERE m.meter_id = NEW.meter_id AND m.site_uid = a.site_uid AND m.mapping_status <> 'confirmed') THEN
|
||||
RAISE EXCEPTION 'assumption_member: % is not an unconfirmed counter of %', NEW.meter_id, a.site_uid;
|
||||
WHERE m.meter_id = NEW.meter_id AND m.holder_uid = a.holder_uid AND m.mapping_status <> 'confirmed') THEN
|
||||
RAISE EXCEPTION 'assumption_member: % is not an unconfirmed counter of %', NEW.meter_id, a.holder_uid;
|
||||
END IF;
|
||||
ELSIF NOT EXISTS (SELECT 1 FROM fm.asset_placement p JOIN fm.asset s ON s.asset_uid = p.asset_uid
|
||||
WHERE p.asset_uid = NEW.asset_uid AND p.site_uid = a.site_uid AND s.kind = 'generator') THEN
|
||||
RAISE EXCEPTION 'assumption_member: % is not a generator placed at %', NEW.asset_uid, a.site_uid;
|
||||
WHERE p.asset_uid = NEW.asset_uid AND p.holder_uid = a.holder_uid AND s.kind = 'generator') THEN
|
||||
RAISE EXCEPTION 'assumption_member: % is not a generator placed at %', NEW.asset_uid, a.holder_uid;
|
||||
END IF;
|
||||
RETURN NEW;
|
||||
END $$;
|
||||
@@ -710,7 +879,7 @@ BEGIN
|
||||
RAISE EXCEPTION 'baseline_input: the inputs of approved version % are immutable', v.code;
|
||||
END IF;
|
||||
IF TG_OP = 'DELETE' THEN RETURN OLD; END IF;
|
||||
IF TG_OP = 'UPDATE' AND (NEW.baseline_id, NEW.site_uid) IS DISTINCT FROM (OLD.baseline_id, OLD.site_uid) THEN
|
||||
IF TG_OP = 'UPDATE' AND (NEW.baseline_id, NEW.holder_uid) IS DISTINCT FROM (OLD.baseline_id, OLD.holder_uid) THEN
|
||||
RAISE EXCEPTION 'baseline_input: an input of % never moves to another version', v.code;
|
||||
END IF;
|
||||
IF v.status NOT IN ('draft','proposed') THEN
|
||||
@@ -736,6 +905,9 @@ CREATE FUNCTION fm.tg_fixed_columns() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||
DECLARE c text;
|
||||
BEGIN
|
||||
FOREACH c IN ARRAY TG_ARGV LOOP
|
||||
IF NOT to_jsonb(NEW) ? c THEN
|
||||
RAISE EXCEPTION '%: tg_fixed_columns names a column it does not have: %', TG_TABLE_NAME, c;
|
||||
END IF;
|
||||
IF to_jsonb(OLD)->c IS DISTINCT FROM to_jsonb(NEW)->c THEN
|
||||
RAISE EXCEPTION '%: % never changes in place', TG_TABLE_NAME, c;
|
||||
END IF;
|
||||
@@ -743,14 +915,62 @@ BEGIN
|
||||
RETURN NEW;
|
||||
END $$;
|
||||
CREATE TRIGGER visit_fixed BEFORE UPDATE ON fm.visit
|
||||
FOR EACH ROW EXECUTE FUNCTION fm.tg_fixed_columns('visit_key','site_uid','visit_on','seq_in_day');
|
||||
FOR EACH ROW EXECUTE FUNCTION fm.tg_fixed_columns('visit_key','holder_uid','visit_on','seq_in_day');
|
||||
CREATE TRIGGER source_document_fixed BEFORE UPDATE ON fm.source_document
|
||||
FOR EACH ROW EXECUTE FUNCTION fm.tg_fixed_columns('sha256','source_family','kind');
|
||||
CREATE TRIGGER asset_fixed BEFORE UPDATE ON fm.asset
|
||||
FOR EACH ROW EXECUTE FUNCTION fm.tg_fixed_columns('asset_uid','kind');
|
||||
CREATE TRIGGER runtime_meter_fixed BEFORE UPDATE ON fm.runtime_meter -- readings follow their meter's site
|
||||
FOR EACH ROW EXECUTE FUNCTION fm.tg_fixed_columns('meter_id','site_uid');
|
||||
FOR EACH ROW EXECUTE FUNCTION fm.tg_fixed_columns('meter_id','holder_uid');
|
||||
CREATE TRIGGER meter_epoch_fixed BEFORE UPDATE ON fm.meter_epoch -- and their epoch's meter
|
||||
FOR EACH ROW EXECUTE FUNCTION fm.tg_fixed_columns('epoch_id','meter_id');
|
||||
CREATE TRIGGER place_fixed BEFORE UPDATE ON fm.place
|
||||
FOR EACH ROW EXECUTE FUNCTION fm.tg_fixed_columns('place_uid');
|
||||
CREATE TRIGGER holder_fixed BEFORE UPDATE ON fm.holder
|
||||
FOR EACH ROW EXECUTE FUNCTION fm.tg_fixed_columns('holder_uid','kind','asset_uid');
|
||||
-- A row's customer never changes in place, on any table.
|
||||
DO $$
|
||||
DECLARE t text;
|
||||
BEGIN
|
||||
FOR t IN SELECT c.table_name FROM information_schema.columns c
|
||||
WHERE c.table_schema = 'fm' AND c.column_name = 'operator_uid' LOOP
|
||||
EXECUTE format('CREATE TRIGGER operator_fixed BEFORE UPDATE ON fm.%I
|
||||
FOR EACH ROW EXECUTE FUNCTION fm.tg_fixed_columns(%L)', t, 'operator_uid');
|
||||
END LOOP;
|
||||
END $$;
|
||||
|
||||
-- 14. A time zone is one PostgreSQL knows: a wrong one would stop every view that uses it.
|
||||
CREATE FUNCTION fm.tg_timezone_known() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||
BEGIN
|
||||
IF NEW.timezone IS NOT NULL AND NOT EXISTS (SELECT 1 FROM pg_timezone_names n WHERE n.name = NEW.timezone) THEN
|
||||
RAISE EXCEPTION '%: unknown time zone %', TG_TABLE_NAME, NEW.timezone;
|
||||
END IF;
|
||||
RETURN NEW;
|
||||
END $$;
|
||||
CREATE TRIGGER operator_timezone BEFORE INSERT OR UPDATE ON fm.operator
|
||||
FOR EACH ROW EXECUTE FUNCTION fm.tg_timezone_known();
|
||||
CREATE TRIGGER place_timezone BEFORE INSERT OR UPDATE ON fm.place
|
||||
FOR EACH ROW EXECUTE FUNCTION fm.tg_timezone_known();
|
||||
|
||||
-- 15. A vehicle holder's asset is a vehicle; only a fixed holder stands at a place.
|
||||
CREATE FUNCTION fm.tg_holder_rules() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||
BEGIN
|
||||
IF NEW.asset_uid IS NOT NULL AND NOT EXISTS (
|
||||
SELECT 1 FROM fm.asset a WHERE a.asset_uid = NEW.asset_uid AND a.kind = 'vehicle') THEN
|
||||
RAISE EXCEPTION 'holder %: % is not a vehicle', NEW.holder_uid, NEW.asset_uid;
|
||||
END IF;
|
||||
RETURN NEW;
|
||||
END $$;
|
||||
CREATE TRIGGER holder_rules BEFORE INSERT OR UPDATE ON fm.holder
|
||||
FOR EACH ROW EXECUTE FUNCTION fm.tg_holder_rules();
|
||||
CREATE FUNCTION fm.tg_holder_place_rules() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||
BEGIN
|
||||
IF NOT EXISTS (SELECT 1 FROM fm.holder h WHERE h.holder_uid = NEW.holder_uid AND h.kind = 'fixed_installation') THEN
|
||||
RAISE EXCEPTION 'holder_place: % is not a fixed installation', NEW.holder_uid;
|
||||
END IF;
|
||||
RETURN NEW;
|
||||
END $$;
|
||||
CREATE TRIGGER holder_place_rules BEFORE INSERT OR UPDATE ON fm.holder_place
|
||||
FOR EACH ROW EXECUTE FUNCTION fm.tg_holder_place_rules();
|
||||
|
||||
COMMIT;
|
||||
@@ -4,6 +4,18 @@
|
||||
BEGIN;
|
||||
SET LOCAL search_path = fm, pg_catalog, public;
|
||||
|
||||
-- Each holder's customer, current place and local day. The current place is found by the customer's
|
||||
-- local day; a holder without a place, or a place without its own zone, uses the customer's zone.
|
||||
CREATE VIEW fm.v_holder_current AS
|
||||
SELECT h.holder_uid, h.operator_uid, hp.place_uid,
|
||||
coalesce(pl.timezone, o.timezone) AS timezone,
|
||||
(now() AT TIME ZONE coalesce(pl.timezone, o.timezone))::date AS today
|
||||
FROM fm.holder h
|
||||
JOIN fm.operator o ON o.operator_uid = h.operator_uid
|
||||
LEFT JOIN fm.holder_place hp ON hp.holder_uid = h.holder_uid
|
||||
AND daterange(coalesce(hp.valid_from, '-infinity'::date), hp.valid_to, '[)') @> (now() AT TIME ZONE o.timezone)::date
|
||||
LEFT JOIN fm.place pl ON pl.place_uid = hp.place_uid;
|
||||
|
||||
CREATE VIEW fm.v_source_rank AS
|
||||
SELECT sha256, kind, CASE kind WHEN 'delivery_form' THEN 1 WHEN 'site_logbook' THEN 2
|
||||
WHEN 'operator_record' THEN 3 WHEN 'operator_workbook' THEN 3 ELSE 9 END AS rank
|
||||
@@ -74,7 +86,7 @@ HAVING count(*) FILTER (WHERE s.claim_status IN ('asserted','reviewed') AND s.li
|
||||
-- v1: tank-level rows are summed only in a later version, and only when every placed tank has one.
|
||||
|
||||
CREATE VIEW fm.v_stock_point AS -- stock on arrival and on departure of each live visit
|
||||
SELECT v.visit_id, v.visit_key, v.site_uid, v.visit_on, v.seq_in_day, v.kind,
|
||||
SELECT v.visit_id, v.visit_key, v.holder_uid, v.visit_on, v.seq_in_day, v.kind,
|
||||
d.delivered_l, coalesce(o.n_external, 0) AS n_other_movements,
|
||||
coalesce(b.litres, st.litres, a.litres - d.delivered_l) AS arrive_l,
|
||||
CASE WHEN coalesce(b.litres, st.litres) IS NOT NULL THEN 'stated'
|
||||
@@ -104,10 +116,10 @@ LEFT JOIN fm.v_visit_stock st ON st.visit_id = v.visit_id AND st.phase = 'standa
|
||||
|
||||
CREATE VIEW fm.v_stock_interval AS -- consecutive stock points per site; no overlap by construction
|
||||
WITH pts AS (
|
||||
SELECT p.*, row_number() OVER (PARTITION BY p.site_uid ORDER BY p.visit_on, p.seq_in_day) AS n
|
||||
SELECT p.*, row_number() OVER (PARTITION BY p.holder_uid ORDER BY p.visit_on, p.seq_in_day) AS n
|
||||
FROM fm.v_stock_point p WHERE p.arrive_l IS NOT NULL OR p.depart_l IS NOT NULL),
|
||||
pr AS (
|
||||
SELECT o.site_uid, o.visit_key || '>' || c.visit_key AS interval_key,
|
||||
SELECT o.holder_uid, o.visit_key || '>' || c.visit_key AS interval_key,
|
||||
o.visit_id AS open_visit_id, c.visit_id AS close_visit_id,
|
||||
o.visit_on AS start_on, c.visit_on AS end_on, c.visit_on - o.visit_on AS days,
|
||||
o.depart_l AS opening_l, o.depart_basis AS opening_basis,
|
||||
@@ -117,8 +129,8 @@ pr AS (
|
||||
coalesce(o.depart_reviewed AND c.arrive_reviewed, false) AS endpoints_reviewed,
|
||||
o.n_other_movements + c.n_other_movements AS n_endpoint_movements,
|
||||
o.visit_on AS o_on, o.seq_in_day AS o_seq, c.visit_on AS c_on, c.seq_in_day AS c_seq
|
||||
FROM pts o JOIN pts c ON c.site_uid = o.site_uid AND c.n = o.n + 1)
|
||||
SELECT p.site_uid, p.interval_key, p.open_visit_id, p.close_visit_id, p.start_on, p.end_on, p.days,
|
||||
FROM pts o JOIN pts c ON c.holder_uid = o.holder_uid AND c.n = o.n + 1)
|
||||
SELECT p.holder_uid, p.interval_key, p.open_visit_id, p.close_visit_id, p.start_on, p.end_on, p.days,
|
||||
p.opening_l, p.opening_basis, p.closing_l, p.closing_basis,
|
||||
coalesce(mid.delivered_l, 0) AS deliveries_between_l, coalesce(mid.out_l, 0) AS out_l,
|
||||
p.opening_l + coalesce(mid.delivered_l,0) - coalesce(mid.out_l,0) - p.closing_l AS consumed_l,
|
||||
@@ -144,7 +156,7 @@ CROSS JOIN LATERAL (
|
||||
FROM fm.v_visit_live v
|
||||
LEFT JOIN fm.v_visit_delivery d ON d.visit_id = v.visit_id
|
||||
LEFT JOIN fm.v_visit_other_movement x ON x.visit_id = v.visit_id
|
||||
WHERE v.site_uid = p.site_uid
|
||||
WHERE v.holder_uid = p.holder_uid
|
||||
AND (v.visit_on, v.seq_in_day) > (p.o_on, p.o_seq)
|
||||
AND (v.visit_on, v.seq_in_day) < (p.c_on, p.c_seq)) mid;
|
||||
|
||||
@@ -180,7 +192,7 @@ WITH gen AS ( -- generators placed at the site at any time in the
|
||||
bool_or(daterange(coalesce(p.occupied_from,'-infinity'::date), p.occupied_to, '[)')
|
||||
@> daterange(i.start_on, i.end_on, '[]')) AS full_cover
|
||||
FROM fm.v_stock_interval i
|
||||
JOIN fm.asset_placement p ON p.site_uid = i.site_uid
|
||||
JOIN fm.asset_placement p ON p.holder_uid = i.holder_uid
|
||||
JOIN fm.asset a ON a.asset_uid = p.asset_uid AND a.kind = 'generator'
|
||||
WHERE daterange(coalesce(p.occupied_from,'-infinity'::date), p.occupied_to, '[)')
|
||||
&& daterange(i.start_on, i.end_on, '[]')
|
||||
@@ -201,13 +213,13 @@ pair AS ( -- every counter read at both endpoints, classified
|
||||
o.all_reviewed AND c.all_reviewed AS reviewed,
|
||||
(o.hours_max > o.hours_min OR c.hours_max > c.hours_min) AS conflict
|
||||
FROM fm.v_stock_interval i
|
||||
JOIN fm.runtime_meter m ON m.site_uid = i.site_uid
|
||||
JOIN fm.runtime_meter m ON m.holder_uid = i.holder_uid
|
||||
JOIN fm.v_visit_reading o ON o.visit_id = i.open_visit_id AND o.meter_id = m.meter_id
|
||||
JOIN fm.v_visit_reading c ON c.visit_id = i.close_visit_id AND c.meter_id = m.meter_id),
|
||||
idle AS ( -- at most one live generator_idle per asset and date (EXCLUDE)
|
||||
SELECT g.interval_key, g.asset_uid, x.code, x.status
|
||||
FROM gen g JOIN fm.v_stock_interval i USING (interval_key)
|
||||
JOIN fm.assumption x ON x.asset_uid = g.asset_uid AND x.site_uid = i.site_uid -- this site's only
|
||||
JOIN fm.assumption x ON x.asset_uid = g.asset_uid AND x.holder_uid = i.holder_uid -- this site's only
|
||||
AND x.topic = 'generator_idle' AND x.status IN ('proposed','accepted')
|
||||
AND daterange(x.applies_from, x.applies_to, '[]') @> daterange(i.start_on, i.end_on, '[]')),
|
||||
per_gen AS (
|
||||
@@ -234,7 +246,7 @@ loose AS ( -- counters read at both ends but not tied to a set
|
||||
SELECT interval_key, array_agg(meter_id ORDER BY meter_id) AS meters
|
||||
FROM pair WHERE tie = 'not_tied' GROUP BY interval_key),
|
||||
cova AS ( -- live meter_coverage assumptions with their exact members
|
||||
SELECT x.assumption_id, x.code, x.site_uid, x.status, x.applies_from, x.applies_to,
|
||||
SELECT x.assumption_id, x.code, x.holder_uid, x.status, x.applies_from, x.applies_to,
|
||||
coalesce(array_agg(m.meter_id ORDER BY m.meter_id)
|
||||
FILTER (WHERE m.member_role = 'counter'), '{}') AS meters,
|
||||
coalesce(array_agg(m.asset_uid ORDER BY m.asset_uid)
|
||||
@@ -254,7 +266,7 @@ unm AS ( -- at most one live coverage assumption per site an
|
||||
FROM loose l
|
||||
JOIN fm.v_stock_interval i USING (interval_key)
|
||||
LEFT JOIN cov cv USING (interval_key)
|
||||
LEFT JOIN cova c ON c.site_uid = i.site_uid
|
||||
LEFT JOIN cova c ON c.holder_uid = i.holder_uid
|
||||
AND daterange(c.applies_from, c.applies_to, '[]') @> daterange(i.start_on, i.end_on, '[]')),
|
||||
sel AS ( -- the validated population; nothing else is summed
|
||||
SELECT p.* FROM pair p
|
||||
@@ -343,7 +355,7 @@ FROM x;
|
||||
-- M4 (L per calendar day) per stock interval: no runtime, so no generator population. The
|
||||
-- population is the site's whole stock. Input to the L/day proposal path only; never a KPI.
|
||||
CREATE VIEW fm.v_m4_interval AS
|
||||
SELECT i.*, 'stock:' || i.site_uid AS population_key,
|
||||
SELECT i.*, 'stock:' || i.holder_uid AS population_key,
|
||||
CASE WHEN i.days > 0 THEN i.consumed_l / i.days END AS l_per_day,
|
||||
CASE WHEN i.stock_status = 'ok' THEN 'eligible' ELSE 'excluded' END AS eligibility,
|
||||
CASE WHEN i.stock_status <> 'ok' THEN 'stock interval: ' || i.stock_status ELSE '' END AS exclusion_reasons,
|
||||
@@ -359,7 +371,7 @@ WITH span AS ( -- the endpoint visits and every visit between them
|
||||
FROM fm.v_stock_interval i
|
||||
JOIN fm.visit o ON o.visit_id = i.open_visit_id
|
||||
JOIN fm.visit c ON c.visit_id = i.close_visit_id
|
||||
JOIN fm.visit v ON v.site_uid = i.site_uid
|
||||
JOIN fm.visit v ON v.holder_uid = i.holder_uid
|
||||
AND (v.visit_on, v.seq_in_day) >= (o.visit_on, o.seq_in_day)
|
||||
AND (v.visit_on, v.seq_in_day) <= (c.visit_on, c.seq_in_day)),
|
||||
stock_deps AS (
|
||||
@@ -379,27 +391,27 @@ runtime_deps AS (
|
||||
FROM fm.hour_meter_reading x WHERE x.visit_id IN (i.open_visit_id, i.close_visit_id)
|
||||
UNION ALL
|
||||
SELECT 'meter:' || m.meter_id || ':' || m.mapping_status || ':' || coalesce(m.asset_uid, '-')
|
||||
FROM fm.runtime_meter m WHERE m.site_uid = i.site_uid
|
||||
FROM fm.runtime_meter m WHERE m.holder_uid = i.holder_uid
|
||||
UNION ALL
|
||||
SELECT 'epoch:' || e.meter_id || ':' || e.epoch_id || ':' || coalesce(e.starts_on::text, '-')
|
||||
|| ':' || e.start_reason || ':' || e.continuity || ':' || coalesce(e.rollover_at::text, '-')
|
||||
FROM fm.meter_epoch e JOIN fm.runtime_meter m ON m.meter_id = e.meter_id
|
||||
WHERE m.site_uid = i.site_uid
|
||||
WHERE m.holder_uid = i.holder_uid
|
||||
UNION ALL -- placements at this site, and anywhere of the assets its meters name
|
||||
SELECT 'placement:' || p.asset_uid || ':' || p.site_uid || ':' || coalesce(p.occupied_from::text, '-')
|
||||
SELECT 'placement:' || p.asset_uid || ':' || coalesce(p.holder_uid, p.place_uid) || ':' || coalesce(p.occupied_from::text, '-')
|
||||
|| ':' || coalesce(p.occupied_to::text, '-')
|
||||
FROM fm.asset_placement p
|
||||
WHERE (p.site_uid = i.site_uid
|
||||
OR p.asset_uid IN (SELECT m.asset_uid FROM fm.runtime_meter m WHERE m.site_uid = i.site_uid))
|
||||
WHERE (p.holder_uid = i.holder_uid
|
||||
OR p.asset_uid IN (SELECT m.asset_uid FROM fm.runtime_meter m WHERE m.holder_uid = i.holder_uid))
|
||||
AND daterange(coalesce(p.occupied_from, '-infinity'::date), p.occupied_to, '[)')
|
||||
&& daterange(i.start_on, i.end_on, '[]')
|
||||
UNION ALL
|
||||
SELECT 'assumption:' || x.code || ':' || x.fingerprint || ':' || x.status
|
||||
FROM fm.assumption x
|
||||
WHERE x.site_uid = i.site_uid AND x.topic IN ('generator_idle','meter_coverage')
|
||||
WHERE x.holder_uid = i.holder_uid AND x.topic IN ('generator_idle','meter_coverage')
|
||||
AND daterange(x.applies_from, x.applies_to, '[]') && daterange(i.start_on, i.end_on, '[]')) d
|
||||
GROUP BY i.interval_key)
|
||||
SELECT i.interval_key, i.site_uid, i.open_visit_id, i.close_visit_id,
|
||||
SELECT i.interval_key, i.holder_uid, i.open_visit_id, i.close_visit_id,
|
||||
coalesce(s.deps, '[]'::jsonb) AS stock_deps, coalesce(r.deps, '[]'::jsonb) AS runtime_deps,
|
||||
encode(sha256(convert_to(coalesce(s.deps, '[]'::jsonb)::text, 'UTF8')), 'hex') AS stock_dep_sha256,
|
||||
encode(sha256(convert_to(coalesce(r.deps, '[]'::jsonb)::text, 'UTF8')), 'hex') AS runtime_dep_sha256
|
||||
@@ -424,11 +436,11 @@ GROUP BY b.baseline_id, b.metric;
|
||||
-- Consecutive readings per meter (live visits), giving hours per day: R1 / public.v_runtime.
|
||||
CREATE VIEW fm.v_runtime_pair AS
|
||||
WITH r AS (
|
||||
SELECT v.site_uid, v.visit_id, v.visit_on, v.seq_in_day, x.meter_id, x.epoch_id, x.n_epochs,
|
||||
SELECT v.holder_uid, v.visit_id, v.visit_on, v.seq_in_day, x.meter_id, x.epoch_id, x.n_epochs,
|
||||
x.n_unassigned_epoch, x.hours, x.all_reviewed,
|
||||
row_number() OVER (PARTITION BY x.meter_id ORDER BY v.visit_on, v.seq_in_day) AS n
|
||||
FROM fm.v_visit_reading x JOIN fm.v_visit_live v ON v.visit_id = x.visit_id)
|
||||
SELECT o.site_uid, o.meter_id, o.visit_id AS open_visit_id, c.visit_id AS close_visit_id,
|
||||
SELECT o.holder_uid, o.meter_id, o.visit_id AS open_visit_id, c.visit_id AS close_visit_id,
|
||||
o.visit_on AS start_on, c.visit_on AS end_on, c.visit_on - o.visit_on AS days,
|
||||
c.hours - o.hours AS runtime_h,
|
||||
CASE WHEN c.visit_on > o.visit_on THEN (c.hours - o.hours) / (c.visit_on - o.visit_on) END AS h_per_day,
|
||||
@@ -443,7 +455,7 @@ FROM r o JOIN r c ON c.meter_id = o.meter_id AND c.n = o.n + 1;
|
||||
-- assumption's dates, divided by that independently sourced runtime assumption (never one derived
|
||||
-- from fuel). The window is the matched one. Reference tab only; never a KPI (design §4.1).
|
||||
CREATE VIEW fm.v_modelled_lph AS
|
||||
SELECT x.site_uid, x.code, x.status, x.basis,
|
||||
SELECT x.holder_uid, x.code, x.status, x.basis,
|
||||
x.value_num AS h_per_day, x.low_num AS h_per_day_low, x.high_num AS h_per_day_high,
|
||||
m4.window_start, m4.window_end, m4.days AS matched_days, m4.fuel_l / NULLIF(m4.days, 0) AS l_per_day,
|
||||
m4.fuel_l / NULLIF(m4.days, 0) / NULLIF(x.value_num, 0) AS modelled_l_per_h,
|
||||
@@ -454,46 +466,48 @@ CROSS JOIN LATERAL (
|
||||
SELECT sum(i.consumed_l) AS fuel_l, sum(i.days) AS days,
|
||||
min(i.start_on) AS window_start, max(i.end_on) AS window_end
|
||||
FROM fm.v_m4_interval i
|
||||
WHERE i.site_uid = x.site_uid AND i.eligibility = 'eligible'
|
||||
WHERE i.holder_uid = x.holder_uid AND i.eligibility = 'eligible'
|
||||
AND daterange(x.applies_from, x.applies_to, '[]') @> daterange(i.start_on, i.end_on, '[]')) m4
|
||||
WHERE x.topic = 'runtime_hours_per_day' AND NOT x.derived_from_fuel AND x.status IN ('proposed','accepted')
|
||||
AND m4.days > 0;
|
||||
|
||||
-- Review entries whose fingerprint matches no live row: the value changed after it was reviewed,
|
||||
-- or the subject was superseded or retracted. The site comes from every row the subject key has
|
||||
-- had, in any status; for an asset's specification or engine total, from every site the asset was
|
||||
-- placed at. So one entry can give one row per site, and none is lost when its subject leaves.
|
||||
-- or the subject was superseded or retracted. The holder comes from every row the subject key has
|
||||
-- had, in any status; for an asset's specification or engine total, from every holder the asset was
|
||||
-- placed on; for a location, from every holder that has stood at the place. So one entry can give
|
||||
-- one row per holder, and none is lost when its subject leaves.
|
||||
CREATE VIEW fm.v_review_stale AS
|
||||
WITH subj AS (
|
||||
SELECT 'claim' AS kind, source_key AS subject_key, assertion_sha256, site_uid,
|
||||
SELECT operator_uid, 'claim' AS kind, source_key AS subject_key, assertion_sha256, holder_uid,
|
||||
claim_status IN ('asserted','reviewed','disputed') AS live
|
||||
FROM fm.fuel_movement
|
||||
UNION ALL SELECT 'claim', source_key, assertion_sha256, site_uid, claim_status IN ('asserted','reviewed','disputed')
|
||||
UNION ALL SELECT operator_uid, 'claim', source_key, assertion_sha256, holder_uid, claim_status IN ('asserted','reviewed','disputed')
|
||||
FROM fm.stock_observation
|
||||
UNION ALL SELECT 'claim', h.source_key, h.assertion_sha256, m.site_uid, h.claim_status IN ('asserted','reviewed','disputed')
|
||||
UNION ALL SELECT h.operator_uid, 'claim', h.source_key, h.assertion_sha256, m.holder_uid, h.claim_status IN ('asserted','reviewed','disputed')
|
||||
FROM fm.hour_meter_reading h JOIN fm.runtime_meter m ON m.meter_id = h.meter_id
|
||||
UNION ALL SELECT 'claim', source_key, assertion_sha256, site_uid, claim_status IN ('asserted','reviewed','disputed')
|
||||
UNION ALL SELECT operator_uid, 'claim', source_key, assertion_sha256, holder_uid, claim_status IN ('asserted','reviewed','disputed')
|
||||
FROM fm.electricity_bill
|
||||
UNION ALL SELECT 'claim', source_key, assertion_sha256, site_uid, claim_status IN ('asserted','reviewed','disputed')
|
||||
UNION ALL SELECT operator_uid, 'claim', source_key, assertion_sha256, holder_uid, claim_status IN ('asserted','reviewed','disputed')
|
||||
FROM fm.imported_calculation
|
||||
UNION ALL SELECT 'claim', e.source_key, e.assertion_sha256, p.site_uid, e.claim_status IN ('asserted','reviewed','disputed')
|
||||
UNION ALL SELECT e.operator_uid, 'claim', e.source_key, e.assertion_sha256, p.holder_uid, e.claim_status IN ('asserted','reviewed','disputed')
|
||||
FROM fm.equipment_spec e LEFT JOIN fm.asset_placement p ON p.asset_uid = e.asset_uid
|
||||
UNION ALL SELECT 'claim', source_key, assertion_sha256, site_uid, claim_status IN ('asserted','reviewed','disputed')
|
||||
UNION ALL SELECT operator_uid, 'claim', source_key, assertion_sha256, holder_uid, claim_status IN ('asserted','reviewed','disputed')
|
||||
FROM fm.runtime_interval
|
||||
UNION ALL SELECT 'claim', t.source_key, t.assertion_sha256, p.site_uid, t.claim_status IN ('asserted','reviewed','disputed')
|
||||
UNION ALL SELECT t.operator_uid, 'claim', t.source_key, t.assertion_sha256, p.holder_uid, t.claim_status IN ('asserted','reviewed','disputed')
|
||||
FROM fm.engine_fuel_total t LEFT JOIN fm.asset_placement p ON p.asset_uid = t.asset_uid
|
||||
UNION ALL SELECT 'location', source_key, assertion_sha256, site_uid, acceptance IN ('pending','accepted','rejected')
|
||||
FROM fm.site_location
|
||||
UNION ALL SELECT 'assumption', code, fingerprint, site_uid, status <> 'superseded'
|
||||
UNION ALL SELECT l.operator_uid, 'location', l.source_key, l.assertion_sha256, hp.holder_uid, l.acceptance IN ('pending','accepted','rejected')
|
||||
FROM fm.place_location l LEFT JOIN fm.holder_place hp ON hp.place_uid = l.place_uid
|
||||
UNION ALL SELECT operator_uid, 'assumption', code, fingerprint, holder_uid, status <> 'superseded'
|
||||
FROM fm.assumption)
|
||||
SELECT r.review_id, r.entry_key, r.subject_kind, r.subject_key, r.assertion_sha256, r.verdict,
|
||||
r.reviewed_on, r.reviewer_role, st.site_uid
|
||||
SELECT r.operator_uid, r.review_id, r.entry_key, r.subject_kind, r.subject_key, r.assertion_sha256, r.verdict,
|
||||
r.reviewed_on, r.reviewer_role, st.holder_uid
|
||||
FROM fm.review r
|
||||
LEFT JOIN LATERAL (SELECT DISTINCT s.site_uid FROM subj s
|
||||
WHERE s.kind = r.subject_kind AND s.subject_key = r.subject_key AND s.site_uid IS NOT NULL) st ON true
|
||||
LEFT JOIN LATERAL (SELECT DISTINCT s.holder_uid FROM subj s
|
||||
WHERE s.operator_uid = r.operator_uid AND s.kind = r.subject_kind
|
||||
AND s.subject_key = r.subject_key AND s.holder_uid IS NOT NULL) st ON true
|
||||
WHERE NOT EXISTS (SELECT 1 FROM subj l
|
||||
WHERE l.live AND l.kind = r.subject_kind AND l.subject_key = r.subject_key
|
||||
AND l.assertion_sha256 = r.assertion_sha256);
|
||||
WHERE l.live AND l.operator_uid = r.operator_uid AND l.kind = r.subject_kind
|
||||
AND l.subject_key = r.subject_key AND l.assertion_sha256 = r.assertion_sha256);
|
||||
|
||||
-- Visits kept for provenance only: every claim superseded or retracted.
|
||||
CREATE VIEW fm.v_visit_provenance_only AS
|
||||
|
||||
@@ -18,16 +18,16 @@ SET LOCAL search_path = fm, pg_catalog, public;
|
||||
-- insufficient_evidence: none of the above (today: every site, since there is no runtime).
|
||||
CREATE VIEW fm.v_site_baseline_status AS
|
||||
WITH cur AS (
|
||||
SELECT DISTINCT ON (b.site_uid) b.*
|
||||
SELECT DISTINCT ON (b.holder_uid) b.*
|
||||
FROM fm.baseline_version b
|
||||
WHERE b.status IN ('approved','proposed')
|
||||
AND b.metric IN ('measured_generator_l_per_h','apparent_l_per_generator_h',
|
||||
'apparent_l_per_set_h','apparent_l_per_site_dg_h') -- never modelled, never per-day
|
||||
ORDER BY b.site_uid, (b.status = 'approved') DESC, b.approved_on DESC NULLS LAST, b.computed_at DESC),
|
||||
ORDER BY b.holder_uid, (b.status = 'approved') DESC, b.approved_on DESC NULLS LAST, b.computed_at DESC),
|
||||
ev AS (
|
||||
SELECT site_uid, count(*) FILTER (WHERE eligibility <> 'excluded') AS n_usable
|
||||
FROM fm.v_lph_interval GROUP BY site_uid)
|
||||
SELECT s.site_uid,
|
||||
SELECT holder_uid, count(*) FILTER (WHERE eligibility <> 'excluded') AS n_usable
|
||||
FROM fm.v_lph_interval GROUP BY holder_uid)
|
||||
SELECT s.holder_uid,
|
||||
CASE WHEN cur.status = 'approved' THEN 'approved'
|
||||
WHEN cur.status = 'proposed' AND cur.meets_policy AND val.inputs_unchanged THEN 'eligible'
|
||||
WHEN cur.status = 'proposed' OR coalesce(ev.n_usable, 0) > 0 THEN 'provisional'
|
||||
@@ -42,37 +42,45 @@ SELECT s.site_uid,
|
||||
cur.metric AS lph_metric, cur.population, cur.window_start AS lph_window_start,
|
||||
cur.window_end AS lph_window_end, cur.conditions, cur.decision_ref,
|
||||
coalesce(ev.n_usable, 0) AS n_usable_intervals
|
||||
FROM fm.site s LEFT JOIN cur USING (site_uid) LEFT JOIN ev USING (site_uid)
|
||||
FROM fm.holder s LEFT JOIN cur USING (holder_uid) LEFT JOIN ev USING (holder_uid)
|
||||
LEFT JOIN fm.v_baseline_validity val ON val.baseline_id = cur.baseline_id;
|
||||
|
||||
-- One row per site, every status: the map dataset and the single source of site-level fields.
|
||||
-- "Current" means the period that contains today's date in Kabul, not "no end date".
|
||||
-- One row per holder, every status: the map dataset and the single source of site-level fields.
|
||||
-- Public column names stay site_* (a telecom site is a place with one fixed holder); internal fm uses
|
||||
-- holder and place. "Current" means the period that contains today in the holder's local time.
|
||||
CREATE VIEW public.v_site AS
|
||||
WITH today AS (SELECT (now() AT TIME ZONE 'Asia/Kabul')::date AS d),
|
||||
alias AS ( -- v1: the operator is ATOMA; later: the site's current operator
|
||||
SELECT a.site_uid, a.label FROM fm.site_alias a, today
|
||||
WHERE a.issuer = 'ATOMA' AND a.is_primary
|
||||
AND daterange(coalesce(a.valid_from, '-infinity'::date), a.valid_to, '[)') @> today.d),
|
||||
pp AS ( -- one programme, chosen explicitly
|
||||
SELECT p.site_uid, p.status FROM fm.pilot_participation p, today
|
||||
WHERE p.programme = 'ATOMA OFMS pilot' AND daterange(p.valid_from, p.valid_to, '[)') @> today.d),
|
||||
-- alias and pp are at most one row per site: the exclusion constraints forbid overlapping primary
|
||||
-- labels per (site, issuer) and overlapping periods per (site, programme).
|
||||
loc AS (SELECT site_uid, latitude, longitude, basis, precision_m FROM fm.site_location WHERE acceptance = 'accepted'),
|
||||
loc_any AS (SELECT site_uid, bool_or(acceptance = 'pending') AS has_pending FROM fm.site_location GROUP BY site_uid),
|
||||
cal AS (SELECT site_uid, sum(consumed_l) AS fuel_l, sum(days) AS days
|
||||
FROM fm.v_stock_interval WHERE stock_status = 'ok' GROUP BY site_uid),
|
||||
latest AS (SELECT site_uid, max(visit_on) AS latest_record_on -- live records only: a fully
|
||||
FROM fm.v_visit_live GROUP BY site_uid) -- retracted visit does not count
|
||||
SELECT s.site_uid::varchar(12) AS site_uid,
|
||||
coalesce(alias.label, s.site_uid)::varchar(20) AS site_code,
|
||||
WITH alias AS ( -- the label the customer's people use
|
||||
SELECT a.holder_uid, a.label FROM fm.holder_alias a
|
||||
JOIN fm.v_holder_current z ON z.holder_uid = a.holder_uid
|
||||
JOIN fm.operator o ON o.operator_uid = z.operator_uid
|
||||
WHERE a.issuer = o.label_issuer AND a.is_primary
|
||||
AND daterange(coalesce(a.valid_from, '-infinity'::date), a.valid_to, '[)') @> z.today),
|
||||
pp AS ( -- the customer's one programme
|
||||
SELECT p.holder_uid, p.status FROM fm.participation p
|
||||
JOIN fm.v_holder_current z ON z.holder_uid = p.holder_uid
|
||||
JOIN fm.operator o ON o.operator_uid = z.operator_uid
|
||||
WHERE p.programme = o.programme AND daterange(p.valid_from, p.valid_to, '[)') @> z.today),
|
||||
-- alias and pp are at most one row per holder: the exclusion constraints forbid overlapping primary
|
||||
-- labels per (holder, issuer) and overlapping periods per (holder, programme).
|
||||
loc AS (SELECT z.holder_uid, l.latitude, l.longitude, l.basis, l.precision_m
|
||||
FROM fm.place_location l JOIN fm.v_holder_current z ON z.place_uid = l.place_uid
|
||||
WHERE l.acceptance = 'accepted'),
|
||||
loc_any AS (SELECT z.holder_uid, bool_or(l.acceptance = 'pending') AS has_pending
|
||||
FROM fm.place_location l JOIN fm.v_holder_current z ON z.place_uid = l.place_uid GROUP BY z.holder_uid),
|
||||
cal AS (SELECT holder_uid, sum(consumed_l) AS fuel_l, sum(days) AS days
|
||||
FROM fm.v_stock_interval WHERE stock_status = 'ok' GROUP BY holder_uid),
|
||||
latest AS (SELECT holder_uid, max(visit_on) AS latest_record_on -- live records only: a fully
|
||||
FROM fm.v_visit_live GROUP BY holder_uid) -- retracted visit does not count
|
||||
SELECT s.holder_uid::varchar(12) AS site_uid,
|
||||
coalesce(alias.label, s.holder_uid)::varchar(20) AS site_code,
|
||||
coalesce(s.name, 'name not recorded')::varchar(120) AS site_name,
|
||||
(coalesce(alias.label, s.site_uid) || ' - ' || coalesce(s.name, 'name not recorded')
|
||||
|| ' - ' || s.site_uid)::varchar(170) AS site_label, -- the one filter value
|
||||
s.zone::varchar(40) AS zone, s.province::varchar(40) AS province, s.district::varchar(60) AS district,
|
||||
(coalesce(alias.label, s.holder_uid) || ' - ' || coalesce(s.name, 'name not recorded')
|
||||
|| ' - ' || s.holder_uid)::varchar(170) AS site_label, -- the one filter value
|
||||
s.operator_uid::varchar(12) AS operator_uid,
|
||||
pl.zone::varchar(40) AS zone, pl.province::varchar(40) AS province, pl.district::varchar(60) AS district,
|
||||
coalesce(pp.status, 'none')::varchar(12) AS pilot_status,
|
||||
loc.latitude::numeric(9,6) AS latitude, loc.longitude::numeric(9,6) AS longitude, -- NULL unless accepted
|
||||
(CASE WHEN loc.site_uid IS NOT NULL THEN 'accepted'
|
||||
(CASE WHEN loc.holder_uid IS NOT NULL THEN 'accepted'
|
||||
WHEN loc_any.has_pending THEN 'awaiting acceptance' ELSE 'missing' END)::varchar(20) AS location_status,
|
||||
loc.basis::varchar(20) AS location_basis, loc.precision_m::integer AS location_precision_m,
|
||||
bs.baseline_status::varchar(24) AS baseline_status,
|
||||
@@ -91,19 +99,22 @@ SELECT s.site_uid::varchar(12) AS site_ui
|
||||
bs.decision_ref::varchar(12) AS baseline_decision_ref,
|
||||
round(cal.fuel_l / NULLIF(cal.days, 0), 1)::numeric(7,1) AS fuel_l_per_calendar_day,
|
||||
latest.latest_record_on
|
||||
FROM fm.site s
|
||||
LEFT JOIN alias USING (site_uid) LEFT JOIN loc USING (site_uid) LEFT JOIN loc_any USING (site_uid)
|
||||
LEFT JOIN pp USING (site_uid) LEFT JOIN fm.v_site_baseline_status bs USING (site_uid)
|
||||
LEFT JOIN cal USING (site_uid) LEFT JOIN latest USING (site_uid);
|
||||
FROM fm.holder s
|
||||
JOIN fm.v_holder_current z ON z.holder_uid = s.holder_uid
|
||||
LEFT JOIN fm.place pl ON pl.place_uid = z.place_uid
|
||||
LEFT JOIN alias ON alias.holder_uid = s.holder_uid LEFT JOIN loc ON loc.holder_uid = s.holder_uid
|
||||
LEFT JOIN loc_any ON loc_any.holder_uid = s.holder_uid LEFT JOIN pp ON pp.holder_uid = s.holder_uid
|
||||
LEFT JOIN fm.v_site_baseline_status bs ON bs.holder_uid = s.holder_uid
|
||||
LEFT JOIN cal ON cal.holder_uid = s.holder_uid LEFT JOIN latest ON latest.holder_uid = s.holder_uid;
|
||||
|
||||
-- Every view below joins public.v_site for site_uid, site_code, site_name and site_label, so one
|
||||
-- Every view below joins public.v_site for site_uid, site_code, site_name, site_label and operator_uid, so one
|
||||
-- Site filter drives every dataset, and casts to plain types (varchar(n), integer,
|
||||
-- numeric(p,s), date, timestamp(0), boolean). Live = asserted, reviewed or disputed.
|
||||
|
||||
-- S2, R3: one row per site and measure; the rows stay when a value is not established.
|
||||
CREATE VIEW public.v_kpi_measures AS
|
||||
WITH cur AS ( -- the site's L/h version; its snapshot only while approved or its inputs unchanged
|
||||
SELECT bs.site_uid, b.baseline_id, b.code, b.fuel_l, b.runtime_h, b.matched_days, b.window_start, b.window_end,
|
||||
SELECT bs.holder_uid, b.baseline_id, b.code, b.fuel_l, b.runtime_h, b.matched_days, b.window_start, b.window_end,
|
||||
(b.status = 'approved' OR coalesce(val.inputs_unchanged, false)) AS shown,
|
||||
' (' || b.status || ')'
|
||||
|| CASE WHEN coalesce(val.inputs_unchanged, false) THEN ''
|
||||
@@ -112,35 +123,35 @@ WITH cur AS ( -- the site's L/h version; its snapshot only while
|
||||
FROM fm.v_site_baseline_status bs JOIN fm.baseline_version b ON b.code = bs.baseline_code
|
||||
LEFT JOIN fm.v_baseline_validity val ON val.baseline_id = b.baseline_id),
|
||||
ninp AS (SELECT baseline_id, count(*) FILTER (WHERE included) AS n FROM fm.baseline_input GROUP BY baseline_id),
|
||||
m4 AS (SELECT site_uid, sum(consumed_l) AS fuel_l, sum(days) AS days, min(start_on) AS first_on, max(end_on) AS last_on
|
||||
FROM fm.v_m4_interval WHERE eligibility = 'eligible' GROUP BY site_uid),
|
||||
span AS (SELECT site_uid, min(visit_on) AS first_on, max(visit_on) AS last_on FROM fm.v_visit_live GROUP BY site_uid),
|
||||
del AS (SELECT v.site_uid, count(*) AS n FROM fm.v_movement_event e JOIN fm.v_visit_live v ON v.visit_id = e.visit_id
|
||||
WHERE e.kind = 'delivery' GROUP BY v.site_uid),
|
||||
rdg AS (SELECT m.site_uid, count(*) AS n FROM fm.hour_meter_reading h JOIN fm.runtime_meter m ON m.meter_id = h.meter_id
|
||||
WHERE h.claim_status IN ('asserted','reviewed','disputed') GROUP BY m.site_uid),
|
||||
kwh AS (SELECT site_uid, sum(kwh) AS kwh, count(kwh) AS n_kwh, min(bill_month) AS first_m, max(bill_month) AS last_m
|
||||
FROM fm.electricity_bill WHERE claim_status IN ('asserted','reviewed') GROUP BY site_uid), -- values never from disputed
|
||||
m4 AS (SELECT holder_uid, sum(consumed_l) AS fuel_l, sum(days) AS days, min(start_on) AS first_on, max(end_on) AS last_on
|
||||
FROM fm.v_m4_interval WHERE eligibility = 'eligible' GROUP BY holder_uid),
|
||||
span AS (SELECT holder_uid, min(visit_on) AS first_on, max(visit_on) AS last_on FROM fm.v_visit_live GROUP BY holder_uid),
|
||||
del AS (SELECT v.holder_uid, count(*) AS n FROM fm.v_movement_event e JOIN fm.v_visit_live v ON v.visit_id = e.visit_id
|
||||
WHERE e.kind = 'delivery' GROUP BY v.holder_uid),
|
||||
rdg AS (SELECT m.holder_uid, count(*) AS n FROM fm.hour_meter_reading h JOIN fm.runtime_meter m ON m.meter_id = h.meter_id
|
||||
WHERE h.claim_status IN ('asserted','reviewed','disputed') GROUP BY m.holder_uid),
|
||||
kwh AS (SELECT holder_uid, sum(kwh) AS kwh, count(kwh) AS n_kwh, min(bill_month) AS first_m, max(bill_month) AS last_m
|
||||
FROM fm.electricity_bill WHERE claim_status IN ('asserted','reviewed') GROUP BY holder_uid), -- values never from disputed
|
||||
lday AS ( -- the L/day reference: the latest approved, else proposed, L/day version
|
||||
SELECT DISTINCT ON (b.site_uid) b.site_uid, b.code, b.status, b.value, b.window_start, b.window_end,
|
||||
SELECT DISTINCT ON (b.holder_uid) b.holder_uid, b.code, b.status, b.value, b.window_start, b.window_end,
|
||||
coalesce(val.inputs_unchanged, false) AS inputs_unchanged
|
||||
FROM fm.baseline_version b LEFT JOIN fm.v_baseline_validity val ON val.baseline_id = b.baseline_id
|
||||
WHERE b.metric = 'fuel_l_per_calendar_day' AND b.status IN ('approved','proposed')
|
||||
ORDER BY b.site_uid, (b.status = 'approved') DESC, b.approved_on DESC NULLS LAST, b.computed_at DESC)
|
||||
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||
ORDER BY b.holder_uid, (b.status = 'approved') DESC, b.approved_on DESC NULLS LAST, b.computed_at DESC)
|
||||
SELECT s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
|
||||
x.ord::integer AS measure_order, x.measure::varchar(60) AS measure,
|
||||
round(x.value, 2)::numeric(12,2) AS value, x.unit::varchar(12) AS unit,
|
||||
(CASE WHEN x.value IS NULL THEN 'not established' ELSE x.kind END)::varchar(16) AS value_kind,
|
||||
x.window_start, x.window_end, x.note::varchar(200) AS note
|
||||
FROM public.v_site s
|
||||
LEFT JOIN cur ON cur.site_uid = s.site_uid
|
||||
LEFT JOIN cur ON cur.holder_uid = s.site_uid
|
||||
LEFT JOIN ninp ON ninp.baseline_id = cur.baseline_id
|
||||
LEFT JOIN m4 ON m4.site_uid = s.site_uid
|
||||
LEFT JOIN span ON span.site_uid = s.site_uid
|
||||
LEFT JOIN del ON del.site_uid = s.site_uid
|
||||
LEFT JOIN rdg ON rdg.site_uid = s.site_uid
|
||||
LEFT JOIN kwh ON kwh.site_uid = s.site_uid
|
||||
LEFT JOIN lday ON lday.site_uid = s.site_uid
|
||||
LEFT JOIN m4 ON m4.holder_uid = s.site_uid
|
||||
LEFT JOIN span ON span.holder_uid = s.site_uid
|
||||
LEFT JOIN del ON del.holder_uid = s.site_uid
|
||||
LEFT JOIN rdg ON rdg.holder_uid = s.site_uid
|
||||
LEFT JOIN kwh ON kwh.holder_uid = s.site_uid
|
||||
LEFT JOIN lday ON lday.holder_uid = s.site_uid
|
||||
CROSS JOIN LATERAL (VALUES
|
||||
(1, 'Matched fuel', CASE WHEN cur.shown THEN cur.fuel_l END, 'L', 'calculated', cur.window_start, cur.window_end,
|
||||
'fuel used in the intervals of L/h baseline version ' || cur.code || cur.state),
|
||||
@@ -175,7 +186,7 @@ CROSS JOIN LATERAL (VALUES
|
||||
|
||||
-- R2: timer readings as recorded (no rows in v1).
|
||||
CREATE VIEW public.v_hour_meter_evidence AS
|
||||
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||
SELECT s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
|
||||
m.source_label::varchar(40) AS meter_label,
|
||||
(CASE WHEN m.asset_uid IS NULL THEN 'unmapped'
|
||||
ELSE m.asset_uid || ' (' || m.mapping_status || ')' END)::varchar(60) AS mapped_set,
|
||||
@@ -186,13 +197,13 @@ SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||
FROM fm.hour_meter_reading h
|
||||
JOIN fm.runtime_meter m ON m.meter_id = h.meter_id
|
||||
JOIN fm.visit v ON v.visit_id = h.visit_id
|
||||
JOIN public.v_site s ON s.site_uid = m.site_uid
|
||||
JOIN public.v_site s ON s.site_uid = m.holder_uid
|
||||
LEFT JOIN fm.meter_epoch e ON e.epoch_id = h.epoch_id
|
||||
WHERE h.claim_status IN ('asserted','reviewed','disputed');
|
||||
|
||||
-- B5: assumption versions in force or proposed; a coverage assumption lists its members.
|
||||
CREATE VIEW public.v_assumption AS
|
||||
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||
SELECT s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
|
||||
x.code::varchar(60) AS code, x.topic::varchar(30) AS topic, x.asset_uid::varchar(40) AS asset_uid,
|
||||
x.value_num::numeric(10,3) AS value, x.low_num::numeric(10,3) AS value_low,
|
||||
x.high_num::numeric(10,3) AS value_high, x.unit::varchar(20) AS unit,
|
||||
@@ -201,7 +212,7 @@ SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||
x.rationale::varchar(400) AS rationale, x.contradicted_by::varchar(200) AS contradicted_by,
|
||||
mem.members::varchar(400) AS members
|
||||
FROM fm.assumption x
|
||||
JOIN public.v_site s ON s.site_uid = x.site_uid
|
||||
JOIN public.v_site s ON s.site_uid = x.holder_uid
|
||||
LEFT JOIN LATERAL (
|
||||
SELECT string_agg(m.member_role || ': ' || coalesce(m.meter_id, m.asset_uid), '; '
|
||||
ORDER BY m.member_role, coalesce(m.meter_id, m.asset_uid)) AS members
|
||||
@@ -210,7 +221,7 @@ WHERE x.status <> 'superseded';
|
||||
|
||||
-- B1: baseline versions, with whether their inputs are unchanged today.
|
||||
CREATE VIEW public.v_baseline_version AS
|
||||
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||
SELECT s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
|
||||
b.code::varchar(60) AS code, b.metric::varchar(40) AS metric, b.status::varchar(12) AS status,
|
||||
b.population::varchar(200) AS population,
|
||||
round(b.value, 3)::numeric(8,3) AS value, round(b.range_low, 3)::numeric(8,3) AS range_low,
|
||||
@@ -222,14 +233,15 @@ SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||
b.evidence_grade::varchar(2) AS evidence_grade, b.conditions::varchar(200) AS conditions,
|
||||
b.season_note::varchar(200) AS season_note, b.approved_on, b.approved_by_role::varchar(40) AS approved_by_role,
|
||||
b.decision_ref::varchar(12) AS decision_ref, coalesce(val.inputs_unchanged, false) AS inputs_unchanged,
|
||||
(b.computed_at AT TIME ZONE 'Asia/Kabul')::timestamp(0) AS computed_at
|
||||
(b.computed_at AT TIME ZONE z.timezone)::timestamp(0) AS computed_at
|
||||
FROM fm.baseline_version b
|
||||
JOIN public.v_site s ON s.site_uid = b.site_uid
|
||||
JOIN public.v_site s ON s.site_uid = b.holder_uid
|
||||
JOIN fm.v_holder_current z ON z.holder_uid = b.holder_uid
|
||||
LEFT JOIN fm.v_baseline_validity val ON val.baseline_id = b.baseline_id;
|
||||
|
||||
-- B2: the inputs of each version, with dependency drift since computation.
|
||||
CREATE VIEW public.v_baseline_input AS
|
||||
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||
SELECT s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
|
||||
b.code::varchar(60) AS baseline_code, b.status::varchar(12) AS baseline_status,
|
||||
(to_char(bi.start_on, 'YYYY-MM-DD') || ' to ' || to_char(bi.end_on, 'YYYY-MM-DD'))::varchar(30) AS interval_label,
|
||||
bi.start_on, bi.end_on, bi.included, bi.exclusion_reason::varchar(300) AS exclusion_reason,
|
||||
@@ -241,42 +253,42 @@ SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||
AND d.runtime_dep_sha256 IS DISTINCT FROM bi.runtime_dep_sha256)) AS dependencies_changed
|
||||
FROM fm.baseline_input bi
|
||||
JOIN fm.baseline_version b ON b.baseline_id = bi.baseline_id
|
||||
JOIN public.v_site s ON s.site_uid = bi.site_uid
|
||||
JOIN public.v_site s ON s.site_uid = bi.holder_uid
|
||||
LEFT JOIN fm.v_interval_dependency d ON d.open_visit_id = bi.open_visit_id AND d.close_visit_id = bi.close_visit_id;
|
||||
|
||||
-- B7: documents the site's figures rest on (the document itself is never served).
|
||||
CREATE VIEW public.v_evidence AS
|
||||
WITH cites AS (
|
||||
SELECT site_uid, source_sha256, 'fuel movement' AS record_type
|
||||
SELECT holder_uid, source_sha256, 'fuel movement' AS record_type
|
||||
FROM fm.fuel_movement WHERE claim_status IN ('asserted','reviewed','disputed')
|
||||
UNION SELECT site_uid, source_sha256, 'stock reading'
|
||||
UNION SELECT holder_uid, source_sha256, 'stock reading'
|
||||
FROM fm.stock_observation WHERE claim_status IN ('asserted','reviewed','disputed')
|
||||
UNION SELECT m.site_uid, h.source_sha256, 'timer reading'
|
||||
UNION SELECT m.holder_uid, h.source_sha256, 'timer reading'
|
||||
FROM fm.hour_meter_reading h JOIN fm.runtime_meter m ON m.meter_id = h.meter_id
|
||||
WHERE h.claim_status IN ('asserted','reviewed','disputed')
|
||||
UNION SELECT site_uid, source_sha256, 'electricity bill'
|
||||
UNION SELECT holder_uid, source_sha256, 'electricity bill'
|
||||
FROM fm.electricity_bill WHERE claim_status IN ('asserted','reviewed','disputed')
|
||||
UNION SELECT site_uid, source_sha256, 'imported calculation'
|
||||
FROM fm.imported_calculation WHERE site_uid IS NOT NULL AND claim_status IN ('asserted','reviewed','disputed')
|
||||
UNION SELECT p.site_uid, e.source_sha256, 'equipment specification'
|
||||
UNION SELECT holder_uid, source_sha256, 'imported calculation'
|
||||
FROM fm.imported_calculation WHERE holder_uid IS NOT NULL AND claim_status IN ('asserted','reviewed','disputed')
|
||||
UNION SELECT p.holder_uid, e.source_sha256, 'equipment specification'
|
||||
FROM fm.equipment_spec e JOIN fm.asset_placement p ON p.asset_uid = e.asset_uid
|
||||
WHERE e.source_sha256 IS NOT NULL AND e.claim_status IN ('asserted','reviewed','disputed')
|
||||
UNION SELECT site_uid, source_sha256, 'assumption'
|
||||
UNION SELECT holder_uid, source_sha256, 'assumption'
|
||||
FROM fm.assumption WHERE source_sha256 IS NOT NULL AND status <> 'superseded')
|
||||
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||
SELECT s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
|
||||
left(d.sha256, 16)::varchar(16) AS document_key, d.title::varchar(200) AS title,
|
||||
d.kind::varchar(20) AS kind, d.issuer::varchar(80) AS issuer, d.document_date,
|
||||
string_agg(c.record_type, ', ' ORDER BY c.record_type)::varchar(200) AS used_for,
|
||||
d.contains_personal_data
|
||||
FROM cites c
|
||||
JOIN fm.source_document d ON d.sha256 = c.source_sha256
|
||||
JOIN public.v_site s ON s.site_uid = c.site_uid
|
||||
GROUP BY s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||
JOIN public.v_site s ON s.site_uid = c.holder_uid
|
||||
GROUP BY s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
|
||||
d.sha256, d.title, d.kind, d.issuer, d.document_date, d.contains_personal_data;
|
||||
|
||||
-- F1, F4: one row per delivery event; quantity NULL = not recorded. No monthly sums.
|
||||
CREATE VIEW public.v_delivery AS
|
||||
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||
SELECT s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
|
||||
e.event_key::varchar(80) AS event_key, v.visit_on AS delivery_date, v.seq_in_day::integer AS seq_in_day,
|
||||
round(e.litres, 1)::numeric(8,1) AS delivered_l,
|
||||
b.unit_price::numeric(8,2) AS unit_price, b.price_currency::varchar(8) AS currency,
|
||||
@@ -284,19 +296,19 @@ SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||
e.n_claims::integer AS n_sources, (e.spread_l > 0) AS sources_disagree, e.all_reviewed AS reviewed
|
||||
FROM fm.v_movement_event e
|
||||
JOIN fm.v_visit_live v ON v.visit_id = e.visit_id
|
||||
JOIN public.v_site s ON s.site_uid = v.site_uid
|
||||
JOIN public.v_site s ON s.site_uid = v.holder_uid
|
||||
CROSS JOIN LATERAL ( -- price and reference of the best-ranked claim;
|
||||
-- a disputed claim gives the reference only, never a price
|
||||
SELECT CASE WHEN m.claim_status <> 'disputed' THEN m.unit_price END AS unit_price,
|
||||
CASE WHEN m.claim_status <> 'disputed' THEN m.price_currency END AS price_currency, m.record_ref
|
||||
FROM fm.fuel_movement m JOIN fm.v_source_rank r ON r.sha256 = m.source_sha256
|
||||
WHERE m.event_key = e.event_key AND m.claim_status IN ('asserted','reviewed','disputed')
|
||||
WHERE m.visit_id = e.visit_id AND m.event_key = e.event_key AND m.claim_status IN ('asserted','reviewed','disputed')
|
||||
ORDER BY (m.claim_status = 'disputed'), r.rank, m.movement_id LIMIT 1) b
|
||||
WHERE e.kind = 'delivery';
|
||||
|
||||
-- F2: stock readings as recorded, plus after-delivery points derived as before + delivered.
|
||||
CREATE VIEW public.v_stock_reading AS
|
||||
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||
SELECT s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
|
||||
v.visit_on AS observed_on, v.seq_in_day::integer AS seq_in_day,
|
||||
(CASE st.phase WHEN 'before_delivery' THEN 'before delivery' WHEN 'after_delivery' THEN 'after delivery'
|
||||
ELSE 'stock check' END)::varchar(20) AS reading_type,
|
||||
@@ -304,20 +316,20 @@ SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||
round(st.spread_l, 1)::numeric(8,1) AS spread_l, st.all_reviewed AS reviewed
|
||||
FROM fm.v_visit_stock st
|
||||
JOIN fm.v_visit_live v ON v.visit_id = st.visit_id
|
||||
JOIN public.v_site s ON s.site_uid = v.site_uid
|
||||
JOIN public.v_site s ON s.site_uid = v.holder_uid
|
||||
UNION ALL
|
||||
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||
SELECT s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
|
||||
p.visit_on, p.seq_in_day::integer, 'after delivery'::varchar(20),
|
||||
round(p.depart_l, 1)::numeric(8,1), 'calculated'::varchar(12),
|
||||
round(p.depart_spread_l, 1)::numeric(8,1), p.depart_reviewed
|
||||
FROM fm.v_stock_point p
|
||||
JOIN public.v_site s ON s.site_uid = p.site_uid
|
||||
JOIN public.v_site s ON s.site_uid = p.holder_uid
|
||||
WHERE p.depart_basis = 'derived';
|
||||
|
||||
-- F3, F4, B3, B4: one row per stock interval, with its L/h evaluation. The approved baseline is set
|
||||
-- beside an interval only when both have the same metric and population (B4 compares like with like).
|
||||
CREATE VIEW public.v_interval AS
|
||||
WITH cur AS (SELECT bs.site_uid, bs.baseline_code, bs.baseline_status, bs.lph_estimate, b.metric, b.population_key
|
||||
WITH cur AS (SELECT bs.holder_uid, bs.baseline_code, bs.baseline_status, bs.lph_estimate, b.metric, b.population_key
|
||||
FROM fm.v_site_baseline_status bs LEFT JOIN fm.baseline_version b ON b.code = bs.baseline_code),
|
||||
inb AS ( -- the inputs of each site's KPI version
|
||||
SELECT bi.open_visit_id, bi.close_visit_id, bi.included
|
||||
@@ -327,12 +339,12 @@ inb AS ( -- the inputs of each site's KPI vers
|
||||
cond AS ( -- assumptions a valid interval depends on
|
||||
SELECT l.interval_key, string_agg(DISTINCT x.code, ', ') AS codes
|
||||
FROM fm.v_lph_interval l
|
||||
JOIN fm.assumption x ON x.site_uid = l.site_uid AND x.status IN ('proposed','accepted')
|
||||
JOIN fm.assumption x ON x.holder_uid = l.holder_uid AND x.status IN ('proposed','accepted')
|
||||
AND x.topic IN ('generator_idle','meter_coverage')
|
||||
AND daterange(x.applies_from, x.applies_to, '[]') @> daterange(l.start_on, l.end_on, '[]')
|
||||
WHERE l.population_valid
|
||||
GROUP BY l.interval_key)
|
||||
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||
SELECT s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
|
||||
(to_char(l.start_on, 'YYYY-MM-DD') || ' to ' || to_char(l.end_on, 'YYYY-MM-DD'))::varchar(30) AS interval_label,
|
||||
l.start_on, l.end_on, l.days::integer AS days,
|
||||
round(l.opening_l, 1)::numeric(9,1) AS opening_l, l.opening_basis::varchar(8) AS opening_basis,
|
||||
@@ -357,15 +369,15 @@ SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||
AND l.population_key = cur.population_key
|
||||
THEN cur.lph_estimate END)::numeric(8,3) AS current_baseline_l_per_h
|
||||
FROM fm.v_lph_interval l
|
||||
JOIN public.v_site s ON s.site_uid = l.site_uid
|
||||
JOIN cur ON cur.site_uid = l.site_uid
|
||||
JOIN public.v_site s ON s.site_uid = l.holder_uid
|
||||
JOIN cur ON cur.holder_uid = l.holder_uid
|
||||
LEFT JOIN inb ON inb.open_visit_id = l.open_visit_id AND inb.close_visit_id = l.close_visit_id
|
||||
LEFT JOIN cond ON cond.interval_key = l.interval_key;
|
||||
|
||||
-- R1: hours per day by meter interval (no rows in v1). Every pair is listed with its flags; runtime
|
||||
-- and h/day only for a valid pair (one epoch, not decreasing, at most 24 h per day).
|
||||
CREATE VIEW public.v_runtime AS
|
||||
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||
SELECT s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
|
||||
m.source_label::varchar(40) AS meter_label,
|
||||
(CASE WHEN m.asset_uid IS NULL THEN 'unmapped'
|
||||
ELSE m.asset_uid || ' (' || m.mapping_status || ')' END)::varchar(60) AS mapped_set,
|
||||
@@ -378,55 +390,56 @@ SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||
FROM fm.v_runtime_pair r
|
||||
CROSS JOIN LATERAL (SELECT coalesce(r.same_epoch AND r.monotonic AND NOT r.over_24h_per_day, false) AS pair_valid) v
|
||||
JOIN fm.runtime_meter m ON m.meter_id = r.meter_id
|
||||
JOIN public.v_site s ON s.site_uid = r.site_uid;
|
||||
JOIN public.v_site s ON s.site_uid = r.holder_uid;
|
||||
|
||||
-- S3 (Gantt): which periods each kind of evidence covers; Kabul midnight as timestamp(0).
|
||||
-- S3 (Gantt): which periods each kind of evidence covers; local midnight as timestamp(0).
|
||||
CREATE VIEW public.v_coverage AS
|
||||
WITH spans AS (
|
||||
SELECT site_uid, '1 Stock intervals' AS lane, stock_status AS status, start_on, end_on
|
||||
SELECT holder_uid, '1 Stock intervals' AS lane, stock_status AS status, start_on, end_on
|
||||
FROM fm.v_stock_interval
|
||||
UNION ALL
|
||||
SELECT r.site_uid, '2 Runtime: ' || m.source_label,
|
||||
SELECT r.holder_uid, '2 Runtime: ' || m.source_label,
|
||||
CASE WHEN r.same_epoch AND r.monotonic AND NOT r.over_24h_per_day THEN 'ok' ELSE 'flagged' END,
|
||||
r.start_on, r.end_on
|
||||
FROM fm.v_runtime_pair r JOIN fm.runtime_meter m ON m.meter_id = r.meter_id
|
||||
UNION ALL
|
||||
SELECT site_uid, '3 Baseline window', baseline_status, lph_window_start, lph_window_end
|
||||
SELECT holder_uid, '3 Baseline window', baseline_status, lph_window_start, lph_window_end
|
||||
FROM fm.v_site_baseline_status WHERE lph_window_start IS NOT NULL
|
||||
UNION ALL
|
||||
SELECT site_uid, '4 Electricity bills',
|
||||
SELECT holder_uid, '4 Electricity bills',
|
||||
CASE WHEN claim_status = 'disputed' THEN 'disputed' WHEN kwh IS NULL THEN 'cost only' ELSE 'kWh' END,
|
||||
coalesce(period_start, bill_month), coalesce(period_end, (bill_month + interval '1 month')::date)
|
||||
FROM fm.electricity_bill WHERE claim_status IN ('asserted','reviewed','disputed'))
|
||||
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||
SELECT s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
|
||||
x.lane::varchar(60) AS lane, x.status::varchar(40) AS status, x.start_on, x.end_on,
|
||||
x.start_on::timestamp(0) AS start_ts, x.end_on::timestamp(0) AS end_ts
|
||||
FROM spans x JOIN public.v_site s ON s.site_uid = x.site_uid;
|
||||
FROM spans x JOIN public.v_site s ON s.site_uid = x.holder_uid;
|
||||
|
||||
-- B9: stock-balance fuel against engine-measured fuel inside the interval. No rows until flow-meter
|
||||
-- data exist. The window runs from the opening visit's departure to the closing visit's arrival
|
||||
-- (Kabul time) when both times are recorded, else from midnight to midnight. covered_h = the hours
|
||||
-- (the holder's local time) when both times are recorded, else from midnight to midnight. covered_h = the hours
|
||||
-- of that window the generator's records cover (live records never overlap per asset). The
|
||||
-- difference is shown only for an `ok` stock interval with both times recorded, in which every
|
||||
-- generator is placed at the site throughout (one placement spans the interval, as in
|
||||
-- fm.v_lph_interval) and covered for the whole window.
|
||||
CREATE VIEW public.v_reconciliation AS
|
||||
WITH gen AS ( -- one row per interval and generator placed at the site at any time in it
|
||||
SELECT l.interval_key, l.site_uid, l.start_on, l.end_on, l.consumed_l, l.stock_status, p.asset_uid,
|
||||
SELECT l.interval_key, l.holder_uid, l.start_on, l.end_on, l.consumed_l, l.stock_status, p.asset_uid,
|
||||
bool_or(daterange(coalesce(p.occupied_from, '-infinity'::date), p.occupied_to, '[)')
|
||||
@> daterange(l.start_on, l.end_on, '[]')) AS placed_throughout,
|
||||
(o.time_out_local IS NOT NULL AND c.time_in_local IS NOT NULL) AS times_known,
|
||||
(l.start_on + coalesce(o.time_out_local, '00:00'::time)) AT TIME ZONE 'Asia/Kabul' AS t0,
|
||||
(l.end_on + coalesce(c.time_in_local, '00:00'::time)) AT TIME ZONE 'Asia/Kabul' AS t1
|
||||
(l.start_on + coalesce(o.time_out_local, '00:00'::time)) AT TIME ZONE z.timezone AS t0,
|
||||
(l.end_on + coalesce(c.time_in_local, '00:00'::time)) AT TIME ZONE z.timezone AS t1
|
||||
FROM fm.v_stock_interval l
|
||||
JOIN fm.visit o ON o.visit_id = l.open_visit_id
|
||||
JOIN fm.visit c ON c.visit_id = l.close_visit_id
|
||||
JOIN fm.asset_placement p ON p.site_uid = l.site_uid
|
||||
JOIN fm.v_holder_current z ON z.holder_uid = l.holder_uid
|
||||
JOIN fm.asset_placement p ON p.holder_uid = l.holder_uid
|
||||
JOIN fm.asset a ON a.asset_uid = p.asset_uid AND a.kind = 'generator'
|
||||
WHERE daterange(coalesce(p.occupied_from, '-infinity'::date), p.occupied_to, '[)')
|
||||
&& daterange(l.start_on, l.end_on, '[]')
|
||||
GROUP BY l.interval_key, l.site_uid, l.start_on, l.end_on, l.consumed_l, l.stock_status, p.asset_uid,
|
||||
o.time_out_local, c.time_in_local),
|
||||
GROUP BY l.interval_key, l.holder_uid, l.start_on, l.end_on, l.consumed_l, l.stock_status, p.asset_uid,
|
||||
o.time_out_local, c.time_in_local, z.timezone),
|
||||
eng AS (
|
||||
SELECT g.interval_key, g.asset_uid, sum(t.net_fuel_l) AS engine_l,
|
||||
sum(extract(epoch FROM t.ended_at - t.started_at)) / 3600 AS covered_h,
|
||||
@@ -440,7 +453,7 @@ tot AS (
|
||||
count(*) FILTER (WHERE g.placed_throughout AND e.full_cover) AS n_complete, sum(e.engine_l) AS engine_all_l
|
||||
FROM gen g LEFT JOIN eng e ON e.interval_key = g.interval_key AND e.asset_uid = g.asset_uid
|
||||
GROUP BY g.interval_key)
|
||||
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||
SELECT s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
|
||||
(to_char(g.start_on, 'YYYY-MM-DD') || ' to ' || to_char(g.end_on, 'YYYY-MM-DD'))::varchar(30) AS interval_label,
|
||||
g.start_on, g.end_on, g.stock_status::varchar(40) AS stock_status, g.times_known AS endpoint_times_known,
|
||||
g.asset_uid::varchar(40) AS asset_uid, g.placed_throughout,
|
||||
@@ -456,60 +469,60 @@ SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||
FROM gen g
|
||||
JOIN eng e ON e.interval_key = g.interval_key AND e.asset_uid = g.asset_uid
|
||||
JOIN tot t ON t.interval_key = g.interval_key
|
||||
JOIN public.v_site s ON s.site_uid = g.site_uid;
|
||||
JOIN public.v_site s ON s.site_uid = g.holder_uid;
|
||||
|
||||
-- B6: one row per site and check; rows stay when ok.
|
||||
CREATE VIEW public.v_data_quality AS
|
||||
WITH c AS (
|
||||
SELECT s.site_uid,
|
||||
(SELECT count(*) FROM fm.stock_observation o WHERE o.site_uid = s.site_uid
|
||||
SELECT s.holder_uid,
|
||||
(SELECT count(*) FROM fm.stock_observation o WHERE o.holder_uid = s.holder_uid
|
||||
AND o.litres_basis = 'placeholder_zero' AND o.claim_status IN ('asserted','reviewed','disputed')) AS n_placeholder,
|
||||
(SELECT count(*) FROM fm.v_movement_event e JOIN fm.visit v ON v.visit_id = e.visit_id
|
||||
WHERE v.site_uid = s.site_uid AND e.spread_l > 0)
|
||||
WHERE v.holder_uid = s.holder_uid AND e.spread_l > 0)
|
||||
+ (SELECT count(*) FROM fm.v_visit_stock st JOIN fm.visit v ON v.visit_id = st.visit_id
|
||||
WHERE v.site_uid = s.site_uid AND st.spread_l > 0) AS n_conflicts,
|
||||
WHERE v.holder_uid = s.holder_uid AND st.spread_l > 0) AS n_conflicts,
|
||||
(SELECT count(*) FROM fm.hour_meter_reading h JOIN fm.runtime_meter m ON m.meter_id = h.meter_id
|
||||
WHERE m.site_uid = s.site_uid AND h.claim_status IN ('asserted','reviewed','disputed')) AS n_readings,
|
||||
(SELECT count(*) FROM fm.runtime_meter m WHERE m.site_uid = s.site_uid AND m.mapping_status <> 'confirmed') AS n_unmapped,
|
||||
WHERE m.holder_uid = s.holder_uid AND h.claim_status IN ('asserted','reviewed','disputed')) AS n_readings,
|
||||
(SELECT count(*) FROM fm.runtime_meter m WHERE m.holder_uid = s.holder_uid AND m.mapping_status <> 'confirmed') AS n_unmapped,
|
||||
(SELECT count(*) FROM fm.asset_placement p JOIN fm.asset a ON a.asset_uid = p.asset_uid
|
||||
WHERE p.site_uid = s.site_uid AND a.kind = 'generator'
|
||||
WHERE p.holder_uid = s.holder_uid AND a.kind = 'generator'
|
||||
AND daterange(coalesce(p.occupied_from, '-infinity'::date), p.occupied_to, '[)')
|
||||
@> (now() AT TIME ZONE 'Asia/Kabul')::date
|
||||
@> (SELECT z.today FROM fm.v_holder_current z WHERE z.holder_uid = s.holder_uid)
|
||||
AND NOT EXISTS (SELECT 1 FROM fm.runtime_meter m JOIN fm.hour_meter_reading h ON h.meter_id = m.meter_id
|
||||
WHERE m.asset_uid = p.asset_uid AND m.mapping_status = 'confirmed'
|
||||
AND h.claim_status IN ('asserted','reviewed'))) AS n_gen_no_readings,
|
||||
(SELECT count(*) FROM fm.fuel_movement x WHERE x.site_uid = s.site_uid AND x.claim_status = 'asserted')
|
||||
+ (SELECT count(*) FROM fm.stock_observation x WHERE x.site_uid = s.site_uid AND x.claim_status = 'asserted')
|
||||
(SELECT count(*) FROM fm.fuel_movement x WHERE x.holder_uid = s.holder_uid AND x.claim_status = 'asserted')
|
||||
+ (SELECT count(*) FROM fm.stock_observation x WHERE x.holder_uid = s.holder_uid AND x.claim_status = 'asserted')
|
||||
+ (SELECT count(*) FROM fm.hour_meter_reading x JOIN fm.runtime_meter m ON m.meter_id = x.meter_id
|
||||
WHERE m.site_uid = s.site_uid AND x.claim_status = 'asserted')
|
||||
+ (SELECT count(*) FROM fm.electricity_bill x WHERE x.site_uid = s.site_uid AND x.claim_status = 'asserted') AS n_unreviewed,
|
||||
(SELECT count(*) FROM fm.v_stock_point p WHERE p.site_uid = s.site_uid
|
||||
WHERE m.holder_uid = s.holder_uid AND x.claim_status = 'asserted')
|
||||
+ (SELECT count(*) FROM fm.electricity_bill x WHERE x.holder_uid = s.holder_uid AND x.claim_status = 'asserted') AS n_unreviewed,
|
||||
(SELECT count(*) FROM fm.v_stock_point p WHERE p.holder_uid = s.holder_uid
|
||||
AND abs(p.after_vs_before_plus_delivered_l) > 0.5) AS n_after_mismatch,
|
||||
(SELECT count(*) FROM fm.v_stock_interval i WHERE i.site_uid = s.site_uid
|
||||
(SELECT count(*) FROM fm.v_stock_interval i WHERE i.holder_uid = s.holder_uid
|
||||
AND i.stock_status IN ('negative','same_day')) AS n_neg_same_day,
|
||||
(SELECT count(*) FROM fm.v_stock_interval i WHERE i.site_uid = s.site_uid
|
||||
(SELECT count(*) FROM fm.v_stock_interval i WHERE i.holder_uid = s.holder_uid
|
||||
AND i.stock_status NOT IN ('ok','negative','same_day')) AS n_gaps,
|
||||
(SELECT count(*) FROM fm.electricity_bill b WHERE b.site_uid = s.site_uid
|
||||
(SELECT count(*) FROM fm.electricity_bill b WHERE b.holder_uid = s.holder_uid
|
||||
AND b.claim_status IN ('asserted','reviewed','disputed')) AS n_bills,
|
||||
(SELECT count(*) FROM fm.electricity_bill b WHERE b.site_uid = s.site_uid
|
||||
(SELECT count(*) FROM fm.electricity_bill b WHERE b.holder_uid = s.holder_uid
|
||||
AND (b.kwh IS NULL OR b.claim_status = 'disputed')
|
||||
AND b.claim_status IN ('asserted','reviewed','disputed')) AS n_bills_no_kwh, -- no usable kWh
|
||||
(SELECT count(*) FROM fm.fuel_movement m WHERE m.site_uid = s.site_uid
|
||||
(SELECT count(*) FROM fm.fuel_movement m WHERE m.holder_uid = s.holder_uid
|
||||
AND m.kind IN ('withdrawal','drain','spill','loss_reported','internal_transfer')
|
||||
AND m.claim_status IN ('asserted','reviewed','disputed')) AS n_transfer_records,
|
||||
(SELECT count(*) FROM fm.baseline_version b JOIN fm.v_baseline_validity v ON v.baseline_id = b.baseline_id
|
||||
WHERE b.site_uid = s.site_uid AND b.status IN ('proposed','approved') AND NOT v.inputs_unchanged) AS n_drift,
|
||||
(SELECT count(*) FROM fm.v_review_stale r WHERE r.site_uid = s.site_uid) AS n_stale,
|
||||
(SELECT count(*) FROM fm.v_visit_provenance_only v WHERE v.site_uid = s.site_uid) AS n_provenance_only,
|
||||
(SELECT count(*) FROM fm.v_lph_interval l WHERE l.site_uid = s.site_uid
|
||||
WHERE b.holder_uid = s.holder_uid AND b.status IN ('proposed','approved') AND NOT v.inputs_unchanged) AS n_drift,
|
||||
(SELECT count(*) FROM fm.v_review_stale r WHERE r.holder_uid = s.holder_uid) AS n_stale,
|
||||
(SELECT count(*) FROM fm.v_visit_provenance_only v WHERE v.holder_uid = s.holder_uid) AS n_provenance_only,
|
||||
(SELECT count(*) FROM fm.v_lph_interval l WHERE l.holder_uid = s.holder_uid
|
||||
AND l.exclusion_reasons LIKE '%confirmed for a generator not placed here%') AS n_unplaced_counter,
|
||||
(SELECT count(*) FROM fm.v_lph_interval l WHERE l.site_uid = s.site_uid
|
||||
(SELECT count(*) FROM fm.v_lph_interval l WHERE l.holder_uid = s.holder_uid
|
||||
AND l.exclusion_reasons LIKE '%meter-coverage assumption % does not name%') AS n_cov_mismatch
|
||||
FROM fm.site s)
|
||||
SELECT vs.site_uid, vs.site_code, vs.site_name, vs.site_label,
|
||||
FROM fm.holder s)
|
||||
SELECT vs.site_uid, vs.site_code, vs.site_name, vs.site_label, vs.operator_uid,
|
||||
x.ord::integer AS check_order, x.code::varchar(40) AS check_code, x.label::varchar(80) AS label,
|
||||
x.status::varchar(10) AS status, x.detail::varchar(300) AS detail
|
||||
FROM c JOIN public.v_site vs ON vs.site_uid = c.site_uid
|
||||
FROM c JOIN public.v_site vs ON vs.site_uid = c.holder_uid
|
||||
CROSS JOIN LATERAL (VALUES
|
||||
(1, 'placeholder_rows', 'Placeholder rows',
|
||||
CASE WHEN c.n_placeholder > 0 THEN 'warning' ELSE 'ok' END,
|
||||
@@ -564,8 +577,8 @@ CROSS JOIN LATERAL (VALUES
|
||||
-- B10: specifications, labelled as such; live rows only. A disputed specification is listed with its
|
||||
-- status and without its value.
|
||||
CREATE VIEW public.v_spec_reference AS
|
||||
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||
p.slot_label::varchar(20) AS slot, e.asset_uid::varchar(40) AS asset_uid, a.kind::varchar(10) AS asset_kind,
|
||||
SELECT s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
|
||||
p.slot_label::varchar(20) AS slot, e.asset_uid::varchar(40) AS asset_uid, a.kind::varchar(20) AS asset_kind,
|
||||
concat_ws(' ', a.make, a.model)::varchar(120) AS make_model,
|
||||
e.attribute::varchar(40) AS attribute, e.load_pct::numeric(5,1) AS load_pct,
|
||||
(CASE WHEN e.claim_status <> 'disputed' THEN e.value_num END)::numeric(12,3) AS value,
|
||||
@@ -575,14 +588,14 @@ SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||
'specification, not measured'::varchar(30) AS value_kind, e.claim_status::varchar(12) AS claim_status
|
||||
FROM fm.equipment_spec e
|
||||
JOIN fm.asset a ON a.asset_uid = e.asset_uid
|
||||
JOIN (SELECT DISTINCT ON (asset_uid, site_uid) asset_uid, site_uid, slot_label
|
||||
FROM fm.asset_placement ORDER BY asset_uid, site_uid, occupied_from DESC NULLS LAST) p ON p.asset_uid = e.asset_uid
|
||||
JOIN public.v_site s ON s.site_uid = p.site_uid
|
||||
JOIN (SELECT DISTINCT ON (asset_uid, holder_uid) asset_uid, holder_uid, slot_label
|
||||
FROM fm.asset_placement ORDER BY asset_uid, holder_uid, occupied_from DESC NULLS LAST) p ON p.asset_uid = e.asset_uid
|
||||
JOIN public.v_site s ON s.site_uid = p.holder_uid
|
||||
WHERE e.claim_status IN ('asserted','reviewed','disputed');
|
||||
|
||||
-- M3 sensitivity table, reference tab only (shown only once Q9 approves a runtime assumption).
|
||||
CREATE VIEW public.v_modelled_lph AS
|
||||
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||
SELECT s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
|
||||
m.code::varchar(60) AS assumption_code, m.status::varchar(12) AS status, m.basis::varchar(20) AS basis,
|
||||
m.h_per_day::numeric(5,2) AS h_per_day, m.h_per_day_low::numeric(5,2) AS h_per_day_low,
|
||||
m.h_per_day_high::numeric(5,2) AS h_per_day_high, m.window_start, m.window_end,
|
||||
@@ -591,31 +604,36 @@ SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||
round(m.modelled_l_per_h_low, 2)::numeric(6,2) AS modelled_l_per_h_low,
|
||||
round(m.modelled_l_per_h_high, 2)::numeric(6,2) AS modelled_l_per_h_high,
|
||||
'modelled: L per calendar day / assumed runtime; not measured'::varchar(70) AS value_kind
|
||||
FROM fm.v_modelled_lph m JOIN public.v_site s ON s.site_uid = m.site_uid;
|
||||
FROM fm.v_modelled_lph m JOIN public.v_site s ON s.site_uid = m.holder_uid;
|
||||
|
||||
-- R4: electricity cost per bill month; cost context only, never joined into fuel metrics. A
|
||||
-- disputed bill stays listed, without amount or kWh.
|
||||
CREATE VIEW public.v_electricity_cost AS
|
||||
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||
SELECT s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
|
||||
b.bill_month, b.period_start, b.period_end, (b.period_start IS NOT NULL) AS period_known,
|
||||
(CASE WHEN b.claim_status <> 'disputed' THEN b.amount END)::numeric(12,2) AS amount, b.currency::varchar(8) AS currency,
|
||||
b.currency_basis::varchar(10) AS currency_basis,
|
||||
(CASE WHEN b.claim_status <> 'disputed' THEN b.kwh END)::numeric(10,1) AS kwh,
|
||||
b.record_ref::varchar(60) AS source_ref, b.claim_status::varchar(12) AS claim_status
|
||||
FROM fm.electricity_bill b JOIN public.v_site s ON s.site_uid = b.site_uid
|
||||
FROM fm.electricity_bill b JOIN public.v_site s ON s.site_uid = b.holder_uid
|
||||
WHERE b.claim_status IN ('asserted','reviewed','disputed');
|
||||
|
||||
-- B8: operator workbook calculations and our assessment; live rows only. A disputed transcription is
|
||||
-- listed with its status and without its formula or value.
|
||||
CREATE VIEW public.v_imported_calculation AS
|
||||
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||
SELECT s.site_uid, s.site_code, s.site_name, s.site_label, s.operator_uid,
|
||||
c.cell_ref::varchar(40) AS cell_ref, c.source_label::varchar(80) AS source_label,
|
||||
(CASE WHEN c.claim_status <> 'disputed' THEN c.formula END)::varchar(200) AS formula,
|
||||
(CASE WHEN c.claim_status <> 'disputed' THEN c.cached_value END)::numeric(14,6) AS cached_value,
|
||||
c.unit_as_labelled::varchar(30) AS unit_as_labelled, c.our_reading::varchar(300) AS our_reading,
|
||||
c.baseline_use::varchar(12) AS baseline_use, c.exclusion_reason::varchar(300) AS exclusion_reason,
|
||||
left(c.source_sha256, 16)::varchar(16) AS document_key, c.claim_status::varchar(12) AS claim_status
|
||||
FROM fm.imported_calculation c JOIN public.v_site s ON s.site_uid = c.site_uid
|
||||
FROM fm.imported_calculation c JOIN public.v_site s ON s.site_uid = c.holder_uid
|
||||
WHERE c.claim_status IN ('asserted','reviewed','disputed');
|
||||
|
||||
-- Which Keycloak group is which customer (PD-21). Only server/ reads this, at sign-in; it holds no
|
||||
-- site data, and no dashboard has a dataset on it.
|
||||
CREATE VIEW public.v_operator AS
|
||||
SELECT operator_uid, name, keycloak_group FROM fm.operator;
|
||||
|
||||
COMMIT;
|
||||
@@ -0,0 +1,91 @@
|
||||
-- 0004_licence.sql: customer licences (PD-20) and plan prices (PD-19), in schema `licence`.
|
||||
-- DRAFT, not applied. Applied after 0003 by a loader acting as fuel_owner (README in this folder).
|
||||
-- fuel_ro has no access: its default privileges cover `public` only. The server gets its own role
|
||||
-- when it starts checking licences. No customer's licence is seeded here; our admins write those.
|
||||
BEGIN;
|
||||
CREATE SCHEMA licence AUTHORIZATION fuel_owner;
|
||||
SET LOCAL search_path = licence, fm, pg_catalog, public; -- fm holds btree_gist
|
||||
|
||||
CREATE TABLE licence.plan (
|
||||
plan_code text PRIMARY KEY,
|
||||
name text NOT NULL
|
||||
);
|
||||
CREATE TABLE licence.plan_price ( -- per sensor per month (PD-19); a change is a new dated row
|
||||
plan_code text NOT NULL REFERENCES licence.plan,
|
||||
payment_term text NOT NULL CHECK (payment_term IN ('annual','monthly')),
|
||||
currency text NOT NULL CHECK (currency ~ '^[A-Z]{3}$'),
|
||||
price numeric(8,2) NOT NULL CHECK (price >= 0),
|
||||
valid_from date NOT NULL, valid_to date, -- NULL to = still valid
|
||||
EXCLUDE USING gist (plan_code WITH =, payment_term WITH =, currency WITH =,
|
||||
daterange(valid_from, valid_to, '[)') WITH &&)
|
||||
);
|
||||
CREATE TABLE licence.plan_entitlement (
|
||||
plan_code text NOT NULL REFERENCES licence.plan,
|
||||
feature text NOT NULL,
|
||||
limit_value integer, -- NULL: no limit
|
||||
PRIMARY KEY (plan_code, feature)
|
||||
);
|
||||
CREATE TABLE licence.licence ( -- a renewal is a new row
|
||||
licence_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
operator_uid text NOT NULL REFERENCES fm.operator,
|
||||
plan_code text NOT NULL REFERENCES licence.plan,
|
||||
payment_term text NOT NULL CHECK (payment_term IN ('annual','monthly')),
|
||||
valid_from date NOT NULL,
|
||||
valid_until date NOT NULL, -- last valid day
|
||||
grace_until date NOT NULL, -- last day of grace; alarms stop after it
|
||||
status text NOT NULL CHECK (status IN ('active','suspended')),
|
||||
contract_ref text NOT NULL, -- the signed agreement or amendment the dates come from
|
||||
created_by text NOT NULL, -- a role, never a person's name
|
||||
created_at timestamptz NOT NULL DEFAULT now(),
|
||||
CHECK (valid_until >= valid_from),
|
||||
CHECK (grace_until >= valid_until),
|
||||
EXCLUDE USING gist (operator_uid WITH =, daterange(valid_from, valid_until, '[]') WITH &&)
|
||||
);
|
||||
CREATE TABLE licence.licence_override (
|
||||
licence_id bigint NOT NULL REFERENCES licence.licence,
|
||||
feature text NOT NULL,
|
||||
limit_value integer, -- NULL: no limit
|
||||
PRIMARY KEY (licence_id, feature)
|
||||
);
|
||||
|
||||
-- Prices decided on 30 Oct 2026 (PD-19). The pilot plan costs nothing on either term.
|
||||
INSERT INTO licence.plan (plan_code, name) VALUES ('pilot', 'Pilot'), ('standard', 'Standard');
|
||||
INSERT INTO licence.plan_price (plan_code, payment_term, currency, price, valid_from) VALUES
|
||||
('pilot', 'annual', 'USD', 0.00, '2026-01-01'),
|
||||
('pilot', 'monthly', 'USD', 0.00, '2026-01-01'),
|
||||
('standard', 'annual', 'USD', 15.00, '2026-10-30'),
|
||||
('standard', 'monthly', 'USD', 17.00, '2026-10-30');
|
||||
|
||||
-- Every change is logged (PD-20), and the log is never changed.
|
||||
CREATE TABLE licence.licence_event (
|
||||
event_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
table_name text NOT NULL,
|
||||
action text NOT NULL CHECK (action IN ('INSERT','UPDATE','DELETE')),
|
||||
old_row jsonb, new_row jsonb,
|
||||
changed_by text NOT NULL DEFAULT current_user,
|
||||
changed_at timestamptz NOT NULL DEFAULT now()
|
||||
);
|
||||
CREATE FUNCTION licence.tg_log_change() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||
BEGIN
|
||||
INSERT INTO licence.licence_event (table_name, action, old_row, new_row)
|
||||
VALUES (TG_TABLE_NAME, TG_OP,
|
||||
CASE WHEN TG_OP <> 'INSERT' THEN to_jsonb(OLD) END,
|
||||
CASE WHEN TG_OP <> 'DELETE' THEN to_jsonb(NEW) END);
|
||||
RETURN NULL;
|
||||
END $$;
|
||||
DO $$
|
||||
DECLARE t text;
|
||||
BEGIN
|
||||
FOREACH t IN ARRAY ARRAY['plan','plan_price','plan_entitlement','licence','licence_override'] LOOP
|
||||
EXECUTE format('CREATE TRIGGER log_change AFTER INSERT OR UPDATE OR DELETE ON licence.%I
|
||||
FOR EACH ROW EXECUTE FUNCTION licence.tg_log_change()', t);
|
||||
END LOOP;
|
||||
END $$;
|
||||
CREATE FUNCTION licence.tg_event_immutable() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||
BEGIN
|
||||
RAISE EXCEPTION 'licence_event: the log is never changed';
|
||||
END $$;
|
||||
CREATE TRIGGER event_immutable BEFORE UPDATE OR DELETE ON licence.licence_event
|
||||
FOR EACH ROW EXECUTE FUNCTION licence.tg_event_immutable();
|
||||
|
||||
COMMIT;
|
||||
@@ -12,30 +12,48 @@ The design and its rationale live in the fuel-monitoring corpus:
|
||||
|
||||
| File | Creates | Readable by `fuel_ro` (Superset, Petrolyst server) |
|
||||
|---|---|---|
|
||||
| `0001_fm_core.sql` | schema `fm`, `btree_gist` in it, 25 tables, constraints, trigger-backed rules 1–13 | no |
|
||||
| `0001_fm_core.sql` | schema `fm`, `btree_gist` in it, 29 tables (customers, assets, places and holders, claims, reviews, baselines), constraints, trigger-backed rules 1–15 | no |
|
||||
| `0002_fm_derivations.sql` | internal views in `fm`: claims → visits → stock intervals → L/h and L/day intervals, dependency hashes, baseline validity | no |
|
||||
| `0003_bi_views.sql` | `fm.v_site_baseline_status` and the 18 `public.v_*` views Superset reads | yes, read-only |
|
||||
| `0003_bi_views.sql` | `fm.v_site_baseline_status` and the 19 `public.v_*` views Superset reads | yes, read-only |
|
||||
| `0004_licence.sql` | schema `licence`: plans, prices, entitlements, customer licences, overrides and their change log (PD-19, PD-20). Holds no customer's licence | no |
|
||||
|
||||
Each file is one transaction. Apply them in order.
|
||||
|
||||
**The shape (PD-16, PD-18).** Every row in `fm` carries its customer, `operator_uid`, except
|
||||
`asset_kind`. A child names its parent and the customer together in a composite foreign key, so it
|
||||
cannot point at another customer's parent. A site is a `place` with one fixed `holder`; holders keep
|
||||
their `SITE-0001` ids and places get `PLC-0001`. Internal `fm` speaks of holder and place. The public
|
||||
views keep their `site_*` column names, because dashboards, the app and the server read them; each
|
||||
gains `operator_uid`. They get renamed, or holder-named siblings, with the first fleet customer.
|
||||
|
||||
**Runtime is generator-only.** An engine can be a placed asset, but rules 8, 9 and 12 refuse a
|
||||
meter or idle assumption on one: the runtime and L/h calculations have not been checked on engines.
|
||||
|
||||
**Who depends on the views.** Superset's dashboards list every column (`petrolyst-insights/dashboards/atoma/columns.tsv`;
|
||||
the import reports drift). The Petrolyst server reads `public.v_site`; its integration test
|
||||
(petrolyst-platform `server/tests/test_integration.py`) names the columns and types it needs. After
|
||||
a change to `0003_bi_views.sql`, run both checks. The migrations stay in this repository (open
|
||||
point 8, closed 30 Oct 2026).
|
||||
|
||||
## Applying (on the server)
|
||||
|
||||
1. Create a loader role as in the repository README, "Data loaders". Its sessions act as
|
||||
`fuel_owner`, so every object belongs to `fuel_owner`. `fuel_ro` can then read the `public`
|
||||
views through the existing default privileges, with no role or grant changes.
|
||||
2. Open the tunnel and apply the three files as the loader:
|
||||
2. Open the tunnel and apply the four files as the loader:
|
||||
|
||||
```sh
|
||||
ssh -L 5433:127.0.0.1:5433 petrolyst
|
||||
for f in 0001_fm_core 0002_fm_derivations 0003_bi_views; do
|
||||
for f in 0001_fm_core 0002_fm_derivations 0003_bi_views 0004_licence; do
|
||||
psql -h localhost -p 5433 -U <loader> -d fuel -v ON_ERROR_STOP=1 -f "$f.sql" || break
|
||||
done
|
||||
```
|
||||
|
||||
3. Check as `fuel_ro`
|
||||
(`docker exec -it petrolyst-fuel-db-1 psql -U postgres -d fuel`, then `SET ROLE fuel_ro;`):
|
||||
- `\dv public.*` lists the 18 `v_*` views;
|
||||
- `SELECT count(*) FROM fm.site;` fails with "permission denied for schema fm".
|
||||
- `\dv public.*` lists the 19 `v_*` views;
|
||||
- `SELECT count(*) FROM fm.holder;` fails with "permission denied for schema fm";
|
||||
- `SELECT count(*) FROM licence.plan;` fails with "permission denied for schema licence".
|
||||
|
||||
The `public` views are owned by `fuel_owner` and run with its rights, so `fuel_ro` never needs,
|
||||
and never gets, read access to `fm`. PostgreSQL's catalog still shows every schema's table and
|
||||
@@ -60,14 +78,15 @@ column names to any role, so Superset's SQL Lab can list fm's names; names only,
|
||||
1. Baseline versions start as draft or proposed. Approval needs `meets_policy` and unchanged
|
||||
inputs. Approved rows and their inputs are immutable; the only change is approved →
|
||||
superseded. An input never moves to another version.
|
||||
2. An internal transfer's tanks are placed at the movement's site on the visit date.
|
||||
3. A claim's site is its visit's site, and a reading's meter belongs to that site.
|
||||
2. An internal transfer's tanks are placed on the movement's holder on the visit date.
|
||||
3. A claim's holder is its visit's holder, and a reading's meter belongs to that holder.
|
||||
4. A reading's epoch belongs to its meter.
|
||||
5. A runtime assumption derived from fuel cannot be accepted. This is a CHECK constraint.
|
||||
6. A baseline input's dates equal its endpoint visits. Included inputs carry the version's
|
||||
population.
|
||||
7. A source document supersedes only one of its own family.
|
||||
8. A confirmed or proposed meter mapping names a generator placed at the meter's site.
|
||||
7. A source document supersedes only one of its own family. The same document under a second
|
||||
customer is refused until document sharing exists.
|
||||
8. A confirmed or proposed meter mapping names a generator placed on the meter's holder.
|
||||
9. Meter-coverage members:
|
||||
- valid when written, and written in the assumption's own transaction;
|
||||
- immutable afterwards;
|
||||
@@ -75,51 +94,34 @@ column names to any role, so Superset's SQL Lab can list fm's names; names only,
|
||||
10. The live claims of one `event_key` share visit, kind and tanks. A superseded or retracted
|
||||
claim keeps its own, so a correction can change them.
|
||||
11. Status comes only from a matching review. Values are never corrected in place.
|
||||
12. An idle assumption names a generator placed at its own site during its dates.
|
||||
13. Identity and classification never change in place: a visit's key, site, date and sequence; a
|
||||
source document's hash, family and kind (its rank); an asset's uid and kind; a meter's id and
|
||||
site; an epoch's id and meter. A wrong visit date is corrected by a new visit, which the
|
||||
superseding claims point to.
|
||||
12. An idle assumption names a generator placed on its own holder during its dates.
|
||||
13. Identity and classification never change in place: a visit's key, holder, date and sequence; a
|
||||
source document's hash, family and kind (its rank); an asset's uid and kind; a place's uid; a
|
||||
holder's uid, kind and asset; a meter's id and holder; an epoch's id and meter; any row's customer.
|
||||
A wrong visit date is corrected by a new visit, which the superseding claims point to. Naming a
|
||||
column a table does not have raises an error.
|
||||
14. A time zone is one PostgreSQL knows.
|
||||
15. A vehicle holder's asset is a vehicle; only a fixed installation stands at a place.
|
||||
|
||||
## Rollback
|
||||
|
||||
Before any data matters, run `DROP SCHEMA fm CASCADE` as the loader. This also drops every
|
||||
`public.v_*` view, and leaves fuel-db as it was before 0001.
|
||||
Before any data matters, run `DROP SCHEMA licence CASCADE; DROP SCHEMA fm CASCADE` as the loader. This
|
||||
also drops every `public.v_*` view, and leaves fuel-db as it was before 0001.
|
||||
|
||||
Once data has been loaded, this deletes it. The corpus remains the source, so a re-load restores
|
||||
it, except baseline versions and their approvals, which live only in fuel-db: export
|
||||
`fm.baseline_version` and `fm.baseline_input` first.
|
||||
|
||||
## Verified locally (2026-10-24, scratch only)
|
||||
## Verified locally (2026-10-30, scratch only)
|
||||
|
||||
With Trino, before its removal on 29 Oct 2026; `trino_ro` is now `fuel_ro`.
|
||||
Not run against the live fuel-db. An earlier set of 150 checks was a scratch script, since removed;
|
||||
these replace it:
|
||||
|
||||
Setup:
|
||||
- `timescale/timescaledb:2.31.0-pg17`, with this repository's `01-roles.sh`;
|
||||
- a loader created by the README recipe;
|
||||
- `trinodb/trino:484`, with this repository's `trino/etc` connected as `trino_ro`.
|
||||
|
||||
Results:
|
||||
- **Apply:** all three files applied as the loader, under the loader's default `search_path`.
|
||||
- **Fixture:** the operator workbook of 21 Oct 2026, the five site records, and synthetic sites
|
||||
for the runtime path.
|
||||
- **Tests:** all 150 checks of the design's §9 pass: 98 written with the migrations, and 52
|
||||
regressions for the implementation inspection's findings. These cover:
|
||||
- figures and statuses;
|
||||
- claims, reviews and revisions;
|
||||
- movement events;
|
||||
- coverage, placement and idle scope;
|
||||
- baseline dependencies and approval;
|
||||
- the L/day path;
|
||||
- current periods;
|
||||
- flow-meter reconciliation (coverage, placement, visit times), runtime validity, modelled-L/h
|
||||
windows, disputed claims supplying no values, the L/day reference, S2's L/h rows following
|
||||
the version's validity, corrections under one `event_key`, fixed meter and epoch owners, and
|
||||
population keys that name the generators a coverage assumption covers;
|
||||
- every refusal listed above.
|
||||
- **Trino:**
|
||||
- all 303 columns of the 18 views have identical names, order and types in PostgreSQL and
|
||||
Trino;
|
||||
- `fm` is unreadable: a query is refused, though its names are listed (see Applying);
|
||||
- writes are refused.
|
||||
- **Not run on the server:** none of this has been run against the live fuel-db.
|
||||
- **Rules:** `../tests/run.sh` applies these files to a throw-away database and runs `tests/rules.sql`
|
||||
and `tests/read_only.sql`, with synthetic data only: each rule with an accepted and a refused case,
|
||||
two customers, a vehicle as asset and holder, licences, and what `fuel_ro` may read.
|
||||
- **Loader:** fuel-monitoring's `make selftest` and `bin/fmdb.py dbtest`.
|
||||
- **Row comparison:** before this rewrite, all 18 views were dumped from a database loaded from the
|
||||
corpus and a recorded review manifest; after it, the same corpus gave identical rows, leaving out
|
||||
`operator_uid`. That one-time comparison covers only data the corpus has; rules with no corpus data
|
||||
rest on `rules.sql`.
|
||||
@@ -0,0 +1,36 @@
|
||||
-- read_only.sql: what fuel_ro may and may not do. Run by run.sh, logged in as fuel_ro.
|
||||
\set ON_ERROR_STOP 1
|
||||
DO $$
|
||||
DECLARE
|
||||
v text; n integer := 0; has boolean;
|
||||
BEGIN
|
||||
FOREACH v IN ARRAY ARRAY['v_site','v_kpi_measures','v_hour_meter_evidence','v_assumption','v_baseline_version',
|
||||
'v_baseline_input','v_evidence','v_delivery','v_stock_reading','v_interval','v_runtime',
|
||||
'v_coverage','v_reconciliation','v_data_quality','v_spec_reference','v_modelled_lph',
|
||||
'v_electricity_cost','v_imported_calculation','v_operator'] LOOP
|
||||
EXECUTE format('SELECT count(*) FROM public.%I', v);
|
||||
SELECT EXISTS (SELECT 1 FROM information_schema.columns c
|
||||
WHERE c.table_schema = 'public' AND c.table_name = v AND c.column_name = 'operator_uid') INTO has;
|
||||
IF NOT has THEN RAISE EXCEPTION '% has no operator_uid', v; END IF;
|
||||
n := n + 1;
|
||||
END LOOP;
|
||||
IF n <> 19 THEN RAISE EXCEPTION 'expected 19 views, checked %', n; END IF;
|
||||
IF (SELECT count(*) FROM information_schema.views WHERE table_schema = 'public' AND table_name LIKE 'v\_%') <> 19 THEN
|
||||
RAISE EXCEPTION 'public holds a view this list does not name';
|
||||
END IF;
|
||||
|
||||
FOREACH v IN ARRAY ARRAY['fm.holder','fm.operator','licence.licence','licence.plan','licence.licence_event'] LOOP
|
||||
BEGIN
|
||||
EXECUTE format('SELECT count(*) FROM %s', v);
|
||||
RAISE EXCEPTION USING ERRCODE = 'TT001', MESSAGE = v || ' is readable';
|
||||
EXCEPTION WHEN insufficient_privilege THEN NULL;
|
||||
END;
|
||||
END LOOP;
|
||||
|
||||
BEGIN
|
||||
EXECUTE 'CREATE TABLE public.written (a integer)';
|
||||
RAISE EXCEPTION USING ERRCODE = 'TT001', MESSAGE = 'fuel_ro created a table';
|
||||
EXCEPTION WHEN insufficient_privilege OR read_only_sql_transaction THEN NULL;
|
||||
END;
|
||||
RAISE NOTICE 'read-only role: 19 views readable, each with operator_uid; fm and licence refused; writes refused';
|
||||
END $$;
|
||||
@@ -0,0 +1,366 @@
|
||||
-- rules.sql: every rule of the fuel-db migrations, each with an accepted and a refused case.
|
||||
-- Synthetic data only. Run by run.sh on a throw-away database, as a loader acting as fuel_owner.
|
||||
-- A refused case names the words its error must hold, so it fails if a row is refused for another
|
||||
-- reason than the rule under test.
|
||||
\set ON_ERROR_STOP 1
|
||||
\o /dev/null
|
||||
SET client_min_messages = warning;
|
||||
SET search_path = fm, pg_catalog, public;
|
||||
|
||||
CREATE SCHEMA rules_test;
|
||||
CREATE TABLE rules_test.result (n bigint GENERATED ALWAYS AS IDENTITY, label text NOT NULL, passed boolean NOT NULL, detail text);
|
||||
|
||||
CREATE FUNCTION rules_test.ok(label text, stmt text) RETURNS void LANGUAGE plpgsql AS $$
|
||||
BEGIN
|
||||
EXECUTE stmt;
|
||||
INSERT INTO rules_test.result (label, passed) VALUES (label, true);
|
||||
EXCEPTION WHEN OTHERS THEN
|
||||
INSERT INTO rules_test.result (label, passed, detail) VALUES (label, false, 'refused: ' || SQLERRM);
|
||||
END $$;
|
||||
|
||||
-- A refused case always rolls back, so a wrongly accepted row cannot disturb the later cases.
|
||||
CREATE FUNCTION rules_test.no(label text, want text, stmt text) RETURNS void LANGUAGE plpgsql AS $$
|
||||
BEGIN
|
||||
BEGIN
|
||||
EXECUTE stmt;
|
||||
RAISE EXCEPTION USING ERRCODE = 'TT001';
|
||||
EXCEPTION
|
||||
WHEN SQLSTATE 'TT001' THEN
|
||||
INSERT INTO rules_test.result (label, passed, detail) VALUES (label, false, 'accepted');
|
||||
WHEN OTHERS THEN
|
||||
INSERT INTO rules_test.result (label, passed, detail)
|
||||
VALUES (label, position(want IN SQLERRM) > 0,
|
||||
CASE WHEN position(want IN SQLERRM) = 0 THEN 'refused for another reason: ' || SQLERRM END);
|
||||
END;
|
||||
END $$;
|
||||
|
||||
CREATE FUNCTION rules_test.eq(label text, query text, want text) RETURNS void LANGUAGE plpgsql AS $$
|
||||
DECLARE got text;
|
||||
BEGIN
|
||||
EXECUTE query INTO got;
|
||||
INSERT INTO rules_test.result (label, passed, detail)
|
||||
VALUES (label, got IS NOT DISTINCT FROM want, format('got %s, want %s', got, want));
|
||||
END $$;
|
||||
|
||||
CREATE FUNCTION rules_test.rec(op text) RETURNS bigint LANGUAGE sql STABLE AS
|
||||
$$ SELECT min(import_record_id) FROM fm.import_record WHERE operator_uid = op $$;
|
||||
CREATE FUNCTION rules_test.batch(op text) RETURNS bigint LANGUAGE sql STABLE AS
|
||||
$$ SELECT min(import_batch_id) FROM fm.import_batch WHERE operator_uid = op $$;
|
||||
CREATE FUNCTION rules_test.visit(k text) RETURNS bigint LANGUAGE sql STABLE AS
|
||||
$$ SELECT visit_id FROM fm.visit WHERE visit_key = k $$;
|
||||
CREATE FUNCTION rules_test.sha(c text) RETURNS text LANGUAGE sql IMMUTABLE AS
|
||||
$$ SELECT repeat(c, 64) $$;
|
||||
|
||||
-- Two customers ------------------------------------------------------------------------------
|
||||
INSERT INTO fm.operator (operator_uid, name, label_issuer, programme, timezone, keycloak_group) VALUES
|
||||
('OPR-0001', 'Customer A', 'Issuer A', 'Programme A', 'Asia/Kabul', '/customer-a'),
|
||||
('OPR-0002', 'Customer B', 'Issuer B', 'Programme B', 'Europe/Berlin', '/customer-b');
|
||||
INSERT INTO fm.source_document (sha256, operator_uid, source_family, kind, issuer, title, corpus_path, contains_personal_data) VALUES
|
||||
(rules_test.sha('a'), 'OPR-0001', 'family-a', 'operator_workbook', 'Issuer A', 'workbook A', 'a.xlsx', false),
|
||||
(rules_test.sha('b'), 'OPR-0002', 'family-b', 'operator_workbook', 'Issuer B', 'workbook B', 'b.xlsx', false);
|
||||
INSERT INTO fm.import_batch (operator_uid, source_kind, source_path, source_sha256, importer_version) VALUES
|
||||
('OPR-0001', 'operator_workbook', 'a.xlsx', rules_test.sha('a'), 'test'),
|
||||
('OPR-0002', 'operator_workbook', 'b.xlsx', rules_test.sha('b'), 'test');
|
||||
INSERT INTO fm.import_record (operator_uid, logical_key, source_locator, payload, payload_sha256, first_batch_id, last_seen_batch_id)
|
||||
SELECT o, 'record-' || o, 'test', '{}', rules_test.sha('c'), rules_test.batch(o), rules_test.batch(o)
|
||||
FROM unnest(ARRAY['OPR-0001','OPR-0002']) o;
|
||||
INSERT INTO fm.place (place_uid, operator_uid, role, zone, province, timezone, import_record_id) VALUES
|
||||
('PLC-0001', 'OPR-0001', 'tower_site', 'z', 'p', NULL, rules_test.rec('OPR-0001')),
|
||||
('PLC-0002', 'OPR-0002', 'depot', 'z', 'p', 'Asia/Tokyo', rules_test.rec('OPR-0002'));
|
||||
INSERT INTO fm.holder (holder_uid, operator_uid, kind, name, import_record_id) VALUES
|
||||
('SITE-0001', 'OPR-0001', 'fixed_installation', 'Holder A', rules_test.rec('OPR-0001')),
|
||||
('SITE-0002', 'OPR-0002', 'fixed_installation', 'Holder B', rules_test.rec('OPR-0002'));
|
||||
INSERT INTO fm.holder_place (holder_uid, operator_uid, place_uid, null_reasons) VALUES
|
||||
('SITE-0001', 'OPR-0001', 'PLC-0001', '{"valid_from": {}}'),
|
||||
('SITE-0002', 'OPR-0002', 'PLC-0002', '{"valid_from": {}}');
|
||||
INSERT INTO fm.holder_alias (holder_uid, operator_uid, label, issuer, is_primary, null_reasons) VALUES
|
||||
('SITE-0001', 'OPR-0001', 'A1', 'Issuer A', true, '{"valid_from": {}}'),
|
||||
('SITE-0002', 'OPR-0002', 'B1', 'Issuer B', true, '{"valid_from": {}}');
|
||||
INSERT INTO fm.participation (holder_uid, operator_uid, programme, status, valid_from) VALUES
|
||||
('SITE-0001', 'OPR-0001', 'Programme A', 'active', '2026-01-01'),
|
||||
('SITE-0002', 'OPR-0002', 'Programme B', 'active', '2026-01-01');
|
||||
INSERT INTO fm.asset (asset_uid, operator_uid, kind, import_record_id) VALUES
|
||||
('AST-A-GEN', 'OPR-0001', 'generator', rules_test.rec('OPR-0001')),
|
||||
('AST-A-GEN2', 'OPR-0001', 'generator', rules_test.rec('OPR-0001')),
|
||||
('AST-A-ENG', 'OPR-0001', 'engine', rules_test.rec('OPR-0001')),
|
||||
('AST-A-T1', 'OPR-0001', 'tank', rules_test.rec('OPR-0001')),
|
||||
('AST-A-T2', 'OPR-0001', 'tank', rules_test.rec('OPR-0001')),
|
||||
('AST-B-GEN', 'OPR-0002', 'generator', rules_test.rec('OPR-0002'));
|
||||
INSERT INTO fm.asset_placement (asset_uid, operator_uid, holder_uid, slot_label, occupied_from, installed_on) VALUES
|
||||
('AST-A-GEN', 'OPR-0001', 'SITE-0001', 'GEN-01', '2025-01-01', '2025-01-01'),
|
||||
('AST-A-GEN2', 'OPR-0001', 'SITE-0001', 'GEN-02', '2025-01-01', '2025-01-01'),
|
||||
('AST-A-ENG', 'OPR-0001', 'SITE-0001', 'ENG-01', '2025-01-01', '2025-01-01'),
|
||||
('AST-A-T1', 'OPR-0001', 'SITE-0001', 'T1', '2025-01-01', '2025-01-01'),
|
||||
('AST-B-GEN', 'OPR-0002', 'SITE-0002', 'GEN-01', '2025-01-01', '2025-01-01');
|
||||
INSERT INTO fm.visit (visit_key, operator_uid, holder_uid, visit_on, kind) VALUES
|
||||
('SITE-0001:2026-01-01:1', 'OPR-0001', 'SITE-0001', '2026-01-01', 'stock_check'),
|
||||
('SITE-0001:2026-01-11:1', 'OPR-0001', 'SITE-0001', '2026-01-11', 'stock_check'),
|
||||
('SITE-0002:2026-01-01:1', 'OPR-0002', 'SITE-0002', '2026-01-01', 'stock_check');
|
||||
INSERT INTO fm.stock_observation (source_key, assertion_sha256, operator_uid, visit_id, holder_uid, scope, phase, litres, litres_basis, source_sha256, record_ref, import_record_id)
|
||||
SELECT 'stock:' || k, rules_test.sha('d'), 'OPR-0001', rules_test.visit(k), 'SITE-0001', 'site_total', 'standalone', l, 'stated_without_level',
|
||||
rules_test.sha('a'), 'test', rules_test.rec('OPR-0001')
|
||||
FROM (VALUES ('SITE-0001:2026-01-01:1', 100.0), ('SITE-0001:2026-01-11:1', 50.0)) x(k, l);
|
||||
|
||||
SELECT rules_test.ok('fixture holds two customers', 'SELECT 1');
|
||||
SELECT rules_test.no('customers: one Keycloak group is one customer', 'duplicate key',
|
||||
'UPDATE fm.operator SET keycloak_group = ''/customer-a'' WHERE operator_uid = ''OPR-0002''');
|
||||
SELECT rules_test.no('customers: the group is a top-level one', 'keycloak_group_check',
|
||||
'UPDATE fm.operator SET keycloak_group = ''/customer-a/team'' WHERE operator_uid = ''OPR-0001''');
|
||||
SELECT rules_test.eq('customers: v_operator gives the group', 'SELECT operator_uid FROM public.v_operator WHERE keycloak_group = ''/customer-b''', 'OPR-0002');
|
||||
|
||||
-- Rule 1: baseline versions --------------------------------------------------------------------
|
||||
\set bv 'INSERT INTO fm.baseline_version (code, operator_uid, holder_uid, metric, population, population_key, window_start, window_end, policy_version, meets_policy, computed_at, computed_by_role'
|
||||
SELECT rules_test.no('rule 1: a version does not start approved', 'starts as draft or proposed',
|
||||
:'bv' || ', status, approved_on, decision_ref) VALUES (''BL-X'', ''OPR-0001'', ''SITE-0001'', ''fuel_l_per_calendar_day'', ''x'', ''stock:SITE-0001'', ''2026-01-01'', ''2026-01-11'', ''BP-1'', true, now(), ''r'', ''approved'', ''2026-01-12'', ''D-1'')');
|
||||
SELECT rules_test.ok('rule 1: a version starts proposed',
|
||||
:'bv' || ', status) VALUES (''BL-1'', ''OPR-0001'', ''SITE-0001'', ''fuel_l_per_calendar_day'', ''x'', ''stock:SITE-0001'', ''2026-01-01'', ''2026-01-11'', ''BP-1'', true, now(), ''r'', ''proposed'')');
|
||||
SELECT rules_test.ok('rule 1: a version below policy starts proposed',
|
||||
:'bv' || ', status) VALUES (''BL-2'', ''OPR-0001'', ''SITE-0001'', ''fuel_l_per_calendar_day'', ''x'', ''stock:SITE-0001'', ''2026-01-01'', ''2026-01-11'', ''BP-1'', false, now(), ''r'', ''proposed'')');
|
||||
SELECT rules_test.no('rule 1: below policy cannot be approved', 'does not meet',
|
||||
'UPDATE fm.baseline_version SET status = ''approved'', approved_on = ''2026-01-12'', decision_ref = ''D-1'' WHERE code = ''BL-2''');
|
||||
SELECT rules_test.no('rule 1: no inputs cannot be approved', 'its inputs changed',
|
||||
'UPDATE fm.baseline_version SET status = ''approved'', approved_on = ''2026-01-12'', decision_ref = ''D-1'' WHERE code = ''BL-1''');
|
||||
|
||||
-- Rule 6: inputs (needed before an approval) ------------------------------------------------------
|
||||
\set bi 'INSERT INTO fm.baseline_input (baseline_id, operator_uid, holder_uid, open_visit_id, close_visit_id, start_on, end_on, included, days, population_key, eligibility_at_computation, dependencies, stock_dep_sha256) SELECT b.baseline_id, ''OPR-0001'', ''SITE-0001'', rules_test.visit(''SITE-0001:2026-01-01:1''), rules_test.visit(''SITE-0001:2026-01-11:1''), '
|
||||
SELECT rules_test.no('rule 6: dates must equal the endpoint visits', 'differ from the endpoint visits',
|
||||
:'bi' || '''2026-01-02'', ''2026-01-11'', true, 10, ''stock:SITE-0001'', ''eligible'', ''{}'', d.stock_dep_sha256 FROM fm.baseline_version b, fm.v_interval_dependency d WHERE b.code = ''BL-1'' AND d.holder_uid = ''SITE-0001''');
|
||||
SELECT rules_test.no('rule 6: an included input carries the population', 'differs from version',
|
||||
:'bi' || '''2026-01-01'', ''2026-01-11'', true, 10, ''stock:OTHER'', ''eligible'', ''{}'', d.stock_dep_sha256 FROM fm.baseline_version b, fm.v_interval_dependency d WHERE b.code = ''BL-1'' AND d.holder_uid = ''SITE-0001''');
|
||||
SELECT rules_test.ok('rule 6: a correct input is accepted',
|
||||
:'bi' || '''2026-01-01'', ''2026-01-11'', true, 10, ''stock:SITE-0001'', ''eligible'', ''{}'', d.stock_dep_sha256 FROM fm.baseline_version b, fm.v_interval_dependency d WHERE b.code = ''BL-1'' AND d.holder_uid = ''SITE-0001''');
|
||||
SELECT rules_test.ok('rule 1: unchanged inputs and policy met can be approved',
|
||||
'UPDATE fm.baseline_version SET status = ''approved'', approved_on = ''2026-01-12'', approved_by_role = ''r'', decision_ref = ''D-1'' WHERE code = ''BL-1''');
|
||||
SELECT rules_test.no('rule 1: an approved version is immutable', 'approved rows are immutable',
|
||||
'UPDATE fm.baseline_version SET value = 1 WHERE code = ''BL-1''');
|
||||
SELECT rules_test.no('rule 1: an approved version is never deleted', 'never deleted',
|
||||
'DELETE FROM fm.baseline_version WHERE code = ''BL-1''');
|
||||
SELECT rules_test.no('rule 1: an approved version''s inputs are immutable', 'are immutable',
|
||||
'UPDATE fm.baseline_input SET days = 9 WHERE included');
|
||||
|
||||
-- Rules 2, 3, 10: claims -----------------------------------------------------------------------
|
||||
\set mv 'INSERT INTO fm.fuel_movement (source_key, assertion_sha256, operator_uid, event_key, visit_id, holder_uid, kind, from_tank_uid, to_tank_uid, litres, litres_state, source_sha256, record_ref, import_record_id) VALUES '
|
||||
SELECT rules_test.no('rule 3: a movement''s holder is its visit''s holder', 'differs from its visit',
|
||||
:'mv' || '(''m0'', rules_test.sha(''e''), ''OPR-0001'', ''e0'', rules_test.visit(''SITE-0001:2026-01-01:1''), ''SITE-0003'', ''delivery'', NULL, NULL, 10, ''stated'', rules_test.sha(''a''), ''r'', rules_test.rec(''OPR-0001''))');
|
||||
SELECT rules_test.no('rule 2: an internal transfer''s tanks are placed', 'must be placed',
|
||||
:'mv' || '(''m1'', rules_test.sha(''e''), ''OPR-0001'', ''e1'', rules_test.visit(''SITE-0001:2026-01-01:1''), ''SITE-0001'', ''internal_transfer'', ''AST-A-T1'', ''AST-A-T2'', 10, ''stated'', rules_test.sha(''a''), ''r'', rules_test.rec(''OPR-0001''))');
|
||||
INSERT INTO fm.asset_placement (asset_uid, operator_uid, holder_uid, slot_label, occupied_from, installed_on)
|
||||
VALUES ('AST-A-T2', 'OPR-0001', 'SITE-0001', 'T2', '2025-01-01', '2025-01-01');
|
||||
SELECT rules_test.ok('rule 2: placed tanks are accepted',
|
||||
:'mv' || '(''m1'', rules_test.sha(''e''), ''OPR-0001'', ''e1'', rules_test.visit(''SITE-0001:2026-01-01:1''), ''SITE-0001'', ''internal_transfer'', ''AST-A-T1'', ''AST-A-T2'', 10, ''stated'', rules_test.sha(''a''), ''r'', rules_test.rec(''OPR-0001''))');
|
||||
SELECT rules_test.no('rule 10: claims of one event share kind and tanks', 'must share visit',
|
||||
:'mv' || '(''m2'', rules_test.sha(''e''), ''OPR-0001'', ''e1'', rules_test.visit(''SITE-0001:2026-01-01:1''), ''SITE-0001'', ''delivery'', NULL, NULL, 10, ''stated'', rules_test.sha(''a''), ''r'', rules_test.rec(''OPR-0001''))');
|
||||
SELECT rules_test.ok('rule 10: a second claim of one event is accepted',
|
||||
:'mv' || '(''m2'', rules_test.sha(''e''), ''OPR-0001'', ''e1'', rules_test.visit(''SITE-0001:2026-01-01:1''), ''SITE-0001'', ''internal_transfer'', ''AST-A-T1'', ''AST-A-T2'', 11, ''stated'', rules_test.sha(''a''), ''r'', rules_test.rec(''OPR-0001''))');
|
||||
SELECT rules_test.no('rule 3: a stock claim''s holder is its visit''s holder', 'differs from its visit',
|
||||
'INSERT INTO fm.stock_observation (source_key, assertion_sha256, operator_uid, visit_id, holder_uid, scope, phase, litres, litres_basis, source_sha256, record_ref, import_record_id) VALUES (''s0'', rules_test.sha(''d''), ''OPR-0001'', rules_test.visit(''SITE-0001:2026-01-01:1''), ''SITE-0003'', ''site_total'', ''standalone'', 1, ''stated_without_level'', rules_test.sha(''a''), ''r'', rules_test.rec(''OPR-0001''))');
|
||||
|
||||
-- Rules 8, 12, 9: meters and assumptions, generator-only (decision 6) ---------------------------
|
||||
\set mt 'INSERT INTO fm.runtime_meter (meter_id, operator_uid, holder_uid, source_label, asset_uid, mapping_status) VALUES '
|
||||
SELECT rules_test.ok('rule 8: a confirmed meter on a placed generator',
|
||||
:'mt' || '(''MTR-A-1'', ''OPR-0001'', ''SITE-0001'', ''Genset 1'', ''AST-A-GEN'', ''confirmed'')');
|
||||
SELECT rules_test.no('rule 8: a meter on an engine is refused', 'is not a generator placed',
|
||||
:'mt' || '(''MTR-A-E'', ''OPR-0001'', ''SITE-0001'', ''Engine'', ''AST-A-ENG'', ''confirmed'')');
|
||||
SELECT rules_test.no('rule 8: a meter on a tank is refused', 'is not a generator placed',
|
||||
:'mt' || '(''MTR-A-T'', ''OPR-0001'', ''SITE-0001'', ''Tank'', ''AST-A-T1'', ''proposed'')');
|
||||
SELECT rules_test.ok('rule 8: an unmapped counter names no asset',
|
||||
:'mt' || '(''MTR-A-2'', ''OPR-0001'', ''SITE-0001'', ''R.H'', NULL, ''unmapped'')');
|
||||
|
||||
INSERT INTO fm.meter_epoch (meter_id, operator_uid, epoch_no, start_reason, continuity) VALUES
|
||||
('MTR-A-1', 'OPR-0001', 1, 'first_seen', 'assumed'),
|
||||
('MTR-A-2', 'OPR-0001', 1, 'first_seen', 'assumed');
|
||||
\set rd 'INSERT INTO fm.hour_meter_reading (source_key, assertion_sha256, operator_uid, visit_id, meter_id, epoch_id, source_field, date_basis, hours, reading_state, transcribed_by, source_sha256, record_ref, import_record_id) VALUES '
|
||||
SELECT rules_test.ok('rules 3, 4: a reading of the visit''s own meter and epoch',
|
||||
:'rd' || '(''h1'', rules_test.sha(''f''), ''OPR-0001'', rules_test.visit(''SITE-0001:2026-01-01:1''), ''MTR-A-1'', (SELECT epoch_id FROM fm.meter_epoch WHERE meter_id = ''MTR-A-1''), ''current'', ''stated'', 10, ''clear'', ''source_operator'', rules_test.sha(''a''), ''r'', rules_test.rec(''OPR-0001''))');
|
||||
SELECT rules_test.no('rule 4: a reading''s epoch belongs to its meter', 'does not belong to meter',
|
||||
:'rd' || '(''h2'', rules_test.sha(''f''), ''OPR-0001'', rules_test.visit(''SITE-0001:2026-01-01:1''), ''MTR-A-1'', (SELECT epoch_id FROM fm.meter_epoch WHERE meter_id = ''MTR-A-2''), ''current'', ''stated'', 10, ''clear'', ''source_operator'', rules_test.sha(''a''), ''r'', rules_test.rec(''OPR-0001''))');
|
||||
|
||||
\set ax 'INSERT INTO fm.assumption (code, fingerprint, operator_uid, holder_uid, asset_uid, topic, basis, derived_from_fuel, status, rationale'
|
||||
SELECT rules_test.ok('rule 12: an idle assumption on a placed generator',
|
||||
:'ax' || ') VALUES (''A-IDLE'', rules_test.sha(''0''), ''OPR-0001'', ''SITE-0001'', ''AST-A-GEN2'', ''generator_idle'', ''our_assumption'', false, ''proposed'', ''r'')');
|
||||
SELECT rules_test.no('rule 12: an idle assumption on an engine is refused', 'is not a generator placed',
|
||||
:'ax' || ') VALUES (''A-IDLE-E'', rules_test.sha(''0''), ''OPR-0001'', ''SITE-0001'', ''AST-A-ENG'', ''generator_idle'', ''our_assumption'', false, ''proposed'', ''r'')');
|
||||
|
||||
-- Rule 5: needs a matching review first, or rule 11 refuses before the CHECK is reached
|
||||
INSERT INTO fm.review (entry_key, operator_uid, subject_kind, subject_key, assertion_sha256, verdict, reviewed_on, reviewer_role, import_batch_id)
|
||||
VALUES ('R-5', 'OPR-0001', 'assumption', 'A-R5', rules_test.sha('5'), 'accepted', '2026-01-02', 'r', rules_test.batch('OPR-0001'));
|
||||
SELECT rules_test.no('rule 5: a runtime derived from fuel cannot be accepted', 'assumption_no_runtime_from_fuel',
|
||||
:'ax' || ', accepted_on, accepted_by_role) VALUES (''A-R5'', rules_test.sha(''5''), ''OPR-0001'', ''SITE-0001'', NULL, ''runtime_hours_per_day'', ''our_assumption'', true, ''accepted'', ''r'', ''2026-01-02'', ''r'')');
|
||||
SELECT rules_test.ok('rule 5: one not derived from fuel can be accepted',
|
||||
:'ax' || ', accepted_on, accepted_by_role) VALUES (''A-R5'', rules_test.sha(''5''), ''OPR-0001'', ''SITE-0001'', NULL, ''runtime_hours_per_day'', ''our_assumption'', false, ''accepted'', ''r'', ''2026-01-02'', ''r'')');
|
||||
|
||||
SELECT rules_test.no('rule 9: members complete at commit', 'as many counters as covered',
|
||||
:'ax' || ') VALUES (''A-COV0'', rules_test.sha(''6''), ''OPR-0001'', ''SITE-0001'', NULL, ''meter_coverage'', ''our_assumption'', false, ''proposed'', ''r''); '
|
||||
'INSERT INTO fm.assumption_member (assumption_id, operator_uid, member_role, meter_id) SELECT assumption_id, ''OPR-0001'', ''counter'', ''MTR-A-2'' FROM fm.assumption WHERE code = ''A-COV0''; '
|
||||
'SET CONSTRAINTS ALL IMMEDIATE');
|
||||
SELECT rules_test.ok('rule 9: valid and complete members in the assumption''s own transaction',
|
||||
:'ax' || ') VALUES (''A-COV'', rules_test.sha(''6''), ''OPR-0001'', ''SITE-0001'', NULL, ''meter_coverage'', ''our_assumption'', false, ''proposed'', ''r''); '
|
||||
'INSERT INTO fm.assumption_member (assumption_id, operator_uid, member_role, meter_id) SELECT assumption_id, ''OPR-0001'', ''counter'', ''MTR-A-2'' FROM fm.assumption WHERE code = ''A-COV''; '
|
||||
'INSERT INTO fm.assumption_member (assumption_id, operator_uid, member_role, asset_uid) SELECT assumption_id, ''OPR-0001'', ''covered_generator'', ''AST-A-GEN'' FROM fm.assumption WHERE code = ''A-COV''; '
|
||||
'SET CONSTRAINTS ALL IMMEDIATE');
|
||||
SELECT rules_test.no('rule 9: members are written in the same transaction', 'written in the transaction',
|
||||
'INSERT INTO fm.assumption_member (assumption_id, operator_uid, member_role, asset_uid) SELECT assumption_id, ''OPR-0001'', ''idle_generator'', ''AST-A-GEN2'' FROM fm.assumption WHERE code = ''A-COV''');
|
||||
SELECT rules_test.no('rule 9: members are immutable', 'members are immutable',
|
||||
'DELETE FROM fm.assumption_member');
|
||||
SELECT rules_test.no('rule 9: an engine cannot be a covered member', 'is not a generator placed',
|
||||
'INSERT INTO fm.assumption (code, fingerprint, operator_uid, holder_uid, topic, basis, derived_from_fuel, status, rationale) '
|
||||
'SELECT ''A-COV2'', rules_test.sha(''7''), ''OPR-0002'', ''SITE-0002'', ''meter_coverage'', ''our_assumption'', false, ''proposed'', ''r''; '
|
||||
'INSERT INTO fm.asset (asset_uid, operator_uid, kind, import_record_id) VALUES (''AST-B-ENG'', ''OPR-0002'', ''engine'', rules_test.rec(''OPR-0002'')); '
|
||||
'INSERT INTO fm.asset_placement (asset_uid, operator_uid, holder_uid, slot_label, occupied_from, installed_on) VALUES (''AST-B-ENG'', ''OPR-0002'', ''SITE-0002'', ''E1'', ''2025-01-01'', ''2025-01-01''); '
|
||||
'INSERT INTO fm.assumption_member (assumption_id, operator_uid, member_role, asset_uid) SELECT assumption_id, ''OPR-0002'', ''covered_generator'', ''AST-B-ENG'' FROM fm.assumption WHERE code = ''A-COV2''');
|
||||
|
||||
-- Rule 7: source documents ---------------------------------------------------------------------
|
||||
\set sd 'INSERT INTO fm.source_document (sha256, operator_uid, source_family, kind, issuer, title, corpus_path, contains_personal_data, supersedes_sha256) VALUES '
|
||||
SELECT rules_test.ok('rule 7: a later version in its own family',
|
||||
:'sd' || '(rules_test.sha(''1''), ''OPR-0001'', ''family-a'', ''operator_workbook'', ''Issuer A'', ''later'', ''a2.xlsx'', false, rules_test.sha(''a''))');
|
||||
SELECT rules_test.no('rule 7: no supersession across families', 'may supersede only',
|
||||
:'sd' || '(rules_test.sha(''2''), ''OPR-0001'', ''family-other'', ''operator_workbook'', ''Issuer A'', ''x'', ''x.xlsx'', false, rules_test.sha(''1''))');
|
||||
SELECT rules_test.no('rule 7: a document cannot be shared with a second customer', 'another customer',
|
||||
:'sd' || '(rules_test.sha(''a''), ''OPR-0002'', ''family-a'', ''operator_workbook'', ''Issuer A'', ''x'', ''x.xlsx'', false, NULL)');
|
||||
|
||||
-- Rule 11: status from a review ------------------------------------------------------------------
|
||||
SELECT rules_test.no('rule 11: no status without a review', 'needs a review',
|
||||
'UPDATE fm.stock_observation SET claim_status = ''reviewed'' WHERE source_key = ''stock:SITE-0001:2026-01-01:1''');
|
||||
SELECT rules_test.no('rule 11: a value is never corrected in place', 'only the status columns',
|
||||
'UPDATE fm.stock_observation SET litres = 101 WHERE source_key = ''stock:SITE-0001:2026-01-01:1''');
|
||||
INSERT INTO fm.review (entry_key, operator_uid, subject_kind, subject_key, assertion_sha256, verdict, reviewed_on, reviewer_role, import_batch_id) VALUES
|
||||
('R-OTHER', 'OPR-0002', 'claim', 'stock:SITE-0001:2026-01-01:1', rules_test.sha('d'), 'reviewed', '2026-01-02', 'r', rules_test.batch('OPR-0002'));
|
||||
SELECT rules_test.no('rule 11: another customer''s review does not count', 'needs a review',
|
||||
'UPDATE fm.stock_observation SET claim_status = ''reviewed'' WHERE source_key = ''stock:SITE-0001:2026-01-01:1''');
|
||||
INSERT INTO fm.review (entry_key, operator_uid, subject_kind, subject_key, assertion_sha256, verdict, reviewed_on, reviewer_role, import_batch_id) VALUES
|
||||
('R-OTHER', 'OPR-0001', 'claim', 'stock:SITE-0001:2026-01-01:1', rules_test.sha('d'), 'reviewed', '2026-01-02', 'r', rules_test.batch('OPR-0001'));
|
||||
SELECT rules_test.ok('rule 11: a matching review gives the status; entry ids are per customer',
|
||||
'UPDATE fm.stock_observation SET claim_status = ''reviewed'' WHERE source_key = ''stock:SITE-0001:2026-01-01:1''');
|
||||
SELECT rules_test.no('review: an entry id is unique within one customer', 'duplicate key',
|
||||
'INSERT INTO fm.review (entry_key, operator_uid, subject_kind, subject_key, assertion_sha256, verdict, reviewed_on, reviewer_role, import_batch_id) VALUES (''R-OTHER'', ''OPR-0001'', ''claim'', ''elsewhere'', rules_test.sha(''d''), ''disputed'', ''2026-01-03'', ''r'', rules_test.batch(''OPR-0001''))');
|
||||
|
||||
-- Rule 13: identity never changes in place -------------------------------------------------------
|
||||
SELECT rules_test.no('rule 13: a visit''s date', 'never changes in place', 'UPDATE fm.visit SET visit_on = ''2026-01-02'' WHERE visit_key = ''SITE-0001:2026-01-01:1''');
|
||||
SELECT rules_test.no('rule 13: a document''s family', 'never changes in place', 'UPDATE fm.source_document SET source_family = ''x'' WHERE sha256 = rules_test.sha(''a'')');
|
||||
SELECT rules_test.no('rule 13: an asset''s kind', 'never changes in place', 'UPDATE fm.asset SET kind = ''engine'' WHERE asset_uid = ''AST-A-GEN''');
|
||||
SELECT rules_test.no('rule 13: a meter''s holder', 'never changes in place', 'UPDATE fm.runtime_meter SET holder_uid = ''SITE-0002'' WHERE meter_id = ''MTR-A-1''');
|
||||
SELECT rules_test.no('rule 13: an epoch''s meter', 'never changes in place', 'UPDATE fm.meter_epoch SET meter_id = ''MTR-A-2'' WHERE meter_id = ''MTR-A-1''');
|
||||
SELECT rules_test.no('rule 13: a holder''s kind', 'never changes in place', 'UPDATE fm.holder SET kind = ''trailer'' WHERE holder_uid = ''SITE-0001''');
|
||||
SELECT rules_test.ok('rule 13: a descriptive field may change', 'UPDATE fm.holder SET name = ''Renamed'' WHERE holder_uid = ''SITE-0001''');
|
||||
CREATE TABLE rules_test.wrong_name (a integer);
|
||||
CREATE TRIGGER wrong_name BEFORE UPDATE ON rules_test.wrong_name FOR EACH ROW EXECUTE FUNCTION fm.tg_fixed_columns('nope');
|
||||
INSERT INTO rules_test.wrong_name VALUES (1);
|
||||
SELECT rules_test.no('rule 13: a wrong column name raises, not a silent pass', 'does not have', 'UPDATE rules_test.wrong_name SET a = 2');
|
||||
|
||||
-- Rules 14, 15 --------------------------------------------------------------------------------
|
||||
SELECT rules_test.no('rule 14: an unknown time zone', 'unknown time zone', 'UPDATE fm.operator SET timezone = ''Asia/Nowhere'' WHERE operator_uid = ''OPR-0001''');
|
||||
SELECT rules_test.no('rule 14: an unknown time zone on a place', 'unknown time zone', 'UPDATE fm.place SET timezone = ''Asia/Nowhere'' WHERE place_uid = ''PLC-0001''');
|
||||
SELECT rules_test.ok('rule 14: a known time zone', 'UPDATE fm.place SET timezone = ''Asia/Kabul'' WHERE place_uid = ''PLC-0001''');
|
||||
|
||||
-- Two customers: a child cannot name another customer's parent -----------------------------------
|
||||
SELECT rules_test.no('customers: a visit on another''s holder', 'foreign key',
|
||||
'INSERT INTO fm.visit (visit_key, operator_uid, holder_uid, visit_on, kind) VALUES (''x'', ''OPR-0002'', ''SITE-0001'', ''2026-02-01'', ''survey'')');
|
||||
SELECT rules_test.no('customers: a claim on another''s visit', 'foreign key',
|
||||
:'mv' || '(''mx'', rules_test.sha(''e''), ''OPR-0002'', ''ex'', rules_test.visit(''SITE-0001:2026-01-01:1''), ''SITE-0001'', ''delivery'', NULL, NULL, 10, ''stated'', rules_test.sha(''b''), ''r'', rules_test.rec(''OPR-0002''))');
|
||||
SELECT rules_test.no('customers: a placement of another''s asset', 'foreign key',
|
||||
'INSERT INTO fm.asset (asset_uid, operator_uid, kind, import_record_id) VALUES (''AST-A-SPARE'', ''OPR-0001'', ''generator'', rules_test.rec(''OPR-0001'')); '
|
||||
'INSERT INTO fm.asset_placement (asset_uid, operator_uid, holder_uid, slot_label, null_reasons) VALUES (''AST-A-SPARE'', ''OPR-0002'', ''SITE-0002'', ''X'', ''{"installed_on": {}}'')');
|
||||
SELECT rules_test.no('customers: a meter on another''s holder', 'foreign key',
|
||||
:'mt' || '(''MTR-X'', ''OPR-0002'', ''SITE-0001'', ''X'', NULL, ''unmapped'')');
|
||||
SELECT rules_test.no('customers: a reading of another''s meter', 'foreign key',
|
||||
:'rd' || '(''hx'', rules_test.sha(''f''), ''OPR-0002'', rules_test.visit(''SITE-0001:2026-01-01:1''), ''MTR-A-1'', NULL, ''current'', ''stated'', 10, ''clear'', ''source_operator'', rules_test.sha(''b''), ''r'', rules_test.rec(''OPR-0002''))');
|
||||
SELECT rules_test.no('customers: a baseline on another''s holder', 'foreign key',
|
||||
:'bv' || ', status) VALUES (''BL-X2'', ''OPR-0002'', ''SITE-0001'', ''fuel_l_per_calendar_day'', ''x'', ''k'', ''2026-01-01'', ''2026-01-11'', ''BP-1'', true, now(), ''r'', ''proposed'')');
|
||||
SELECT rules_test.no('customers: an input of another''s baseline', 'foreign key',
|
||||
'INSERT INTO fm.baseline_input (baseline_id, operator_uid, holder_uid, open_visit_id, close_visit_id, start_on, end_on, included, exclusion_reason, eligibility_at_computation, dependencies, stock_dep_sha256) SELECT b.baseline_id, ''OPR-0002'', ''SITE-0002'', rules_test.visit(''SITE-0002:2026-01-01:1''), rules_test.visit(''SITE-0002:2026-01-01:1''), ''2026-01-01'', ''2026-01-01'', false, ''x'', ''excluded'', ''{}'', ''x'' FROM fm.baseline_version b WHERE b.code = ''BL-2''');
|
||||
SELECT rules_test.no('customers: a claim citing another''s document', 'foreign key',
|
||||
:'mv' || '(''my'', rules_test.sha(''e''), ''OPR-0002'', ''ey'', rules_test.visit(''SITE-0002:2026-01-01:1''), ''SITE-0002'', ''delivery'', NULL, NULL, 10, ''stated'', rules_test.sha(''a''), ''r'', rules_test.rec(''OPR-0002''))');
|
||||
SELECT rules_test.no('customers: a holder standing at another''s place', 'foreign key',
|
||||
'INSERT INTO fm.holder (holder_uid, operator_uid, kind, import_record_id) VALUES (''SITE-0004'', ''OPR-0002'', ''fixed_installation'', rules_test.rec(''OPR-0002'')); '
|
||||
'INSERT INTO fm.holder_place (holder_uid, operator_uid, place_uid, valid_from) VALUES (''SITE-0004'', ''OPR-0002'', ''PLC-0001'', ''2030-01-01'')');
|
||||
\set me 'INSERT INTO fm.fuel_movement (source_key, assertion_sha256, operator_uid, event_key, visit_id, holder_uid, kind, litres, litres_state, unit_price, price_currency, source_sha256, record_ref, import_record_id) VALUES '
|
||||
SELECT rules_test.ok('customers: one event key under the first customer',
|
||||
:'me' || '(''shared-a'', rules_test.sha(''e''), ''OPR-0001'', ''shared-event'', rules_test.visit(''SITE-0001:2026-01-11:1''), ''SITE-0001'', ''delivery'', 10, ''stated'', 70, ''AAA'', rules_test.sha(''a''), ''ref-a'', rules_test.rec(''OPR-0001''))');
|
||||
SELECT rules_test.ok('customers: the same event key under the second, with another kind (rule 10 is per customer)',
|
||||
:'me' || '(''shared-b'', rules_test.sha(''e''), ''OPR-0002'', ''shared-event'', rules_test.visit(''SITE-0002:2026-01-01:1''), ''SITE-0002'', ''withdrawal'', 20, ''stated'', 90, ''BBB'', rules_test.sha(''b''), ''ref-b'', rules_test.rec(''OPR-0002''))');
|
||||
SELECT rules_test.ok('customers: and as a delivery',
|
||||
:'me' || '(''shared-c'', rules_test.sha(''e''), ''OPR-0002'', ''shared-event-2'', rules_test.visit(''SITE-0002:2026-01-01:1''), ''SITE-0002'', ''delivery'', 20, ''stated'', 90, ''BBB'', rules_test.sha(''b''), ''ref-b'', rules_test.rec(''OPR-0002''))');
|
||||
SELECT rules_test.ok('customers: the first again, under the second''s delivery key',
|
||||
:'me' || '(''shared-d'', rules_test.sha(''e''), ''OPR-0001'', ''shared-event-2'', rules_test.visit(''SITE-0001:2026-01-11:1''), ''SITE-0001'', ''delivery'', 10, ''stated'', 70, ''AAA'', rules_test.sha(''a''), ''ref-a'', rules_test.rec(''OPR-0001''))');
|
||||
SELECT rules_test.eq('customers: v_delivery gives each its own price and reference under one event key',
|
||||
'SELECT string_agg(operator_uid || '':'' || unit_price || '':'' || currency || '':'' || source_ref, '','' ORDER BY operator_uid) FROM public.v_delivery WHERE event_key = ''shared-event-2''',
|
||||
'OPR-0001:70.00:AAA:ref-a,OPR-0002:90.00:BBB:ref-b');
|
||||
SELECT rules_test.no('customers: a row''s customer cannot change: holder', 'never changes in place', 'UPDATE fm.holder SET operator_uid = ''OPR-0002'' WHERE holder_uid = ''SITE-0001''');
|
||||
SELECT rules_test.no('customers: a row''s customer cannot change: review', 'never changes in place', 'UPDATE fm.review SET operator_uid = ''OPR-0002'' WHERE entry_key = ''R-5''');
|
||||
SELECT rules_test.no('customers: a row''s customer cannot change: batch', 'never changes in place', 'UPDATE fm.import_batch SET operator_uid = ''OPR-0002'' WHERE import_batch_id = rules_test.batch(''OPR-0001'')');
|
||||
SELECT rules_test.eq('customers: every table but asset_kind has the fixed-customer trigger',
|
||||
'SELECT string_agg(c.relname, '','' ORDER BY c.relname) FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace WHERE n.nspname = ''fm'' AND c.relkind = ''r'' AND NOT EXISTS (SELECT 1 FROM pg_trigger t WHERE t.tgrelid = c.oid AND t.tgname = ''operator_fixed'')',
|
||||
'asset_kind');
|
||||
SELECT rules_test.eq('customers: v_site gives each holder its customer',
|
||||
'SELECT string_agg(site_uid || ''='' || operator_uid, '','' ORDER BY site_uid) FROM public.v_site', 'SITE-0001=OPR-0001,SITE-0002=OPR-0002');
|
||||
SELECT rules_test.eq('customers: labels and programme come from each customer''s record',
|
||||
'SELECT string_agg(site_code || '':'' || pilot_status, '','' ORDER BY site_uid) FROM public.v_site', 'A1:active,B1:active');
|
||||
SELECT rules_test.eq('customers: local day uses the place''s zone, else the customer''s',
|
||||
'SELECT string_agg(holder_uid || '':'' || timezone, '','' ORDER BY holder_uid) FROM fm.v_holder_current', 'SITE-0001:Asia/Kabul,SITE-0002:Asia/Tokyo');
|
||||
|
||||
-- A vehicle is both an asset and a holder (PD-18) -------------------------------------------------
|
||||
INSERT INTO fm.asset (asset_uid, operator_uid, kind, import_record_id) VALUES
|
||||
('AST-VEH-1', 'OPR-0002', 'vehicle', rules_test.rec('OPR-0002')),
|
||||
('AST-VEH-T', 'OPR-0002', 'vehicle_tank', rules_test.rec('OPR-0002')),
|
||||
('AST-VEH-E', 'OPR-0002', 'engine', rules_test.rec('OPR-0002'));
|
||||
\set hd 'INSERT INTO fm.holder (holder_uid, operator_uid, kind, asset_uid, import_record_id) VALUES '
|
||||
SELECT rules_test.no('vehicle: a vehicle holder needs its asset', 'holder_check', :'hd' || '(''VEH-0001'', ''OPR-0002'', ''vehicle'', NULL, rules_test.rec(''OPR-0002''))');
|
||||
SELECT rules_test.no('vehicle: a fixed holder has no asset', 'holder_check', :'hd' || '(''SITE-0009'', ''OPR-0002'', ''fixed_installation'', ''AST-VEH-1'', rules_test.rec(''OPR-0002''))');
|
||||
SELECT rules_test.no('rule 15: a vehicle holder''s asset is a vehicle', 'is not a vehicle', :'hd' || '(''VEH-0001'', ''OPR-0002'', ''vehicle'', ''AST-VEH-T'', rules_test.rec(''OPR-0002''))');
|
||||
SELECT rules_test.ok('vehicle: asset and holder', :'hd' || '(''VEH-0001'', ''OPR-0002'', ''vehicle'', ''AST-VEH-1'', rules_test.rec(''OPR-0002''))');
|
||||
SELECT rules_test.no('rule 15: a vehicle stands at no place', 'not a fixed installation',
|
||||
'INSERT INTO fm.holder_place (holder_uid, operator_uid, place_uid, valid_from) VALUES (''VEH-0001'', ''OPR-0002'', ''PLC-0002'', ''2026-01-01'')');
|
||||
SELECT rules_test.ok('vehicle: a tank placed on it',
|
||||
'INSERT INTO fm.asset_placement (asset_uid, operator_uid, holder_uid, slot_label, occupied_from, installed_on) VALUES (''AST-VEH-T'', ''OPR-0002'', ''VEH-0001'', ''C1'', ''2026-01-01'', ''2026-01-01'')');
|
||||
SELECT rules_test.ok('vehicle: an engine placed on it',
|
||||
'INSERT INTO fm.asset_placement (asset_uid, operator_uid, holder_uid, slot_label, occupied_from, installed_on) VALUES (''AST-VEH-E'', ''OPR-0002'', ''VEH-0001'', ''E1'', ''2026-01-01'', ''2026-01-01'')');
|
||||
SELECT rules_test.no('rule 8: a meter on the vehicle''s engine is refused', 'is not a generator placed',
|
||||
:'mt' || '(''MTR-V-E'', ''OPR-0002'', ''VEH-0001'', ''Engine'', ''AST-VEH-E'', ''confirmed'')');
|
||||
SELECT rules_test.ok('vehicle: a visit', 'INSERT INTO fm.visit (visit_key, operator_uid, holder_uid, visit_on, kind) VALUES (''VEH-0001:2026-02-01:1'', ''OPR-0002'', ''VEH-0001'', ''2026-02-01'', ''delivery'')');
|
||||
SELECT rules_test.ok('vehicle: a delivery on its visit',
|
||||
:'mv' || '(''m-veh'', rules_test.sha(''e''), ''OPR-0002'', ''e-veh'', rules_test.visit(''VEH-0001:2026-02-01:1''), ''VEH-0001'', ''delivery'', NULL, NULL, 200, ''stated'', rules_test.sha(''b''), ''r'', rules_test.rec(''OPR-0002''))');
|
||||
SELECT rules_test.eq('vehicle: the delivery reaches v_delivery with its customer',
|
||||
'SELECT operator_uid || '':'' || delivered_l FROM public.v_delivery WHERE site_uid = ''VEH-0001''', 'OPR-0002:200.0');
|
||||
SELECT rules_test.eq('vehicle: v_site lists it, with no place', 'SELECT coalesce(province, ''no place'') FROM public.v_site WHERE site_uid = ''VEH-0001''', 'no place');
|
||||
|
||||
-- Placement: in or on a holder, or at a place, exactly one -----------------------------------------
|
||||
\set pl 'INSERT INTO fm.asset_placement (asset_uid, operator_uid, holder_uid, place_uid, slot_label, occupied_from, installed_on) VALUES '
|
||||
INSERT INTO fm.asset (asset_uid, operator_uid, kind, import_record_id) VALUES
|
||||
('AST-B-T1', 'OPR-0002', 'tank', rules_test.rec('OPR-0002')), ('AST-B-T2', 'OPR-0002', 'tank', rules_test.rec('OPR-0002'));
|
||||
SELECT rules_test.no('placement: not both', 'asset_placement_check', :'pl' || '(''AST-B-T1'', ''OPR-0002'', ''SITE-0002'', ''PLC-0002'', ''P1'', ''2026-01-01'', ''2026-01-01'')');
|
||||
SELECT rules_test.no('placement: not neither', 'asset_placement_check', :'pl' || '(''AST-B-T1'', ''OPR-0002'', NULL, NULL, ''P1'', ''2026-01-01'', ''2026-01-01'')');
|
||||
SELECT rules_test.ok('placement: at a place', :'pl' || '(''AST-B-T1'', ''OPR-0002'', NULL, ''PLC-0002'', ''P1'', ''2026-01-01'', ''2026-01-01'')');
|
||||
SELECT rules_test.no('placement: one slot at a place at a time', 'conflicting key', :'pl' || '(''AST-B-T2'', ''OPR-0002'', NULL, ''PLC-0002'', ''P1'', ''2026-06-01'', ''2026-06-01'')');
|
||||
SELECT rules_test.no('placement: one slot on a holder at a time', 'conflicting key', :'pl' || '(''AST-B-T2'', ''OPR-0002'', ''SITE-0002'', NULL, ''GEN-01'', ''2026-06-01'', ''2026-06-01'')');
|
||||
SELECT rules_test.no('placement: an asset is in one place at a time', 'conflicting key', :'pl' || '(''AST-B-T1'', ''OPR-0002'', ''SITE-0002'', NULL, ''P2'', ''2026-06-01'', ''2026-06-01'')');
|
||||
SELECT rules_test.eq('placement: a place-held tank does not break the views',
|
||||
'SELECT count(*)::text FROM public.v_spec_reference WHERE asset_uid = ''AST-B-T1''', '0');
|
||||
|
||||
-- Licences -------------------------------------------------------------------------------------
|
||||
\set li 'INSERT INTO licence.licence (operator_uid, plan_code, payment_term, valid_from, valid_until, grace_until, status, contract_ref, created_by) VALUES '
|
||||
SELECT rules_test.eq('licence: PD-19''s prices are seeded',
|
||||
'SELECT string_agg(payment_term || ''='' || price, '','' ORDER BY payment_term) FROM licence.plan_price WHERE plan_code = ''standard''', 'annual=15.00,monthly=17.00');
|
||||
SELECT rules_test.eq('licence: no customer''s licence is seeded', 'SELECT count(*)::text FROM licence.licence', '0');
|
||||
SELECT rules_test.eq('licence: seeds are not in the log', 'SELECT count(*)::text FROM licence.licence_event', '0');
|
||||
SELECT rules_test.ok('licence: a first licence', :'li' || '(''OPR-0001'', ''pilot'', ''annual'', ''2026-01-01'', ''2026-12-31'', ''2027-01-30'', ''active'', ''agreement'', ''our admin'')');
|
||||
SELECT rules_test.no('licence: overlapping dates refused', 'conflicting key', :'li' || '(''OPR-0001'', ''standard'', ''annual'', ''2026-12-31'', ''2027-12-31'', ''2028-01-30'', ''active'', ''agreement'', ''our admin'')');
|
||||
SELECT rules_test.ok('licence: a renewal may start during the old grace', :'li' || '(''OPR-0001'', ''standard'', ''annual'', ''2027-01-01'', ''2027-12-31'', ''2028-01-30'', ''active'', ''agreement'', ''our admin'')');
|
||||
SELECT rules_test.ok('licence: another customer may share the dates', :'li' || '(''OPR-0002'', ''standard'', ''monthly'', ''2026-01-01'', ''2026-12-31'', ''2027-01-30'', ''active'', ''agreement'', ''our admin'')');
|
||||
SELECT rules_test.no('licence: grace at least until expiry', 'licence_check', :'li' || '(''OPR-0002'', ''standard'', ''annual'', ''2028-01-01'', ''2028-12-31'', ''2028-12-30'', ''active'', ''agreement'', ''our admin'')');
|
||||
SELECT rules_test.no('licence: an unknown customer', 'foreign key', :'li' || '(''OPR-0009'', ''standard'', ''annual'', ''2028-01-01'', ''2028-12-31'', ''2029-01-30'', ''active'', ''agreement'', ''our admin'')');
|
||||
SELECT rules_test.eq('licence: every change writes an event', 'SELECT count(*)::text FROM licence.licence_event WHERE table_name = ''licence'' AND action = ''INSERT''', '3');
|
||||
SELECT rules_test.ok('licence: a suspension', 'UPDATE licence.licence SET status = ''suspended'' WHERE operator_uid = ''OPR-0002''');
|
||||
SELECT rules_test.eq('licence: the event holds the old and new row',
|
||||
'SELECT old_row->>''status'' || ''>'' || (new_row->>''status'') FROM licence.licence_event WHERE action = ''UPDATE''', 'active>suspended');
|
||||
SELECT rules_test.no('licence: the log is never changed', 'never changed', 'DELETE FROM licence.licence_event');
|
||||
|
||||
-- Result ------------------------------------------------------------------------------------------
|
||||
\o
|
||||
\pset format unaligned
|
||||
\pset tuples_only on
|
||||
SELECT CASE WHEN passed THEN 'ok ' ELSE 'FAIL ' END || label || CASE WHEN NOT passed THEN ' - ' || coalesce(detail, '') ELSE '' END
|
||||
FROM rules_test.result ORDER BY n;
|
||||
SELECT 'rules: ' || count(*) || ' checks, ' || count(*) FILTER (WHERE NOT passed) || ' failed' FROM rules_test.result;
|
||||
DO $$ BEGIN
|
||||
IF EXISTS (SELECT 1 FROM rules_test.result WHERE NOT passed) THEN RAISE EXCEPTION 'rule tests failed'; END IF;
|
||||
END $$;
|
||||
Executable
+48
@@ -0,0 +1,48 @@
|
||||
#!/usr/bin/env bash
|
||||
# Applies the migrations to a throw-away database and runs the rule tests. Synthetic data only.
|
||||
# Needs a running fuel-db container with this repository's roles (fuel_owner, fuel_ro), e.g. the
|
||||
# local stack: FUEL_DB_CONTAINER=fmdb-scratch-db stacks/fuel-monitoring/fuel-db/tests/run.sh
|
||||
# Never point it at the server: it creates a role and a database.
|
||||
set -euo pipefail
|
||||
|
||||
here=$(cd "$(dirname "$0")" && pwd)
|
||||
container=${FUEL_DB_CONTAINER:?set FUEL_DB_CONTAINER to a local fuel-db container}
|
||||
db=fuel_rules_$$
|
||||
loader=fuel_rules_loader
|
||||
|
||||
psql_as() {
|
||||
local user=$1
|
||||
shift
|
||||
docker exec -i "$container" psql -X -q -v ON_ERROR_STOP=1 -U "$user" "$@"
|
||||
}
|
||||
|
||||
cleanup() {
|
||||
psql_as postgres -d postgres -c "DROP DATABASE IF EXISTS $db;" >/dev/null
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
psql_as postgres -d postgres -c "CREATE DATABASE $db;" >/dev/null
|
||||
psql_as postgres -d postgres >/dev/null <<SQL
|
||||
DO \$\$ BEGIN
|
||||
IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = '$loader') THEN
|
||||
CREATE ROLE $loader LOGIN;
|
||||
GRANT fuel_owner TO $loader WITH INHERIT FALSE, SET TRUE;
|
||||
ALTER ROLE $loader SET role TO 'fuel_owner';
|
||||
END IF;
|
||||
END \$\$;
|
||||
ALTER DATABASE $db OWNER TO fuel_owner;
|
||||
GRANT CONNECT ON DATABASE $db TO fuel_ro;
|
||||
SQL
|
||||
psql_as postgres -d "$db" >/dev/null <<SQL
|
||||
REVOKE CREATE ON SCHEMA public FROM PUBLIC;
|
||||
GRANT USAGE ON SCHEMA public TO fuel_ro;
|
||||
ALTER DEFAULT PRIVILEGES FOR ROLE fuel_owner IN SCHEMA public GRANT SELECT ON TABLES TO fuel_ro;
|
||||
SQL
|
||||
|
||||
for file in "$here"/../migrations/*.sql; do
|
||||
psql_as "$loader" -d "$db" <"$file" >/dev/null
|
||||
done
|
||||
echo "migrations applied: $(cd "$here/../migrations" && echo *.sql)"
|
||||
|
||||
psql_as "$loader" -d "$db" <"$here/rules.sql"
|
||||
psql_as fuel_ro -d "$db" <"$here/read_only.sql" 2>&1 | sed 's/^psql:[^ ]* NOTICE: //'
|
||||
@@ -10,23 +10,6 @@ set -euo pipefail
|
||||
superset db upgrade
|
||||
superset init
|
||||
|
||||
# Trino was removed on 29 Oct 2026. An instance still holding its connection
|
||||
# "Fuel (Trino)" gets it renamed, so its datasets and dashboards keep their ids;
|
||||
# the next step points it at fuel-db. Drop this once no instance has it.
|
||||
/app/.venv/bin/python - <<'PY'
|
||||
from superset.app import create_app
|
||||
|
||||
with create_app().app_context():
|
||||
from superset import db
|
||||
from superset.models.core import Database
|
||||
|
||||
old = db.session.query(Database).filter_by(database_name="Fuel (Trino)").one_or_none()
|
||||
if old and not db.session.query(Database).filter_by(database_name="Fuel").count():
|
||||
old.database_name = "Fuel"
|
||||
db.session.commit()
|
||||
print('renamed the connection "Fuel (Trino)" to "Fuel"')
|
||||
PY
|
||||
|
||||
# No password here: superset_config.py reads it from /run/secrets at connect time.
|
||||
superset set-database-uri \
|
||||
--database_name "Fuel" \
|
||||
|
||||
@@ -69,7 +69,7 @@ services:
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.5
|
||||
command:
|
||||
- --provider=keycloak-oidc
|
||||
- --oidc-issuer-url=${KEYCLOAK_REALM_URL:-https://iam.thedigitalbridge.io/realms/thedigitalbridge}
|
||||
- --oidc-issuer-url=${KEYCLOAK_REALM_URL:-https://iam.thedigitalbridge.io/realms/petrolyst}
|
||||
- --client-id=petrolyst
|
||||
- --client-secret-file=/run/secrets/keycloak_petrolyst_client_secret
|
||||
- --cookie-secret-file=/run/secrets/oauth2_proxy_cookie_secret
|
||||
@@ -114,6 +114,6 @@ volumes:
|
||||
|
||||
secrets:
|
||||
keycloak_petrolyst_client_secret:
|
||||
file: ${PETROLYST_ROOT:-/opt/petrolyst}/secrets/shared/keycloak_petrolyst_client_secret
|
||||
file: ${PETROLYST_ROOT:-/opt/petrolyst}/secrets/shared/keycloak_petrolyst_realm_client_secret
|
||||
oauth2_proxy_cookie_secret:
|
||||
file: ${PETROLYST_ROOT:-/opt/petrolyst}/secrets/traefik/oauth2_proxy_cookie_secret
|
||||
Reference in new issue
Block a user