Compare commits
2
Commits
7d01f66f84
...
bc27bf2356
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
bc27bf2356 | ||
|
|
bf96edc828 |
No files matched your search
@@ -1,57 +0,0 @@
|
|||||||
# Every push to main deploys everything: Traefik first, then each stack under
|
|
||||||
# stacks/. The runner only uploads; the server validates, deploys and
|
|
||||||
# smoke-tests (deploy/deploy.sh), so this job needs no Docker of its own.
|
|
||||||
name: deploy
|
|
||||||
|
|
||||||
on:
|
|
||||||
push:
|
|
||||||
branches: [main]
|
|
||||||
workflow_dispatch:
|
|
||||||
|
|
||||||
# Serializes CI runs; deploy.sh's host lock also covers manual rollbacks.
|
|
||||||
concurrency:
|
|
||||||
group: deploy-petrolyst
|
|
||||||
cancel-in-progress: false
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
deploy:
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
timeout-minutes: 45
|
|
||||||
env:
|
|
||||||
DEPLOY_HOST: ${{ secrets.DEPLOY_HOST }}
|
|
||||||
steps:
|
|
||||||
- uses: actions/checkout@v4
|
|
||||||
|
|
||||||
- name: Lint shell scripts
|
|
||||||
run: |
|
|
||||||
if ! command -v shellcheck >/dev/null; then
|
|
||||||
SUDO=$(command -v sudo || true)
|
|
||||||
$SUDO apt-get update -qq && $SUDO apt-get install -y -qq shellcheck
|
|
||||||
fi
|
|
||||||
shellcheck -x host/*.sh deploy/*.sh stacks/*/superset/init.sh stacks/*/fuel-db/init/*.sh
|
|
||||||
|
|
||||||
- name: Install the deploy key
|
|
||||||
env:
|
|
||||||
DEPLOY_SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
|
|
||||||
DEPLOY_KNOWN_HOSTS: ${{ secrets.DEPLOY_KNOWN_HOSTS }}
|
|
||||||
run: |
|
|
||||||
install -d -m 0700 "$HOME/.ssh"
|
|
||||||
printf '%s\n' "$DEPLOY_SSH_KEY" > "$HOME/.ssh/deploy_key"
|
|
||||||
chmod 0600 "$HOME/.ssh/deploy_key"
|
|
||||||
printf '%s\n' "$DEPLOY_KNOWN_HOSTS" > "$HOME/.ssh/known_hosts"
|
|
||||||
command -v rsync >/dev/null || { SUDO=$(command -v sudo || true); $SUDO apt-get install -y -qq rsync; }
|
|
||||||
|
|
||||||
- name: Upload and deploy
|
|
||||||
run: |
|
|
||||||
ssh_cmd="ssh -i $HOME/.ssh/deploy_key -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes -o UserKnownHostsFile=$HOME/.ssh/known_hosts"
|
|
||||||
# A fresh directory per run; only deploy.sh, under the host lock,
|
|
||||||
# turns it into releases/<sha>. --omit-dir-times: the upload's age
|
|
||||||
# must be the server's, because retention prunes uploads by mtime.
|
|
||||||
incoming="/opt/petrolyst/incoming/${GITHUB_SHA}-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT:-1}-$(date -u +%s)"
|
|
||||||
rsync -az --omit-dir-times --exclude .git -e "$ssh_cmd" ./ "deploy@${DEPLOY_HOST}:${incoming}/"
|
|
||||||
$ssh_cmd "deploy@${DEPLOY_HOST}" "${incoming}/deploy/deploy.sh" "$GITHUB_SHA" "$incoming"
|
|
||||||
|
|
||||||
- name: Check from outside
|
|
||||||
run: |
|
|
||||||
test "$(curl -fsS --max-time 20 https://atoma.petrolyst.com/health)" = OK
|
|
||||||
curl -fsS --max-time 20 -o /dev/null -w '%{http_code}\n' https://traefik.petrolyst.com/ | grep -qx 302
|
|
||||||
@@ -1,7 +1,6 @@
|
|||||||
# petrolyst-infra
|
# petrolyst-infra
|
||||||
|
|
||||||
Everything that runs on the petrolyst server (109.199.127.84, Ubuntu 24.04), deployed on every
|
Everything that runs on the petrolyst server (109.199.127.84, Ubuntu 24.04):
|
||||||
push to `main`:
|
|
||||||
|
|
||||||
- **Traefik** at the edge for every project on the server. Its dashboard is at
|
- **Traefik** at the edge for every project on the server. Its dashboard is at
|
||||||
https://traefik.petrolyst.com, behind Keycloak.
|
https://traefik.petrolyst.com, behind Keycloak.
|
||||||
@@ -9,6 +8,9 @@ push to `main`:
|
|||||||
store `fuel-db` (Postgres 17 with TimescaleDB) through Trino. Sign-in goes through Keycloak
|
store `fuel-db` (Postgres 17 with TimescaleDB) through Trino. Sign-in goes through Keycloak
|
||||||
client `petrolyst`, and the UI carries the Petrolyst brand.
|
client `petrolyst`, and the UI carries the Petrolyst brand.
|
||||||
|
|
||||||
|
**Deploys are manual.** A person deploys a clean commit of `main` from the Mac ("Manual deploy"
|
||||||
|
under Operating). There is no CI runner, and a push to Gitea deploys nothing.
|
||||||
|
|
||||||
Customization and deployment are separate, as with media-insights and media-intelligence-docker:
|
Customization and deployment are separate, as with media-insights and media-intelligence-docker:
|
||||||
|
|
||||||
| Folder | Holds |
|
| Folder | Holds |
|
||||||
@@ -16,19 +18,19 @@ Customization and deployment are separate, as with media-insights and media-inte
|
|||||||
| `petrolyst-insights/` | how Superset looks and behaves for Petrolyst: config, Keycloak login code, branding, patches, dashboards. See its README. |
|
| `petrolyst-insights/` | how Superset looks and behaves for Petrolyst: config, Keycloak login code, branding, patches, dashboards. See its README. |
|
||||||
| `traefik/` | edge proxy, TLS, the `sso` middleware (oauth2-proxy) that guards the dashboard |
|
| `traefik/` | edge proxy, TLS, the `sso` middleware (oauth2-proxy) that guards the dashboard |
|
||||||
| `stacks/fuel-monitoring/` | Compose stack, Superset Dockerfile, Trino config and catalog, fuel-db roles |
|
| `stacks/fuel-monitoring/` | Compose stack, Superset Dockerfile, Trino config and catalog, fuel-db roles |
|
||||||
| `deploy/` | `deploy.sh` (run by CI), `rollback.sh` (manual), `lib.sh` (shared) |
|
| `deploy/` | `deploy.sh` (run on the server by a manual deploy), `rollback.sh` (manual), `lib.sh` (shared) |
|
||||||
| `host/` | one-time server setup: `bootstrap.sh`, `secrets-init.sh` |
|
| `host/` | one-time server setup: `bootstrap.sh`, `secrets-init.sh` |
|
||||||
| `keycloak/` | the `petrolyst` client for Keycloak's partial import |
|
| `keycloak/` | the `petrolyst` client for Keycloak's partial import |
|
||||||
| `.gitea/workflows/deploy.yml` | the pipeline |
|
|
||||||
|
|
||||||
No credential is in this repository. Secrets exist only on the server, under
|
No credential is in this repository. Secrets exist only on the server, under
|
||||||
`/opt/petrolyst/secrets/`; CI holds only the SSH deploy key.
|
`/opt/petrolyst/secrets/`. A deploy needs only root SSH access to the server (`ssh petrolyst`).
|
||||||
|
|
||||||
## How a deploy works
|
## How a deploy works
|
||||||
|
|
||||||
1. A push to `main` (or a manual run) starts `.gitea/workflows/deploy.yml` on the act_runner. It
|
1. A person runs "Manual deploy" (Operating, below) from a clean checkout of `main`: the scripts
|
||||||
lints the scripts, rsyncs the checkout to a fresh `/opt/petrolyst/incoming/<sha>-<run>/`, and
|
are linted, `git archive` sends exactly that commit to a fresh
|
||||||
calls `deploy/deploy.sh <sha> <that dir>` over SSH as user `deploy`.
|
`/opt/petrolyst/incoming/<sha>-manual-<time>/`, and `deploy/deploy.sh <sha> <that dir>` runs
|
||||||
|
there as user `deploy` (root hands over with `sudo -u deploy`).
|
||||||
2. `deploy.sh` takes the host lock `/opt/petrolyst/deploy.lock`, which it shares with
|
2. `deploy.sh` takes the host lock `/opt/petrolyst/deploy.lock`, which it shares with
|
||||||
`rollback.sh`, and keeps it until it exits.
|
`rollback.sh`, and keeps it until it exits.
|
||||||
3. It finalizes the upload. A new commit is moved to `releases/<sha>`, and the release is made
|
3. It finalizes the upload. A new commit is moved to `releases/<sha>`, and the release is made
|
||||||
@@ -52,7 +54,7 @@ No credential is in this repository. Secrets exist only on the server, under
|
|||||||
`superset`, writes `backups/superset-meta_<old tag>_<revision>_<time>.dump`, then runs
|
`superset`, writes `backups/superset-meta_<old tag>_<revision>_<time>.dump`, then runs
|
||||||
`superset-init` (`db upgrade`, `init`, registers "Fuel (Trino)").
|
`superset-init` (`db upgrade`, `init`, registers "Fuel (Trino)").
|
||||||
- If the image is unchanged and the schema is at its head, it does none of this.
|
- If the image is unchanged and the schema is at its head, it does none of this.
|
||||||
8. Smoke tests. A failure turns CI red and prints the stacks' state.
|
8. Smoke tests. A failure ends the deploy with an error and prints the stacks' state.
|
||||||
- `/health` answers through Traefik with a valid certificate.
|
- `/health` answers through Traefik with a valid certificate.
|
||||||
- `/login/` leads to Keycloak with `client_id=petrolyst` and PKCE.
|
- `/login/` leads to Keycloak with `client_id=petrolyst` and PKCE.
|
||||||
- The dashboard redirects to Keycloak.
|
- The dashboard redirects to Keycloak.
|
||||||
@@ -63,14 +65,14 @@ No credential is in this repository. Secrets exist only on the server, under
|
|||||||
- the 3 newest Superset images, plus any image that a kept release or backup names;
|
- the 3 newest Superset images, plus any image that a kept release or backup names;
|
||||||
- uploads older than a day.
|
- uploads older than a day.
|
||||||
|
|
||||||
The runner then checks both hosts from outside.
|
The manual deploy then checks both hosts from outside.
|
||||||
|
|
||||||
## Server paths
|
## Server paths
|
||||||
|
|
||||||
| Path | Owner | What |
|
| Path | Owner | What |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `/opt/petrolyst/infra` | deploy | live tree (the promoted release) |
|
| `/opt/petrolyst/infra` | deploy | live tree (the promoted release) |
|
||||||
| `/opt/petrolyst/incoming/<sha>-<run>` | deploy | one CI upload; only `deploy.sh` turns it into a release |
|
| `/opt/petrolyst/incoming/<sha>-manual-<time>` | deploy | one upload; only `deploy.sh` turns it into a release |
|
||||||
| `/opt/petrolyst/releases/<sha>` | deploy, read-only | finalized release plus `ARTIFACTS`; the last 5 are kept |
|
| `/opt/petrolyst/releases/<sha>` | deploy, read-only | finalized release plus `ARTIFACTS`; the last 5 are kept |
|
||||||
| `/opt/petrolyst/secrets/{shared,traefik,fuel-monitoring}/` | root:deploy 0750, files 0444 | secrets, mounted as `/run/secrets/<name>` |
|
| `/opt/petrolyst/secrets/{shared,traefik,fuel-monitoring}/` | root:deploy 0750, files 0444 | secrets, mounted as `/run/secrets/<name>` |
|
||||||
| `/opt/petrolyst/env/fuel-monitoring.env` | root:deploy 0440 | `FUEL_DB_TRINO_PASSWORD` for Trino's `${ENV:…}` |
|
| `/opt/petrolyst/env/fuel-monitoring.env` | root:deploy 0440 | `FUEL_DB_TRINO_PASSWORD` for Trino's `${ENV:…}` |
|
||||||
@@ -81,24 +83,9 @@ The runner then checks both hosts from outside.
|
|||||||
|
|
||||||
### 1. Gitea
|
### 1. Gitea
|
||||||
|
|
||||||
1. Create the private repository `Petrolyst/petrolyst-infra` and push this repository's
|
Create the private repository `Petrolyst/petrolyst-infra` and push this repository's `main` to
|
||||||
`main` to it. Enable Actions under Settings → Repository → Units.
|
it. Gitea only keeps the history: deploys are manual, so it needs no runner, no Actions and no
|
||||||
2. Runner. Check Site/Org Administration → Actions → Runners. If none is registered, run
|
deploy secrets.
|
||||||
[Gitea Runner](https://docs.gitea.com/runner/installation/docker/) as the container
|
|
||||||
`docker.io/gitea/runner:5.2.2-dind` (privileged, with `/data` and `/var/lib/docker` volumes),
|
|
||||||
for example on the Gitea host (164.68.121.117):
|
|
||||||
- register it as an **organization runner** for `Petrolyst`: take the token from
|
|
||||||
Settings → Actions → Runners → Create new Runner;
|
|
||||||
- set `GITEA_RUNNER_LABELS=ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latest`;
|
|
||||||
- note its egress IPv4 (`curl -4 -s https://ifconfig.me`).
|
|
||||||
3. Generate a deploy key on your Mac, used only for this:
|
|
||||||
`ssh-keygen -t ed25519 -N '' -C petrolyst-ci -f petrolyst-ci`
|
|
||||||
4. Add repository secrets (Settings → Actions → Secrets):
|
|
||||||
- `DEPLOY_SSH_KEY`: contents of `petrolyst-ci` (the private key)
|
|
||||||
- `DEPLOY_HOST`: `109.199.127.84`
|
|
||||||
- `DEPLOY_KNOWN_HOSTS`: output of `ssh-keyscan -t ed25519 109.199.127.84`. Compare its
|
|
||||||
fingerprint (`ssh-keygen -lf <file>`) with `ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub`
|
|
||||||
run on the server.
|
|
||||||
|
|
||||||
### 2. Keycloak (realm `thedigitalbridge` on iam.thedigitalbridge.io)
|
### 2. Keycloak (realm `thedigitalbridge` on iam.thedigitalbridge.io)
|
||||||
|
|
||||||
@@ -132,24 +119,21 @@ The server needs only the two host scripts; it never gets Gitea credentials.
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
# on the Mac, in this repository
|
# on the Mac, in this repository
|
||||||
scp host/bootstrap.sh host/secrets-init.sh petrolyst-ci.pub petrolyst:/root/
|
scp host/bootstrap.sh host/secrets-init.sh petrolyst:/root/
|
||||||
ssh petrolyst
|
ssh petrolyst
|
||||||
# on the server, as root
|
# on the server, as root
|
||||||
/root/bootstrap.sh --ci-key /root/petrolyst-ci.pub --runner-ip <runner egress IP>
|
/root/bootstrap.sh
|
||||||
/root/secrets-init.sh # generates every secret; reports the missing Keycloak one
|
/root/secrets-init.sh # generates every secret; reports the missing Keycloak one
|
||||||
/root/secrets-init.sh keycloak-secret # paste the secret from Keycloak step 2
|
/root/secrets-init.sh keycloak-secret # paste the secret from Keycloak step 2
|
||||||
```
|
```
|
||||||
|
|
||||||
No runner yet? Pass `--runner-ip 127.0.0.1`. That installs the CI key but leaves it unusable from
|
|
||||||
outside. Deploy manually until a runner exists (see "Manual deploy" below). When the runner is
|
|
||||||
running, run `bootstrap.sh` again with its egress IP.
|
|
||||||
|
|
||||||
`bootstrap.sh` does the following:
|
`bootstrap.sh` does the following:
|
||||||
- installs Docker from Docker's apt repository;
|
- installs Docker from Docker's apt repository;
|
||||||
- adds 4 GB swap;
|
- adds 4 GB swap;
|
||||||
- turns on ufw for 22, 80 and 443 only;
|
- turns on ufw for 22, 80 and 443 only;
|
||||||
- turns off SSH password login and root password login;
|
- turns off SSH password login and root password login;
|
||||||
- creates the `deploy` user, whose key works only from the runner IP;
|
- creates the `deploy` user, which runs `deploy.sh`. It has no SSH key and is reached only
|
||||||
|
through `sudo -u deploy` from root; a re-run removes the CI key an earlier version installed;
|
||||||
- creates `/opt/petrolyst` and the Docker network `proxy`.
|
- creates `/opt/petrolyst` and the Docker network `proxy`.
|
||||||
|
|
||||||
**Before closing your root session**, check from a second terminal that `ssh petrolyst` still
|
**Before closing your root session**, check from a second terminal that `ssh petrolyst` still
|
||||||
@@ -157,8 +141,7 @@ works.
|
|||||||
|
|
||||||
### 4. First deploy
|
### 4. First deploy
|
||||||
|
|
||||||
Run the workflow manually (Actions → deploy → Run workflow) or push to `main`. Without a runner,
|
Use "Manual deploy" below. The first run builds the Superset image and takes the fresh-install
|
||||||
use "Manual deploy" below. The first run builds the Superset image and takes the fresh-install
|
|
||||||
path. Let's Encrypt needs about a minute
|
path. Let's Encrypt needs about a minute
|
||||||
for the two certificates; the smoke tests wait up to two minutes.
|
for the two certificates; the smoke tests wait up to two minutes.
|
||||||
|
|
||||||
@@ -169,7 +152,7 @@ becomes the first Superset admin; there is no local admin account.
|
|||||||
|
|
||||||
| Task | How |
|
| Task | How |
|
||||||
|---|---|
|
|---|---|
|
||||||
| Deploy | push to `main` (without a runner: "Manual deploy" below) |
|
| Deploy | "Manual deploy" below, from a clean `main` |
|
||||||
| Re-deploy a kept release | `sudo -u deploy /opt/petrolyst/infra/deploy/deploy.sh <sha>` |
|
| Re-deploy a kept release | `sudo -u deploy /opt/petrolyst/infra/deploy/deploy.sh <sha>` |
|
||||||
| Roll back | see below |
|
| Roll back | see below |
|
||||||
| Traefik dashboard | https://traefik.petrolyst.com (role `traefik-admin`) |
|
| Traefik dashboard | https://traefik.petrolyst.com (role `traefik-admin`) |
|
||||||
@@ -178,21 +161,41 @@ becomes the first Superset admin; there is no local admin account.
|
|||||||
| Logs | `docker compose -p fuel-monitoring logs -f superset` (or `-p traefik`) |
|
| Logs | `docker compose -p fuel-monitoring logs -f superset` (or `-p traefik`) |
|
||||||
| Superset break-glass | `docker exec -it fuel-monitoring-superset-1 superset fab ...` |
|
| Superset break-glass | `docker exec -it fuel-monitoring-superset-1 superset fab ...` |
|
||||||
|
|
||||||
**Manual deploy** (no runner): commit first, so that `git status` is clean. Then run this on the
|
**Manual deploy**, the only way to deploy (there is no CI runner). On the Mac, in this
|
||||||
Mac, in this repository:
|
repository:
|
||||||
|
|
||||||
```bash
|
1. Commit on `main`, so that `git status` is clean. The live commit is in
|
||||||
sha=$(git rev-parse HEAD)
|
`/opt/petrolyst/state/deployed-sha` (`ssh petrolyst cat /opt/petrolyst/state/deployed-sha`).
|
||||||
dir=/opt/petrolyst/incoming/$sha-manual-$(date -u +%s)
|
2. Lint the shell scripts. Warnings and errors must be zero; the info notes on the deliberate
|
||||||
git archive --format=tar "$sha" |
|
`test && test || die` guards in `deploy/lib.sh` (SC2015) are filtered out:
|
||||||
ssh petrolyst "install -d -o deploy -g deploy -m 0755 $dir && tar -x --no-same-owner -C $dir && chown -R deploy:deploy $dir"
|
|
||||||
ssh petrolyst "sudo -u deploy $dir/deploy/deploy.sh $sha $dir"
|
|
||||||
```
|
|
||||||
|
|
||||||
`git archive` sends exactly the commit, never uncommitted or ignored files such as the deploy
|
```bash
|
||||||
key. The server then runs the same `deploy.sh` that CI runs, under the same lock. A later push
|
docker run --rm -v "$PWD:/mnt" -w /mnt koalaman/shellcheck:v0.10.0 -S warning -x \
|
||||||
queues a CI run that waits for a runner. Cancel stale waiting runs under Actions before you
|
host/*.sh deploy/*.sh stacks/*/superset/init.sh stacks/*/fuel-db/init/*.sh
|
||||||
register one.
|
```
|
||||||
|
|
||||||
|
3. Deploy:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sha=$(git rev-parse HEAD)
|
||||||
|
dir=/opt/petrolyst/incoming/$sha-manual-$(date -u +%s)
|
||||||
|
git archive --format=tar "$sha" |
|
||||||
|
ssh petrolyst "install -d -o deploy -g deploy -m 0755 $dir && tar -x --no-same-owner -C $dir && chown -R deploy:deploy $dir"
|
||||||
|
ssh petrolyst "sudo -u deploy $dir/deploy/deploy.sh $sha $dir"
|
||||||
|
```
|
||||||
|
|
||||||
|
4. Check both hosts from outside:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
test "$(curl -fsS --max-time 20 https://atoma.petrolyst.com/health)" = OK && echo atoma ok
|
||||||
|
curl -fsS --max-time 20 -o /dev/null -w '%{http_code}\n' https://traefik.petrolyst.com/ # 302
|
||||||
|
```
|
||||||
|
|
||||||
|
5. Push `main` to Gitea, so that the repository matches what is live.
|
||||||
|
|
||||||
|
`git archive` sends exactly the commit, never uncommitted or ignored files. The server runs
|
||||||
|
`deploy.sh` under the host lock it shares with `rollback.sh`, and records the live commit in
|
||||||
|
`/opt/petrolyst/state/deployed-sha`.
|
||||||
|
|
||||||
**Rollback** (`deploy/rollback.sh`, manual, under the same lock as deploys):
|
**Rollback** (`deploy/rollback.sh`, manual, under the same lock as deploys):
|
||||||
|
|
||||||
@@ -291,8 +294,7 @@ Results:
|
|||||||
Not verified locally:
|
Not verified locally:
|
||||||
- Let's Encrypt issuance;
|
- Let's Encrypt issuance;
|
||||||
- a real Keycloak sign-in, OTP and logout (they need the client from setup step 2);
|
- a real Keycloak sign-in, OTP and logout (they need the client from setup step 2);
|
||||||
- the theme as rendered in a browser;
|
- the theme as rendered in a browser.
|
||||||
- the Gitea runner.
|
|
||||||
|
|
||||||
## Open items
|
## Open items
|
||||||
|
|
||||||
|
|||||||
+6
-6
@@ -1,6 +1,6 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# Server-side deploy, run as the non-root deploy user:
|
# Server-side deploy, run as the non-root deploy user:
|
||||||
# deploy.sh <sha> <incoming-dir> CI: finalize the upload as releases/<sha>, then deploy it
|
# deploy.sh <sha> <incoming-dir> manual deploy: finalize the upload as releases/<sha>, then deploy it
|
||||||
# deploy.sh <sha> re-deploy a release that is already finalized
|
# deploy.sh <sha> re-deploy a release that is already finalized
|
||||||
#
|
#
|
||||||
# Order: lock, finalize, artifacts, preflight, image build and downgrade guard
|
# Order: lock, finalize, artifacts, preflight, image build and downgrade guard
|
||||||
@@ -11,7 +11,7 @@ set -euo pipefail
|
|||||||
# shellcheck source=deploy/lib.sh
|
# shellcheck source=deploy/lib.sh
|
||||||
. "$(dirname "$(readlink -f "$0")")/lib.sh"
|
. "$(dirname "$(readlink -f "$0")")/lib.sh"
|
||||||
|
|
||||||
# finalize_release <sha> <upload>: CI never writes into releases/. A new SHA is
|
# finalize_release <sha> <upload>: an upload never writes into releases/. A new SHA is
|
||||||
# moved in atomically; a re-run of a deployed SHA must match it byte for byte
|
# moved in atomically; a re-run of a deployed SHA must match it byte for byte
|
||||||
# and leaves the finalized release, ARTIFACTS included, untouched.
|
# and leaves the finalized release, ARTIFACTS included, untouched.
|
||||||
finalize_release() {
|
finalize_release() {
|
||||||
@@ -100,10 +100,10 @@ retention() {
|
|||||||
rm -rf "$dir"
|
rm -rf "$dir"
|
||||||
log "removed release ${dir##*/}"
|
log "removed release ${dir##*/}"
|
||||||
done < <(find "$RELEASES" -mindepth 1 -maxdepth 1 -type d -printf '%T@ %p\n' | sort -rn | tail -n +6 | cut -d' ' -f2-)
|
done < <(find "$RELEASES" -mindepth 1 -maxdepth 1 -type d -printf '%T@ %p\n' | sort -rn | tail -n +6 | cut -d' ' -f2-)
|
||||||
# Uploads older than a day were never finalized (interrupted CI runs). Their
|
# Uploads older than a day were never finalized (interrupted deploys). The
|
||||||
# mtime is the server's (the workflow's rsync omits directory times), and a
|
# upload directory is created on the server when the upload starts, and its
|
||||||
# CI job lives at most 45 minutes, so an upload being sent or waiting for the
|
# mtime is the server's, so an upload being sent or waiting for the lock is
|
||||||
# lock is never this old.
|
# never this old.
|
||||||
while IFS= read -r -d '' dir; do
|
while IFS= read -r -d '' dir; do
|
||||||
chmod -R u+w "$dir"
|
chmod -R u+w "$dir"
|
||||||
rm -rf "$dir"
|
rm -rf "$dir"
|
||||||
|
|||||||
+2
-2
@@ -57,8 +57,8 @@ die() { log "ERROR: $*"; exit 1; }
|
|||||||
# is absolute, so nothing needs the caller's directory.
|
# is absolute, so nothing needs the caller's directory.
|
||||||
cd /
|
cd /
|
||||||
|
|
||||||
# acquire_lock <seconds>: deploy.sh and rollback.sh share this lock, so a CI
|
# acquire_lock <seconds>: deploy.sh and rollback.sh share this lock, so a
|
||||||
# deploy and a manual rollback never run at the same time. Held until exit.
|
# deploy and a rollback never run at the same time. Held until exit.
|
||||||
acquire_lock() {
|
acquire_lock() {
|
||||||
exec 9>>"$LOCK_FILE"
|
exec 9>>"$LOCK_FILE"
|
||||||
flock -w "$1" 9 || die "another deploy or rollback holds $LOCK_FILE (waited $1 s)"
|
flock -w "$1" 9 || die "another deploy or rollback holds $LOCK_FILE (waited $1 s)"
|
||||||
|
|||||||
+8
-23
@@ -2,27 +2,17 @@
|
|||||||
# One-time host preparation for the petrolyst server (Ubuntu 24.04), run as root.
|
# One-time host preparation for the petrolyst server (Ubuntu 24.04), run as root.
|
||||||
# Idempotent: re-running converges to the same state.
|
# Idempotent: re-running converges to the same state.
|
||||||
#
|
#
|
||||||
# bootstrap.sh --ci-key <public-key-file> --runner-ip <ip|any>
|
# bootstrap.sh
|
||||||
#
|
#
|
||||||
# --ci-key public half of the dedicated CI deploy key (DEPLOY_SSH_KEY in Gitea)
|
# Deploys are manual: a person runs them over root SSH, and root hands over to
|
||||||
# --runner-ip egress IP of the Gitea act_runner; the deploy key works only from
|
# the deploy user (README, "Manual deploy"). There is no CI runner, so deploy
|
||||||
# there ("any" drops that restriction, e.g. before the runner exists)
|
# has no SSH key of its own.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
die() { echo "ERROR: $*" >&2; exit 1; }
|
die() { echo "ERROR: $*" >&2; exit 1; }
|
||||||
[ "$(id -u)" = 0 ] || die "run as root"
|
[ "$(id -u)" = 0 ] || die "run as root"
|
||||||
|
|
||||||
ci_key='' runner_ip=''
|
[ $# -eq 0 ] || die "bootstrap.sh takes no arguments (deploys are manual; there is no CI key)"
|
||||||
while [ $# -gt 0 ]; do
|
|
||||||
case $1 in
|
|
||||||
--ci-key) ci_key=$2; shift 2 ;;
|
|
||||||
--runner-ip) runner_ip=$2; shift 2 ;;
|
|
||||||
*) die "unknown argument $1" ;;
|
|
||||||
esac
|
|
||||||
done
|
|
||||||
[ -s "$ci_key" ] || die "--ci-key <public-key-file> is required"
|
|
||||||
[ -n "$runner_ip" ] || die "--runner-ip <ip|any> is required"
|
|
||||||
grep -qE '^ssh-(ed25519|rsa) ' "$ci_key" || die "$ci_key does not look like an SSH public key"
|
|
||||||
|
|
||||||
ROOT=/opt/petrolyst
|
ROOT=/opt/petrolyst
|
||||||
|
|
||||||
@@ -69,15 +59,10 @@ ufw default allow outgoing >/dev/null
|
|||||||
for port in 22 80 443; do ufw allow "$port/tcp" >/dev/null; done
|
for port in 22 80 443; do ufw allow "$port/tcp" >/dev/null; done
|
||||||
ufw --force enable >/dev/null
|
ufw --force enable >/dev/null
|
||||||
|
|
||||||
echo "== deploy user (CI)"
|
echo "== deploy user (runs deploy.sh; reached only through sudo from root)"
|
||||||
id deploy >/dev/null 2>&1 || useradd --create-home --shell /bin/bash deploy
|
id deploy >/dev/null 2>&1 || useradd --create-home --shell /bin/bash deploy
|
||||||
usermod -aG docker deploy # docker access makes deploy root-equivalent; the key restrictions below matter
|
usermod -aG docker deploy # docker access makes deploy root-equivalent, so it gets no SSH key
|
||||||
install -d -o deploy -g deploy -m 0700 /home/deploy/.ssh
|
rm -f /home/deploy/.ssh/authorized_keys # the former CI key, if an earlier bootstrap installed it
|
||||||
options=restrict
|
|
||||||
[ "$runner_ip" = any ] || options="restrict,from=\"$runner_ip\""
|
|
||||||
printf '%s %s\n' "$options" "$(cat "$ci_key")" >/home/deploy/.ssh/authorized_keys
|
|
||||||
chown deploy:deploy /home/deploy/.ssh/authorized_keys
|
|
||||||
chmod 0600 /home/deploy/.ssh/authorized_keys
|
|
||||||
|
|
||||||
echo "== SSH: keys only"
|
echo "== SSH: keys only"
|
||||||
[ -s /root/.ssh/authorized_keys ] || die "root has no authorized_keys; refusing to turn off password login"
|
[ -s /root/.ssh/authorized_keys ] || die "root has no authorized_keys; refusing to turn off password login"
|
||||||
|
|||||||
@@ -0,0 +1,272 @@
|
|||||||
|
# ATOMA dashboards: Sites Map and Site Detail
|
||||||
|
|
||||||
|
**Status: proposed, not built.**
|
||||||
|
- `build_bundle.py` (phase P2) will generate both dashboards, every dataset and every chart from
|
||||||
|
this specification, as one Superset import bundle.
|
||||||
|
- Nothing is imported into the live Superset without the owner's go-ahead (phase P3).
|
||||||
|
|
||||||
|
**Datasets.** The `public.v_*` views of
|
||||||
|
`stacks/fuel-monitoring/fuel-db/migrations/0003_bi_views.sql`, read through Trino as
|
||||||
|
`fuel.public`.
|
||||||
|
|
||||||
|
**Design and rationale** live in the fuel-monitoring corpus, in
|
||||||
|
`.context/plans/2026-10-24-fuel-db-baseline-dashboard.md`:
|
||||||
|
- its §6 is this specification;
|
||||||
|
- the site-specific facts left out here (positions, distances, pilot sites) are there too;
|
||||||
|
- Q-numbers refer to its §7, and phases P0–P5 to its §8.
|
||||||
|
|
||||||
|
Two dashboards replace the earlier tab-per-site design; nothing in them is specific to a site.
|
||||||
|
|
||||||
|
## 1. Capabilities of the installed Superset 6.1.0
|
||||||
|
|
||||||
|
Source: read-only inspection of `config.py`, `superset_config.py` and the frontend bundle on
|
||||||
|
24 Oct 2026. Items marked *(local test)* are confirmed in P2 before the design relies on them.
|
||||||
|
|
||||||
|
| Requirement | What 6.1.0 provides | Choice |
|
||||||
|
|---|---|---|
|
||||||
|
| Map chart | `deck_scatter` (deck.gl Scatterplot): a modern plugin with the InteractiveChart behaviour. Also: deck_multi, deck_screengrid / hex / grid, country_map, world_map, cartodiagram, legacy `mapbox` | **deck.gl Scatterplot** |
|
||||||
|
| Basemap | `DECKGL_BASE_MAP` unset → OSM raster (`tile.openstreetmap.org`), OSM Swiss topo, Mapbox styles. `MAPBOX_API_KEY` is empty, so the Mapbox styles fail. CSP `connect-src` already allows the OSM host; Traefik sets no CSP | set `DECKGL_BASE_MAP` to **OSM only**; "© OpenStreetMap contributors" in a Markdown under the map |
|
||||||
|
| OSM terms | no key, no cost; tile usage policy: light use, attribution, no bulk download, a valid Referer (ours sends the origin) | fine for an internal user base; viewers' browsers reveal the viewed area (tile coordinates) and their IP to OSMF, not our data. Self-hosted tiles later if needed |
|
||||||
|
| Tooltip | "Tooltip contents" (drag columns) plus "Customize tooltips template" (Handlebars). `js_tooltip` exists but `ENABLE_JAVASCRIPT_CONTROLS` is False | contents and template; **no JavaScript** |
|
||||||
|
| Marker colour | "Categorical color" by a dimension, from the colour scheme; a fixed colour per label through the dashboard's `label_colors` *(local test)* | colour by `baseline_status_label`. Fallback: `deck_multi` with one fixed-colour scatter layer per status |
|
||||||
|
| Marker size | fixed radius, with minimum = maximum pixel radius | uniform |
|
||||||
|
| Viewport | `autozoom` fits the filtered points | on |
|
||||||
|
| Clustering | none in deck.gl; only the legacy `mapbox` chart (`clustering_radius`, needs a Mapbox key) | **no clustering** (§3) |
|
||||||
|
| Marker click | emits a **cross-filter** on the latitude and longitude columns: the exact point, or a box around the click that shrinks as you zoom in. It applies to other charts on the same dashboard | cross-filters the Sites list |
|
||||||
|
| Navigation | **no chart-to-dashboard navigation.** A dashboard reads `?native_filters=<rison>` into preselected native filter values (also `native_filters_key`). Table cells render sanitized HTML (`allow_render_html`) | an "Open site detail" link column in the Sites list |
|
||||||
|
| Native filters | Select (single or multi, required, searchable, "first value by default"), Time range; scoping per chart and tab; cross-filter scoping per chart | §4, §5 |
|
||||||
|
|
||||||
|
No custom frontend and no patch are needed.
|
||||||
|
|
||||||
|
## 2. Navigation flow
|
||||||
|
|
||||||
|
```
|
||||||
|
ATOMA — Sites (landing, slug atoma-sites)
|
||||||
|
filters: Site search · Zone · Province · Pilot status (default: active) · Baseline status
|
||||||
|
M1 map: markers fitted to the filtered sites ──hover──► tooltip
|
||||||
|
└─click marker──► cross-filter: M3 Sites list shows that site (or the few under the click)
|
||||||
|
M3 Sites list ─────────────"Open site detail"──► ATOMA — Site Detail ?native_filters=(site preset)
|
||||||
|
M4/M5 Sites without an accepted location ──────"Open site detail"──► same
|
||||||
|
ATOMA — Site Detail (slug atoma-site-detail)
|
||||||
|
filters: Site (exactly one, searchable) · Viewing period
|
||||||
|
tabs: Summary · Fuel & Deliveries · Runtime & Electricity · Baseline & Evidence
|
||||||
|
header link "← Sites map"
|
||||||
|
```
|
||||||
|
|
||||||
|
## 3. Sites Map (landing dashboard)
|
||||||
|
|
||||||
|
```
|
||||||
|
[Filter bar: Site search · Zone · Province · Pilot status · Baseline status]
|
||||||
|
[M1 Map (deck.gl Scatterplot) ................................ w8][M2 Status key + attribution w4]
|
||||||
|
[M3 Sites list (table with link column) ..................................................... w12]
|
||||||
|
[M4 Sites without accepted location: count w3][M5 Sites without accepted location (table) .. w9]
|
||||||
|
```
|
||||||
|
|
||||||
|
| # | Question | Type | Dataset | Content | Missing or insufficient |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| M1 | Where are the sites, and what is each one's baseline status? | deck.gl Scatterplot; lat/lon from `latitude`, `longitude`; uniform size; colour = `baseline_status_label`; autozoom | v_site | tooltip: `site_code`, `site_name`, `lph_estimate` ("not available" when NULL), `lph_method`, `baseline_status_label`, `baseline_validity_note` (when set), `latest_record_on` | sites without an accepted location have NULL lat/lon and are not plotted; M4/M5 show them |
|
||||||
|
| M2 | What do the colours mean? | Markdown (static) | — | the status definitions below; "Colours show evidence status, not performance"; OSM attribution | — |
|
||||||
|
| M3 | Which sites match, and how do I open one? | Table; search box; server pagination | v_site | code, name, province, pilot status, baseline status, L/h (+ method), L/day, latest record, location status, `detail_link` | marker click narrows it; "not available" shown as text |
|
||||||
|
| M4 | How many sites lack an accepted location? | Big Number Total, COUNT(*) with saved filter `location_status <> 'accepted'` | v_site | count; follows the dashboard filters | 0 is shown as 0 |
|
||||||
|
| M5 | Which sites are they? | Table; search box | v_site, same saved filter | code, name, province, `location_status` (awaiting acceptance · missing), baseline status, `detail_link` | — |
|
||||||
|
|
||||||
|
**Filters** (all on `v_site`, scoped to M1, M3, M4 and M5):
|
||||||
|
- Site search: Select on `site_label`, multi, searchable. The label holds code, name and uid, so
|
||||||
|
code or name both match.
|
||||||
|
- Zone, Province: Select.
|
||||||
|
- Pilot status: default `active`.
|
||||||
|
- Baseline status: Select on `baseline_status_label`.
|
||||||
|
|
||||||
|
The cross-filter from M1 is scoped **to M3 only**. M4 and M5 hold the sites without coordinates and
|
||||||
|
must not be emptied by it.
|
||||||
|
|
||||||
|
**Baseline status and marker colours.** Colours are neutral categorical hues, with no red or green
|
||||||
|
and no reference to L/h:
|
||||||
|
|
||||||
|
| Status | Definition (`fm.v_site_baseline_status`) | Colour |
|
||||||
|
|---|---|---|
|
||||||
|
| Approved baseline | an approved `baseline_version` with a decision reference. The **only** state that says "approved" | teal #009688 |
|
||||||
|
| Eligible – not yet approved | no approval; the latest proposed KPI version has `meets_policy` (BP-1: ≥ 2 eligible G1 inputs, ≥ 60 matched days, no conditions, one population and metric) | blue #3987e5 |
|
||||||
|
| Provisional | no approval; a proposed version below BP-1, **or** usable intervals with no version yet (no estimate shown, method "no baseline version proposed") | amber #c98500 |
|
||||||
|
| Insufficient evidence | no usable interval. **Every pilot site while no runtime evidence exists** | slate #64748B |
|
||||||
|
|
||||||
|
**Nearby sites.**
|
||||||
|
- Sites a few kilometres apart separate at zoom ≥ 11 (≈ 60 px per 2–4 km at zoom 12) and coincide
|
||||||
|
at country zoom (≈ 2 km per pixel at zoom 6). The pilot's own distances are in the design doc.
|
||||||
|
- Handling:
|
||||||
|
- autozoom to the filtered sites (the default "active" filter opens on the pilot area);
|
||||||
|
- small uniform markers (min = max radius, about 7 px) with an outline;
|
||||||
|
- a click near overlapping markers selects every site in the click box, and M3 lists them;
|
||||||
|
- no clustering.
|
||||||
|
- With hundreds of sites, a `deck_multi` density layer (screengrid) for low zoom is a later option.
|
||||||
|
|
||||||
|
## 4. Site Detail (one dashboard, any site)
|
||||||
|
|
||||||
|
**Filters**
|
||||||
|
- **Site** (`NATIVE_FILTER-site`):
|
||||||
|
- Select on `v_site.site_label`, single value, **required**, searchable, no default;
|
||||||
|
- scope: every chart on every tab;
|
||||||
|
- every detail dataset carries `site_label` from `v_site`, so this one filter drives all of
|
||||||
|
them *(local test)*.
|
||||||
|
- **Viewing period** (`NATIVE_FILTER-period`):
|
||||||
|
- Time range, default no filter;
|
||||||
|
- scope: the time-series charts on Fuel & Deliveries and Runtime & Electricity only;
|
||||||
|
- Summary and Baseline & Evidence are outside its scope, so a viewing range never changes a
|
||||||
|
baseline's fixed window.
|
||||||
|
|
||||||
|
**Exactly one site, and never network totals:**
|
||||||
|
1. A required, single-value filter *(local test: with the value cleared, no chart queries
|
||||||
|
unfiltered data)*.
|
||||||
|
2. The header (D0) and KPI (S1) are Handlebars templates that render a site only when exactly one
|
||||||
|
row arrives. Otherwise they show "Select one site".
|
||||||
|
3. No Big Number or aggregate tile for site figures. They are computed per site in `v_site`,
|
||||||
|
never aggregated by Superset.
|
||||||
|
4. Fallback, if (1) fails its test: "Select first filter value by default", so the Site filter
|
||||||
|
is never empty.
|
||||||
|
|
||||||
|
```
|
||||||
|
[D0 Site header: code + name (large) · province · pilot status · baseline status · "← Sites map"]
|
||||||
|
Summary [S1 L/h KPI card w6][S2 Supporting measures w6]
|
||||||
|
[S3 Evidence coverage timeline (Gantt) w12]
|
||||||
|
Fuel & Deliveries [F1 Deliveries w6][F2 Recorded stock w6]
|
||||||
|
[F3 Fuel per calendar day by interval w12]
|
||||||
|
[F4 Interval table w12]
|
||||||
|
Runtime & Electricity[R1 Runtime per day by meter interval w6][R2 Hour-meter evidence w6]
|
||||||
|
[R3 Grid energy, kWh w6][R4 Electricity cost, AFN w6]
|
||||||
|
Baseline & Evidence [B1 Baseline versions w6][B2 Inputs of each version w6]
|
||||||
|
[B3 All intervals: eligibility and exclusions w12]
|
||||||
|
[B4 Interval L/h against baseline w12]
|
||||||
|
[B5 Assumptions w6][B6 Missing data and quality checks w6]
|
||||||
|
[B7 Evidence references w6][B8 Workbook calculations, assessed w6]
|
||||||
|
[B9 Stock against engine-measured fuel w6][B10 Equipment specifications w6]
|
||||||
|
```
|
||||||
|
|
||||||
|
| # | Question it answers | Type | Dataset | Dims / measures (units) | Grain | When data are missing or insufficient |
|
||||||
|
|---|---|---|---|---|---|---|
|
||||||
|
| D0 | Which site am I looking at? | Handlebars | v_site | code and name (large), province, pilot status, baseline status, location status, link "← Sites map" | site | "Select one site" unless exactly one row |
|
||||||
|
| S1 | What is the site's L/h, by which method, over which window, how strong? | Handlebars | v_site | `lph_estimate`, `lph_method`, `baseline_status_label`, `baseline_validity_note`, window, decision; L/day whole record | site | Keeps its position. **In v1 every site shows "Insufficient runtime evidence: no hour-meter readings"**, plus L/day labelled "per calendar day, not per running hour". "Eligible" and "Provisional" carry "not approved" |
|
||||||
|
| S2 | What supports it? | Table | v_kpi_measures | measure, value, unit, `value_kind` (recorded · calculated · assumed · not established), window. Two L/day rows: the whole record (moves with new records) and the L/day reference version (fixed window, its status and any input drift in the note) | site × measure | Rows stay and show "not established" |
|
||||||
|
| S3 | Which periods are covered by which evidence, and where are the gaps? | Gantt | v_coverage | lane; start/end; colour = status | span | Empty lanes are visible; nothing is interpolated |
|
||||||
|
| F1 | When and how much was delivered? | Bar (time axis, **day** grain) | v_delivery (one row per delivery event) | `delivery_date`; **SUM(`delivered_l`)**, titled "Litres delivered per day (all deliveries that day)"; tooltip COUNT(deliveries) | day | Bars only on delivery dates; subtitle "deliveries are not consumption"; F4 lists the individual deliveries |
|
||||||
|
| F2 | What stock was recorded? | Scatter (no lines) | v_stock_reading | `observed_on`; litres; series = reading type × `value_kind` | observation | Points only; nothing between readings |
|
||||||
|
| F3 | How much fuel was used per calendar day in each observed interval? | Bar (categorical x, ordered by start) | v_interval | `interval_label`; MAX(`fuel_l_per_day`) (L/d); colour = `stock_status` | stock interval | Excluded intervals greyed, with the reason in the tooltip; no monthly spreading |
|
||||||
|
| F4 | Interval detail | Table | v_interval | dates, opening, deliveries, closing, used, L/d, status | interval | Shows the reasons |
|
||||||
|
| R1 | How many hours did each generator run per day? | Bar (categorical) | v_runtime | interval × meter; MAX(h/day). h/day is NULL for an invalid pair (`pair_valid` false: two epochs, a decreasing timer or over 24 h/day); such pairs show no bar and are listed with their flags in the tooltip | meter interval | **v1: empty**, with "No hour-meter readings" in the description |
|
||||||
|
| R2 | What runtime evidence exists? | Table | v_hour_meter_evidence | meter, mapped set, date, hours, state, epoch | reading | **v1: no rows**, "No hour-meter readings" |
|
||||||
|
| R3 | How much grid energy did the site receive? | Handlebars → bar once kWh exist | v_kpi_measures (kWh row) | kWh per bill month | month | "No kWh records; bills are cost only and are not converted to fuel or hours" |
|
||||||
|
| R4 | What did grid electricity cost? | Bar (month) + table | v_electricity_cost | bill month; amount (AFN, inferred) | month | Own chart, never on a fuel axis. A disputed bill is listed without amount or kWh |
|
||||||
|
| B1 | Which baseline versions exist, and who approved them? | Table | v_baseline_version | code, status, value, window, grade, decision | version | "No baseline defined" |
|
||||||
|
| B2 | What went into each version? | Table | v_baseline_input | interval, included, reason, fuel, runtime, drift | version × interval | empty until a version exists |
|
||||||
|
| B3 | Which intervals are usable, and why not? | Table | v_interval | eligibility, exclusion reasons, grade, conditions, in current baseline | interval | v1: every interval excluded, "no timer reading at both ends" (and "a generator has neither a counter nor an idle assumption") |
|
||||||
|
| B4 | What L/h did each interval give, against the approved baseline? | Mixed: bars + line | v_interval | x `interval_label`; A: MAX(`l_per_h`) where eligibility ≠ excluded; B: MAX(`current_baseline_l_per_h`) (approved value only, and only beside intervals of the same metric and population, `baseline_comparable`) | interval | v1: no bars; description points to S1 |
|
||||||
|
| B5 | Which assumptions apply? | Table | v_assumption | code, topic, value and range, basis, status, rationale | assumption | "None recorded" |
|
||||||
|
| B6 | What is missing or inconsistent? | Table | v_data_quality | check, status, detail | check | Lists the gaps: no runtime, placeholder rows, no kWh, no transfer records, location awaiting acceptance |
|
||||||
|
| B7 | Which documents do the figures rest on? | Table | v_evidence | title, kind, issuer, date, used for | document | — |
|
||||||
|
| B8 | What did the operator workbook calculate, and is it usable? | Table | v_imported_calculation | label, formula, value, our reading, use, reason, `claim_status` | cell | Shows "excluded" with the reason for "actual L/h" and "CP L/h". A disputed transcription shows its status, without formula or value |
|
||||||
|
| B9 | Does stock depletion match engine-measured fuel? | Table | v_reconciliation | stock L, engine L, `covered_h` of `interval_h`, difference L and %. The window runs from the opening visit's departure to the closing visit's arrival. The difference is shown only for an `ok` stock interval with both visit times recorded, in which every generator is placed at the site throughout and covered for the whole window | interval × generator | "No flow-meter data yet"; partial coverage, unknown visit times or a mid-interval placement show the hours covered and no difference |
|
||||||
|
| B10 | What does the equipment specification say? | Table | v_spec_reference | attribute, value, unit, basis, applicability, `claim_status` | spec | Titled "Manufacturer and nameplate data, not site performance". A disputed specification shows its status, without value |
|
||||||
|
|
||||||
|
Modelled L/h (M3) gets a table on Baseline & Evidence only if Q9 approves a runtime assumption. It
|
||||||
|
pools only the usable intervals inside that assumption's dates, and shows that window and its days.
|
||||||
|
|
||||||
|
## 5. Datasets and filter mapping
|
||||||
|
|
||||||
|
Physical datasets, one per `public.v_*` view on Trino `fuel.public`. Every dataset has the same
|
||||||
|
four site columns from `v_site`: `site_uid`, `site_code`, `site_name`, `site_label`.
|
||||||
|
|
||||||
|
| Dataset | Grain | Used by | Native filters that apply |
|
||||||
|
|---|---|---|---|
|
||||||
|
| v_site | site | M1–M5, D0, S1 | landing: all five filters, plus the M1 cross-filter (on M3); detail: Site |
|
||||||
|
| v_kpi_measures | site × measure | S2, R3 | Site |
|
||||||
|
| v_coverage | site × lane × span | S3 | Site |
|
||||||
|
| v_delivery | delivery | F1 | Site, Viewing period |
|
||||||
|
| v_stock_reading | observation | F2 | Site, Viewing period |
|
||||||
|
| v_interval | stock interval | F3, F4, B3, B4 | Site; Viewing period on F3 and F4 only |
|
||||||
|
| v_runtime | meter interval | R1 | Site, Viewing period |
|
||||||
|
| v_hour_meter_evidence | reading | R2 | Site, Viewing period |
|
||||||
|
| v_electricity_cost | bill month | R4 | Site, Viewing period |
|
||||||
|
| v_baseline_version, v_baseline_input | version (× interval) | B1, B2 | Site |
|
||||||
|
| v_assumption, v_data_quality, v_evidence, v_imported_calculation, v_reconciliation, v_spec_reference | per view | B5–B10 | Site |
|
||||||
|
|
||||||
|
**Metrics.** Saved metrics enforce pooling, with the same eligibility filter on numerator and
|
||||||
|
denominator, e.g. `pooled_l_per_h = SUM(CASE WHEN eligibility <> 'excluded' THEN fuel_used_l END)
|
||||||
|
/ NULLIF(SUM(CASE WHEN eligibility <> 'excluded' THEN runtime_h END), 0)`. A chart that uses it
|
||||||
|
is grouped or filtered by `metric` and `population_key` (both in `v_interval`), so it never pools
|
||||||
|
across methods or generator sets. The site's L/h figure itself comes only from a baseline version
|
||||||
|
(`v_site`), never from this metric. There is no AVG metric on ratio columns.
|
||||||
|
|
||||||
|
**Link column.** `detail_link` is a calculated column on the `v_site` dataset, defined in the
|
||||||
|
bundle in Trino SQL. The native filter id stays inside Superset's own configuration:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
'<a href="/superset/dashboard/atoma-site-detail/?native_filters='
|
||||||
|
|| url_encode('(NATIVE_FILTER-site:(id:NATIVE_FILTER-site,'
|
||||||
|
|| 'extraFormData:(filters:!((col:site_label,op:IN,val:!(' || q || ')))),'
|
||||||
|
|| 'filterState:(value:!(' || q || ')),ownState:()))')
|
||||||
|
|| '">Open site detail</a>'
|
||||||
|
-- q = '''' || replace(replace(site_label, '!', '!!'), '''', '!''') || '''' -- rison string quoting
|
||||||
|
```
|
||||||
|
|
||||||
|
- The exact rison state shape is a *(local test)*.
|
||||||
|
- Fallback: the link opens Site Detail with the Site filter expanded (`expand_filters=1`) and the
|
||||||
|
site's label shown beside the link, to pick by search.
|
||||||
|
|
||||||
|
## 6. Build as code, scale, access
|
||||||
|
|
||||||
|
**Build**
|
||||||
|
- `petrolyst-insights/dashboards/atoma/build_bundle.py` writes one import bundle containing both
|
||||||
|
dashboards, every dataset and every chart:
|
||||||
|
- stable UUIDs (uuid5), so a re-import updates instead of duplicating;
|
||||||
|
- fixed native-filter ids (`NATIVE_FILTER-site`, `-period`, `-search`, …);
|
||||||
|
- `label_colors` for the four statuses;
|
||||||
|
- the existing "Fuel (Trino)" database UUID, read from an export first.
|
||||||
|
- Import: `superset import-dashboards`.
|
||||||
|
|
||||||
|
**Adding a site**
|
||||||
|
- Import its `site.yaml` and its records. Nothing in either dashboard names a site.
|
||||||
|
- New sites appear in M1 (once their location is accepted) or in M5, in M3, and in the Site filter.
|
||||||
|
|
||||||
|
**Hundreds of sites**
|
||||||
|
- `v_site` has one row per site, and the scatter layer handles thousands of points.
|
||||||
|
- The Select filters load the distinct labels; M3 and M5 paginate on the server.
|
||||||
|
|
||||||
|
**Superset configuration change** (petrolyst-insights; image rebuild in P3):
|
||||||
|
`DECKGL_BASE_MAP = [["tile://https://tile.openstreetmap.org/{z}/{x}/{y}.png", "OpenStreetMap"]]`.
|
||||||
|
The CSP needs no change *(local test: no CSP violation in the browser console)*.
|
||||||
|
|
||||||
|
**Access, internal first.** Both dashboards for Admin and Alpha only. No Gamma role until the
|
||||||
|
ATOMA-facing edit.
|
||||||
|
|
||||||
|
## 7. Acceptance checks
|
||||||
|
|
||||||
|
Run in P2 against a local Superset, and repeated as a smoke test in P3.
|
||||||
|
|
||||||
|
1. **New site, no new objects.** Count dashboards, charts and datasets in the metadata database.
|
||||||
|
Then import a synthetic fourth site with records and an accepted location, and a fifth with no
|
||||||
|
location. The counts are unchanged; site 4 appears in M1, M3 and the Site filter; site 5
|
||||||
|
appears in M5, and M4 goes up by 1.
|
||||||
|
2. **Site switching.** In Site Detail, select pilot site A, then site B. D0 shows the new code and
|
||||||
|
name; every chart's query (View query, or chart-data API) carries `site_label = …<site B>…`; no
|
||||||
|
chart keeps site A's rows.
|
||||||
|
3. **Clearing the selection.** With the Site filter cleared, no chart shows multi-site data; D0 and
|
||||||
|
S1 show "Select one site". If not, switch to the §4 fallback and re-test.
|
||||||
|
4. **Unavailable L/h.** For each pilot site:
|
||||||
|
- the tooltip and M3 say "not available" with the method "no runtime evidence";
|
||||||
|
- S1 shows the insufficient-evidence state plus L/day;
|
||||||
|
- B3 shows every interval excluded, with its reason.
|
||||||
|
5. **Missing coordinates.** A site without an accepted location is not plotted, is listed in M5
|
||||||
|
and counted in M4, and its link opens its detail.
|
||||||
|
6. **Filter propagation.**
|
||||||
|
- On the landing page, each filter changes M1, M3, M4 and M5, and the default "active" shows
|
||||||
|
exactly the active pilot sites.
|
||||||
|
- A marker click filters M3 only.
|
||||||
|
- In Site Detail, the Viewing period changes F1–F3 and R1, R2, R4, but not S1 or B1–B4.
|
||||||
|
7. **Detail links.** For each active pilot site, "Open site detail" in M3 opens Site Detail with
|
||||||
|
exactly that site selected, and D0 matches. Same from M5.
|
||||||
|
8. **Colours.**
|
||||||
|
- Fixtures in each status show the four colours.
|
||||||
|
- "Eligible" never renders as "Approved" unless the fixture has an approved version with a
|
||||||
|
decision reference.
|
||||||
|
- Marker size is identical across sites.
|
||||||
|
9. **Basemap.** OSM tiles load, there are no CSP violations in the console, and the attribution
|
||||||
|
is visible.
|
||||||
@@ -0,0 +1,756 @@
|
|||||||
|
-- 0001_fm_core.sql: fuel-db schema `fm` (tables, constraints, trigger-backed rules).
|
||||||
|
-- DRAFT, not applied. Applied by hand by a loader acting as fuel_owner (README in this folder);
|
||||||
|
-- deploy.sh never touches fuel-db. Design and rationale: fuel-monitoring
|
||||||
|
-- .context/plans/2026-10-24-fuel-db-baseline-dashboard.md (sections cited as "design §n").
|
||||||
|
-- Order: 0001, 0002, 0003, each in its own transaction.
|
||||||
|
BEGIN;
|
||||||
|
CREATE SCHEMA fm AUTHORIZATION fuel_owner;
|
||||||
|
CREATE EXTENSION IF NOT EXISTS btree_gist WITH SCHEMA fm; -- trusted extension; for EXCLUDE on text
|
||||||
|
-- Default GiST operator classes are found by type, independent of search_path; fm is put first
|
||||||
|
-- anyway, and the local test applies this file in a session whose search_path is the loader's default.
|
||||||
|
SET LOCAL search_path = fm, pg_catalog, public;
|
||||||
|
|
||||||
|
-- Provenance ------------------------------------------------------------------------------
|
||||||
|
CREATE TABLE fm.source_document (
|
||||||
|
sha256 text PRIMARY KEY CHECK (sha256 ~ '^[0-9a-f]{64}$'),
|
||||||
|
source_family text NOT NULL, -- stable across versions: '<operator>-<workbook>', 'site-yaml:SITE-0001'
|
||||||
|
legacy_key16 text UNIQUE CHECK (legacy_key16 ~ '^[0-9a-f]{16}$'), -- catalog.yaml key
|
||||||
|
kind text NOT NULL CHECK (kind IN ('delivery_form','site_logbook','operator_workbook',
|
||||||
|
'operator_record','survey_workbook','photo','datasheet','mail','our_analysis')),
|
||||||
|
issuer text NOT NULL, -- organisation or role, never a person
|
||||||
|
title text NOT NULL,
|
||||||
|
document_date date,
|
||||||
|
corpus_path text, -- NULL when held outside the repository
|
||||||
|
transcription_path text, -- the corpus note that transcribes it
|
||||||
|
contains_personal_data boolean NOT NULL,
|
||||||
|
supersedes_sha256 text UNIQUE REFERENCES fm.source_document,
|
||||||
|
null_reasons jsonb NOT NULL DEFAULT '{}',
|
||||||
|
CHECK (corpus_path IS NOT NULL OR transcription_path IS NOT NULL)
|
||||||
|
);
|
||||||
|
CREATE TABLE fm.import_batch (
|
||||||
|
import_batch_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||||
|
source_kind text NOT NULL CHECK (source_kind IN ('site_yaml','operator_workbook','catalog','review_manifest','baseline_run')),
|
||||||
|
source_path text NOT NULL,
|
||||||
|
source_sha256 text NOT NULL CHECK (source_sha256 ~ '^[0-9a-f]{64}$'),
|
||||||
|
corpus_commit text,
|
||||||
|
importer_version text NOT NULL,
|
||||||
|
started_at timestamptz NOT NULL DEFAULT now(),
|
||||||
|
finished_at timestamptz,
|
||||||
|
status text NOT NULL DEFAULT 'running' CHECK (status IN ('running','loaded','failed')),
|
||||||
|
report jsonb NOT NULL DEFAULT '{}' -- counts, unmapped paths, conflicts
|
||||||
|
);
|
||||||
|
CREATE TABLE fm.import_record (
|
||||||
|
import_record_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||||
|
logical_key text NOT NULL, -- = source_key: '<family>:delivery:SITE-0001:2026-01-31:1'
|
||||||
|
source_locator text NOT NULL, -- YAML path or sheet!range as read
|
||||||
|
payload jsonb NOT NULL, -- original values, strings as written, 'UNKNOWN' kept
|
||||||
|
payload_sha256 text NOT NULL,
|
||||||
|
first_batch_id bigint NOT NULL REFERENCES fm.import_batch,
|
||||||
|
last_seen_batch_id bigint NOT NULL REFERENCES fm.import_batch,
|
||||||
|
UNIQUE (logical_key, payload_sha256)
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Identity --------------------------------------------------------------------------------
|
||||||
|
CREATE TABLE fm.site (
|
||||||
|
site_uid text PRIMARY KEY CHECK (site_uid ~ '^SITE-[0-9]{4}$'),
|
||||||
|
name text, name_on_plate text,
|
||||||
|
zone text NOT NULL, province text NOT NULL, district text,
|
||||||
|
timezone text NOT NULL DEFAULT 'Asia/Kabul',
|
||||||
|
import_record_id bigint NOT NULL REFERENCES fm.import_record, -- full site.yaml payload
|
||||||
|
null_reasons jsonb NOT NULL DEFAULT '{}'
|
||||||
|
);
|
||||||
|
CREATE TABLE fm.site_location ( -- coordinates are claims; the map uses only an accepted one
|
||||||
|
location_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||||
|
source_key text NOT NULL, -- 'site-yaml:SITE-0001:coordinates'
|
||||||
|
assertion_sha256 text NOT NULL CHECK (assertion_sha256 ~ '^[0-9a-f]{64}$'), -- site, lat, lon, precision, basis
|
||||||
|
site_uid text NOT NULL REFERENCES fm.site,
|
||||||
|
latitude numeric(9,6) NOT NULL CHECK (latitude BETWEEN -90 AND 90),
|
||||||
|
longitude numeric(9,6) NOT NULL CHECK (longitude BETWEEN -180 AND 180),
|
||||||
|
precision_m integer, -- e.g. 110 where the source rounds to 0.001 deg
|
||||||
|
basis text NOT NULL CHECK (basis IN ('operator_plate','operator_workbook','field_gps','geocoded')),
|
||||||
|
source_note text NOT NULL, -- 'operator site plate photo <catalog key>'
|
||||||
|
acceptance text NOT NULL DEFAULT 'pending' CHECK (acceptance IN ('pending','accepted','rejected','superseded','retracted')),
|
||||||
|
accepted_on date, accepted_by_role text, acceptance_ref text, -- copied from the matching fm.review row
|
||||||
|
superseded_by bigint REFERENCES fm.site_location,
|
||||||
|
import_record_id bigint NOT NULL REFERENCES fm.import_record,
|
||||||
|
CHECK (acceptance <> 'accepted' OR (accepted_on IS NOT NULL AND accepted_by_role IS NOT NULL)),
|
||||||
|
CHECK (acceptance <> 'superseded' OR superseded_by IS NOT NULL)
|
||||||
|
);
|
||||||
|
CREATE UNIQUE INDEX site_location_live_key ON fm.site_location (source_key)
|
||||||
|
WHERE acceptance IN ('pending','accepted','rejected');
|
||||||
|
CREATE UNIQUE INDEX site_location_one_accepted ON fm.site_location (site_uid) WHERE acceptance = 'accepted';
|
||||||
|
CREATE TABLE fm.site_alias (
|
||||||
|
site_alias_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||||
|
site_uid text NOT NULL REFERENCES fm.site,
|
||||||
|
label text NOT NULL, issuer text NOT NULL,
|
||||||
|
valid_from date, valid_to date, -- NULL from = unknown start; NULL to = still valid
|
||||||
|
is_primary boolean NOT NULL DEFAULT false,
|
||||||
|
null_reasons jsonb NOT NULL DEFAULT '{}',
|
||||||
|
CHECK (valid_from IS NOT NULL OR null_reasons ? 'valid_from'),
|
||||||
|
EXCLUDE USING gist (issuer WITH =, label WITH =,
|
||||||
|
daterange(coalesce(valid_from,'-infinity'::date), valid_to, '[)') WITH &&),
|
||||||
|
EXCLUDE USING gist (site_uid WITH =, issuer WITH =,
|
||||||
|
daterange(coalesce(valid_from,'-infinity'::date), valid_to, '[)') WITH &&) WHERE (is_primary)
|
||||||
|
);
|
||||||
|
CREATE TABLE fm.pilot_participation (
|
||||||
|
site_uid text NOT NULL REFERENCES fm.site,
|
||||||
|
programme text NOT NULL DEFAULT 'ATOMA OFMS pilot',
|
||||||
|
status text NOT NULL CHECK (status IN ('active','reference','candidate','withdrawn')),
|
||||||
|
valid_from date NOT NULL, valid_to date,
|
||||||
|
decision_ref text, note text,
|
||||||
|
EXCLUDE USING gist (site_uid WITH =, programme WITH =, daterange(valid_from, valid_to, '[)') WITH &&)
|
||||||
|
);
|
||||||
|
CREATE TABLE fm.asset (
|
||||||
|
asset_uid text PRIMARY KEY CHECK (asset_uid ~ '^AST-[A-Z0-9-]+$'),
|
||||||
|
kind text NOT NULL CHECK (kind IN ('generator','tank')),
|
||||||
|
make text, model text, serial text, year_of_manufacture smallint,
|
||||||
|
import_record_id bigint NOT NULL REFERENCES fm.import_record,
|
||||||
|
null_reasons jsonb NOT NULL DEFAULT '{}'
|
||||||
|
);
|
||||||
|
CREATE TABLE fm.asset_placement (
|
||||||
|
placement_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||||
|
asset_uid text NOT NULL REFERENCES fm.asset,
|
||||||
|
site_uid text NOT NULL REFERENCES fm.site,
|
||||||
|
slot_label text NOT NULL, -- 'GEN-01': a position label, not an identity
|
||||||
|
occupied_from date, occupied_to date, -- NULL from = before any record (conservative)
|
||||||
|
earliest_evidence_on date,
|
||||||
|
installed_on date,
|
||||||
|
null_reasons jsonb NOT NULL DEFAULT '{}',
|
||||||
|
CHECK (installed_on IS NOT NULL OR null_reasons ? 'installed_on'),
|
||||||
|
EXCLUDE USING gist (asset_uid WITH =,
|
||||||
|
daterange(coalesce(occupied_from,'-infinity'::date), occupied_to, '[)') WITH &&),
|
||||||
|
EXCLUDE USING gist (site_uid WITH =, slot_label WITH =,
|
||||||
|
daterange(coalesce(occupied_from,'-infinity'::date), occupied_to, '[)') WITH &&)
|
||||||
|
);
|
||||||
|
CREATE TABLE fm.runtime_meter (
|
||||||
|
meter_id text PRIMARY KEY, -- ours, stable: 'MTR-SITE-0001-GENSET1'
|
||||||
|
site_uid text NOT NULL REFERENCES fm.site,
|
||||||
|
source_label text NOT NULL, -- as written: 'Genset 1', 'R.H'
|
||||||
|
asset_uid text REFERENCES fm.asset,
|
||||||
|
mapping_status text NOT NULL CHECK (mapping_status IN ('confirmed','proposed','unmapped')),
|
||||||
|
mapping_basis text,
|
||||||
|
UNIQUE (site_uid, source_label),
|
||||||
|
CHECK ((mapping_status = 'unmapped') = (asset_uid IS NULL))
|
||||||
|
);
|
||||||
|
CREATE TABLE fm.meter_epoch (
|
||||||
|
epoch_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||||
|
meter_id text NOT NULL REFERENCES fm.runtime_meter,
|
||||||
|
epoch_no smallint NOT NULL,
|
||||||
|
starts_on date, -- NULL: began before the first reading held
|
||||||
|
start_reason text NOT NULL CHECK (start_reason IN ('first_seen','display_replaced','controller_replaced','reset','set_replaced')),
|
||||||
|
continuity text NOT NULL CHECK (continuity IN ('confirmed','assumed','uncertain')),
|
||||||
|
rollover_at numeric(9,1),
|
||||||
|
source_sha256 text REFERENCES fm.source_document, note text,
|
||||||
|
UNIQUE (meter_id, epoch_no)
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Operations (claims) ---------------------------------------------------------------------
|
||||||
|
CREATE TABLE fm.visit (
|
||||||
|
visit_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||||
|
visit_key text NOT NULL UNIQUE, -- 'SITE-0001:2026-01-31:1'
|
||||||
|
site_uid text NOT NULL REFERENCES fm.site,
|
||||||
|
visit_on date NOT NULL, -- site-local calendar date
|
||||||
|
seq_in_day smallint NOT NULL DEFAULT 1,
|
||||||
|
time_in_local time, time_out_local time, -- as written; NULL when the source has a date only
|
||||||
|
kind text NOT NULL CHECK (kind IN ('delivery','stock_check','maintenance','survey')),
|
||||||
|
-- descriptive only: views use fm.v_visit_live and the live claims
|
||||||
|
UNIQUE (site_uid, visit_on, seq_in_day),
|
||||||
|
UNIQUE (visit_id, site_uid) -- target of site-checked composite foreign keys
|
||||||
|
);
|
||||||
|
-- A visit is never deleted. Once all its claims are superseded or retracted it stays for provenance
|
||||||
|
-- only, and drops out of every analytical view (fm.v_visit_live).
|
||||||
|
-- Claim columns, repeated in each claim table:
|
||||||
|
-- source_key, assertion_sha256, source_sha256, record_ref, import_record_id, claim_status,
|
||||||
|
-- superseded_by, review_note, null_reasons
|
||||||
|
CREATE TABLE fm.fuel_movement (
|
||||||
|
movement_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||||
|
source_key text NOT NULL, -- '<family>:delivery:SITE-0001:2026-01-31:1'
|
||||||
|
assertion_sha256 text NOT NULL CHECK (assertion_sha256 ~ '^[0-9a-f]{64}$'),
|
||||||
|
event_key text NOT NULL, -- the physical movement: 'SITE-0001:2026-01-31:1:delivery:1';
|
||||||
|
-- claims sharing it are alternative reports of ONE movement
|
||||||
|
visit_id bigint NOT NULL REFERENCES fm.visit,
|
||||||
|
site_uid text NOT NULL REFERENCES fm.site,
|
||||||
|
kind text NOT NULL CHECK (kind IN ('delivery','external_in','internal_transfer',
|
||||||
|
'withdrawal','drain','spill','loss_reported')),
|
||||||
|
from_tank_uid text REFERENCES fm.asset, to_tank_uid text REFERENCES fm.asset,
|
||||||
|
source_tank_label text,
|
||||||
|
litres numeric(8,1) CHECK (litres >= 0),
|
||||||
|
litres_state text NOT NULL CHECK (litres_state IN ('stated','converted_from_level','converted_from_mass','not_recorded','illegible')),
|
||||||
|
density_kg_per_l numeric(5,3),
|
||||||
|
unit_price numeric(8,2), price_currency text,
|
||||||
|
source_sha256 text NOT NULL REFERENCES fm.source_document,
|
||||||
|
record_ref text NOT NULL, -- 'form <no>' | '<sheet>!J20' | 'logbook p2 r1'
|
||||||
|
import_record_id bigint NOT NULL REFERENCES fm.import_record,
|
||||||
|
claim_status text NOT NULL DEFAULT 'asserted' CHECK (claim_status IN ('asserted','reviewed','disputed','superseded','retracted')),
|
||||||
|
superseded_by bigint REFERENCES fm.fuel_movement,
|
||||||
|
review_note text,
|
||||||
|
null_reasons jsonb NOT NULL DEFAULT '{}',
|
||||||
|
CHECK ((litres IS NULL) = (litres_state IN ('not_recorded','illegible'))),
|
||||||
|
CHECK (claim_status <> 'superseded' OR superseded_by IS NOT NULL),
|
||||||
|
CHECK (kind <> 'internal_transfer' OR (from_tank_uid IS NOT NULL AND to_tank_uid IS NOT NULL))
|
||||||
|
);
|
||||||
|
CREATE UNIQUE INDEX fuel_movement_live_key ON fm.fuel_movement (source_key)
|
||||||
|
WHERE claim_status IN ('asserted','reviewed','disputed');
|
||||||
|
CREATE INDEX ON fm.fuel_movement (visit_id, event_key);
|
||||||
|
CREATE TABLE fm.stock_observation (
|
||||||
|
stock_obs_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||||
|
source_key text NOT NULL,
|
||||||
|
assertion_sha256 text NOT NULL CHECK (assertion_sha256 ~ '^[0-9a-f]{64}$'),
|
||||||
|
visit_id bigint NOT NULL REFERENCES fm.visit,
|
||||||
|
site_uid text NOT NULL REFERENCES fm.site,
|
||||||
|
scope text NOT NULL CHECK (scope IN ('site_total','tank')),
|
||||||
|
tank_uid text REFERENCES fm.asset, -- only when the source identifies the tank
|
||||||
|
source_tank_label text, -- 'T1' as written, even when unmapped
|
||||||
|
phase text NOT NULL CHECK (phase IN ('before_delivery','after_delivery','standalone')),
|
||||||
|
level_cm numeric(6,1),
|
||||||
|
litres numeric(8,1),
|
||||||
|
litres_basis text NOT NULL CHECK (litres_basis IN ('gauge_converted_by_source','stated_without_level',
|
||||||
|
'stated_total','level_only','not_recorded','illegible','placeholder_zero')),
|
||||||
|
resolution_l numeric(6,1), -- ±1 cm of dip at this level, from tank geometry
|
||||||
|
resolution_basis text,
|
||||||
|
source_sha256 text NOT NULL REFERENCES fm.source_document,
|
||||||
|
record_ref text NOT NULL,
|
||||||
|
import_record_id bigint NOT NULL REFERENCES fm.import_record,
|
||||||
|
claim_status text NOT NULL DEFAULT 'asserted' CHECK (claim_status IN ('asserted','reviewed','disputed','superseded','retracted')),
|
||||||
|
superseded_by bigint REFERENCES fm.stock_observation,
|
||||||
|
review_note text,
|
||||||
|
null_reasons jsonb NOT NULL DEFAULT '{}',
|
||||||
|
CHECK (scope = 'site_total' OR tank_uid IS NOT NULL OR source_tank_label IS NOT NULL),
|
||||||
|
CHECK ((litres IS NULL) = (litres_basis IN ('level_only','not_recorded','illegible','placeholder_zero'))),
|
||||||
|
CHECK (claim_status <> 'superseded' OR superseded_by IS NOT NULL)
|
||||||
|
);
|
||||||
|
CREATE UNIQUE INDEX stock_obs_live_key ON fm.stock_observation (source_key)
|
||||||
|
WHERE claim_status IN ('asserted','reviewed','disputed');
|
||||||
|
CREATE INDEX ON fm.stock_observation (visit_id, phase);
|
||||||
|
CREATE TABLE fm.hour_meter_reading (
|
||||||
|
reading_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||||
|
source_key text NOT NULL,
|
||||||
|
assertion_sha256 text NOT NULL CHECK (assertion_sha256 ~ '^[0-9a-f]{64}$'),
|
||||||
|
visit_id bigint NOT NULL REFERENCES fm.visit,
|
||||||
|
meter_id text NOT NULL REFERENCES fm.runtime_meter,
|
||||||
|
epoch_id bigint REFERENCES fm.meter_epoch, -- NULL = not yet judged; excluded from runtime
|
||||||
|
source_field text NOT NULL CHECK (source_field IN ('current','previous','logbook_rh','display_photo')),
|
||||||
|
date_basis text NOT NULL CHECK (date_basis IN ('stated','inferred_previous_visit','capture_date_unknown')),
|
||||||
|
hours numeric(9,1),
|
||||||
|
resolution_h numeric(4,1) NOT NULL DEFAULT 1.0,
|
||||||
|
reading_state text NOT NULL CHECK (reading_state IN ('clear','best_reading','ambiguous','illegible','crossed_out','not_recorded')),
|
||||||
|
alternatives text, -- competing readings of one illegible value, '|'-separated
|
||||||
|
transcribed_by text NOT NULL CHECK (transcribed_by IN ('source_operator','our_reading')),
|
||||||
|
source_sha256 text NOT NULL REFERENCES fm.source_document,
|
||||||
|
record_ref text NOT NULL,
|
||||||
|
import_record_id bigint NOT NULL REFERENCES fm.import_record,
|
||||||
|
claim_status text NOT NULL DEFAULT 'asserted' CHECK (claim_status IN ('asserted','reviewed','disputed','superseded','retracted')),
|
||||||
|
superseded_by bigint REFERENCES fm.hour_meter_reading,
|
||||||
|
review_note text,
|
||||||
|
null_reasons jsonb NOT NULL DEFAULT '{}',
|
||||||
|
CHECK ((hours IS NULL) = (reading_state IN ('ambiguous','illegible','crossed_out','not_recorded'))),
|
||||||
|
CHECK (claim_status <> 'superseded' OR superseded_by IS NOT NULL)
|
||||||
|
);
|
||||||
|
CREATE UNIQUE INDEX hour_reading_live_key ON fm.hour_meter_reading (source_key)
|
||||||
|
WHERE claim_status IN ('asserted','reviewed','disputed');
|
||||||
|
CREATE INDEX ON fm.hour_meter_reading (visit_id, meter_id);
|
||||||
|
CREATE TABLE fm.runtime_interval ( -- future: controller / ATS logs, telemetry
|
||||||
|
runtime_interval_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||||
|
source_key text NOT NULL, -- live-unique, as for every claim
|
||||||
|
assertion_sha256 text NOT NULL CHECK (assertion_sha256 ~ '^[0-9a-f]{64}$'),
|
||||||
|
site_uid text NOT NULL REFERENCES fm.site,
|
||||||
|
asset_uid text REFERENCES fm.asset,
|
||||||
|
scope text NOT NULL CHECK (scope IN ('generator','site_any_generator')),
|
||||||
|
started_at timestamptz NOT NULL, ended_at timestamptz NOT NULL CHECK (ended_at > started_at),
|
||||||
|
source_kind text NOT NULL CHECK (source_kind IN ('controller_event_log','ats_log','gateway_telemetry','manual_log')),
|
||||||
|
source_sha256 text REFERENCES fm.source_document,
|
||||||
|
claim_status text NOT NULL DEFAULT 'asserted' CHECK (claim_status IN ('asserted','reviewed','disputed','superseded','retracted')),
|
||||||
|
superseded_by bigint REFERENCES fm.runtime_interval,
|
||||||
|
CHECK ((scope = 'generator') = (asset_uid IS NOT NULL)),
|
||||||
|
CHECK (claim_status <> 'superseded' OR superseded_by IS NOT NULL),
|
||||||
|
EXCLUDE USING gist (site_uid WITH =, scope WITH =, (coalesce(asset_uid,'')) WITH =, source_kind WITH =,
|
||||||
|
tstzrange(started_at, ended_at) WITH &&) WHERE (claim_status IN ('asserted','reviewed'))
|
||||||
|
);
|
||||||
|
CREATE UNIQUE INDEX runtime_interval_live_key ON fm.runtime_interval (source_key)
|
||||||
|
WHERE claim_status IN ('asserted','reviewed','disputed');
|
||||||
|
CREATE TABLE fm.engine_fuel_total ( -- future: flow-meter net fuel per period
|
||||||
|
total_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||||
|
source_key text NOT NULL, -- live-unique, as for every claim
|
||||||
|
assertion_sha256 text NOT NULL CHECK (assertion_sha256 ~ '^[0-9a-f]{64}$'),
|
||||||
|
asset_uid text NOT NULL REFERENCES fm.asset,
|
||||||
|
started_at timestamptz NOT NULL, ended_at timestamptz NOT NULL CHECK (ended_at > started_at),
|
||||||
|
net_fuel_l numeric(10,2) NOT NULL, -- feed − return as the device reports
|
||||||
|
engine_hours numeric(8,2), -- same device, if reported
|
||||||
|
source_kind text NOT NULL CHECK (source_kind IN ('differential_flow_meter','single_flow_meter','controller_fuel_counter')),
|
||||||
|
source_sha256 text REFERENCES fm.source_document,
|
||||||
|
claim_status text NOT NULL DEFAULT 'asserted' CHECK (claim_status IN ('asserted','reviewed','disputed','superseded','retracted')),
|
||||||
|
superseded_by bigint REFERENCES fm.engine_fuel_total,
|
||||||
|
CHECK (claim_status <> 'superseded' OR superseded_by IS NOT NULL),
|
||||||
|
EXCLUDE USING gist (asset_uid WITH =, tstzrange(started_at, ended_at) WITH &&)
|
||||||
|
WHERE (claim_status IN ('asserted','reviewed'))
|
||||||
|
);
|
||||||
|
CREATE UNIQUE INDEX engine_fuel_total_live_key ON fm.engine_fuel_total (source_key)
|
||||||
|
WHERE claim_status IN ('asserted','reviewed','disputed');
|
||||||
|
CREATE TABLE fm.electricity_bill (
|
||||||
|
bill_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||||
|
source_key text NOT NULL,
|
||||||
|
assertion_sha256 text NOT NULL CHECK (assertion_sha256 ~ '^[0-9a-f]{64}$'),
|
||||||
|
site_uid text NOT NULL REFERENCES fm.site,
|
||||||
|
bill_month date NOT NULL CHECK (extract(day FROM bill_month) = 1), -- month as the source assigns it
|
||||||
|
period_start date, period_end date,
|
||||||
|
amount numeric(12,2),
|
||||||
|
currency text, currency_basis text CHECK (currency_basis IN ('stated','inferred')),
|
||||||
|
kwh numeric(10,1),
|
||||||
|
source_sha256 text NOT NULL REFERENCES fm.source_document,
|
||||||
|
record_ref text NOT NULL,
|
||||||
|
import_record_id bigint NOT NULL REFERENCES fm.import_record,
|
||||||
|
claim_status text NOT NULL DEFAULT 'asserted' CHECK (claim_status IN ('asserted','reviewed','disputed','superseded','retracted')),
|
||||||
|
superseded_by bigint REFERENCES fm.electricity_bill,
|
||||||
|
null_reasons jsonb NOT NULL DEFAULT '{}',
|
||||||
|
CHECK (kwh IS NOT NULL OR null_reasons ? 'kwh'),
|
||||||
|
CHECK (period_start IS NOT NULL OR null_reasons ? 'period_start')
|
||||||
|
);
|
||||||
|
CREATE UNIQUE INDEX bill_live_key ON fm.electricity_bill (source_key)
|
||||||
|
WHERE claim_status IN ('asserted','reviewed','disputed');
|
||||||
|
|
||||||
|
-- Specifications, assumptions, imported calculations -------------------------------------
|
||||||
|
CREATE TABLE fm.equipment_spec ( -- versioned like a claim: corrections supersede
|
||||||
|
spec_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||||
|
source_key text NOT NULL, -- 'site-yaml:SITE-0001:AST-…:rated_prime_kw'
|
||||||
|
assertion_sha256 text NOT NULL CHECK (assertion_sha256 ~ '^[0-9a-f]{64}$'),
|
||||||
|
asset_uid text NOT NULL REFERENCES fm.asset,
|
||||||
|
attribute text NOT NULL, -- rated_prime_kw, capacity_l, fuel_rate_at_load, …
|
||||||
|
load_pct numeric(5,1),
|
||||||
|
value_num numeric(12,3), value_text text, unit text,
|
||||||
|
basis text NOT NULL CHECK (basis IN ('nameplate','manufacturer_datasheet','derived_from_datasheet',
|
||||||
|
'operator_stated','our_inference','geometry')),
|
||||||
|
applicability text NOT NULL DEFAULT 'confirmed' CHECK (applicability IN ('confirmed','conditional','unconfirmed')),
|
||||||
|
applicability_note text,
|
||||||
|
source_sha256 text REFERENCES fm.source_document,
|
||||||
|
import_record_id bigint NOT NULL REFERENCES fm.import_record,
|
||||||
|
claim_status text NOT NULL DEFAULT 'asserted' CHECK (claim_status IN ('asserted','reviewed','disputed','superseded','retracted')),
|
||||||
|
superseded_by bigint REFERENCES fm.equipment_spec,
|
||||||
|
CHECK (claim_status <> 'superseded' OR superseded_by IS NOT NULL),
|
||||||
|
CHECK ((attribute IN ('fuel_rate_at_load','specific_fuel_at_load')) = (load_pct IS NOT NULL))
|
||||||
|
);
|
||||||
|
CREATE UNIQUE INDEX equipment_spec_live_key ON fm.equipment_spec (source_key)
|
||||||
|
WHERE claim_status IN ('asserted','reviewed','disputed');
|
||||||
|
CREATE TABLE fm.assumption ( -- one row per version; a change is a new version
|
||||||
|
assumption_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||||
|
code text NOT NULL UNIQUE, -- 'A-SITE-0001-GEN02-IDLE-v1'
|
||||||
|
fingerprint text NOT NULL CHECK (fingerprint ~ '^[0-9a-f]{64}$'), -- all fields below + sorted members
|
||||||
|
created_xact xid8 NOT NULL DEFAULT pg_current_xact_id(), -- rule 9: members only in this transaction
|
||||||
|
site_uid text NOT NULL REFERENCES fm.site,
|
||||||
|
asset_uid text REFERENCES fm.asset,
|
||||||
|
topic text NOT NULL CHECK (topic IN ('runtime_hours_per_day','generator_idle','meter_coverage',
|
||||||
|
'tank_linkage','no_unrecorded_movements','density','other')),
|
||||||
|
-- meter_coverage: its members name the exact counters,
|
||||||
|
-- the sets they cover together and the idle sets
|
||||||
|
value_num numeric(10,3), low_num numeric(10,3), high_num numeric(10,3), unit text,
|
||||||
|
applies_from date, applies_to date,
|
||||||
|
basis text NOT NULL CHECK (basis IN ('operator_stated','our_assumption','evidence_based')),
|
||||||
|
derived_from_fuel boolean NOT NULL, -- true: never a runtime input to an L/h figure
|
||||||
|
status text NOT NULL CHECK (status IN ('proposed','accepted','rejected','superseded')),
|
||||||
|
accepted_on date, accepted_by_role text, decision_ref text, -- copied from the matching fm.review row
|
||||||
|
supersedes_id bigint UNIQUE REFERENCES fm.assumption,
|
||||||
|
rationale text NOT NULL, contradicted_by text,
|
||||||
|
source_sha256 text REFERENCES fm.source_document,
|
||||||
|
CHECK (status <> 'accepted' OR (accepted_on IS NOT NULL AND accepted_by_role IS NOT NULL)),
|
||||||
|
CONSTRAINT assumption_no_runtime_from_fuel -- rule 5
|
||||||
|
CHECK (NOT (derived_from_fuel AND topic = 'runtime_hours_per_day' AND status = 'accepted')),
|
||||||
|
CHECK (topic <> 'generator_idle' OR asset_uid IS NOT NULL),
|
||||||
|
CHECK (topic <> 'meter_coverage' OR asset_uid IS NULL), -- its sets are in assumption_member
|
||||||
|
EXCLUDE USING gist (site_uid WITH =, daterange(applies_from, applies_to, '[]') WITH &&)
|
||||||
|
WHERE (topic = 'meter_coverage' AND status IN ('proposed','accepted')),
|
||||||
|
EXCLUDE USING gist (asset_uid WITH =, daterange(applies_from, applies_to, '[]') WITH &&)
|
||||||
|
WHERE (topic = 'generator_idle' AND status IN ('proposed','accepted'))
|
||||||
|
);
|
||||||
|
CREATE TABLE fm.assumption_member ( -- exact membership of a meter_coverage assumption
|
||||||
|
assumption_id bigint NOT NULL REFERENCES fm.assumption,
|
||||||
|
member_role text NOT NULL CHECK (member_role IN ('counter','covered_generator','idle_generator')),
|
||||||
|
meter_id text REFERENCES fm.runtime_meter,
|
||||||
|
asset_uid text REFERENCES fm.asset,
|
||||||
|
CHECK ((member_role = 'counter') = (meter_id IS NOT NULL)),
|
||||||
|
CHECK ((member_role = 'counter') = (asset_uid IS NULL))
|
||||||
|
);
|
||||||
|
CREATE UNIQUE INDEX assumption_member_meter ON fm.assumption_member (assumption_id, meter_id) WHERE meter_id IS NOT NULL;
|
||||||
|
CREATE UNIQUE INDEX assumption_member_asset ON fm.assumption_member (assumption_id, asset_uid) WHERE asset_uid IS NOT NULL;
|
||||||
|
-- a set is either covered or idle within one assumption
|
||||||
|
CREATE TABLE fm.imported_calculation ( -- versioned like a claim: a revised formula supersedes
|
||||||
|
calc_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||||
|
source_key text NOT NULL, -- '<family>:calc:SITE-0001:<label>'
|
||||||
|
assertion_sha256 text NOT NULL CHECK (assertion_sha256 ~ '^[0-9a-f]{64}$'),
|
||||||
|
source_sha256 text NOT NULL REFERENCES fm.source_document,
|
||||||
|
cell_ref text NOT NULL, -- '<sheet>!K2' | '<survey item>'
|
||||||
|
site_uid text REFERENCES fm.site,
|
||||||
|
source_label text NOT NULL, -- as written in the source
|
||||||
|
formula text, cached_value numeric(14,6), unit_as_labelled text,
|
||||||
|
our_reading text NOT NULL, -- what it actually computes
|
||||||
|
baseline_use text NOT NULL CHECK (baseline_use IN ('excluded','context_only')),
|
||||||
|
exclusion_reason text,
|
||||||
|
import_record_id bigint NOT NULL REFERENCES fm.import_record,
|
||||||
|
claim_status text NOT NULL DEFAULT 'asserted' CHECK (claim_status IN ('asserted','reviewed','disputed','superseded','retracted')),
|
||||||
|
superseded_by bigint REFERENCES fm.imported_calculation,
|
||||||
|
CHECK (claim_status <> 'superseded' OR superseded_by IS NOT NULL),
|
||||||
|
CHECK (baseline_use <> 'excluded' OR exclusion_reason IS NOT NULL)
|
||||||
|
);
|
||||||
|
CREATE UNIQUE INDEX imported_calc_live_key ON fm.imported_calculation (source_key)
|
||||||
|
WHERE claim_status IN ('asserted','reviewed','disputed');
|
||||||
|
|
||||||
|
-- Reviews ---------------------------------------------------------------------------------
|
||||||
|
CREATE TABLE fm.review ( -- one applied entry of the corpus review manifest
|
||||||
|
review_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||||
|
entry_key text NOT NULL UNIQUE, -- the manifest entry's own id
|
||||||
|
subject_kind text NOT NULL CHECK (subject_kind IN ('claim','location','assumption')),
|
||||||
|
subject_key text NOT NULL, -- source_key, or the assumption code
|
||||||
|
assertion_sha256 text NOT NULL CHECK (assertion_sha256 ~ '^[0-9a-f]{64}$'), -- what was actually reviewed
|
||||||
|
checked_against_sha256 text REFERENCES fm.source_document, -- the file version compared with
|
||||||
|
verdict text NOT NULL CHECK (verdict IN ('reviewed','disputed','accepted','rejected')),
|
||||||
|
reviewed_on date NOT NULL, reviewer_role text NOT NULL, decision_ref text, note text,
|
||||||
|
import_batch_id bigint NOT NULL REFERENCES fm.import_batch,
|
||||||
|
CHECK ((subject_kind = 'claim') = (verdict IN ('reviewed','disputed')))
|
||||||
|
);
|
||||||
|
CREATE INDEX ON fm.review (subject_key, assertion_sha256);
|
||||||
|
|
||||||
|
-- Baseline --------------------------------------------------------------------------------
|
||||||
|
CREATE TABLE fm.baseline_version (
|
||||||
|
baseline_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||||
|
code text NOT NULL UNIQUE, -- 'BL-SITE-0001-APGH-v1'
|
||||||
|
site_uid text NOT NULL REFERENCES fm.site,
|
||||||
|
metric text NOT NULL CHECK (metric IN ('measured_generator_l_per_h','apparent_l_per_generator_h',
|
||||||
|
'apparent_l_per_set_h','apparent_l_per_site_dg_h','modelled_l_per_h','fuel_l_per_calendar_day')),
|
||||||
|
population text NOT NULL, -- human-readable: 'GEN-01 (AST-XXX-GEN-0001)'
|
||||||
|
population_key text NOT NULL, -- sorted asset uids, 'meter:<id>' for counters not tied
|
||||||
|
-- to a set plus 'covers:<asset uid>' for the sets they
|
||||||
|
-- cover, or 'stock:<site_uid>' for fuel_l_per_calendar_day
|
||||||
|
window_start date NOT NULL, window_end date NOT NULL CHECK (window_end > window_start),
|
||||||
|
policy_version text NOT NULL, -- 'BP-1'
|
||||||
|
meets_policy boolean NOT NULL DEFAULT false,-- set by `baseline propose` from the inputs (BP-1 test)
|
||||||
|
policy_check text, -- which BP-1 conditions failed, if any
|
||||||
|
value numeric(8,3), range_low numeric(8,3), range_high numeric(8,3),
|
||||||
|
fuel_l numeric(10,1), runtime_h numeric(10,1), matched_days integer,
|
||||||
|
conditions text, -- assumption codes the value depends on
|
||||||
|
evidence_grade text CHECK (evidence_grade IN ('G1','G2')),
|
||||||
|
atoma_tier text CHECK (atoma_tier IN ('A','B','C')), -- descriptive only (Baseline Verification Plan)
|
||||||
|
season_note text,
|
||||||
|
status text NOT NULL CHECK (status IN ('draft','proposed','approved','superseded','rejected')),
|
||||||
|
computed_at timestamptz NOT NULL, computed_by_role text NOT NULL,
|
||||||
|
approved_on date, approved_by_role text, decision_ref text,
|
||||||
|
supersedes_id bigint REFERENCES fm.baseline_version,
|
||||||
|
UNIQUE (baseline_id, site_uid),
|
||||||
|
CHECK (status <> 'approved' OR (approved_on IS NOT NULL AND decision_ref IS NOT NULL))
|
||||||
|
);
|
||||||
|
CREATE UNIQUE INDEX baseline_one_approved ON fm.baseline_version (site_uid, metric) WHERE status = 'approved';
|
||||||
|
CREATE TABLE fm.baseline_input ( -- an interval is identified by its two endpoint visits
|
||||||
|
baseline_id bigint NOT NULL,
|
||||||
|
site_uid text NOT NULL,
|
||||||
|
open_visit_id bigint NOT NULL,
|
||||||
|
close_visit_id bigint NOT NULL,
|
||||||
|
start_on date NOT NULL, end_on date NOT NULL CHECK (end_on >= start_on),
|
||||||
|
included boolean NOT NULL, exclusion_reason text,
|
||||||
|
fuel_l numeric(10,1), runtime_h numeric(10,1), days integer, population_key text, -- snapshot at computation
|
||||||
|
eligibility_at_computation text NOT NULL, evidence_grade_at_computation text,
|
||||||
|
dependencies jsonb NOT NULL, -- {"stock": [...], "runtime": [...]} from fm.v_interval_dependency
|
||||||
|
stock_dep_sha256 text NOT NULL, runtime_dep_sha256 text, -- their hashes; runtime NULL for L/day versions
|
||||||
|
PRIMARY KEY (baseline_id, open_visit_id, close_visit_id),
|
||||||
|
FOREIGN KEY (baseline_id, site_uid) REFERENCES fm.baseline_version (baseline_id, site_uid),
|
||||||
|
FOREIGN KEY (open_visit_id, site_uid) REFERENCES fm.visit (visit_id, site_uid), -- both endpoints belong
|
||||||
|
FOREIGN KEY (close_visit_id, site_uid) REFERENCES fm.visit (visit_id, site_uid), -- to the baseline's site
|
||||||
|
CHECK (included OR exclusion_reason IS NOT NULL),
|
||||||
|
CHECK (NOT included OR end_on > start_on), -- a same-day interval can only be an excluded input
|
||||||
|
EXCLUDE USING gist (baseline_id WITH =, daterange(start_on, end_on, '[)') WITH &&) WHERE (included)
|
||||||
|
);
|
||||||
|
-- Trigger-backed rules ------------------------------------------------------------------------
|
||||||
|
-- Numbered as in the design (fuel-monitoring .context/plans/2026-10-24-fuel-db-baseline-dashboard.md).
|
||||||
|
-- Rule 5 is the CHECK assumption_no_runtime_from_fuel above. Rules 6 and 1 together cover
|
||||||
|
-- baseline_input; `baseline propose` stays the only writer by convention.
|
||||||
|
|
||||||
|
-- Claims are corrected by supersession, never in place (design §2.1). Only these columns may change.
|
||||||
|
CREATE FUNCTION fm.mutable_columns(tbl text) RETURNS text[] LANGUAGE sql IMMUTABLE AS $$
|
||||||
|
SELECT CASE tbl
|
||||||
|
WHEN 'site_location' THEN ARRAY['acceptance','accepted_on','accepted_by_role','acceptance_ref','superseded_by']
|
||||||
|
WHEN 'assumption' THEN ARRAY['status','accepted_on','accepted_by_role','decision_ref','contradicted_by']
|
||||||
|
ELSE ARRAY['claim_status','superseded_by','review_note']
|
||||||
|
END $$;
|
||||||
|
|
||||||
|
-- 11. Status only from a matching review, and no in-place correction.
|
||||||
|
CREATE FUNCTION fm.tg_status_from_review() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||||
|
DECLARE
|
||||||
|
j jsonb := to_jsonb(NEW);
|
||||||
|
v_kind text; v_subject text; v_fp text; v_verdict text; r fm.review;
|
||||||
|
BEGIN
|
||||||
|
IF TG_OP = 'UPDATE' AND (to_jsonb(OLD) - fm.mutable_columns(TG_TABLE_NAME))
|
||||||
|
IS DISTINCT FROM (j - fm.mutable_columns(TG_TABLE_NAME)) THEN
|
||||||
|
RAISE EXCEPTION '%: only the status columns change in place; a corrected value is a new row that supersedes this one',
|
||||||
|
TG_TABLE_NAME;
|
||||||
|
END IF;
|
||||||
|
IF TG_TABLE_NAME = 'site_location' THEN
|
||||||
|
v_kind := 'location'; v_subject := j->>'source_key'; v_fp := j->>'assertion_sha256'; v_verdict := j->>'acceptance';
|
||||||
|
IF v_verdict NOT IN ('accepted','rejected') THEN RETURN NEW; END IF;
|
||||||
|
ELSIF TG_TABLE_NAME = 'assumption' THEN
|
||||||
|
v_kind := 'assumption'; v_subject := j->>'code'; v_fp := j->>'fingerprint'; v_verdict := j->>'status';
|
||||||
|
IF v_verdict NOT IN ('accepted','rejected') THEN RETURN NEW; END IF;
|
||||||
|
ELSE
|
||||||
|
v_kind := 'claim'; v_subject := j->>'source_key'; v_fp := j->>'assertion_sha256'; v_verdict := j->>'claim_status';
|
||||||
|
IF v_verdict NOT IN ('reviewed','disputed') THEN RETURN NEW; END IF;
|
||||||
|
END IF;
|
||||||
|
SELECT * INTO r FROM fm.review x
|
||||||
|
WHERE x.subject_kind = v_kind AND x.subject_key = v_subject AND x.assertion_sha256 = v_fp AND x.verdict = v_verdict
|
||||||
|
ORDER BY x.reviewed_on DESC, x.review_id DESC LIMIT 1;
|
||||||
|
IF NOT FOUND THEN
|
||||||
|
RAISE EXCEPTION '%: status % for % needs a review of this exact assertion (%)',
|
||||||
|
TG_TABLE_NAME, v_verdict, v_subject, left(v_fp, 16);
|
||||||
|
END IF;
|
||||||
|
IF v_verdict = 'accepted' AND ((j->>'accepted_on')::date IS DISTINCT FROM r.reviewed_on
|
||||||
|
OR j->>'accepted_by_role' IS DISTINCT FROM r.reviewer_role) THEN
|
||||||
|
RAISE EXCEPTION '%: acceptance date and role must be copied from review %', TG_TABLE_NAME, r.entry_key;
|
||||||
|
END IF;
|
||||||
|
RETURN NEW;
|
||||||
|
END $$;
|
||||||
|
|
||||||
|
DO $$
|
||||||
|
DECLARE t text;
|
||||||
|
BEGIN
|
||||||
|
FOREACH t IN ARRAY ARRAY['fuel_movement','stock_observation','hour_meter_reading','electricity_bill',
|
||||||
|
'equipment_spec','imported_calculation','runtime_interval','engine_fuel_total',
|
||||||
|
'site_location','assumption'] LOOP
|
||||||
|
EXECUTE format('CREATE TRIGGER status_from_review BEFORE INSERT OR UPDATE ON fm.%I
|
||||||
|
FOR EACH ROW EXECUTE FUNCTION fm.tg_status_from_review()', t);
|
||||||
|
END LOOP;
|
||||||
|
END $$;
|
||||||
|
|
||||||
|
-- 2, 3, 10. A movement claim: same site as its visit; an internal transfer's two tanks placed at
|
||||||
|
-- that site on the visit date; the live claims of one event_key share visit, kind and tanks. A
|
||||||
|
-- superseded or retracted claim keeps its own, so a correction can change them.
|
||||||
|
CREATE FUNCTION fm.tg_fuel_movement_rules() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||||
|
DECLARE v fm.visit;
|
||||||
|
BEGIN
|
||||||
|
SELECT * INTO v FROM fm.visit WHERE visit_id = NEW.visit_id;
|
||||||
|
IF v.site_uid <> NEW.site_uid THEN
|
||||||
|
RAISE EXCEPTION 'fuel_movement %: site % differs from its visit''s site %', NEW.source_key, NEW.site_uid, v.site_uid;
|
||||||
|
END IF;
|
||||||
|
IF NEW.kind = 'internal_transfer' AND EXISTS (
|
||||||
|
SELECT 1 FROM unnest(ARRAY[NEW.from_tank_uid, NEW.to_tank_uid]) t(tank)
|
||||||
|
WHERE NOT EXISTS (SELECT 1 FROM fm.asset_placement p
|
||||||
|
WHERE p.asset_uid = t.tank AND p.site_uid = NEW.site_uid
|
||||||
|
AND daterange(coalesce(p.occupied_from,'-infinity'::date), p.occupied_to, '[)') @> v.visit_on)) THEN
|
||||||
|
RAISE EXCEPTION 'fuel_movement %: both tanks of an internal transfer must be placed at % on %', NEW.source_key, NEW.site_uid, v.visit_on;
|
||||||
|
END IF;
|
||||||
|
IF NEW.claim_status IN ('asserted','reviewed','disputed') AND EXISTS (SELECT 1 FROM fm.fuel_movement m
|
||||||
|
WHERE m.event_key = NEW.event_key AND m.movement_id <> NEW.movement_id
|
||||||
|
AND m.claim_status IN ('asserted','reviewed','disputed')
|
||||||
|
AND (m.visit_id, m.kind, m.from_tank_uid, m.to_tank_uid)
|
||||||
|
IS DISTINCT FROM (NEW.visit_id, NEW.kind, NEW.from_tank_uid, NEW.to_tank_uid)) THEN
|
||||||
|
RAISE EXCEPTION 'fuel_movement %: claims of event % must share visit, kind and tanks', NEW.source_key, NEW.event_key;
|
||||||
|
END IF;
|
||||||
|
RETURN NEW;
|
||||||
|
END $$;
|
||||||
|
CREATE TRIGGER fuel_movement_rules BEFORE INSERT OR UPDATE ON fm.fuel_movement
|
||||||
|
FOR EACH ROW EXECUTE FUNCTION fm.tg_fuel_movement_rules();
|
||||||
|
|
||||||
|
-- 3. A stock claim belongs to its visit's site.
|
||||||
|
CREATE FUNCTION fm.tg_stock_observation_rules() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||||
|
BEGIN
|
||||||
|
IF NOT EXISTS (SELECT 1 FROM fm.visit v WHERE v.visit_id = NEW.visit_id AND v.site_uid = NEW.site_uid) THEN
|
||||||
|
RAISE EXCEPTION 'stock_observation %: site % differs from its visit''s site', NEW.source_key, NEW.site_uid;
|
||||||
|
END IF;
|
||||||
|
RETURN NEW;
|
||||||
|
END $$;
|
||||||
|
CREATE TRIGGER stock_observation_rules BEFORE INSERT OR UPDATE ON fm.stock_observation
|
||||||
|
FOR EACH ROW EXECUTE FUNCTION fm.tg_stock_observation_rules();
|
||||||
|
|
||||||
|
-- 3, 4. A reading's meter belongs to its visit's site; its epoch belongs to its meter.
|
||||||
|
CREATE FUNCTION fm.tg_hour_meter_reading_rules() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||||
|
BEGIN
|
||||||
|
IF NOT EXISTS (SELECT 1 FROM fm.visit v JOIN fm.runtime_meter m ON m.site_uid = v.site_uid
|
||||||
|
WHERE v.visit_id = NEW.visit_id AND m.meter_id = NEW.meter_id) THEN
|
||||||
|
RAISE EXCEPTION 'hour_meter_reading %: meter % is not a meter of the visit''s site', NEW.source_key, NEW.meter_id;
|
||||||
|
END IF;
|
||||||
|
IF NEW.epoch_id IS NOT NULL AND NOT EXISTS (
|
||||||
|
SELECT 1 FROM fm.meter_epoch e WHERE e.epoch_id = NEW.epoch_id AND e.meter_id = NEW.meter_id) THEN
|
||||||
|
RAISE EXCEPTION 'hour_meter_reading %: epoch % does not belong to meter %', NEW.source_key, NEW.epoch_id, NEW.meter_id;
|
||||||
|
END IF;
|
||||||
|
RETURN NEW;
|
||||||
|
END $$;
|
||||||
|
CREATE TRIGGER hour_meter_reading_rules BEFORE INSERT OR UPDATE ON fm.hour_meter_reading
|
||||||
|
FOR EACH ROW EXECUTE FUNCTION fm.tg_hour_meter_reading_rules();
|
||||||
|
|
||||||
|
-- 7. Supersession stays within one source family.
|
||||||
|
CREATE FUNCTION fm.tg_source_document_rules() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||||
|
BEGIN
|
||||||
|
IF NEW.supersedes_sha256 IS NOT NULL AND NOT EXISTS (
|
||||||
|
SELECT 1 FROM fm.source_document d WHERE d.sha256 = NEW.supersedes_sha256 AND d.source_family = NEW.source_family) THEN
|
||||||
|
RAISE EXCEPTION 'source_document %: it may supersede only a document of family %', left(NEW.sha256, 16), NEW.source_family;
|
||||||
|
END IF;
|
||||||
|
RETURN NEW;
|
||||||
|
END $$;
|
||||||
|
CREATE TRIGGER source_document_rules BEFORE INSERT OR UPDATE ON fm.source_document
|
||||||
|
FOR EACH ROW EXECUTE FUNCTION fm.tg_source_document_rules();
|
||||||
|
|
||||||
|
-- 8. A confirmed or proposed meter mapping names a generator placed at the meter's site at some
|
||||||
|
-- time. Whether the placement spans a given interval is fm.v_lph_interval's check.
|
||||||
|
CREATE FUNCTION fm.tg_runtime_meter_rules() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||||
|
BEGIN
|
||||||
|
IF NEW.mapping_status IN ('confirmed','proposed') AND NOT EXISTS (
|
||||||
|
SELECT 1 FROM fm.asset_placement p JOIN fm.asset a ON a.asset_uid = p.asset_uid
|
||||||
|
WHERE p.asset_uid = NEW.asset_uid AND p.site_uid = NEW.site_uid AND a.kind = 'generator') THEN
|
||||||
|
RAISE EXCEPTION 'runtime_meter %: % is not a generator placed at %', NEW.meter_id, NEW.asset_uid, NEW.site_uid;
|
||||||
|
END IF;
|
||||||
|
RETURN NEW;
|
||||||
|
END $$;
|
||||||
|
CREATE TRIGGER runtime_meter_rules BEFORE INSERT OR UPDATE ON fm.runtime_meter
|
||||||
|
FOR EACH ROW EXECUTE FUNCTION fm.tg_runtime_meter_rules();
|
||||||
|
|
||||||
|
-- 12. A generator_idle assumption names a generator placed at its own site during its dates.
|
||||||
|
CREATE FUNCTION fm.tg_assumption_rules() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||||
|
BEGIN
|
||||||
|
IF NEW.topic = 'generator_idle' AND NOT EXISTS (
|
||||||
|
SELECT 1 FROM fm.asset_placement p JOIN fm.asset a ON a.asset_uid = p.asset_uid
|
||||||
|
WHERE p.asset_uid = NEW.asset_uid AND p.site_uid = NEW.site_uid AND a.kind = 'generator'
|
||||||
|
AND daterange(coalesce(p.occupied_from,'-infinity'::date), p.occupied_to, '[)')
|
||||||
|
&& daterange(NEW.applies_from, NEW.applies_to, '[]')) THEN
|
||||||
|
RAISE EXCEPTION 'assumption %: % is not a generator placed at % during the assumption''s dates',
|
||||||
|
NEW.code, NEW.asset_uid, NEW.site_uid;
|
||||||
|
END IF;
|
||||||
|
RETURN NEW;
|
||||||
|
END $$;
|
||||||
|
CREATE TRIGGER assumption_rules BEFORE INSERT OR UPDATE ON fm.assumption
|
||||||
|
FOR EACH ROW EXECUTE FUNCTION fm.tg_assumption_rules();
|
||||||
|
|
||||||
|
-- 9. Members of a meter_coverage assumption: valid when written, written in the assumption's own
|
||||||
|
-- transaction, immutable afterwards, and complete at commit.
|
||||||
|
CREATE FUNCTION fm.tg_assumption_member_rules() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||||
|
DECLARE a fm.assumption;
|
||||||
|
BEGIN
|
||||||
|
IF TG_OP <> 'INSERT' THEN
|
||||||
|
RAISE EXCEPTION 'assumption_member: members are immutable; write a new assumption version';
|
||||||
|
END IF;
|
||||||
|
SELECT * INTO a FROM fm.assumption WHERE assumption_id = NEW.assumption_id;
|
||||||
|
IF a.topic <> 'meter_coverage' THEN
|
||||||
|
RAISE EXCEPTION 'assumption_member: % is not a meter_coverage assumption', a.code;
|
||||||
|
END IF;
|
||||||
|
IF a.created_xact <> pg_current_xact_id() THEN
|
||||||
|
RAISE EXCEPTION 'assumption_member: members of % are written in the transaction that created it', a.code;
|
||||||
|
END IF;
|
||||||
|
IF NEW.member_role = 'counter' THEN
|
||||||
|
IF NOT EXISTS (SELECT 1 FROM fm.runtime_meter m
|
||||||
|
WHERE m.meter_id = NEW.meter_id AND m.site_uid = a.site_uid AND m.mapping_status <> 'confirmed') THEN
|
||||||
|
RAISE EXCEPTION 'assumption_member: % is not an unconfirmed counter of %', NEW.meter_id, a.site_uid;
|
||||||
|
END IF;
|
||||||
|
ELSIF NOT EXISTS (SELECT 1 FROM fm.asset_placement p JOIN fm.asset s ON s.asset_uid = p.asset_uid
|
||||||
|
WHERE p.asset_uid = NEW.asset_uid AND p.site_uid = a.site_uid AND s.kind = 'generator') THEN
|
||||||
|
RAISE EXCEPTION 'assumption_member: % is not a generator placed at %', NEW.asset_uid, a.site_uid;
|
||||||
|
END IF;
|
||||||
|
RETURN NEW;
|
||||||
|
END $$;
|
||||||
|
CREATE TRIGGER assumption_member_rules BEFORE INSERT OR UPDATE OR DELETE ON fm.assumption_member
|
||||||
|
FOR EACH ROW EXECUTE FUNCTION fm.tg_assumption_member_rules();
|
||||||
|
|
||||||
|
CREATE FUNCTION fm.tg_coverage_members_complete() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||||
|
DECLARE n_counter integer; n_covered integer; code text;
|
||||||
|
BEGIN
|
||||||
|
SELECT x.code,
|
||||||
|
count(*) FILTER (WHERE m.member_role = 'counter'),
|
||||||
|
count(*) FILTER (WHERE m.member_role = 'covered_generator')
|
||||||
|
INTO code, n_counter, n_covered
|
||||||
|
FROM fm.assumption x LEFT JOIN fm.assumption_member m ON m.assumption_id = x.assumption_id
|
||||||
|
WHERE x.assumption_id = NEW.assumption_id GROUP BY x.code;
|
||||||
|
IF n_counter < 1 OR n_counter <> n_covered THEN
|
||||||
|
RAISE EXCEPTION 'assumption %: a meter_coverage assumption needs as many counters as covered generators, at least one (% vs %)',
|
||||||
|
code, n_counter, n_covered;
|
||||||
|
END IF;
|
||||||
|
RETURN NULL;
|
||||||
|
END $$;
|
||||||
|
CREATE CONSTRAINT TRIGGER coverage_members_complete AFTER INSERT ON fm.assumption
|
||||||
|
DEFERRABLE INITIALLY DEFERRED FOR EACH ROW WHEN (NEW.topic = 'meter_coverage')
|
||||||
|
EXECUTE FUNCTION fm.tg_coverage_members_complete();
|
||||||
|
CREATE CONSTRAINT TRIGGER coverage_members_complete AFTER INSERT ON fm.assumption_member
|
||||||
|
DEFERRABLE INITIALLY DEFERRED FOR EACH ROW
|
||||||
|
EXECUTE FUNCTION fm.tg_coverage_members_complete();
|
||||||
|
|
||||||
|
-- 1. A version starts as draft or proposed. Approval needs meets_policy and unchanged inputs
|
||||||
|
-- (fm.v_baseline_validity, created in 0002). An approved version is immutable; the only change
|
||||||
|
-- is approved -> superseded.
|
||||||
|
CREATE FUNCTION fm.tg_baseline_version_rules() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||||
|
BEGIN
|
||||||
|
IF TG_OP = 'INSERT' THEN
|
||||||
|
IF NEW.status NOT IN ('draft','proposed') THEN
|
||||||
|
RAISE EXCEPTION 'baseline_version %: a version starts as draft or proposed; approval is a later update', NEW.code;
|
||||||
|
END IF;
|
||||||
|
RETURN NEW;
|
||||||
|
END IF;
|
||||||
|
IF TG_OP = 'DELETE' THEN
|
||||||
|
IF OLD.approved_on IS NOT NULL THEN
|
||||||
|
RAISE EXCEPTION 'baseline_version %: an approved version is never deleted', OLD.code;
|
||||||
|
END IF;
|
||||||
|
RETURN OLD;
|
||||||
|
END IF;
|
||||||
|
IF OLD.approved_on IS NOT NULL
|
||||||
|
AND NOT (OLD.status = 'approved' AND NEW.status = 'superseded'
|
||||||
|
AND (to_jsonb(OLD) - 'status') = (to_jsonb(NEW) - 'status')) THEN
|
||||||
|
RAISE EXCEPTION 'baseline_version %: approved rows are immutable; the only change is status -> superseded', OLD.code;
|
||||||
|
END IF;
|
||||||
|
IF NEW.status = 'approved' AND OLD.status <> 'approved' THEN
|
||||||
|
IF NOT NEW.meets_policy THEN
|
||||||
|
RAISE EXCEPTION 'baseline_version %: does not meet % (%)', NEW.code, NEW.policy_version, coalesce(NEW.policy_check, 'no check recorded');
|
||||||
|
END IF;
|
||||||
|
IF NOT coalesce((SELECT inputs_unchanged FROM fm.v_baseline_validity WHERE baseline_id = NEW.baseline_id), false) THEN
|
||||||
|
RAISE EXCEPTION 'baseline_version %: its inputs changed since computation; re-propose', NEW.code;
|
||||||
|
END IF;
|
||||||
|
END IF;
|
||||||
|
RETURN NEW;
|
||||||
|
END $$;
|
||||||
|
CREATE TRIGGER baseline_version_rules BEFORE INSERT OR UPDATE OR DELETE ON fm.baseline_version
|
||||||
|
FOR EACH ROW EXECUTE FUNCTION fm.tg_baseline_version_rules();
|
||||||
|
|
||||||
|
-- 1, 6. Inputs: written only while the version is draft or proposed, never moved to another
|
||||||
|
-- version, never changed once their version has been approved; dates equal the endpoint
|
||||||
|
-- visits; included rows carry the version's population.
|
||||||
|
CREATE FUNCTION fm.tg_baseline_input_rules() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||||
|
DECLARE v fm.baseline_version; o date; c date;
|
||||||
|
BEGIN
|
||||||
|
SELECT * INTO v FROM fm.baseline_version
|
||||||
|
WHERE baseline_id = CASE WHEN TG_OP = 'INSERT' THEN NEW.baseline_id ELSE OLD.baseline_id END;
|
||||||
|
IF TG_OP <> 'INSERT' AND v.approved_on IS NOT NULL THEN
|
||||||
|
RAISE EXCEPTION 'baseline_input: the inputs of approved version % are immutable', v.code;
|
||||||
|
END IF;
|
||||||
|
IF TG_OP = 'DELETE' THEN RETURN OLD; END IF;
|
||||||
|
IF TG_OP = 'UPDATE' AND (NEW.baseline_id, NEW.site_uid) IS DISTINCT FROM (OLD.baseline_id, OLD.site_uid) THEN
|
||||||
|
RAISE EXCEPTION 'baseline_input: an input of % never moves to another version', v.code;
|
||||||
|
END IF;
|
||||||
|
IF v.status NOT IN ('draft','proposed') THEN
|
||||||
|
RAISE EXCEPTION 'baseline_input: version % is %; inputs are written only while it is draft or proposed', v.code, v.status;
|
||||||
|
END IF;
|
||||||
|
SELECT visit_on INTO o FROM fm.visit WHERE visit_id = NEW.open_visit_id;
|
||||||
|
SELECT visit_on INTO c FROM fm.visit WHERE visit_id = NEW.close_visit_id;
|
||||||
|
IF NEW.start_on <> o OR NEW.end_on <> c THEN
|
||||||
|
RAISE EXCEPTION 'baseline_input: dates % - % differ from the endpoint visits % - %', NEW.start_on, NEW.end_on, o, c;
|
||||||
|
END IF;
|
||||||
|
IF NEW.included AND NEW.population_key IS DISTINCT FROM v.population_key THEN
|
||||||
|
RAISE EXCEPTION 'baseline_input: included population % differs from version %''s %', NEW.population_key, v.code, v.population_key;
|
||||||
|
END IF;
|
||||||
|
RETURN NEW;
|
||||||
|
END $$;
|
||||||
|
CREATE TRIGGER baseline_input_rules BEFORE INSERT OR UPDATE OR DELETE ON fm.baseline_input
|
||||||
|
FOR EACH ROW EXECUTE FUNCTION fm.tg_baseline_input_rules();
|
||||||
|
|
||||||
|
-- 13. Identity and classification never change in place: the derivations and the dependency
|
||||||
|
-- hashes (0002) rest on them. A wrong visit date is corrected by a new visit, which the
|
||||||
|
-- superseding claims point to.
|
||||||
|
CREATE FUNCTION fm.tg_fixed_columns() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||||
|
DECLARE c text;
|
||||||
|
BEGIN
|
||||||
|
FOREACH c IN ARRAY TG_ARGV LOOP
|
||||||
|
IF to_jsonb(OLD)->c IS DISTINCT FROM to_jsonb(NEW)->c THEN
|
||||||
|
RAISE EXCEPTION '%: % never changes in place', TG_TABLE_NAME, c;
|
||||||
|
END IF;
|
||||||
|
END LOOP;
|
||||||
|
RETURN NEW;
|
||||||
|
END $$;
|
||||||
|
CREATE TRIGGER visit_fixed BEFORE UPDATE ON fm.visit
|
||||||
|
FOR EACH ROW EXECUTE FUNCTION fm.tg_fixed_columns('visit_key','site_uid','visit_on','seq_in_day');
|
||||||
|
CREATE TRIGGER source_document_fixed BEFORE UPDATE ON fm.source_document
|
||||||
|
FOR EACH ROW EXECUTE FUNCTION fm.tg_fixed_columns('sha256','source_family','kind');
|
||||||
|
CREATE TRIGGER asset_fixed BEFORE UPDATE ON fm.asset
|
||||||
|
FOR EACH ROW EXECUTE FUNCTION fm.tg_fixed_columns('asset_uid','kind');
|
||||||
|
CREATE TRIGGER runtime_meter_fixed BEFORE UPDATE ON fm.runtime_meter -- readings follow their meter's site
|
||||||
|
FOR EACH ROW EXECUTE FUNCTION fm.tg_fixed_columns('meter_id','site_uid');
|
||||||
|
CREATE TRIGGER meter_epoch_fixed BEFORE UPDATE ON fm.meter_epoch -- and their epoch's meter
|
||||||
|
FOR EACH ROW EXECUTE FUNCTION fm.tg_fixed_columns('epoch_id','meter_id');
|
||||||
|
|
||||||
|
COMMIT;
|
||||||
@@ -0,0 +1,503 @@
|
|||||||
|
-- 0002_fm_derivations.sql: internal derivations in schema `fm` (not visible to Trino).
|
||||||
|
-- DRAFT, not applied. Applied after 0001 by a loader acting as fuel_owner (README in this folder).
|
||||||
|
-- Design: fuel-monitoring .context/plans/2026-10-24-fuel-db-baseline-dashboard.md, §4 and §5.1.
|
||||||
|
BEGIN;
|
||||||
|
SET LOCAL search_path = fm, pg_catalog, public;
|
||||||
|
|
||||||
|
CREATE VIEW fm.v_source_rank AS
|
||||||
|
SELECT sha256, kind, CASE kind WHEN 'delivery_form' THEN 1 WHEN 'site_logbook' THEN 2
|
||||||
|
WHEN 'operator_record' THEN 3 WHEN 'operator_workbook' THEN 3 ELSE 9 END AS rank
|
||||||
|
FROM fm.source_document;
|
||||||
|
|
||||||
|
-- The analytical visit set: visits that still carry a live claim. A visit whose claims were all
|
||||||
|
-- superseded or retracted stays in fm.visit for provenance only and is invisible from here on.
|
||||||
|
CREATE VIEW fm.v_visit_live AS
|
||||||
|
SELECT v.* FROM fm.visit v
|
||||||
|
WHERE EXISTS (SELECT 1 FROM fm.fuel_movement m WHERE m.visit_id = v.visit_id
|
||||||
|
AND m.claim_status IN ('asserted','reviewed','disputed'))
|
||||||
|
OR EXISTS (SELECT 1 FROM fm.stock_observation s WHERE s.visit_id = v.visit_id
|
||||||
|
AND s.claim_status IN ('asserted','reviewed','disputed'))
|
||||||
|
OR EXISTS (SELECT 1 FROM fm.hour_meter_reading h WHERE h.visit_id = v.visit_id
|
||||||
|
AND h.claim_status IN ('asserted','reviewed','disputed'));
|
||||||
|
|
||||||
|
-- One row per physical movement (event_key; trigger rule 10 keeps kind and tanks equal within it).
|
||||||
|
-- Value = the best-ranked asserted or reviewed claim. The spread covers every live claim, and a
|
||||||
|
-- disputed claim makes the event unreviewed. No value = an unquantified movement.
|
||||||
|
CREATE VIEW fm.v_movement_event AS
|
||||||
|
SELECT m.visit_id, m.event_key, m.kind, m.from_tank_uid, m.to_tank_uid,
|
||||||
|
(array_agg(m.litres ORDER BY r.rank, m.movement_id)
|
||||||
|
FILTER (WHERE m.claim_status IN ('asserted','reviewed') AND m.litres IS NOT NULL))[1] AS litres,
|
||||||
|
coalesce(max(m.litres) - min(m.litres), 0) AS spread_l,
|
||||||
|
bool_and(m.claim_status = 'reviewed') AS all_reviewed,
|
||||||
|
count(*) AS n_claims
|
||||||
|
FROM fm.fuel_movement m JOIN fm.v_source_rank r ON r.sha256 = m.source_sha256
|
||||||
|
WHERE m.claim_status IN ('asserted','reviewed','disputed')
|
||||||
|
GROUP BY m.visit_id, m.event_key, m.kind, m.from_tank_uid, m.to_tank_uid;
|
||||||
|
|
||||||
|
-- Quantified site-level deliveries per visit, summed over events (never over claims).
|
||||||
|
CREATE VIEW fm.v_visit_delivery AS
|
||||||
|
SELECT visit_id, sum(litres) AS delivered_l, sum(spread_l) AS delivered_spread_l,
|
||||||
|
bool_and(all_reviewed) AS all_reviewed, count(*) AS n_events
|
||||||
|
FROM fm.v_movement_event
|
||||||
|
WHERE kind = 'delivery' AND to_tank_uid IS NULL AND litres IS NOT NULL
|
||||||
|
GROUP BY visit_id;
|
||||||
|
|
||||||
|
-- Every other external movement event (internal transfers cancel out at site level). Between two
|
||||||
|
-- endpoints they are booked; AT an endpoint visit their order relative to the stock readings is
|
||||||
|
-- unknown, so the interval is excluded instead of guessed (see v_stock_interval).
|
||||||
|
CREATE VIEW fm.v_visit_other_movement AS
|
||||||
|
SELECT visit_id,
|
||||||
|
sum(litres) FILTER (WHERE kind IN ('withdrawal','drain','spill','loss_reported')) AS out_l,
|
||||||
|
coalesce(sum(spread_l) FILTER (WHERE kind IN ('withdrawal','drain','spill','loss_reported')), 0) AS out_spread_l,
|
||||||
|
count(*) AS n_external,
|
||||||
|
count(*) FILTER (WHERE litres IS NULL OR kind = 'external_in'
|
||||||
|
OR (kind = 'delivery' AND to_tank_uid IS NOT NULL)) AS n_unquantified,
|
||||||
|
bool_and(all_reviewed) AS all_reviewed
|
||||||
|
FROM fm.v_movement_event
|
||||||
|
WHERE kind <> 'internal_transfer'
|
||||||
|
AND NOT (kind = 'delivery' AND to_tank_uid IS NULL AND litres IS NOT NULL)
|
||||||
|
GROUP BY visit_id;
|
||||||
|
|
||||||
|
CREATE VIEW fm.v_visit_stock AS -- site-level stock per visit and phase; same claim rules
|
||||||
|
SELECT s.visit_id, s.phase,
|
||||||
|
(array_agg(s.litres ORDER BY r.rank, s.stock_obs_id)
|
||||||
|
FILTER (WHERE s.claim_status IN ('asserted','reviewed') AND s.litres IS NOT NULL))[1] AS litres,
|
||||||
|
max(s.litres) - min(s.litres) AS spread_l,
|
||||||
|
max(s.resolution_l) FILTER (WHERE s.litres IS NOT NULL) AS resolution_l,
|
||||||
|
bool_and(s.claim_status = 'reviewed') AS all_reviewed, -- every live claim, with a number or not
|
||||||
|
count(*) AS n_claims
|
||||||
|
FROM fm.stock_observation s JOIN fm.v_source_rank r ON r.sha256 = s.source_sha256
|
||||||
|
WHERE s.claim_status IN ('asserted','reviewed','disputed') AND s.scope = 'site_total'
|
||||||
|
GROUP BY s.visit_id, s.phase
|
||||||
|
HAVING count(*) FILTER (WHERE s.claim_status IN ('asserted','reviewed') AND s.litres IS NOT NULL) > 0;
|
||||||
|
-- no asserted or reviewed number (only disputed, illegible or placeholder claims): no reading
|
||||||
|
-- v1: tank-level rows are summed only in a later version, and only when every placed tank has one.
|
||||||
|
|
||||||
|
CREATE VIEW fm.v_stock_point AS -- stock on arrival and on departure of each live visit
|
||||||
|
SELECT v.visit_id, v.visit_key, v.site_uid, v.visit_on, v.seq_in_day, v.kind,
|
||||||
|
d.delivered_l, coalesce(o.n_external, 0) AS n_other_movements,
|
||||||
|
coalesce(b.litres, st.litres, a.litres - d.delivered_l) AS arrive_l,
|
||||||
|
CASE WHEN coalesce(b.litres, st.litres) IS NOT NULL THEN 'stated'
|
||||||
|
WHEN a.litres - d.delivered_l IS NOT NULL THEN 'derived' END AS arrive_basis,
|
||||||
|
CASE WHEN b.litres IS NOT NULL THEN b.resolution_l WHEN st.litres IS NOT NULL THEN st.resolution_l
|
||||||
|
ELSE a.resolution_l END AS arrive_res_l,
|
||||||
|
CASE WHEN b.litres IS NOT NULL THEN b.spread_l WHEN st.litres IS NOT NULL THEN st.spread_l
|
||||||
|
ELSE a.spread_l + d.delivered_spread_l END AS arrive_spread_l,
|
||||||
|
CASE WHEN b.litres IS NOT NULL THEN b.all_reviewed WHEN st.litres IS NOT NULL THEN st.all_reviewed
|
||||||
|
ELSE a.all_reviewed AND d.all_reviewed END AS arrive_reviewed,
|
||||||
|
coalesce(a.litres, st.litres, b.litres + d.delivered_l) AS depart_l,
|
||||||
|
CASE WHEN coalesce(a.litres, st.litres) IS NOT NULL THEN 'stated'
|
||||||
|
WHEN b.litres + d.delivered_l IS NOT NULL THEN 'derived' END AS depart_basis,
|
||||||
|
CASE WHEN a.litres IS NOT NULL THEN a.resolution_l WHEN st.litres IS NOT NULL THEN st.resolution_l
|
||||||
|
ELSE b.resolution_l END AS depart_res_l,
|
||||||
|
CASE WHEN a.litres IS NOT NULL THEN a.spread_l WHEN st.litres IS NOT NULL THEN st.spread_l
|
||||||
|
ELSE b.spread_l + d.delivered_spread_l END AS depart_spread_l,
|
||||||
|
CASE WHEN a.litres IS NOT NULL THEN a.all_reviewed WHEN st.litres IS NOT NULL THEN st.all_reviewed
|
||||||
|
ELSE b.all_reviewed AND d.all_reviewed END AS depart_reviewed,
|
||||||
|
a.litres - (b.litres + d.delivered_l) AS after_vs_before_plus_delivered_l -- QA
|
||||||
|
FROM fm.v_visit_live v
|
||||||
|
LEFT JOIN fm.v_visit_delivery d ON d.visit_id = v.visit_id
|
||||||
|
LEFT JOIN fm.v_visit_other_movement o ON o.visit_id = v.visit_id
|
||||||
|
LEFT JOIN fm.v_visit_stock b ON b.visit_id = v.visit_id AND b.phase = 'before_delivery'
|
||||||
|
LEFT JOIN fm.v_visit_stock a ON a.visit_id = v.visit_id AND a.phase = 'after_delivery'
|
||||||
|
LEFT JOIN fm.v_visit_stock st ON st.visit_id = v.visit_id AND st.phase = 'standalone';
|
||||||
|
|
||||||
|
CREATE VIEW fm.v_stock_interval AS -- consecutive stock points per site; no overlap by construction
|
||||||
|
WITH pts AS (
|
||||||
|
SELECT p.*, row_number() OVER (PARTITION BY p.site_uid ORDER BY p.visit_on, p.seq_in_day) AS n
|
||||||
|
FROM fm.v_stock_point p WHERE p.arrive_l IS NOT NULL OR p.depart_l IS NOT NULL),
|
||||||
|
pr AS (
|
||||||
|
SELECT o.site_uid, o.visit_key || '>' || c.visit_key AS interval_key,
|
||||||
|
o.visit_id AS open_visit_id, c.visit_id AS close_visit_id,
|
||||||
|
o.visit_on AS start_on, c.visit_on AS end_on, c.visit_on - o.visit_on AS days,
|
||||||
|
o.depart_l AS opening_l, o.depart_basis AS opening_basis,
|
||||||
|
c.arrive_l AS closing_l, c.arrive_basis AS closing_basis,
|
||||||
|
coalesce(o.depart_res_l,0) + coalesce(c.arrive_res_l,0) AS endpoint_res_l,
|
||||||
|
coalesce(o.depart_spread_l,0) + coalesce(c.arrive_spread_l,0) AS endpoint_spread_l,
|
||||||
|
coalesce(o.depart_reviewed AND c.arrive_reviewed, false) AS endpoints_reviewed,
|
||||||
|
o.n_other_movements + c.n_other_movements AS n_endpoint_movements,
|
||||||
|
o.visit_on AS o_on, o.seq_in_day AS o_seq, c.visit_on AS c_on, c.seq_in_day AS c_seq
|
||||||
|
FROM pts o JOIN pts c ON c.site_uid = o.site_uid AND c.n = o.n + 1)
|
||||||
|
SELECT p.site_uid, p.interval_key, p.open_visit_id, p.close_visit_id, p.start_on, p.end_on, p.days,
|
||||||
|
p.opening_l, p.opening_basis, p.closing_l, p.closing_basis,
|
||||||
|
coalesce(mid.delivered_l, 0) AS deliveries_between_l, coalesce(mid.out_l, 0) AS out_l,
|
||||||
|
p.opening_l + coalesce(mid.delivered_l,0) - coalesce(mid.out_l,0) - p.closing_l AS consumed_l,
|
||||||
|
p.endpoint_res_l + p.endpoint_spread_l + coalesce(mid.spread_l,0) AS consumed_unc_l,
|
||||||
|
(p.endpoint_spread_l + coalesce(mid.spread_l,0)) > 0 AS inputs_conflict, -- any disagreement, as v_data_quality
|
||||||
|
p.endpoints_reviewed AND coalesce(mid.all_reviewed, true) AS inputs_reviewed,
|
||||||
|
CASE WHEN p.opening_l IS NULL OR p.closing_l IS NULL THEN 'no_opening_or_closing'
|
||||||
|
WHEN p.n_endpoint_movements > 0 THEN 'endpoint_movement_order_unknown'
|
||||||
|
WHEN mid.n_unknown > 0 THEN 'unquantified_movement'
|
||||||
|
WHEN p.days = 0 THEN 'same_day'
|
||||||
|
WHEN p.opening_l + coalesce(mid.delivered_l,0) - coalesce(mid.out_l,0) - p.closing_l
|
||||||
|
< -(p.endpoint_res_l + p.endpoint_spread_l + coalesce(mid.spread_l,0)) THEN 'negative'
|
||||||
|
ELSE 'ok' END AS stock_status
|
||||||
|
-- mid: live visits strictly between the endpoints. A visit counts only through its live movement
|
||||||
|
-- events, never through its kind.
|
||||||
|
FROM pr p
|
||||||
|
CROSS JOIN LATERAL (
|
||||||
|
SELECT sum(d.delivered_l) AS delivered_l,
|
||||||
|
sum(coalesce(d.delivered_spread_l, 0) + coalesce(x.out_spread_l, 0)) AS spread_l,
|
||||||
|
sum(x.out_l) AS out_l,
|
||||||
|
bool_and(coalesce(d.all_reviewed, true) AND coalesce(x.all_reviewed, true)) AS all_reviewed,
|
||||||
|
coalesce(sum(x.n_unquantified), 0) AS n_unknown
|
||||||
|
FROM fm.v_visit_live v
|
||||||
|
LEFT JOIN fm.v_visit_delivery d ON d.visit_id = v.visit_id
|
||||||
|
LEFT JOIN fm.v_visit_other_movement x ON x.visit_id = v.visit_id
|
||||||
|
WHERE v.site_uid = p.site_uid
|
||||||
|
AND (v.visit_on, v.seq_in_day) > (p.o_on, p.o_seq)
|
||||||
|
AND (v.visit_on, v.seq_in_day) < (p.c_on, p.c_seq)) mid;
|
||||||
|
|
||||||
|
CREATE VIEW fm.v_visit_reading AS -- one reading per visit and meter, with the claims' spread
|
||||||
|
SELECT h.visit_id, h.meter_id,
|
||||||
|
min(h.epoch_id) FILTER (WHERE h.hours IS NOT NULL) AS epoch_id,
|
||||||
|
count(DISTINCT h.epoch_id) FILTER (WHERE h.hours IS NOT NULL) AS n_epochs,
|
||||||
|
count(*) FILTER (WHERE h.hours IS NOT NULL AND h.epoch_id IS NULL) AS n_unassigned_epoch,
|
||||||
|
(array_agg(h.hours ORDER BY r.rank, h.reading_id)
|
||||||
|
FILTER (WHERE h.claim_status IN ('asserted','reviewed') AND h.hours IS NOT NULL))[1] AS hours,
|
||||||
|
min(h.hours) AS hours_min, max(h.hours) AS hours_max,
|
||||||
|
max(h.resolution_h) FILTER (WHERE h.hours IS NOT NULL) AS resolution_h,
|
||||||
|
bool_or(h.date_basis <> 'stated') FILTER (WHERE h.hours IS NOT NULL) AS date_inferred,
|
||||||
|
bool_and(h.claim_status = 'reviewed') AS all_reviewed -- every live claim, with a number or not
|
||||||
|
FROM fm.hour_meter_reading h JOIN fm.v_source_rank r ON r.sha256 = h.source_sha256
|
||||||
|
WHERE h.claim_status IN ('asserted','reviewed','disputed')
|
||||||
|
GROUP BY h.visit_id, h.meter_id
|
||||||
|
HAVING count(*) FILTER (WHERE h.claim_status IN ('asserted','reviewed') AND h.hours IS NOT NULL) > 0;
|
||||||
|
-- no asserted or reviewed number (only disputed or illegible claims): no reading
|
||||||
|
|
||||||
|
-- M2 per stock interval. Only a validated population is summed (CTE sel):
|
||||||
|
-- * a counter CONFIRMED for a generator placed at the site in the interval, when that generator
|
||||||
|
-- has exactly one such counter and no idle assumption;
|
||||||
|
-- * counters not tied to a set (unmapped, or mapping only proposed), only when the one live
|
||||||
|
-- meter_coverage assumption for the dates names exactly these counters, exactly the generators
|
||||||
|
-- left uncovered and exactly the idle sets. Such intervals give only the summed-set-hour metric.
|
||||||
|
-- A counter confirmed for a generator NOT placed at the site in the interval is never summed and
|
||||||
|
-- excludes the interval. Runtime and L/h are reported only for a valid population.
|
||||||
|
CREATE VIEW fm.v_lph_interval AS
|
||||||
|
WITH gen AS ( -- generators placed at the site at any time in the interval; full_cover =
|
||||||
|
-- one placement spans the whole interval, both endpoint dates included
|
||||||
|
SELECT i.interval_key, p.asset_uid,
|
||||||
|
bool_or(daterange(coalesce(p.occupied_from,'-infinity'::date), p.occupied_to, '[)')
|
||||||
|
@> daterange(i.start_on, i.end_on, '[]')) AS full_cover
|
||||||
|
FROM fm.v_stock_interval i
|
||||||
|
JOIN fm.asset_placement p ON p.site_uid = i.site_uid
|
||||||
|
JOIN fm.asset a ON a.asset_uid = p.asset_uid AND a.kind = 'generator'
|
||||||
|
WHERE daterange(coalesce(p.occupied_from,'-infinity'::date), p.occupied_to, '[)')
|
||||||
|
&& daterange(i.start_on, i.end_on, '[]')
|
||||||
|
GROUP BY i.interval_key, p.asset_uid),
|
||||||
|
pair AS ( -- every counter read at both endpoints, classified
|
||||||
|
SELECT i.interval_key, m.meter_id,
|
||||||
|
CASE WHEN m.mapping_status = 'confirmed' THEN m.asset_uid END AS asset_uid,
|
||||||
|
CASE WHEN m.mapping_status <> 'confirmed' THEN 'not_tied'
|
||||||
|
WHEN EXISTS (SELECT 1 FROM gen g
|
||||||
|
WHERE g.interval_key = i.interval_key AND g.asset_uid = m.asset_uid) THEN 'tied'
|
||||||
|
ELSE 'tied_elsewhere' END AS tie,
|
||||||
|
(o.n_epochs = 1 AND c.n_epochs = 1 AND o.n_unassigned_epoch = 0 AND c.n_unassigned_epoch = 0
|
||||||
|
AND o.epoch_id = c.epoch_id) AS same_epoch,
|
||||||
|
c.hours - o.hours AS runtime_h,
|
||||||
|
c.hours_min - o.hours_max - o.resolution_h - c.resolution_h AS runtime_low_h,
|
||||||
|
c.hours_max - o.hours_min + o.resolution_h + c.resolution_h AS runtime_high_h,
|
||||||
|
o.date_inferred OR c.date_inferred AS date_inferred,
|
||||||
|
o.all_reviewed AND c.all_reviewed AS reviewed,
|
||||||
|
(o.hours_max > o.hours_min OR c.hours_max > c.hours_min) AS conflict
|
||||||
|
FROM fm.v_stock_interval i
|
||||||
|
JOIN fm.runtime_meter m ON m.site_uid = i.site_uid
|
||||||
|
JOIN fm.v_visit_reading o ON o.visit_id = i.open_visit_id AND o.meter_id = m.meter_id
|
||||||
|
JOIN fm.v_visit_reading c ON c.visit_id = i.close_visit_id AND c.meter_id = m.meter_id),
|
||||||
|
idle AS ( -- at most one live generator_idle per asset and date (EXCLUDE)
|
||||||
|
SELECT g.interval_key, g.asset_uid, x.code, x.status
|
||||||
|
FROM gen g JOIN fm.v_stock_interval i USING (interval_key)
|
||||||
|
JOIN fm.assumption x ON x.asset_uid = g.asset_uid AND x.site_uid = i.site_uid -- this site's only
|
||||||
|
AND x.topic = 'generator_idle' AND x.status IN ('proposed','accepted')
|
||||||
|
AND daterange(x.applies_from, x.applies_to, '[]') @> daterange(i.start_on, i.end_on, '[]')),
|
||||||
|
per_gen AS (
|
||||||
|
SELECT g.interval_key, g.asset_uid, g.full_cover,
|
||||||
|
count(DISTINCT p.meter_id) AS n_counters,
|
||||||
|
count(DISTINCT d.code) > 0 AS is_idle,
|
||||||
|
coalesce(bool_or(d.status = 'proposed'), false) AS idle_proposed
|
||||||
|
FROM gen g
|
||||||
|
LEFT JOIN pair p ON p.interval_key = g.interval_key AND p.asset_uid = g.asset_uid AND p.tie = 'tied'
|
||||||
|
LEFT JOIN idle d ON d.interval_key = g.interval_key AND d.asset_uid = g.asset_uid
|
||||||
|
GROUP BY g.interval_key, g.asset_uid, g.full_cover),
|
||||||
|
cov AS (
|
||||||
|
SELECT interval_key, count(*) AS n_generators,
|
||||||
|
count(*) FILTER (WHERE NOT full_cover) AS n_partial,
|
||||||
|
count(*) FILTER (WHERE n_counters > 1) AS n_double_counted,
|
||||||
|
count(*) FILTER (WHERE n_counters >= 1 AND is_idle) AS n_metered_and_idle,
|
||||||
|
count(*) FILTER (WHERE n_counters = 0 AND NOT is_idle) AS n_uncovered,
|
||||||
|
coalesce(array_agg(asset_uid ORDER BY asset_uid)
|
||||||
|
FILTER (WHERE n_counters = 0 AND NOT is_idle), '{}') AS uncovered_set,
|
||||||
|
coalesce(array_agg(asset_uid ORDER BY asset_uid) FILTER (WHERE is_idle), '{}') AS idle_set,
|
||||||
|
bool_or(idle_proposed) AS idle_proposed
|
||||||
|
FROM per_gen GROUP BY interval_key),
|
||||||
|
loose AS ( -- counters read at both ends but not tied to a set
|
||||||
|
SELECT interval_key, array_agg(meter_id ORDER BY meter_id) AS meters
|
||||||
|
FROM pair WHERE tie = 'not_tied' GROUP BY interval_key),
|
||||||
|
cova AS ( -- live meter_coverage assumptions with their exact members
|
||||||
|
SELECT x.assumption_id, x.code, x.site_uid, x.status, x.applies_from, x.applies_to,
|
||||||
|
coalesce(array_agg(m.meter_id ORDER BY m.meter_id)
|
||||||
|
FILTER (WHERE m.member_role = 'counter'), '{}') AS meters,
|
||||||
|
coalesce(array_agg(m.asset_uid ORDER BY m.asset_uid)
|
||||||
|
FILTER (WHERE m.member_role = 'covered_generator'), '{}') AS covered,
|
||||||
|
coalesce(array_agg(m.asset_uid ORDER BY m.asset_uid)
|
||||||
|
FILTER (WHERE m.member_role = 'idle_generator'), '{}') AS idle_set
|
||||||
|
FROM fm.assumption x JOIN fm.assumption_member m ON m.assumption_id = x.assumption_id
|
||||||
|
WHERE x.topic = 'meter_coverage' AND x.status IN ('proposed','accepted')
|
||||||
|
GROUP BY x.assumption_id),
|
||||||
|
unm AS ( -- at most one live coverage assumption per site and date (EXCLUDE)
|
||||||
|
SELECT l.interval_key, cardinality(l.meters) AS n_unmapped, c.code AS cov_code, c.status AS cov_status,
|
||||||
|
c.covered AS cov_covered,
|
||||||
|
coalesce(c.meters = l.meters
|
||||||
|
AND c.covered = coalesce(cv.uncovered_set, '{}')
|
||||||
|
AND c.idle_set = coalesce(cv.idle_set, '{}')
|
||||||
|
AND cardinality(c.meters) = cardinality(c.covered), false) AS cov_matches
|
||||||
|
FROM loose l
|
||||||
|
JOIN fm.v_stock_interval i USING (interval_key)
|
||||||
|
LEFT JOIN cov cv USING (interval_key)
|
||||||
|
LEFT JOIN cova c ON c.site_uid = i.site_uid
|
||||||
|
AND daterange(c.applies_from, c.applies_to, '[]') @> daterange(i.start_on, i.end_on, '[]')),
|
||||||
|
sel AS ( -- the validated population; nothing else is summed
|
||||||
|
SELECT p.* FROM pair p
|
||||||
|
JOIN per_gen g ON g.interval_key = p.interval_key AND g.asset_uid = p.asset_uid
|
||||||
|
WHERE p.tie = 'tied' AND g.n_counters = 1 AND NOT g.is_idle
|
||||||
|
UNION ALL
|
||||||
|
SELECT p.* FROM pair p JOIN unm u ON u.interval_key = p.interval_key
|
||||||
|
WHERE p.tie = 'not_tied' AND u.cov_matches),
|
||||||
|
agg AS (
|
||||||
|
SELECT interval_key, count(*) AS n_meters, sum(runtime_h) AS runtime_h,
|
||||||
|
sum(runtime_low_h) AS runtime_low_h, sum(runtime_high_h) AS runtime_high_h,
|
||||||
|
bool_and(same_epoch) AS same_epoch, bool_and(runtime_h >= 0) AS monotonic,
|
||||||
|
max(runtime_h) AS max_meter_h, bool_or(date_inferred) AS date_inferred,
|
||||||
|
bool_and(reviewed) AS runtime_reviewed, bool_or(conflict) AS runtime_conflict,
|
||||||
|
string_agg(coalesce(asset_uid, 'meter:' || meter_id), ','
|
||||||
|
ORDER BY coalesce(asset_uid, 'meter:' || meter_id)) AS population_key
|
||||||
|
FROM sel GROUP BY interval_key),
|
||||||
|
rd AS ( -- what was read at both ends, before validation
|
||||||
|
SELECT interval_key, count(*) FILTER (WHERE tie = 'tied_elsewhere') AS n_tied_elsewhere
|
||||||
|
FROM pair GROUP BY interval_key),
|
||||||
|
pv AS ( -- valid = the summed population covers every placed generator exactly once
|
||||||
|
SELECT rd.interval_key,
|
||||||
|
coalesce(cv.n_generators > 0 AND rd.n_tied_elsewhere = 0 AND cv.n_partial = 0
|
||||||
|
AND cv.n_double_counted = 0 AND cv.n_metered_and_idle = 0
|
||||||
|
AND CASE WHEN u.interval_key IS NULL THEN cv.n_uncovered = 0 ELSE u.cov_matches END
|
||||||
|
AND a.n_meters >= 1, false) AS population_valid
|
||||||
|
FROM rd LEFT JOIN cov cv USING (interval_key) LEFT JOIN unm u USING (interval_key)
|
||||||
|
LEFT JOIN agg a USING (interval_key)),
|
||||||
|
x AS (
|
||||||
|
SELECT i.*,
|
||||||
|
coalesce(pv.population_valid, false) AS population_valid,
|
||||||
|
a.n_meters, u.cov_code,
|
||||||
|
CASE WHEN pv.population_valid THEN a.runtime_h END AS runtime_h,
|
||||||
|
CASE WHEN pv.population_valid THEN a.runtime_low_h END AS runtime_low_h,
|
||||||
|
CASE WHEN pv.population_valid THEN a.runtime_high_h END AS runtime_high_h,
|
||||||
|
CASE WHEN pv.population_valid THEN -- counters not tied to a set also name the sets they cover
|
||||||
|
(SELECT string_agg(k, ',' ORDER BY k)
|
||||||
|
FROM unnest(string_to_array(a.population_key, ',')
|
||||||
|
|| CASE WHEN u.cov_matches THEN array(SELECT 'covers:' || g FROM unnest(u.cov_covered) g)
|
||||||
|
ELSE '{}'::text[] END) k) END AS population_key,
|
||||||
|
a.date_inferred, a.runtime_reviewed, a.runtime_conflict,
|
||||||
|
coalesce(cv.n_generators, 0) AS n_generators, coalesce(u.n_unmapped, 0) AS n_unmapped,
|
||||||
|
coalesce(cv.idle_proposed, false)
|
||||||
|
OR (coalesce(u.cov_matches, false) AND u.cov_status = 'proposed') AS depends_on_proposed,
|
||||||
|
concat_ws('; ',
|
||||||
|
CASE WHEN i.stock_status <> 'ok' THEN 'stock interval: ' || i.stock_status END,
|
||||||
|
CASE WHEN rd.interval_key IS NULL THEN 'no timer reading at both ends' END,
|
||||||
|
CASE WHEN coalesce(cv.n_generators, 0) = 0 THEN 'no generator placed at the site' END,
|
||||||
|
CASE WHEN cv.n_partial > 0
|
||||||
|
THEN 'a generator was placed at or removed from the site within the interval' END,
|
||||||
|
CASE WHEN rd.n_tied_elsewhere > 0
|
||||||
|
THEN 'a counter is confirmed for a generator not placed here in the interval' END,
|
||||||
|
CASE WHEN cv.n_double_counted > 0 THEN 'two counters for one generator' END,
|
||||||
|
CASE WHEN cv.n_metered_and_idle > 0 THEN 'a generator is both metered and assumed idle' END,
|
||||||
|
CASE WHEN u.interval_key IS NULL AND cv.n_uncovered > 0
|
||||||
|
THEN 'a generator has neither a counter nor an idle assumption' END,
|
||||||
|
CASE WHEN u.interval_key IS NOT NULL AND u.cov_code IS NULL
|
||||||
|
THEN 'counters not tied to sets, and no meter-coverage assumption for the whole interval' END,
|
||||||
|
CASE WHEN u.cov_code IS NOT NULL AND NOT u.cov_matches
|
||||||
|
THEN 'meter-coverage assumption ' || u.cov_code
|
||||||
|
|| ' does not name exactly these counters, uncovered sets and idle sets' END,
|
||||||
|
CASE WHEN NOT a.same_epoch THEN 'readings not in one meter epoch' END,
|
||||||
|
CASE WHEN NOT a.monotonic THEN 'timer went backwards' END,
|
||||||
|
CASE WHEN a.max_meter_h > 24 * i.days THEN 'over 24 h/day on one meter' END,
|
||||||
|
CASE WHEN a.runtime_h = 0 AND i.consumed_l > 0 THEN 'fuel used with zero runtime' END,
|
||||||
|
CASE WHEN a.runtime_h = 0 AND i.consumed_l <= 0 THEN 'no operation in the interval' END,
|
||||||
|
CASE WHEN i.days < 7 THEN 'shorter than 7 days' END) AS exclusion_reasons
|
||||||
|
FROM fm.v_stock_interval i
|
||||||
|
LEFT JOIN rd USING (interval_key) LEFT JOIN agg a USING (interval_key)
|
||||||
|
LEFT JOIN cov cv USING (interval_key) LEFT JOIN unm u USING (interval_key)
|
||||||
|
LEFT JOIN pv USING (interval_key))
|
||||||
|
SELECT x.*,
|
||||||
|
CASE WHEN NOT population_valid THEN NULL
|
||||||
|
WHEN n_unmapped = 0 AND n_meters = 1 THEN 'apparent_l_per_generator_h'
|
||||||
|
ELSE 'apparent_l_per_set_h' END AS metric,
|
||||||
|
CASE WHEN runtime_h > 0 THEN consumed_l / runtime_h END AS l_per_h,
|
||||||
|
CASE WHEN runtime_high_h > 0 THEN (consumed_l - consumed_unc_l) / runtime_high_h END AS l_per_h_low,
|
||||||
|
CASE WHEN runtime_low_h > 0 THEN (consumed_l + consumed_unc_l) / runtime_low_h END AS l_per_h_high,
|
||||||
|
-- NULL beside a non-NULL l_per_h = unbounded
|
||||||
|
CASE WHEN exclusion_reasons <> '' OR NOT population_valid THEN 'excluded'
|
||||||
|
WHEN depends_on_proposed THEN 'conditional' ELSE 'eligible' END AS eligibility,
|
||||||
|
CASE WHEN opening_basis = 'stated' AND closing_basis = 'stated' AND inputs_reviewed AND NOT inputs_conflict
|
||||||
|
AND NOT date_inferred AND runtime_reviewed AND NOT runtime_conflict THEN 'G1' ELSE 'G2' END AS evidence_grade
|
||||||
|
FROM x;
|
||||||
|
|
||||||
|
-- M4 (L per calendar day) per stock interval: no runtime, so no generator population. The
|
||||||
|
-- population is the site's whole stock. Input to the L/day proposal path only; never a KPI.
|
||||||
|
CREATE VIEW fm.v_m4_interval AS
|
||||||
|
SELECT i.*, 'stock:' || i.site_uid AS population_key,
|
||||||
|
CASE WHEN i.days > 0 THEN i.consumed_l / i.days END AS l_per_day,
|
||||||
|
CASE WHEN i.stock_status = 'ok' THEN 'eligible' ELSE 'excluded' END AS eligibility,
|
||||||
|
CASE WHEN i.stock_status <> 'ok' THEN 'stock interval: ' || i.stock_status ELSE '' END AS exclusion_reasons,
|
||||||
|
CASE WHEN i.inputs_reviewed AND NOT i.inputs_conflict THEN 'G1' ELSE 'G2' END AS evidence_grade
|
||||||
|
FROM fm.v_stock_interval i;
|
||||||
|
|
||||||
|
-- Everything an interval's value and eligibility rest on, as sorted lists, with their hashes. Claims
|
||||||
|
-- are listed in every status, so a review, supersession or retraction changes the hash. `baseline
|
||||||
|
-- propose` snapshots both lists; fm.v_baseline_validity compares them with today's state.
|
||||||
|
CREATE VIEW fm.v_interval_dependency AS
|
||||||
|
WITH span AS ( -- the endpoint visits and every visit between them, live or not
|
||||||
|
SELECT i.interval_key, v.visit_id
|
||||||
|
FROM fm.v_stock_interval i
|
||||||
|
JOIN fm.visit o ON o.visit_id = i.open_visit_id
|
||||||
|
JOIN fm.visit c ON c.visit_id = i.close_visit_id
|
||||||
|
JOIN fm.visit v ON v.site_uid = i.site_uid
|
||||||
|
AND (v.visit_on, v.seq_in_day) >= (o.visit_on, o.seq_in_day)
|
||||||
|
AND (v.visit_on, v.seq_in_day) <= (c.visit_on, c.seq_in_day)),
|
||||||
|
stock_deps AS (
|
||||||
|
SELECT s.interval_key, jsonb_agg(d.item ORDER BY d.item) AS deps
|
||||||
|
FROM span s CROSS JOIN LATERAL (
|
||||||
|
SELECT 'stock:' || x.source_key || ':' || x.assertion_sha256 || ':' || x.claim_status AS item
|
||||||
|
FROM fm.stock_observation x WHERE x.visit_id = s.visit_id
|
||||||
|
UNION ALL
|
||||||
|
SELECT 'movement:' || x.event_key || ':' || x.assertion_sha256 || ':' || x.claim_status
|
||||||
|
FROM fm.fuel_movement x WHERE x.visit_id = s.visit_id) d
|
||||||
|
GROUP BY s.interval_key),
|
||||||
|
runtime_deps AS (
|
||||||
|
SELECT i.interval_key, jsonb_agg(d.item ORDER BY d.item) AS deps
|
||||||
|
FROM fm.v_stock_interval i CROSS JOIN LATERAL (
|
||||||
|
SELECT 'reading:' || x.meter_id || ':' || x.assertion_sha256 || ':' || x.claim_status
|
||||||
|
|| ':' || coalesce(x.epoch_id::text, '-') AS item
|
||||||
|
FROM fm.hour_meter_reading x WHERE x.visit_id IN (i.open_visit_id, i.close_visit_id)
|
||||||
|
UNION ALL
|
||||||
|
SELECT 'meter:' || m.meter_id || ':' || m.mapping_status || ':' || coalesce(m.asset_uid, '-')
|
||||||
|
FROM fm.runtime_meter m WHERE m.site_uid = i.site_uid
|
||||||
|
UNION ALL
|
||||||
|
SELECT 'epoch:' || e.meter_id || ':' || e.epoch_id || ':' || coalesce(e.starts_on::text, '-')
|
||||||
|
|| ':' || e.start_reason || ':' || e.continuity || ':' || coalesce(e.rollover_at::text, '-')
|
||||||
|
FROM fm.meter_epoch e JOIN fm.runtime_meter m ON m.meter_id = e.meter_id
|
||||||
|
WHERE m.site_uid = i.site_uid
|
||||||
|
UNION ALL -- placements at this site, and anywhere of the assets its meters name
|
||||||
|
SELECT 'placement:' || p.asset_uid || ':' || p.site_uid || ':' || coalesce(p.occupied_from::text, '-')
|
||||||
|
|| ':' || coalesce(p.occupied_to::text, '-')
|
||||||
|
FROM fm.asset_placement p
|
||||||
|
WHERE (p.site_uid = i.site_uid
|
||||||
|
OR p.asset_uid IN (SELECT m.asset_uid FROM fm.runtime_meter m WHERE m.site_uid = i.site_uid))
|
||||||
|
AND daterange(coalesce(p.occupied_from, '-infinity'::date), p.occupied_to, '[)')
|
||||||
|
&& daterange(i.start_on, i.end_on, '[]')
|
||||||
|
UNION ALL
|
||||||
|
SELECT 'assumption:' || x.code || ':' || x.fingerprint || ':' || x.status
|
||||||
|
FROM fm.assumption x
|
||||||
|
WHERE x.site_uid = i.site_uid AND x.topic IN ('generator_idle','meter_coverage')
|
||||||
|
AND daterange(x.applies_from, x.applies_to, '[]') && daterange(i.start_on, i.end_on, '[]')) d
|
||||||
|
GROUP BY i.interval_key)
|
||||||
|
SELECT i.interval_key, i.site_uid, i.open_visit_id, i.close_visit_id,
|
||||||
|
coalesce(s.deps, '[]'::jsonb) AS stock_deps, coalesce(r.deps, '[]'::jsonb) AS runtime_deps,
|
||||||
|
encode(sha256(convert_to(coalesce(s.deps, '[]'::jsonb)::text, 'UTF8')), 'hex') AS stock_dep_sha256,
|
||||||
|
encode(sha256(convert_to(coalesce(r.deps, '[]'::jsonb)::text, 'UTF8')), 'hex') AS runtime_dep_sha256
|
||||||
|
FROM fm.v_stock_interval i
|
||||||
|
LEFT JOIN stock_deps s USING (interval_key) LEFT JOIN runtime_deps r USING (interval_key);
|
||||||
|
|
||||||
|
-- Does each version still rest on what it was computed from? An input whose interval no longer
|
||||||
|
-- exists, or whose dependency hash differs, makes the whole version "inputs changed".
|
||||||
|
CREATE VIEW fm.v_baseline_validity AS
|
||||||
|
SELECT b.baseline_id,
|
||||||
|
count(bi.baseline_id) > 0
|
||||||
|
AND coalesce(bool_and(d.interval_key IS NOT NULL
|
||||||
|
AND d.stock_dep_sha256 = bi.stock_dep_sha256
|
||||||
|
AND (b.metric = 'fuel_l_per_calendar_day'
|
||||||
|
OR d.runtime_dep_sha256 = bi.runtime_dep_sha256)), false) AS inputs_unchanged
|
||||||
|
FROM fm.baseline_version b
|
||||||
|
LEFT JOIN fm.baseline_input bi ON bi.baseline_id = b.baseline_id
|
||||||
|
LEFT JOIN fm.v_interval_dependency d
|
||||||
|
ON d.open_visit_id = bi.open_visit_id AND d.close_visit_id = bi.close_visit_id
|
||||||
|
GROUP BY b.baseline_id, b.metric;
|
||||||
|
|
||||||
|
-- Consecutive readings per meter (live visits), giving hours per day: R1 / public.v_runtime.
|
||||||
|
CREATE VIEW fm.v_runtime_pair AS
|
||||||
|
WITH r AS (
|
||||||
|
SELECT v.site_uid, v.visit_id, v.visit_on, v.seq_in_day, x.meter_id, x.epoch_id, x.n_epochs,
|
||||||
|
x.n_unassigned_epoch, x.hours, x.all_reviewed,
|
||||||
|
row_number() OVER (PARTITION BY x.meter_id ORDER BY v.visit_on, v.seq_in_day) AS n
|
||||||
|
FROM fm.v_visit_reading x JOIN fm.v_visit_live v ON v.visit_id = x.visit_id)
|
||||||
|
SELECT o.site_uid, o.meter_id, o.visit_id AS open_visit_id, c.visit_id AS close_visit_id,
|
||||||
|
o.visit_on AS start_on, c.visit_on AS end_on, c.visit_on - o.visit_on AS days,
|
||||||
|
c.hours - o.hours AS runtime_h,
|
||||||
|
CASE WHEN c.visit_on > o.visit_on THEN (c.hours - o.hours) / (c.visit_on - o.visit_on) END AS h_per_day,
|
||||||
|
(o.n_epochs = 1 AND c.n_epochs = 1 AND o.n_unassigned_epoch = 0 AND c.n_unassigned_epoch = 0
|
||||||
|
AND o.epoch_id = c.epoch_id) AS same_epoch,
|
||||||
|
c.hours >= o.hours AS monotonic,
|
||||||
|
c.hours - o.hours > 24 * (c.visit_on - o.visit_on) AS over_24h_per_day,
|
||||||
|
o.all_reviewed AND c.all_reviewed AS reviewed
|
||||||
|
FROM r o JOIN r c ON c.meter_id = o.meter_id AND c.n = o.n + 1;
|
||||||
|
|
||||||
|
-- M3, modelled L/h: L per calendar day over the usable intervals that lie wholly inside the
|
||||||
|
-- assumption's dates, divided by that independently sourced runtime assumption (never one derived
|
||||||
|
-- from fuel). The window is the matched one. Reference tab only; never a KPI (design §4.1).
|
||||||
|
CREATE VIEW fm.v_modelled_lph AS
|
||||||
|
SELECT x.site_uid, x.code, x.status, x.basis,
|
||||||
|
x.value_num AS h_per_day, x.low_num AS h_per_day_low, x.high_num AS h_per_day_high,
|
||||||
|
m4.window_start, m4.window_end, m4.days AS matched_days, m4.fuel_l / NULLIF(m4.days, 0) AS l_per_day,
|
||||||
|
m4.fuel_l / NULLIF(m4.days, 0) / NULLIF(x.value_num, 0) AS modelled_l_per_h,
|
||||||
|
m4.fuel_l / NULLIF(m4.days, 0) / NULLIF(x.high_num, 0) AS modelled_l_per_h_low,
|
||||||
|
m4.fuel_l / NULLIF(m4.days, 0) / NULLIF(x.low_num, 0) AS modelled_l_per_h_high
|
||||||
|
FROM fm.assumption x
|
||||||
|
CROSS JOIN LATERAL (
|
||||||
|
SELECT sum(i.consumed_l) AS fuel_l, sum(i.days) AS days,
|
||||||
|
min(i.start_on) AS window_start, max(i.end_on) AS window_end
|
||||||
|
FROM fm.v_m4_interval i
|
||||||
|
WHERE i.site_uid = x.site_uid AND i.eligibility = 'eligible'
|
||||||
|
AND daterange(x.applies_from, x.applies_to, '[]') @> daterange(i.start_on, i.end_on, '[]')) m4
|
||||||
|
WHERE x.topic = 'runtime_hours_per_day' AND NOT x.derived_from_fuel AND x.status IN ('proposed','accepted')
|
||||||
|
AND m4.days > 0;
|
||||||
|
|
||||||
|
-- Review entries whose fingerprint matches no live row: the value changed after it was reviewed,
|
||||||
|
-- or the subject was superseded or retracted. The site comes from every row the subject key has
|
||||||
|
-- had, in any status; for an asset's specification or engine total, from every site the asset was
|
||||||
|
-- placed at. So one entry can give one row per site, and none is lost when its subject leaves.
|
||||||
|
CREATE VIEW fm.v_review_stale AS
|
||||||
|
WITH subj AS (
|
||||||
|
SELECT 'claim' AS kind, source_key AS subject_key, assertion_sha256, site_uid,
|
||||||
|
claim_status IN ('asserted','reviewed','disputed') AS live
|
||||||
|
FROM fm.fuel_movement
|
||||||
|
UNION ALL SELECT 'claim', source_key, assertion_sha256, site_uid, claim_status IN ('asserted','reviewed','disputed')
|
||||||
|
FROM fm.stock_observation
|
||||||
|
UNION ALL SELECT 'claim', h.source_key, h.assertion_sha256, m.site_uid, h.claim_status IN ('asserted','reviewed','disputed')
|
||||||
|
FROM fm.hour_meter_reading h JOIN fm.runtime_meter m ON m.meter_id = h.meter_id
|
||||||
|
UNION ALL SELECT 'claim', source_key, assertion_sha256, site_uid, claim_status IN ('asserted','reviewed','disputed')
|
||||||
|
FROM fm.electricity_bill
|
||||||
|
UNION ALL SELECT 'claim', source_key, assertion_sha256, site_uid, claim_status IN ('asserted','reviewed','disputed')
|
||||||
|
FROM fm.imported_calculation
|
||||||
|
UNION ALL SELECT 'claim', e.source_key, e.assertion_sha256, p.site_uid, e.claim_status IN ('asserted','reviewed','disputed')
|
||||||
|
FROM fm.equipment_spec e LEFT JOIN fm.asset_placement p ON p.asset_uid = e.asset_uid
|
||||||
|
UNION ALL SELECT 'claim', source_key, assertion_sha256, site_uid, claim_status IN ('asserted','reviewed','disputed')
|
||||||
|
FROM fm.runtime_interval
|
||||||
|
UNION ALL SELECT 'claim', t.source_key, t.assertion_sha256, p.site_uid, t.claim_status IN ('asserted','reviewed','disputed')
|
||||||
|
FROM fm.engine_fuel_total t LEFT JOIN fm.asset_placement p ON p.asset_uid = t.asset_uid
|
||||||
|
UNION ALL SELECT 'location', source_key, assertion_sha256, site_uid, acceptance IN ('pending','accepted','rejected')
|
||||||
|
FROM fm.site_location
|
||||||
|
UNION ALL SELECT 'assumption', code, fingerprint, site_uid, status <> 'superseded'
|
||||||
|
FROM fm.assumption)
|
||||||
|
SELECT r.review_id, r.entry_key, r.subject_kind, r.subject_key, r.assertion_sha256, r.verdict,
|
||||||
|
r.reviewed_on, r.reviewer_role, st.site_uid
|
||||||
|
FROM fm.review r
|
||||||
|
LEFT JOIN LATERAL (SELECT DISTINCT s.site_uid FROM subj s
|
||||||
|
WHERE s.kind = r.subject_kind AND s.subject_key = r.subject_key AND s.site_uid IS NOT NULL) st ON true
|
||||||
|
WHERE NOT EXISTS (SELECT 1 FROM subj l
|
||||||
|
WHERE l.live AND l.kind = r.subject_kind AND l.subject_key = r.subject_key
|
||||||
|
AND l.assertion_sha256 = r.assertion_sha256);
|
||||||
|
|
||||||
|
-- Visits kept for provenance only: every claim superseded or retracted.
|
||||||
|
CREATE VIEW fm.v_visit_provenance_only AS
|
||||||
|
SELECT v.* FROM fm.visit v
|
||||||
|
WHERE NOT EXISTS (SELECT 1 FROM fm.v_visit_live l WHERE l.visit_id = v.visit_id);
|
||||||
|
|
||||||
|
COMMIT;
|
||||||
@@ -0,0 +1,621 @@
|
|||||||
|
-- 0003_bi_views.sql: the views Superset reads through Trino (schema `public`), plus
|
||||||
|
-- fm.v_site_baseline_status. Owned by fuel_owner, so trino_ro can read them through the existing
|
||||||
|
-- default privileges; they run with the owner's rights, so Trino never reads `fm` (it lists fm's
|
||||||
|
-- table names, as it lists every schema's, but any query on them is refused).
|
||||||
|
-- DRAFT, not applied. Applied after 0002 by a loader acting as fuel_owner (README in this folder).
|
||||||
|
-- Design: fuel-monitoring .context/plans/2026-10-24-fuel-db-baseline-dashboard.md, §5.2 and §6.
|
||||||
|
BEGIN;
|
||||||
|
SET LOCAL search_path = fm, pg_catalog, public;
|
||||||
|
|
||||||
|
-- Status, estimate and method come from ONE baseline_version, never from ad-hoc pooling of intervals
|
||||||
|
-- (exploratory interval figures stay on the Baseline & Evidence tab):
|
||||||
|
-- approved : the latest approved KPI version; a decision is recorded. The only state that says "approved".
|
||||||
|
-- If its inputs changed since, it stays approved and baseline_validity_note says so.
|
||||||
|
-- eligible : no approval; the latest proposed KPI version passed BP-1 when `baseline propose`
|
||||||
|
-- computed it from its inputs (meets_policy), and those inputs are unchanged today.
|
||||||
|
-- provisional : no approval; a proposed version below BP-1, or usable intervals but no version yet
|
||||||
|
-- (then no estimate is shown).
|
||||||
|
-- insufficient_evidence: none of the above (today: every site, since there is no runtime).
|
||||||
|
CREATE VIEW fm.v_site_baseline_status AS
|
||||||
|
WITH cur AS (
|
||||||
|
SELECT DISTINCT ON (b.site_uid) b.*
|
||||||
|
FROM fm.baseline_version b
|
||||||
|
WHERE b.status IN ('approved','proposed')
|
||||||
|
AND b.metric IN ('measured_generator_l_per_h','apparent_l_per_generator_h',
|
||||||
|
'apparent_l_per_set_h','apparent_l_per_site_dg_h') -- never modelled, never per-day
|
||||||
|
ORDER BY b.site_uid, (b.status = 'approved') DESC, b.approved_on DESC NULLS LAST, b.computed_at DESC),
|
||||||
|
ev AS (
|
||||||
|
SELECT site_uid, count(*) FILTER (WHERE eligibility <> 'excluded') AS n_usable
|
||||||
|
FROM fm.v_lph_interval GROUP BY site_uid)
|
||||||
|
SELECT s.site_uid,
|
||||||
|
CASE WHEN cur.status = 'approved' THEN 'approved'
|
||||||
|
WHEN cur.status = 'proposed' AND cur.meets_policy AND val.inputs_unchanged THEN 'eligible'
|
||||||
|
WHEN cur.status = 'proposed' OR coalesce(ev.n_usable, 0) > 0 THEN 'provisional'
|
||||||
|
ELSE 'insufficient_evidence' END AS baseline_status,
|
||||||
|
CASE WHEN cur.baseline_id IS NOT NULL AND NOT coalesce(val.inputs_unchanged, false)
|
||||||
|
THEN CASE cur.status WHEN 'approved' THEN 'Inputs changed since approval: review needed'
|
||||||
|
ELSE 'Inputs changed since computation: re-propose' END END AS baseline_validity_note,
|
||||||
|
cur.code AS baseline_code,
|
||||||
|
CASE WHEN cur.status = 'approved' OR val.inputs_unchanged THEN cur.value END AS lph_estimate,
|
||||||
|
CASE WHEN cur.status = 'approved' OR val.inputs_unchanged THEN cur.range_low END AS range_low,
|
||||||
|
CASE WHEN cur.status = 'approved' OR val.inputs_unchanged THEN cur.range_high END AS range_high,
|
||||||
|
cur.metric AS lph_metric, cur.population, cur.window_start AS lph_window_start,
|
||||||
|
cur.window_end AS lph_window_end, cur.conditions, cur.decision_ref,
|
||||||
|
coalesce(ev.n_usable, 0) AS n_usable_intervals
|
||||||
|
FROM fm.site s LEFT JOIN cur USING (site_uid) LEFT JOIN ev USING (site_uid)
|
||||||
|
LEFT JOIN fm.v_baseline_validity val ON val.baseline_id = cur.baseline_id;
|
||||||
|
|
||||||
|
-- One row per site, every status: the map dataset and the single source of site-level fields.
|
||||||
|
-- "Current" means the period that contains today's date in Kabul, not "no end date".
|
||||||
|
CREATE VIEW public.v_site AS
|
||||||
|
WITH today AS (SELECT (now() AT TIME ZONE 'Asia/Kabul')::date AS d),
|
||||||
|
alias AS ( -- v1: the operator is ATOMA; later: the site's current operator
|
||||||
|
SELECT a.site_uid, a.label FROM fm.site_alias a, today
|
||||||
|
WHERE a.issuer = 'ATOMA' AND a.is_primary
|
||||||
|
AND daterange(coalesce(a.valid_from, '-infinity'::date), a.valid_to, '[)') @> today.d),
|
||||||
|
pp AS ( -- one programme, chosen explicitly
|
||||||
|
SELECT p.site_uid, p.status FROM fm.pilot_participation p, today
|
||||||
|
WHERE p.programme = 'ATOMA OFMS pilot' AND daterange(p.valid_from, p.valid_to, '[)') @> today.d),
|
||||||
|
-- alias and pp are at most one row per site: the exclusion constraints forbid overlapping primary
|
||||||
|
-- labels per (site, issuer) and overlapping periods per (site, programme).
|
||||||
|
loc AS (SELECT site_uid, latitude, longitude, basis, precision_m FROM fm.site_location WHERE acceptance = 'accepted'),
|
||||||
|
loc_any AS (SELECT site_uid, bool_or(acceptance = 'pending') AS has_pending FROM fm.site_location GROUP BY site_uid),
|
||||||
|
cal AS (SELECT site_uid, sum(consumed_l) AS fuel_l, sum(days) AS days
|
||||||
|
FROM fm.v_stock_interval WHERE stock_status = 'ok' GROUP BY site_uid),
|
||||||
|
latest AS (SELECT site_uid, max(visit_on) AS latest_record_on -- live records only: a fully
|
||||||
|
FROM fm.v_visit_live GROUP BY site_uid) -- retracted visit does not count
|
||||||
|
SELECT s.site_uid::varchar(12) AS site_uid,
|
||||||
|
coalesce(alias.label, s.site_uid)::varchar(20) AS site_code,
|
||||||
|
coalesce(s.name, 'name not recorded')::varchar(120) AS site_name,
|
||||||
|
(coalesce(alias.label, s.site_uid) || ' - ' || coalesce(s.name, 'name not recorded')
|
||||||
|
|| ' - ' || s.site_uid)::varchar(170) AS site_label, -- the one filter value
|
||||||
|
s.zone::varchar(40) AS zone, s.province::varchar(40) AS province, s.district::varchar(60) AS district,
|
||||||
|
coalesce(pp.status, 'none')::varchar(12) AS pilot_status,
|
||||||
|
loc.latitude::numeric(9,6) AS latitude, loc.longitude::numeric(9,6) AS longitude, -- NULL unless accepted
|
||||||
|
(CASE WHEN loc.site_uid IS NOT NULL THEN 'accepted'
|
||||||
|
WHEN loc_any.has_pending THEN 'awaiting acceptance' ELSE 'missing' END)::varchar(20) AS location_status,
|
||||||
|
loc.basis::varchar(20) AS location_basis, loc.precision_m::integer AS location_precision_m,
|
||||||
|
bs.baseline_status::varchar(24) AS baseline_status,
|
||||||
|
(CASE bs.baseline_status WHEN 'approved' THEN 'Approved baseline'
|
||||||
|
WHEN 'eligible' THEN 'Eligible - not yet approved' WHEN 'provisional' THEN 'Provisional'
|
||||||
|
ELSE 'Insufficient evidence' END)::varchar(40) AS baseline_status_label,
|
||||||
|
bs.baseline_validity_note::varchar(60) AS baseline_validity_note,
|
||||||
|
round(bs.lph_estimate, 2)::numeric(6,2) AS lph_estimate,
|
||||||
|
(CASE bs.lph_metric WHEN 'apparent_l_per_generator_h' THEN 'Stock balance per generator running hour'
|
||||||
|
WHEN 'apparent_l_per_set_h' THEN 'Stock balance per summed generator-hour'
|
||||||
|
WHEN 'apparent_l_per_site_dg_h' THEN 'Stock balance per hour with any generator running'
|
||||||
|
WHEN 'measured_generator_l_per_h' THEN 'Measured engine fuel per running hour'
|
||||||
|
ELSE CASE WHEN bs.n_usable_intervals > 0 THEN 'Not available: no baseline version proposed'
|
||||||
|
ELSE 'Not available: no runtime evidence' END END)::varchar(60) AS lph_method,
|
||||||
|
bs.lph_window_start, bs.lph_window_end, bs.conditions::varchar(200) AS baseline_conditions,
|
||||||
|
bs.decision_ref::varchar(12) AS baseline_decision_ref,
|
||||||
|
round(cal.fuel_l / NULLIF(cal.days, 0), 1)::numeric(7,1) AS fuel_l_per_calendar_day,
|
||||||
|
latest.latest_record_on
|
||||||
|
FROM fm.site s
|
||||||
|
LEFT JOIN alias USING (site_uid) LEFT JOIN loc USING (site_uid) LEFT JOIN loc_any USING (site_uid)
|
||||||
|
LEFT JOIN pp USING (site_uid) LEFT JOIN fm.v_site_baseline_status bs USING (site_uid)
|
||||||
|
LEFT JOIN cal USING (site_uid) LEFT JOIN latest USING (site_uid);
|
||||||
|
|
||||||
|
-- Every view below joins public.v_site for site_uid, site_code, site_name and site_label, so one
|
||||||
|
-- Site filter drives every dataset, and casts to Trino-safe types (varchar(n), integer,
|
||||||
|
-- numeric(p,s), date, timestamp(0), boolean). Live = asserted, reviewed or disputed.
|
||||||
|
|
||||||
|
-- S2, R3: one row per site and measure; the rows stay when a value is not established.
|
||||||
|
CREATE VIEW public.v_kpi_measures AS
|
||||||
|
WITH cur AS ( -- the site's L/h version; its snapshot only while approved or its inputs unchanged
|
||||||
|
SELECT bs.site_uid, b.baseline_id, b.code, b.fuel_l, b.runtime_h, b.matched_days, b.window_start, b.window_end,
|
||||||
|
(b.status = 'approved' OR coalesce(val.inputs_unchanged, false)) AS shown,
|
||||||
|
' (' || b.status || ')'
|
||||||
|
|| CASE WHEN coalesce(val.inputs_unchanged, false) THEN ''
|
||||||
|
WHEN b.status = 'approved' THEN '; inputs changed since approval: review needed'
|
||||||
|
ELSE '; inputs changed since computation: re-propose' END AS state
|
||||||
|
FROM fm.v_site_baseline_status bs JOIN fm.baseline_version b ON b.code = bs.baseline_code
|
||||||
|
LEFT JOIN fm.v_baseline_validity val ON val.baseline_id = b.baseline_id),
|
||||||
|
ninp AS (SELECT baseline_id, count(*) FILTER (WHERE included) AS n FROM fm.baseline_input GROUP BY baseline_id),
|
||||||
|
m4 AS (SELECT site_uid, sum(consumed_l) AS fuel_l, sum(days) AS days, min(start_on) AS first_on, max(end_on) AS last_on
|
||||||
|
FROM fm.v_m4_interval WHERE eligibility = 'eligible' GROUP BY site_uid),
|
||||||
|
span AS (SELECT site_uid, min(visit_on) AS first_on, max(visit_on) AS last_on FROM fm.v_visit_live GROUP BY site_uid),
|
||||||
|
del AS (SELECT v.site_uid, count(*) AS n FROM fm.v_movement_event e JOIN fm.v_visit_live v ON v.visit_id = e.visit_id
|
||||||
|
WHERE e.kind = 'delivery' GROUP BY v.site_uid),
|
||||||
|
rdg AS (SELECT m.site_uid, count(*) AS n FROM fm.hour_meter_reading h JOIN fm.runtime_meter m ON m.meter_id = h.meter_id
|
||||||
|
WHERE h.claim_status IN ('asserted','reviewed','disputed') GROUP BY m.site_uid),
|
||||||
|
kwh AS (SELECT site_uid, sum(kwh) AS kwh, count(kwh) AS n_kwh, min(bill_month) AS first_m, max(bill_month) AS last_m
|
||||||
|
FROM fm.electricity_bill WHERE claim_status IN ('asserted','reviewed') GROUP BY site_uid), -- values never from disputed
|
||||||
|
lday AS ( -- the L/day reference: the latest approved, else proposed, L/day version
|
||||||
|
SELECT DISTINCT ON (b.site_uid) b.site_uid, b.code, b.status, b.value, b.window_start, b.window_end,
|
||||||
|
coalesce(val.inputs_unchanged, false) AS inputs_unchanged
|
||||||
|
FROM fm.baseline_version b LEFT JOIN fm.v_baseline_validity val ON val.baseline_id = b.baseline_id
|
||||||
|
WHERE b.metric = 'fuel_l_per_calendar_day' AND b.status IN ('approved','proposed')
|
||||||
|
ORDER BY b.site_uid, (b.status = 'approved') DESC, b.approved_on DESC NULLS LAST, b.computed_at DESC)
|
||||||
|
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||||
|
x.ord::integer AS measure_order, x.measure::varchar(60) AS measure,
|
||||||
|
round(x.value, 2)::numeric(12,2) AS value, x.unit::varchar(12) AS unit,
|
||||||
|
(CASE WHEN x.value IS NULL THEN 'not established' ELSE x.kind END)::varchar(16) AS value_kind,
|
||||||
|
x.window_start, x.window_end, x.note::varchar(200) AS note
|
||||||
|
FROM public.v_site s
|
||||||
|
LEFT JOIN cur ON cur.site_uid = s.site_uid
|
||||||
|
LEFT JOIN ninp ON ninp.baseline_id = cur.baseline_id
|
||||||
|
LEFT JOIN m4 ON m4.site_uid = s.site_uid
|
||||||
|
LEFT JOIN span ON span.site_uid = s.site_uid
|
||||||
|
LEFT JOIN del ON del.site_uid = s.site_uid
|
||||||
|
LEFT JOIN rdg ON rdg.site_uid = s.site_uid
|
||||||
|
LEFT JOIN kwh ON kwh.site_uid = s.site_uid
|
||||||
|
LEFT JOIN lday ON lday.site_uid = s.site_uid
|
||||||
|
CROSS JOIN LATERAL (VALUES
|
||||||
|
(1, 'Matched fuel', CASE WHEN cur.shown THEN cur.fuel_l END, 'L', 'calculated', cur.window_start, cur.window_end,
|
||||||
|
'fuel used in the intervals of L/h baseline version ' || cur.code || cur.state),
|
||||||
|
(2, 'Matched runtime', CASE WHEN cur.shown THEN cur.runtime_h END, 'h', 'calculated', cur.window_start, cur.window_end,
|
||||||
|
'generator running hours in the same intervals' || coalesce(cur.state, '')),
|
||||||
|
(3, 'Runtime per day', CASE WHEN cur.shown THEN cur.runtime_h / NULLIF(cur.matched_days, 0) END, 'h/day', 'calculated',
|
||||||
|
cur.window_start, cur.window_end, 'matched runtime / matched days' || coalesce(cur.state, '')),
|
||||||
|
(4, 'Intervals used', CASE WHEN cur.shown THEN ninp.n::numeric END, 'intervals', 'calculated', cur.window_start, cur.window_end,
|
||||||
|
'included inputs of the L/h baseline version' || coalesce(cur.state, '')),
|
||||||
|
(5, 'Fuel per calendar day, whole record', m4.fuel_l / NULLIF(m4.days, 0), 'L/day', 'calculated',
|
||||||
|
m4.first_on, m4.last_on, 'stock balance over all usable intervals / calendar days; not per running hour'),
|
||||||
|
(6, 'L/day reference (baseline version)',
|
||||||
|
CASE WHEN lday.status = 'approved' OR lday.inputs_unchanged THEN lday.value END, 'L/day', 'calculated',
|
||||||
|
lday.window_start, lday.window_end,
|
||||||
|
CASE WHEN lday.code IS NULL THEN 'no L/day baseline version'
|
||||||
|
ELSE 'version ' || lday.code || ' (' || lday.status || '), fixed window; not per running hour'
|
||||||
|
|| CASE WHEN lday.inputs_unchanged THEN ''
|
||||||
|
WHEN lday.status = 'approved' THEN '; inputs changed since approval: review needed'
|
||||||
|
ELSE '; inputs changed since computation: re-propose' END END),
|
||||||
|
(7, 'Days covered by usable stock intervals', m4.days::numeric, 'days', 'calculated', m4.first_on, m4.last_on,
|
||||||
|
'compare with the record span'),
|
||||||
|
(8, 'Record span', (span.last_on - span.first_on)::numeric, 'days', 'recorded', span.first_on, span.last_on,
|
||||||
|
'first to last live record'),
|
||||||
|
(9, 'Deliveries recorded', coalesce(del.n, 0)::numeric, 'deliveries', 'recorded', span.first_on, span.last_on,
|
||||||
|
'one per delivery event'),
|
||||||
|
(10, 'Timer readings', coalesce(rdg.n, 0)::numeric, 'readings', 'recorded', NULL::date, NULL::date,
|
||||||
|
CASE WHEN coalesce(rdg.n, 0) = 0 THEN 'no hour-meter readings' END),
|
||||||
|
(11, 'Grid energy', kwh.kwh, 'kWh', 'recorded', kwh.first_m, kwh.last_m,
|
||||||
|
CASE WHEN coalesce(kwh.n_kwh, 0) = 0
|
||||||
|
THEN 'no kWh records; bills are cost only and are not converted to fuel or hours' END)
|
||||||
|
) AS x(ord, measure, value, unit, kind, window_start, window_end, note);
|
||||||
|
|
||||||
|
-- R2: timer readings as recorded (no rows in v1).
|
||||||
|
CREATE VIEW public.v_hour_meter_evidence AS
|
||||||
|
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||||
|
m.source_label::varchar(40) AS meter_label,
|
||||||
|
(CASE WHEN m.asset_uid IS NULL THEN 'unmapped'
|
||||||
|
ELSE m.asset_uid || ' (' || m.mapping_status || ')' END)::varchar(60) AS mapped_set,
|
||||||
|
v.visit_on AS reading_date, h.date_basis::varchar(30) AS date_basis,
|
||||||
|
round(h.hours, 1)::numeric(9,1) AS hours, h.reading_state::varchar(16) AS reading_state,
|
||||||
|
h.source_field::varchar(16) AS source_field, e.epoch_no::integer AS epoch_no,
|
||||||
|
h.claim_status::varchar(12) AS claim_status
|
||||||
|
FROM fm.hour_meter_reading h
|
||||||
|
JOIN fm.runtime_meter m ON m.meter_id = h.meter_id
|
||||||
|
JOIN fm.visit v ON v.visit_id = h.visit_id
|
||||||
|
JOIN public.v_site s ON s.site_uid = m.site_uid
|
||||||
|
LEFT JOIN fm.meter_epoch e ON e.epoch_id = h.epoch_id
|
||||||
|
WHERE h.claim_status IN ('asserted','reviewed','disputed');
|
||||||
|
|
||||||
|
-- B5: assumption versions in force or proposed; a coverage assumption lists its members.
|
||||||
|
CREATE VIEW public.v_assumption AS
|
||||||
|
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||||
|
x.code::varchar(60) AS code, x.topic::varchar(30) AS topic, x.asset_uid::varchar(40) AS asset_uid,
|
||||||
|
x.value_num::numeric(10,3) AS value, x.low_num::numeric(10,3) AS value_low,
|
||||||
|
x.high_num::numeric(10,3) AS value_high, x.unit::varchar(20) AS unit,
|
||||||
|
x.applies_from, x.applies_to, x.basis::varchar(20) AS basis, x.derived_from_fuel,
|
||||||
|
x.status::varchar(12) AS status, x.accepted_on, x.decision_ref::varchar(12) AS decision_ref,
|
||||||
|
x.rationale::varchar(400) AS rationale, x.contradicted_by::varchar(200) AS contradicted_by,
|
||||||
|
mem.members::varchar(400) AS members
|
||||||
|
FROM fm.assumption x
|
||||||
|
JOIN public.v_site s ON s.site_uid = x.site_uid
|
||||||
|
LEFT JOIN LATERAL (
|
||||||
|
SELECT string_agg(m.member_role || ': ' || coalesce(m.meter_id, m.asset_uid), '; '
|
||||||
|
ORDER BY m.member_role, coalesce(m.meter_id, m.asset_uid)) AS members
|
||||||
|
FROM fm.assumption_member m WHERE m.assumption_id = x.assumption_id) mem ON true
|
||||||
|
WHERE x.status <> 'superseded';
|
||||||
|
|
||||||
|
-- B1: baseline versions, with whether their inputs are unchanged today.
|
||||||
|
CREATE VIEW public.v_baseline_version AS
|
||||||
|
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||||
|
b.code::varchar(60) AS code, b.metric::varchar(40) AS metric, b.status::varchar(12) AS status,
|
||||||
|
b.population::varchar(200) AS population,
|
||||||
|
round(b.value, 3)::numeric(8,3) AS value, round(b.range_low, 3)::numeric(8,3) AS range_low,
|
||||||
|
round(b.range_high, 3)::numeric(8,3) AS range_high,
|
||||||
|
(CASE WHEN b.metric = 'fuel_l_per_calendar_day' THEN 'L/day' ELSE 'L/h' END)::varchar(8) AS unit,
|
||||||
|
b.window_start, b.window_end, round(b.fuel_l, 1)::numeric(10,1) AS fuel_l,
|
||||||
|
round(b.runtime_h, 1)::numeric(10,1) AS runtime_h, b.matched_days::integer AS matched_days,
|
||||||
|
b.policy_version::varchar(10) AS policy_version, b.meets_policy, b.policy_check::varchar(300) AS policy_check,
|
||||||
|
b.evidence_grade::varchar(2) AS evidence_grade, b.conditions::varchar(200) AS conditions,
|
||||||
|
b.season_note::varchar(200) AS season_note, b.approved_on, b.approved_by_role::varchar(40) AS approved_by_role,
|
||||||
|
b.decision_ref::varchar(12) AS decision_ref, coalesce(val.inputs_unchanged, false) AS inputs_unchanged,
|
||||||
|
(b.computed_at AT TIME ZONE 'Asia/Kabul')::timestamp(0) AS computed_at
|
||||||
|
FROM fm.baseline_version b
|
||||||
|
JOIN public.v_site s ON s.site_uid = b.site_uid
|
||||||
|
LEFT JOIN fm.v_baseline_validity val ON val.baseline_id = b.baseline_id;
|
||||||
|
|
||||||
|
-- B2: the inputs of each version, with dependency drift since computation.
|
||||||
|
CREATE VIEW public.v_baseline_input AS
|
||||||
|
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||||
|
b.code::varchar(60) AS baseline_code, b.status::varchar(12) AS baseline_status,
|
||||||
|
(to_char(bi.start_on, 'YYYY-MM-DD') || ' to ' || to_char(bi.end_on, 'YYYY-MM-DD'))::varchar(30) AS interval_label,
|
||||||
|
bi.start_on, bi.end_on, bi.included, bi.exclusion_reason::varchar(300) AS exclusion_reason,
|
||||||
|
round(bi.fuel_l, 1)::numeric(10,1) AS fuel_l, round(bi.runtime_h, 1)::numeric(10,1) AS runtime_h,
|
||||||
|
bi.days::integer AS days, bi.eligibility_at_computation::varchar(12) AS eligibility_at_computation,
|
||||||
|
bi.evidence_grade_at_computation::varchar(2) AS evidence_grade_at_computation,
|
||||||
|
(d.interval_key IS NULL OR d.stock_dep_sha256 <> bi.stock_dep_sha256
|
||||||
|
OR (b.metric <> 'fuel_l_per_calendar_day'
|
||||||
|
AND d.runtime_dep_sha256 IS DISTINCT FROM bi.runtime_dep_sha256)) AS dependencies_changed
|
||||||
|
FROM fm.baseline_input bi
|
||||||
|
JOIN fm.baseline_version b ON b.baseline_id = bi.baseline_id
|
||||||
|
JOIN public.v_site s ON s.site_uid = bi.site_uid
|
||||||
|
LEFT JOIN fm.v_interval_dependency d ON d.open_visit_id = bi.open_visit_id AND d.close_visit_id = bi.close_visit_id;
|
||||||
|
|
||||||
|
-- B7: documents the site's figures rest on (the document itself is never served).
|
||||||
|
CREATE VIEW public.v_evidence AS
|
||||||
|
WITH cites AS (
|
||||||
|
SELECT site_uid, source_sha256, 'fuel movement' AS record_type
|
||||||
|
FROM fm.fuel_movement WHERE claim_status IN ('asserted','reviewed','disputed')
|
||||||
|
UNION SELECT site_uid, source_sha256, 'stock reading'
|
||||||
|
FROM fm.stock_observation WHERE claim_status IN ('asserted','reviewed','disputed')
|
||||||
|
UNION SELECT m.site_uid, h.source_sha256, 'timer reading'
|
||||||
|
FROM fm.hour_meter_reading h JOIN fm.runtime_meter m ON m.meter_id = h.meter_id
|
||||||
|
WHERE h.claim_status IN ('asserted','reviewed','disputed')
|
||||||
|
UNION SELECT site_uid, source_sha256, 'electricity bill'
|
||||||
|
FROM fm.electricity_bill WHERE claim_status IN ('asserted','reviewed','disputed')
|
||||||
|
UNION SELECT site_uid, source_sha256, 'imported calculation'
|
||||||
|
FROM fm.imported_calculation WHERE site_uid IS NOT NULL AND claim_status IN ('asserted','reviewed','disputed')
|
||||||
|
UNION SELECT p.site_uid, e.source_sha256, 'equipment specification'
|
||||||
|
FROM fm.equipment_spec e JOIN fm.asset_placement p ON p.asset_uid = e.asset_uid
|
||||||
|
WHERE e.source_sha256 IS NOT NULL AND e.claim_status IN ('asserted','reviewed','disputed')
|
||||||
|
UNION SELECT site_uid, source_sha256, 'assumption'
|
||||||
|
FROM fm.assumption WHERE source_sha256 IS NOT NULL AND status <> 'superseded')
|
||||||
|
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||||
|
left(d.sha256, 16)::varchar(16) AS document_key, d.title::varchar(200) AS title,
|
||||||
|
d.kind::varchar(20) AS kind, d.issuer::varchar(80) AS issuer, d.document_date,
|
||||||
|
string_agg(c.record_type, ', ' ORDER BY c.record_type)::varchar(200) AS used_for,
|
||||||
|
d.contains_personal_data
|
||||||
|
FROM cites c
|
||||||
|
JOIN fm.source_document d ON d.sha256 = c.source_sha256
|
||||||
|
JOIN public.v_site s ON s.site_uid = c.site_uid
|
||||||
|
GROUP BY s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||||
|
d.sha256, d.title, d.kind, d.issuer, d.document_date, d.contains_personal_data;
|
||||||
|
|
||||||
|
-- F1, F4: one row per delivery event; quantity NULL = not recorded. No monthly sums.
|
||||||
|
CREATE VIEW public.v_delivery AS
|
||||||
|
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||||
|
e.event_key::varchar(80) AS event_key, v.visit_on AS delivery_date, v.seq_in_day::integer AS seq_in_day,
|
||||||
|
round(e.litres, 1)::numeric(8,1) AS delivered_l,
|
||||||
|
b.unit_price::numeric(8,2) AS unit_price, b.price_currency::varchar(8) AS currency,
|
||||||
|
b.record_ref::varchar(60) AS source_ref,
|
||||||
|
e.n_claims::integer AS n_sources, (e.spread_l > 0) AS sources_disagree, e.all_reviewed AS reviewed
|
||||||
|
FROM fm.v_movement_event e
|
||||||
|
JOIN fm.v_visit_live v ON v.visit_id = e.visit_id
|
||||||
|
JOIN public.v_site s ON s.site_uid = v.site_uid
|
||||||
|
CROSS JOIN LATERAL ( -- price and reference of the best-ranked claim;
|
||||||
|
-- a disputed claim gives the reference only, never a price
|
||||||
|
SELECT CASE WHEN m.claim_status <> 'disputed' THEN m.unit_price END AS unit_price,
|
||||||
|
CASE WHEN m.claim_status <> 'disputed' THEN m.price_currency END AS price_currency, m.record_ref
|
||||||
|
FROM fm.fuel_movement m JOIN fm.v_source_rank r ON r.sha256 = m.source_sha256
|
||||||
|
WHERE m.event_key = e.event_key AND m.claim_status IN ('asserted','reviewed','disputed')
|
||||||
|
ORDER BY (m.claim_status = 'disputed'), r.rank, m.movement_id LIMIT 1) b
|
||||||
|
WHERE e.kind = 'delivery';
|
||||||
|
|
||||||
|
-- F2: stock readings as recorded, plus after-delivery points derived as before + delivered.
|
||||||
|
CREATE VIEW public.v_stock_reading AS
|
||||||
|
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||||
|
v.visit_on AS observed_on, v.seq_in_day::integer AS seq_in_day,
|
||||||
|
(CASE st.phase WHEN 'before_delivery' THEN 'before delivery' WHEN 'after_delivery' THEN 'after delivery'
|
||||||
|
ELSE 'stock check' END)::varchar(20) AS reading_type,
|
||||||
|
round(st.litres, 1)::numeric(8,1) AS litres, 'recorded'::varchar(12) AS value_kind,
|
||||||
|
round(st.spread_l, 1)::numeric(8,1) AS spread_l, st.all_reviewed AS reviewed
|
||||||
|
FROM fm.v_visit_stock st
|
||||||
|
JOIN fm.v_visit_live v ON v.visit_id = st.visit_id
|
||||||
|
JOIN public.v_site s ON s.site_uid = v.site_uid
|
||||||
|
UNION ALL
|
||||||
|
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||||
|
p.visit_on, p.seq_in_day::integer, 'after delivery'::varchar(20),
|
||||||
|
round(p.depart_l, 1)::numeric(8,1), 'calculated'::varchar(12),
|
||||||
|
round(p.depart_spread_l, 1)::numeric(8,1), p.depart_reviewed
|
||||||
|
FROM fm.v_stock_point p
|
||||||
|
JOIN public.v_site s ON s.site_uid = p.site_uid
|
||||||
|
WHERE p.depart_basis = 'derived';
|
||||||
|
|
||||||
|
-- F3, F4, B3, B4: one row per stock interval, with its L/h evaluation. The approved baseline is set
|
||||||
|
-- beside an interval only when both have the same metric and population (B4 compares like with like).
|
||||||
|
CREATE VIEW public.v_interval AS
|
||||||
|
WITH cur AS (SELECT bs.site_uid, bs.baseline_code, bs.baseline_status, bs.lph_estimate, b.metric, b.population_key
|
||||||
|
FROM fm.v_site_baseline_status bs LEFT JOIN fm.baseline_version b ON b.code = bs.baseline_code),
|
||||||
|
inb AS ( -- the inputs of each site's KPI version
|
||||||
|
SELECT bi.open_visit_id, bi.close_visit_id, bi.included
|
||||||
|
FROM fm.baseline_input bi
|
||||||
|
JOIN fm.baseline_version b ON b.baseline_id = bi.baseline_id
|
||||||
|
JOIN cur ON cur.baseline_code = b.code),
|
||||||
|
cond AS ( -- assumptions a valid interval depends on
|
||||||
|
SELECT l.interval_key, string_agg(DISTINCT x.code, ', ') AS codes
|
||||||
|
FROM fm.v_lph_interval l
|
||||||
|
JOIN fm.assumption x ON x.site_uid = l.site_uid AND x.status IN ('proposed','accepted')
|
||||||
|
AND x.topic IN ('generator_idle','meter_coverage')
|
||||||
|
AND daterange(x.applies_from, x.applies_to, '[]') @> daterange(l.start_on, l.end_on, '[]')
|
||||||
|
WHERE l.population_valid
|
||||||
|
GROUP BY l.interval_key)
|
||||||
|
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||||
|
(to_char(l.start_on, 'YYYY-MM-DD') || ' to ' || to_char(l.end_on, 'YYYY-MM-DD'))::varchar(30) AS interval_label,
|
||||||
|
l.start_on, l.end_on, l.days::integer AS days,
|
||||||
|
round(l.opening_l, 1)::numeric(9,1) AS opening_l, l.opening_basis::varchar(8) AS opening_basis,
|
||||||
|
round(l.deliveries_between_l, 1)::numeric(9,1) AS deliveries_l,
|
||||||
|
round(l.out_l, 1)::numeric(9,1) AS withdrawals_l,
|
||||||
|
round(l.closing_l, 1)::numeric(9,1) AS closing_l, l.closing_basis::varchar(8) AS closing_basis,
|
||||||
|
round(l.consumed_l, 1)::numeric(9,1) AS fuel_used_l, round(l.consumed_unc_l, 1)::numeric(9,1) AS fuel_used_unc_l,
|
||||||
|
round(l.consumed_l / NULLIF(l.days, 0), 1)::numeric(8,1) AS fuel_l_per_day,
|
||||||
|
l.stock_status::varchar(40) AS stock_status,
|
||||||
|
round(l.runtime_h, 1)::numeric(9,1) AS runtime_h,
|
||||||
|
round(l.runtime_h / NULLIF(l.days, 0), 2)::numeric(6,2) AS runtime_h_per_day,
|
||||||
|
coalesce(l.n_meters, 0)::integer AS n_meters, l.metric::varchar(40) AS metric,
|
||||||
|
l.population_key::varchar(200) AS population_key, -- pool only within one metric and population
|
||||||
|
round(l.l_per_h, 3)::numeric(8,3) AS l_per_h, round(l.l_per_h_low, 3)::numeric(8,3) AS l_per_h_low,
|
||||||
|
round(l.l_per_h_high, 3)::numeric(8,3) AS l_per_h_high,
|
||||||
|
l.eligibility::varchar(12) AS eligibility, nullif(l.exclusion_reasons, '')::varchar(600) AS exclusion_reasons,
|
||||||
|
l.evidence_grade::varchar(2) AS evidence_grade, cond.codes::varchar(200) AS conditions,
|
||||||
|
coalesce(inb.included, false) AS in_current_baseline,
|
||||||
|
coalesce(cur.baseline_status = 'approved' AND l.metric = cur.metric
|
||||||
|
AND l.population_key = cur.population_key, false) AS baseline_comparable,
|
||||||
|
(CASE WHEN cur.baseline_status = 'approved' AND l.metric = cur.metric
|
||||||
|
AND l.population_key = cur.population_key
|
||||||
|
THEN cur.lph_estimate END)::numeric(8,3) AS current_baseline_l_per_h
|
||||||
|
FROM fm.v_lph_interval l
|
||||||
|
JOIN public.v_site s ON s.site_uid = l.site_uid
|
||||||
|
JOIN cur ON cur.site_uid = l.site_uid
|
||||||
|
LEFT JOIN inb ON inb.open_visit_id = l.open_visit_id AND inb.close_visit_id = l.close_visit_id
|
||||||
|
LEFT JOIN cond ON cond.interval_key = l.interval_key;
|
||||||
|
|
||||||
|
-- R1: hours per day by meter interval (no rows in v1). Every pair is listed with its flags; runtime
|
||||||
|
-- and h/day only for a valid pair (one epoch, not decreasing, at most 24 h per day).
|
||||||
|
CREATE VIEW public.v_runtime AS
|
||||||
|
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||||
|
m.source_label::varchar(40) AS meter_label,
|
||||||
|
(CASE WHEN m.asset_uid IS NULL THEN 'unmapped'
|
||||||
|
ELSE m.asset_uid || ' (' || m.mapping_status || ')' END)::varchar(60) AS mapped_set,
|
||||||
|
(m.source_label || ': ' || to_char(r.start_on, 'YYYY-MM-DD') || ' to '
|
||||||
|
|| to_char(r.end_on, 'YYYY-MM-DD'))::varchar(80) AS interval_label,
|
||||||
|
r.start_on, r.end_on, r.days::integer AS days,
|
||||||
|
(CASE WHEN v.pair_valid THEN round(r.runtime_h, 1) END)::numeric(9,1) AS runtime_h,
|
||||||
|
(CASE WHEN v.pair_valid THEN round(r.h_per_day, 2) END)::numeric(6,2) AS h_per_day,
|
||||||
|
v.pair_valid, r.same_epoch, r.monotonic, r.over_24h_per_day, r.reviewed
|
||||||
|
FROM fm.v_runtime_pair r
|
||||||
|
CROSS JOIN LATERAL (SELECT coalesce(r.same_epoch AND r.monotonic AND NOT r.over_24h_per_day, false) AS pair_valid) v
|
||||||
|
JOIN fm.runtime_meter m ON m.meter_id = r.meter_id
|
||||||
|
JOIN public.v_site s ON s.site_uid = r.site_uid;
|
||||||
|
|
||||||
|
-- S3 (Gantt): which periods each kind of evidence covers; Kabul midnight as timestamp(0).
|
||||||
|
CREATE VIEW public.v_coverage AS
|
||||||
|
WITH spans AS (
|
||||||
|
SELECT site_uid, '1 Stock intervals' AS lane, stock_status AS status, start_on, end_on
|
||||||
|
FROM fm.v_stock_interval
|
||||||
|
UNION ALL
|
||||||
|
SELECT r.site_uid, '2 Runtime: ' || m.source_label,
|
||||||
|
CASE WHEN r.same_epoch AND r.monotonic AND NOT r.over_24h_per_day THEN 'ok' ELSE 'flagged' END,
|
||||||
|
r.start_on, r.end_on
|
||||||
|
FROM fm.v_runtime_pair r JOIN fm.runtime_meter m ON m.meter_id = r.meter_id
|
||||||
|
UNION ALL
|
||||||
|
SELECT site_uid, '3 Baseline window', baseline_status, lph_window_start, lph_window_end
|
||||||
|
FROM fm.v_site_baseline_status WHERE lph_window_start IS NOT NULL
|
||||||
|
UNION ALL
|
||||||
|
SELECT site_uid, '4 Electricity bills',
|
||||||
|
CASE WHEN claim_status = 'disputed' THEN 'disputed' WHEN kwh IS NULL THEN 'cost only' ELSE 'kWh' END,
|
||||||
|
coalesce(period_start, bill_month), coalesce(period_end, (bill_month + interval '1 month')::date)
|
||||||
|
FROM fm.electricity_bill WHERE claim_status IN ('asserted','reviewed','disputed'))
|
||||||
|
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||||
|
x.lane::varchar(60) AS lane, x.status::varchar(40) AS status, x.start_on, x.end_on,
|
||||||
|
x.start_on::timestamp(0) AS start_ts, x.end_on::timestamp(0) AS end_ts
|
||||||
|
FROM spans x JOIN public.v_site s ON s.site_uid = x.site_uid;
|
||||||
|
|
||||||
|
-- B9: stock-balance fuel against engine-measured fuel inside the interval. No rows until flow-meter
|
||||||
|
-- data exist. The window runs from the opening visit's departure to the closing visit's arrival
|
||||||
|
-- (Kabul time) when both times are recorded, else from midnight to midnight. covered_h = the hours
|
||||||
|
-- of that window the generator's records cover (live records never overlap per asset). The
|
||||||
|
-- difference is shown only for an `ok` stock interval with both times recorded, in which every
|
||||||
|
-- generator is placed at the site throughout (one placement spans the interval, as in
|
||||||
|
-- fm.v_lph_interval) and covered for the whole window.
|
||||||
|
CREATE VIEW public.v_reconciliation AS
|
||||||
|
WITH gen AS ( -- one row per interval and generator placed at the site at any time in it
|
||||||
|
SELECT l.interval_key, l.site_uid, l.start_on, l.end_on, l.consumed_l, l.stock_status, p.asset_uid,
|
||||||
|
bool_or(daterange(coalesce(p.occupied_from, '-infinity'::date), p.occupied_to, '[)')
|
||||||
|
@> daterange(l.start_on, l.end_on, '[]')) AS placed_throughout,
|
||||||
|
(o.time_out_local IS NOT NULL AND c.time_in_local IS NOT NULL) AS times_known,
|
||||||
|
(l.start_on + coalesce(o.time_out_local, '00:00'::time)) AT TIME ZONE 'Asia/Kabul' AS t0,
|
||||||
|
(l.end_on + coalesce(c.time_in_local, '00:00'::time)) AT TIME ZONE 'Asia/Kabul' AS t1
|
||||||
|
FROM fm.v_stock_interval l
|
||||||
|
JOIN fm.visit o ON o.visit_id = l.open_visit_id
|
||||||
|
JOIN fm.visit c ON c.visit_id = l.close_visit_id
|
||||||
|
JOIN fm.asset_placement p ON p.site_uid = l.site_uid
|
||||||
|
JOIN fm.asset a ON a.asset_uid = p.asset_uid AND a.kind = 'generator'
|
||||||
|
WHERE daterange(coalesce(p.occupied_from, '-infinity'::date), p.occupied_to, '[)')
|
||||||
|
&& daterange(l.start_on, l.end_on, '[]')
|
||||||
|
GROUP BY l.interval_key, l.site_uid, l.start_on, l.end_on, l.consumed_l, l.stock_status, p.asset_uid,
|
||||||
|
o.time_out_local, c.time_in_local),
|
||||||
|
eng AS (
|
||||||
|
SELECT g.interval_key, g.asset_uid, sum(t.net_fuel_l) AS engine_l,
|
||||||
|
sum(extract(epoch FROM t.ended_at - t.started_at)) / 3600 AS covered_h,
|
||||||
|
sum(extract(epoch FROM t.ended_at - t.started_at)) >= extract(epoch FROM g.t1 - g.t0) AS full_cover
|
||||||
|
FROM gen g JOIN fm.engine_fuel_total t ON t.asset_uid = g.asset_uid
|
||||||
|
AND t.claim_status IN ('asserted','reviewed')
|
||||||
|
AND t.started_at >= g.t0 AND t.ended_at <= g.t1
|
||||||
|
GROUP BY g.interval_key, g.asset_uid, g.t0, g.t1),
|
||||||
|
tot AS (
|
||||||
|
SELECT g.interval_key, count(*) AS n_generators,
|
||||||
|
count(*) FILTER (WHERE g.placed_throughout AND e.full_cover) AS n_complete, sum(e.engine_l) AS engine_all_l
|
||||||
|
FROM gen g LEFT JOIN eng e ON e.interval_key = g.interval_key AND e.asset_uid = g.asset_uid
|
||||||
|
GROUP BY g.interval_key)
|
||||||
|
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||||
|
(to_char(g.start_on, 'YYYY-MM-DD') || ' to ' || to_char(g.end_on, 'YYYY-MM-DD'))::varchar(30) AS interval_label,
|
||||||
|
g.start_on, g.end_on, g.stock_status::varchar(40) AS stock_status, g.times_known AS endpoint_times_known,
|
||||||
|
g.asset_uid::varchar(40) AS asset_uid, g.placed_throughout,
|
||||||
|
round(g.consumed_l, 1)::numeric(9,1) AS stock_balance_l, round(e.engine_l, 1)::numeric(9,1) AS engine_l,
|
||||||
|
round(e.covered_h, 1)::numeric(9,1) AS covered_h,
|
||||||
|
round(extract(epoch FROM g.t1 - g.t0) / 3600, 1)::numeric(9,1) AS interval_h,
|
||||||
|
round(t.engine_all_l, 1)::numeric(9,1) AS engine_all_generators_l,
|
||||||
|
(t.n_complete = t.n_generators) AS all_generators_metered,
|
||||||
|
(CASE WHEN t.n_complete = t.n_generators AND g.times_known AND g.stock_status = 'ok'
|
||||||
|
THEN round(g.consumed_l - t.engine_all_l, 1) END)::numeric(9,1) AS difference_l,
|
||||||
|
(CASE WHEN t.n_complete = t.n_generators AND g.times_known AND g.stock_status = 'ok'
|
||||||
|
THEN round(100 * (g.consumed_l - t.engine_all_l) / NULLIF(t.engine_all_l, 0), 1) END)::numeric(7,1) AS difference_pct
|
||||||
|
FROM gen g
|
||||||
|
JOIN eng e ON e.interval_key = g.interval_key AND e.asset_uid = g.asset_uid
|
||||||
|
JOIN tot t ON t.interval_key = g.interval_key
|
||||||
|
JOIN public.v_site s ON s.site_uid = g.site_uid;
|
||||||
|
|
||||||
|
-- B6: one row per site and check; rows stay when ok.
|
||||||
|
CREATE VIEW public.v_data_quality AS
|
||||||
|
WITH c AS (
|
||||||
|
SELECT s.site_uid,
|
||||||
|
(SELECT count(*) FROM fm.stock_observation o WHERE o.site_uid = s.site_uid
|
||||||
|
AND o.litres_basis = 'placeholder_zero' AND o.claim_status IN ('asserted','reviewed','disputed')) AS n_placeholder,
|
||||||
|
(SELECT count(*) FROM fm.v_movement_event e JOIN fm.visit v ON v.visit_id = e.visit_id
|
||||||
|
WHERE v.site_uid = s.site_uid AND e.spread_l > 0)
|
||||||
|
+ (SELECT count(*) FROM fm.v_visit_stock st JOIN fm.visit v ON v.visit_id = st.visit_id
|
||||||
|
WHERE v.site_uid = s.site_uid AND st.spread_l > 0) AS n_conflicts,
|
||||||
|
(SELECT count(*) FROM fm.hour_meter_reading h JOIN fm.runtime_meter m ON m.meter_id = h.meter_id
|
||||||
|
WHERE m.site_uid = s.site_uid AND h.claim_status IN ('asserted','reviewed','disputed')) AS n_readings,
|
||||||
|
(SELECT count(*) FROM fm.runtime_meter m WHERE m.site_uid = s.site_uid AND m.mapping_status <> 'confirmed') AS n_unmapped,
|
||||||
|
(SELECT count(*) FROM fm.asset_placement p JOIN fm.asset a ON a.asset_uid = p.asset_uid
|
||||||
|
WHERE p.site_uid = s.site_uid AND a.kind = 'generator'
|
||||||
|
AND daterange(coalesce(p.occupied_from, '-infinity'::date), p.occupied_to, '[)')
|
||||||
|
@> (now() AT TIME ZONE 'Asia/Kabul')::date
|
||||||
|
AND NOT EXISTS (SELECT 1 FROM fm.runtime_meter m JOIN fm.hour_meter_reading h ON h.meter_id = m.meter_id
|
||||||
|
WHERE m.asset_uid = p.asset_uid AND m.mapping_status = 'confirmed'
|
||||||
|
AND h.claim_status IN ('asserted','reviewed'))) AS n_gen_no_readings,
|
||||||
|
(SELECT count(*) FROM fm.fuel_movement x WHERE x.site_uid = s.site_uid AND x.claim_status = 'asserted')
|
||||||
|
+ (SELECT count(*) FROM fm.stock_observation x WHERE x.site_uid = s.site_uid AND x.claim_status = 'asserted')
|
||||||
|
+ (SELECT count(*) FROM fm.hour_meter_reading x JOIN fm.runtime_meter m ON m.meter_id = x.meter_id
|
||||||
|
WHERE m.site_uid = s.site_uid AND x.claim_status = 'asserted')
|
||||||
|
+ (SELECT count(*) FROM fm.electricity_bill x WHERE x.site_uid = s.site_uid AND x.claim_status = 'asserted') AS n_unreviewed,
|
||||||
|
(SELECT count(*) FROM fm.v_stock_point p WHERE p.site_uid = s.site_uid
|
||||||
|
AND abs(p.after_vs_before_plus_delivered_l) > 0.5) AS n_after_mismatch,
|
||||||
|
(SELECT count(*) FROM fm.v_stock_interval i WHERE i.site_uid = s.site_uid
|
||||||
|
AND i.stock_status IN ('negative','same_day')) AS n_neg_same_day,
|
||||||
|
(SELECT count(*) FROM fm.v_stock_interval i WHERE i.site_uid = s.site_uid
|
||||||
|
AND i.stock_status NOT IN ('ok','negative','same_day')) AS n_gaps,
|
||||||
|
(SELECT count(*) FROM fm.electricity_bill b WHERE b.site_uid = s.site_uid
|
||||||
|
AND b.claim_status IN ('asserted','reviewed','disputed')) AS n_bills,
|
||||||
|
(SELECT count(*) FROM fm.electricity_bill b WHERE b.site_uid = s.site_uid
|
||||||
|
AND (b.kwh IS NULL OR b.claim_status = 'disputed')
|
||||||
|
AND b.claim_status IN ('asserted','reviewed','disputed')) AS n_bills_no_kwh, -- no usable kWh
|
||||||
|
(SELECT count(*) FROM fm.fuel_movement m WHERE m.site_uid = s.site_uid
|
||||||
|
AND m.kind IN ('withdrawal','drain','spill','loss_reported','internal_transfer')
|
||||||
|
AND m.claim_status IN ('asserted','reviewed','disputed')) AS n_transfer_records,
|
||||||
|
(SELECT count(*) FROM fm.baseline_version b JOIN fm.v_baseline_validity v ON v.baseline_id = b.baseline_id
|
||||||
|
WHERE b.site_uid = s.site_uid AND b.status IN ('proposed','approved') AND NOT v.inputs_unchanged) AS n_drift,
|
||||||
|
(SELECT count(*) FROM fm.v_review_stale r WHERE r.site_uid = s.site_uid) AS n_stale,
|
||||||
|
(SELECT count(*) FROM fm.v_visit_provenance_only v WHERE v.site_uid = s.site_uid) AS n_provenance_only,
|
||||||
|
(SELECT count(*) FROM fm.v_lph_interval l WHERE l.site_uid = s.site_uid
|
||||||
|
AND l.exclusion_reasons LIKE '%confirmed for a generator not placed here%') AS n_unplaced_counter,
|
||||||
|
(SELECT count(*) FROM fm.v_lph_interval l WHERE l.site_uid = s.site_uid
|
||||||
|
AND l.exclusion_reasons LIKE '%meter-coverage assumption % does not name%') AS n_cov_mismatch
|
||||||
|
FROM fm.site s)
|
||||||
|
SELECT vs.site_uid, vs.site_code, vs.site_name, vs.site_label,
|
||||||
|
x.ord::integer AS check_order, x.code::varchar(40) AS check_code, x.label::varchar(80) AS label,
|
||||||
|
x.status::varchar(10) AS status, x.detail::varchar(300) AS detail
|
||||||
|
FROM c JOIN public.v_site vs ON vs.site_uid = c.site_uid
|
||||||
|
CROSS JOIN LATERAL (VALUES
|
||||||
|
(1, 'placeholder_rows', 'Placeholder rows',
|
||||||
|
CASE WHEN c.n_placeholder > 0 THEN 'warning' ELSE 'ok' END,
|
||||||
|
c.n_placeholder || ' row(s) imported as "no stock reading", never as 0'),
|
||||||
|
(2, 'claim_conflicts', 'Conflicting claims',
|
||||||
|
CASE WHEN c.n_conflicts > 0 THEN 'conflict' ELSE 'ok' END,
|
||||||
|
c.n_conflicts || ' movement or stock reading(s) with disagreeing sources'),
|
||||||
|
(3, 'runtime_evidence', 'Runtime evidence',
|
||||||
|
CASE WHEN c.n_readings = 0 THEN 'missing' ELSE 'ok' END,
|
||||||
|
CASE WHEN c.n_readings = 0 THEN 'no hour-meter readings: no L/h can be calculated'
|
||||||
|
ELSE c.n_readings || ' timer reading(s)' END),
|
||||||
|
(4, 'location', 'Location',
|
||||||
|
CASE vs.location_status WHEN 'accepted' THEN 'ok' WHEN 'awaiting acceptance' THEN 'warning' ELSE 'missing' END,
|
||||||
|
vs.location_status::text),
|
||||||
|
(5, 'unmapped_meters', 'Counters not tied to a generator',
|
||||||
|
CASE WHEN c.n_unmapped > 0 THEN 'warning' ELSE 'ok' END, c.n_unmapped || ' counter(s)'),
|
||||||
|
(6, 'generators_without_readings', 'Generators without timer readings',
|
||||||
|
CASE WHEN c.n_gen_no_readings > 0 THEN 'missing' ELSE 'ok' END, c.n_gen_no_readings || ' generator(s) placed now'),
|
||||||
|
(7, 'unreviewed_claims', 'Claims not yet reviewed against the originals',
|
||||||
|
CASE WHEN c.n_unreviewed > 0 THEN 'warning' ELSE 'ok' END, c.n_unreviewed || ' claim(s)'),
|
||||||
|
(8, 'after_vs_before_plus_delivered', 'Stated after-delivery stock differs from before + delivered',
|
||||||
|
CASE WHEN c.n_after_mismatch > 0 THEN 'conflict' ELSE 'ok' END, c.n_after_mismatch || ' visit(s)'),
|
||||||
|
(9, 'negative_or_same_day_intervals', 'Negative or same-day stock intervals',
|
||||||
|
CASE WHEN c.n_neg_same_day > 0 THEN 'warning' ELSE 'ok' END, c.n_neg_same_day || ' interval(s)'),
|
||||||
|
(10, 'interval_gaps', 'Stock intervals excluded for missing or unordered data',
|
||||||
|
CASE WHEN c.n_gaps > 0 THEN 'warning' ELSE 'ok' END, c.n_gaps || ' interval(s)'),
|
||||||
|
(11, 'no_kwh', 'Grid energy (kWh)',
|
||||||
|
CASE WHEN c.n_bills = 0 OR c.n_bills_no_kwh > 0 THEN 'missing' ELSE 'ok' END,
|
||||||
|
CASE WHEN c.n_bills = 0 THEN 'no electricity bills'
|
||||||
|
WHEN c.n_bills_no_kwh > 0 THEN c.n_bills_no_kwh || ' bill(s) without usable kWh (cost only or disputed)'
|
||||||
|
ELSE 'kWh on every bill' END),
|
||||||
|
(12, 'transfer_records', 'Transfer, drain or loss records (HD-05)',
|
||||||
|
CASE WHEN c.n_transfer_records = 0 THEN 'warning' ELSE 'ok' END,
|
||||||
|
CASE WHEN c.n_transfer_records = 0 THEN 'none recorded: every stock balance assumes no unrecorded movements'
|
||||||
|
ELSE c.n_transfer_records || ' record(s)' END),
|
||||||
|
(13, 'baseline_input_drift', 'Baseline inputs changed since computation',
|
||||||
|
CASE WHEN c.n_drift > 0 THEN 'warning' ELSE 'ok' END, c.n_drift || ' proposed or approved version(s)'),
|
||||||
|
(14, 'approval_status', 'Baseline approval',
|
||||||
|
CASE WHEN vs.baseline_status = 'approved' THEN 'ok' ELSE 'warning' END,
|
||||||
|
vs.baseline_status_label || coalesce(' - ' || vs.baseline_validity_note, '')),
|
||||||
|
(15, 'stale_reviews', 'Review entries for values changed since',
|
||||||
|
CASE WHEN c.n_stale > 0 THEN 'warning' ELSE 'ok' END, c.n_stale || ' entr(y/ies)'),
|
||||||
|
(16, 'provenance_only_visits', 'Visits kept for provenance only',
|
||||||
|
CASE WHEN c.n_provenance_only > 0 THEN 'warning' ELSE 'ok' END,
|
||||||
|
c.n_provenance_only || ' visit(s) whose records were all retracted or superseded'),
|
||||||
|
(17, 'counter_on_unplaced_generator', 'Counters confirmed for a generator not placed here',
|
||||||
|
CASE WHEN c.n_unplaced_counter > 0 THEN 'conflict' ELSE 'ok' END, c.n_unplaced_counter || ' interval(s)'),
|
||||||
|
(18, 'coverage_assumption_mismatch', 'Meter-coverage assumption does not match the counters read',
|
||||||
|
CASE WHEN c.n_cov_mismatch > 0 THEN 'conflict' ELSE 'ok' END, c.n_cov_mismatch || ' interval(s)')
|
||||||
|
) AS x(ord, code, label, status, detail);
|
||||||
|
|
||||||
|
-- B10: specifications, labelled as such; live rows only. A disputed specification is listed with its
|
||||||
|
-- status and without its value.
|
||||||
|
CREATE VIEW public.v_spec_reference AS
|
||||||
|
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||||
|
p.slot_label::varchar(20) AS slot, e.asset_uid::varchar(40) AS asset_uid, a.kind::varchar(10) AS asset_kind,
|
||||||
|
concat_ws(' ', a.make, a.model)::varchar(120) AS make_model,
|
||||||
|
e.attribute::varchar(40) AS attribute, e.load_pct::numeric(5,1) AS load_pct,
|
||||||
|
(CASE WHEN e.claim_status <> 'disputed' THEN e.value_num END)::numeric(12,3) AS value,
|
||||||
|
(CASE WHEN e.claim_status <> 'disputed' THEN e.value_text END)::varchar(120) AS value_text, e.unit::varchar(20) AS unit,
|
||||||
|
e.basis::varchar(30) AS basis, e.applicability::varchar(12) AS applicability,
|
||||||
|
e.applicability_note::varchar(300) AS applicability_note,
|
||||||
|
'specification, not measured'::varchar(30) AS value_kind, e.claim_status::varchar(12) AS claim_status
|
||||||
|
FROM fm.equipment_spec e
|
||||||
|
JOIN fm.asset a ON a.asset_uid = e.asset_uid
|
||||||
|
JOIN (SELECT DISTINCT ON (asset_uid, site_uid) asset_uid, site_uid, slot_label
|
||||||
|
FROM fm.asset_placement ORDER BY asset_uid, site_uid, occupied_from DESC NULLS LAST) p ON p.asset_uid = e.asset_uid
|
||||||
|
JOIN public.v_site s ON s.site_uid = p.site_uid
|
||||||
|
WHERE e.claim_status IN ('asserted','reviewed','disputed');
|
||||||
|
|
||||||
|
-- M3 sensitivity table, reference tab only (shown only once Q9 approves a runtime assumption).
|
||||||
|
CREATE VIEW public.v_modelled_lph AS
|
||||||
|
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||||
|
m.code::varchar(60) AS assumption_code, m.status::varchar(12) AS status, m.basis::varchar(20) AS basis,
|
||||||
|
m.h_per_day::numeric(5,2) AS h_per_day, m.h_per_day_low::numeric(5,2) AS h_per_day_low,
|
||||||
|
m.h_per_day_high::numeric(5,2) AS h_per_day_high, m.window_start, m.window_end,
|
||||||
|
m.matched_days::integer AS matched_days, round(m.l_per_day, 1)::numeric(7,1) AS l_per_day,
|
||||||
|
round(m.modelled_l_per_h, 2)::numeric(6,2) AS modelled_l_per_h,
|
||||||
|
round(m.modelled_l_per_h_low, 2)::numeric(6,2) AS modelled_l_per_h_low,
|
||||||
|
round(m.modelled_l_per_h_high, 2)::numeric(6,2) AS modelled_l_per_h_high,
|
||||||
|
'modelled: L per calendar day / assumed runtime; not measured'::varchar(70) AS value_kind
|
||||||
|
FROM fm.v_modelled_lph m JOIN public.v_site s ON s.site_uid = m.site_uid;
|
||||||
|
|
||||||
|
-- R4: electricity cost per bill month; cost context only, never joined into fuel metrics. A
|
||||||
|
-- disputed bill stays listed, without amount or kWh.
|
||||||
|
CREATE VIEW public.v_electricity_cost AS
|
||||||
|
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||||
|
b.bill_month, b.period_start, b.period_end, (b.period_start IS NOT NULL) AS period_known,
|
||||||
|
(CASE WHEN b.claim_status <> 'disputed' THEN b.amount END)::numeric(12,2) AS amount, b.currency::varchar(8) AS currency,
|
||||||
|
b.currency_basis::varchar(10) AS currency_basis,
|
||||||
|
(CASE WHEN b.claim_status <> 'disputed' THEN b.kwh END)::numeric(10,1) AS kwh,
|
||||||
|
b.record_ref::varchar(60) AS source_ref, b.claim_status::varchar(12) AS claim_status
|
||||||
|
FROM fm.electricity_bill b JOIN public.v_site s ON s.site_uid = b.site_uid
|
||||||
|
WHERE b.claim_status IN ('asserted','reviewed','disputed');
|
||||||
|
|
||||||
|
-- B8: operator workbook calculations and our assessment; live rows only. A disputed transcription is
|
||||||
|
-- listed with its status and without its formula or value.
|
||||||
|
CREATE VIEW public.v_imported_calculation AS
|
||||||
|
SELECT s.site_uid, s.site_code, s.site_name, s.site_label,
|
||||||
|
c.cell_ref::varchar(40) AS cell_ref, c.source_label::varchar(80) AS source_label,
|
||||||
|
(CASE WHEN c.claim_status <> 'disputed' THEN c.formula END)::varchar(200) AS formula,
|
||||||
|
(CASE WHEN c.claim_status <> 'disputed' THEN c.cached_value END)::numeric(14,6) AS cached_value,
|
||||||
|
c.unit_as_labelled::varchar(30) AS unit_as_labelled, c.our_reading::varchar(300) AS our_reading,
|
||||||
|
c.baseline_use::varchar(12) AS baseline_use, c.exclusion_reason::varchar(300) AS exclusion_reason,
|
||||||
|
left(c.source_sha256, 16)::varchar(16) AS document_key, c.claim_status::varchar(12) AS claim_status
|
||||||
|
FROM fm.imported_calculation c JOIN public.v_site s ON s.site_uid = c.site_uid
|
||||||
|
WHERE c.claim_status IN ('asserted','reviewed','disputed');
|
||||||
|
|
||||||
|
COMMIT;
|
||||||
@@ -0,0 +1,122 @@
|
|||||||
|
# fuel-db migrations
|
||||||
|
|
||||||
|
**Status: DRAFT, not applied.** Nothing here runs automatically: `deploy.sh` never touches
|
||||||
|
fuel-db, and the compose file mounts only `../init/01-roles.sh`. A person applies these files by
|
||||||
|
hand, once the owner gives the go-ahead (phase P3 of the design).
|
||||||
|
|
||||||
|
The design and its rationale live in the fuel-monitoring corpus:
|
||||||
|
`.context/plans/2026-10-24-fuel-db-baseline-dashboard.md`. The SQL comments cite it as
|
||||||
|
"design §n". This folder carries no site data.
|
||||||
|
|
||||||
|
## Files
|
||||||
|
|
||||||
|
| File | Creates | Visible to Trino |
|
||||||
|
|---|---|---|
|
||||||
|
| `0001_fm_core.sql` | schema `fm`, `btree_gist` in it, 25 tables, constraints, trigger-backed rules 1–13 | no |
|
||||||
|
| `0002_fm_derivations.sql` | internal views in `fm`: claims → visits → stock intervals → L/h and L/day intervals, dependency hashes, baseline validity | no |
|
||||||
|
| `0003_bi_views.sql` | `fm.v_site_baseline_status` and the 18 `public.v_*` views Superset reads | yes, read-only |
|
||||||
|
|
||||||
|
Each file is one transaction. Apply them in order.
|
||||||
|
|
||||||
|
## Applying (on the server)
|
||||||
|
|
||||||
|
1. Create a loader role as in the repository README, "Data loaders". Its sessions act as
|
||||||
|
`fuel_owner`, so every object belongs to `fuel_owner`. `trino_ro` can then read the `public`
|
||||||
|
views through the existing default privileges, with no role or grant changes.
|
||||||
|
2. Open the tunnel and apply the three files as the loader:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
ssh -L 5433:127.0.0.1:5433 petrolyst
|
||||||
|
for f in 0001_fm_core 0002_fm_derivations 0003_bi_views; do
|
||||||
|
psql -h localhost -p 5433 -U <loader> -d fuel -v ON_ERROR_STOP=1 -f "$f.sql" || break
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
3. Check through Trino (`docker exec -it fuel-monitoring-trino-1 trino`):
|
||||||
|
- `SHOW TABLES FROM fuel.public` lists the 18 `v_*` views;
|
||||||
|
- `SELECT count(*) FROM fuel.fm.site` fails with "permission denied for schema fm".
|
||||||
|
|
||||||
|
The `public` views are owned by `fuel_owner` and run with its rights, so Trino never needs, and
|
||||||
|
never gets, read access to `fm`. Trino's metadata still lists fm's table and column names, as it
|
||||||
|
lists every schema in fuel-db (TimescaleDB's internal ones included); names only, no rows.
|
||||||
|
|
||||||
|
## Data
|
||||||
|
|
||||||
|
- Nobody edits `fm.*` by hand. Rows arrive only through the corpus loader (`fmdb.py export` /
|
||||||
|
`load`, phase P1, not yet written), in one transaction per load.
|
||||||
|
- Every value is a claim:
|
||||||
|
- a correction marks the old row `superseded`, inserts the new row under the same `source_key`,
|
||||||
|
and points `superseded_by` at it;
|
||||||
|
- the old row may point at itself until the new row exists;
|
||||||
|
- a removed record is `retracted`, never deleted.
|
||||||
|
- Review status, coordinate acceptance and assumption acceptance come only from an `fm.review` row
|
||||||
|
that matches the exact `assertion_sha256` (rule 11).
|
||||||
|
- A baseline version is inserted as `proposed`. `approved` is a later update, refused unless the
|
||||||
|
version meets its policy and its inputs are unchanged (rule 1).
|
||||||
|
|
||||||
|
## Trigger-backed rules (in 0001)
|
||||||
|
|
||||||
|
1. Baseline versions start as draft or proposed. Approval needs `meets_policy` and unchanged
|
||||||
|
inputs. Approved rows and their inputs are immutable; the only change is approved →
|
||||||
|
superseded. An input never moves to another version.
|
||||||
|
2. An internal transfer's tanks are placed at the movement's site on the visit date.
|
||||||
|
3. A claim's site is its visit's site, and a reading's meter belongs to that site.
|
||||||
|
4. A reading's epoch belongs to its meter.
|
||||||
|
5. A runtime assumption derived from fuel cannot be accepted. This is a CHECK constraint.
|
||||||
|
6. A baseline input's dates equal its endpoint visits. Included inputs carry the version's
|
||||||
|
population.
|
||||||
|
7. A source document supersedes only one of its own family.
|
||||||
|
8. A confirmed or proposed meter mapping names a generator placed at the meter's site.
|
||||||
|
9. Meter-coverage members:
|
||||||
|
- valid when written, and written in the assumption's own transaction;
|
||||||
|
- immutable afterwards;
|
||||||
|
- complete at commit: as many counters as covered sets.
|
||||||
|
10. The live claims of one `event_key` share visit, kind and tanks. A superseded or retracted
|
||||||
|
claim keeps its own, so a correction can change them.
|
||||||
|
11. Status comes only from a matching review. Values are never corrected in place.
|
||||||
|
12. An idle assumption names a generator placed at its own site during its dates.
|
||||||
|
13. Identity and classification never change in place: a visit's key, site, date and sequence; a
|
||||||
|
source document's hash, family and kind (its rank); an asset's uid and kind; a meter's id and
|
||||||
|
site; an epoch's id and meter. A wrong visit date is corrected by a new visit, which the
|
||||||
|
superseding claims point to.
|
||||||
|
|
||||||
|
## Rollback
|
||||||
|
|
||||||
|
Before any data matters, run `DROP SCHEMA fm CASCADE` as the loader. This also drops every
|
||||||
|
`public.v_*` view, and leaves fuel-db as it was before 0001.
|
||||||
|
|
||||||
|
Once data has been loaded, this deletes it. The corpus remains the source, so a re-load restores
|
||||||
|
it, except baseline versions and their approvals, which live only in fuel-db: export
|
||||||
|
`fm.baseline_version` and `fm.baseline_input` first.
|
||||||
|
|
||||||
|
## Verified locally (2026-10-24, scratch only)
|
||||||
|
|
||||||
|
Setup:
|
||||||
|
- `timescale/timescaledb:2.31.0-pg17`, with this repository's `01-roles.sh`;
|
||||||
|
- a loader created by the README recipe;
|
||||||
|
- `trinodb/trino:484`, with this repository's `trino/etc` connected as `trino_ro`.
|
||||||
|
|
||||||
|
Results:
|
||||||
|
- **Apply:** all three files applied as the loader, under the loader's default `search_path`.
|
||||||
|
- **Fixture:** the operator workbook of 21 Oct 2026, the five site records, and synthetic sites
|
||||||
|
for the runtime path.
|
||||||
|
- **Tests:** all 150 checks of the design's §9 pass: 98 written with the migrations, and 52
|
||||||
|
regressions for the implementation inspection's findings. These cover:
|
||||||
|
- figures and statuses;
|
||||||
|
- claims, reviews and revisions;
|
||||||
|
- movement events;
|
||||||
|
- coverage, placement and idle scope;
|
||||||
|
- baseline dependencies and approval;
|
||||||
|
- the L/day path;
|
||||||
|
- current periods;
|
||||||
|
- flow-meter reconciliation (coverage, placement, visit times), runtime validity, modelled-L/h
|
||||||
|
windows, disputed claims supplying no values, the L/day reference, S2's L/h rows following
|
||||||
|
the version's validity, corrections under one `event_key`, fixed meter and epoch owners, and
|
||||||
|
population keys that name the generators a coverage assumption covers;
|
||||||
|
- every refusal listed above.
|
||||||
|
- **Trino:**
|
||||||
|
- all 303 columns of the 18 views have identical names, order and types in PostgreSQL and
|
||||||
|
Trino;
|
||||||
|
- `fm` is unreadable: a query is refused, though its names are listed (see Applying);
|
||||||
|
- writes are refused.
|
||||||
|
- **Not run on the server:** none of this has been run against the live fuel-db.
|
||||||
Reference in new issue
Block a user