Remove Trino: Superset reads fuel-db directly as fuel_ro (PD-14)
trino/ deleted; the role trino_ro becomes fuel_ro, its password read from a secret file at connect time; dashboards' SQL and column list for PostgreSQL; the Compose project is renamed petrolyst, with volumes pinned to their old names. AGENTS.md points to the architecture context. Not deployed: the next deploy needs README's one-time steps first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
686318dfac
commit
8405706ed8
25 files changed
+453
-410
No files matched your search
@@ -4,6 +4,10 @@ Deployment of the petrolyst server (atoma.petrolyst.com) and the Superset custom
|
|||||||
Runbook: README.md. Workspace rules: `../AGENTS.md`; shared context: `../.context/` (server and
|
Runbook: README.md. Workspace rules: `../AGENTS.md`; shared context: `../.context/` (server and
|
||||||
identity facts: `deployment.md`).
|
identity facts: `deployment.md`).
|
||||||
|
|
||||||
|
**Architecture:** read `../.context/architecture.md` before planning or building. This repository
|
||||||
|
runs persistence (fuel-db and its migrations) and Superset; name the bands a change touches and
|
||||||
|
keep the rules there; a change against a rule or an open conflict goes to the owner first.
|
||||||
|
|
||||||
## Layout
|
## Layout
|
||||||
- `stacks/`, `traefik/`, `deploy/`, `host/`: how and where things run.
|
- `stacks/`, `traefik/`, `deploy/`, `host/`: how and where things run.
|
||||||
- `petrolyst-insights/`: what Superset looks like and does (config, security manager, the guest
|
- `petrolyst-insights/`: what Superset looks like and does (config, security manager, the guest
|
||||||
|
|||||||
@@ -5,7 +5,8 @@ Everything that runs on the petrolyst server (109.199.127.84, Ubuntu 24.04):
|
|||||||
- **Traefik** at the edge for every project on the server. Its dashboard is at
|
- **Traefik** at the edge for every project on the server. Its dashboard is at
|
||||||
https://traefik.petrolyst.com, behind Keycloak.
|
https://traefik.petrolyst.com, behind Keycloak.
|
||||||
- **fuel-monitoring** (ATOMA): Superset at https://atoma.petrolyst.com, reading the operational
|
- **fuel-monitoring** (ATOMA): Superset at https://atoma.petrolyst.com, reading the operational
|
||||||
store `fuel-db` (Postgres 17 with TimescaleDB) through Trino. Sign-in goes through Keycloak
|
store `fuel-db` (Postgres 17 with TimescaleDB) directly, as the read-only role `fuel_ro`
|
||||||
|
(Trino was removed on 29 Oct 2026). Sign-in goes through Keycloak
|
||||||
client `petrolyst`, and the UI carries the Petrolyst brand.
|
client `petrolyst`, and the UI carries the Petrolyst brand.
|
||||||
|
|
||||||
**Deploys are manual.** A person deploys a clean commit of `main` from the Mac ("Manual deploy"
|
**Deploys are manual.** A person deploys a clean commit of `main` from the Mac ("Manual deploy"
|
||||||
@@ -17,7 +18,7 @@ Customization and deployment are separate, as with media-insights and media-inte
|
|||||||
|---|---|
|
|---|---|
|
||||||
| `petrolyst-insights/` | how Superset looks and behaves for Petrolyst: config, Keycloak login code, branding, patches, dashboards. See its README. |
|
| `petrolyst-insights/` | how Superset looks and behaves for Petrolyst: config, Keycloak login code, branding, patches, dashboards. See its README. |
|
||||||
| `traefik/` | edge proxy, TLS, the `sso` middleware (oauth2-proxy) that guards the dashboard |
|
| `traefik/` | edge proxy, TLS, the `sso` middleware (oauth2-proxy) that guards the dashboard |
|
||||||
| `stacks/fuel-monitoring/` | Compose stack, Superset Dockerfile, Trino config and catalog, fuel-db roles |
|
| `stacks/fuel-monitoring/` | Compose stack, Superset Dockerfile and init, fuel-db roles and migrations |
|
||||||
| `deploy/` | `deploy.sh` (run on the server by a manual deploy), `rollback.sh` (manual), `lib.sh` (shared) |
|
| `deploy/` | `deploy.sh` (run on the server by a manual deploy), `rollback.sh` (manual), `lib.sh` (shared) |
|
||||||
| `host/` | one-time server setup: `bootstrap.sh`, `secrets-init.sh` |
|
| `host/` | one-time server setup: `bootstrap.sh`, `secrets-init.sh` |
|
||||||
| `keycloak/` | the `petrolyst` client for Keycloak's partial import |
|
| `keycloak/` | the `petrolyst` client for Keycloak's partial import |
|
||||||
@@ -38,10 +39,10 @@ No credential is in this repository. Secrets exist only on the server, under
|
|||||||
4. It resolves the artifacts and writes them to `releases/<sha>/ARTIFACTS`:
|
4. It resolves the artifacts and writes them to `releases/<sha>/ARTIFACTS`:
|
||||||
- `SUPERSET_IMAGE_TAG` is a hash of exactly the Superset image's build inputs. They include
|
- `SUPERSET_IMAGE_TAG` is a hash of exactly the Superset image's build inputs. They include
|
||||||
`superset/init.sh`, so changing that file runs `superset-init` again.
|
`superset/init.sh`, so changing that file runs `superset-init` again.
|
||||||
- `TRINO_CONFIG_HASH` and `TRAEFIK_CONFIG_HASH` are hashes of the bind-mounted config.
|
- `TRAEFIK_CONFIG_HASH` is a hash of Traefik's bind-mounted static config.
|
||||||
- Compose recreates a container only when its value changes.
|
- Compose recreates a container only when its value changes.
|
||||||
5. Preflight, before anything live changes:
|
5. Preflight, before anything live changes:
|
||||||
- every secret and env file exists;
|
- every secret file exists;
|
||||||
- `petrolyst-insights/` holds no deployment files;
|
- `petrolyst-insights/` holds no deployment files;
|
||||||
- `docker compose config` passes for every stack;
|
- `docker compose config` passes for every stack;
|
||||||
- the Superset image is built if it is new;
|
- the Superset image is built if it is new;
|
||||||
@@ -52,13 +53,14 @@ No credential is in this repository. Secrets exist only on the server, under
|
|||||||
- An empty database is a fresh install: it runs `superset-init`.
|
- An empty database is a fresh install: it runs `superset-init`.
|
||||||
- If the image changed, or the database is not at the image's migration head: it stops
|
- If the image changed, or the database is not at the image's migration head: it stops
|
||||||
`superset`, writes `backups/superset-meta_<old tag>_<revision>_<time>.dump`, then runs
|
`superset`, writes `backups/superset-meta_<old tag>_<revision>_<time>.dump`, then runs
|
||||||
`superset-init` (`db upgrade`, `init`, registers "Fuel (Trino)").
|
`superset-init` (`db upgrade`, `init`, registers "Fuel": fuel-db as `fuel_ro`, its password
|
||||||
|
read from `/run/secrets` at connect time, never stored in Superset).
|
||||||
- If the image is unchanged and the schema is at its head, it does none of this.
|
- If the image is unchanged and the schema is at its head, it does none of this.
|
||||||
8. Smoke tests. A failure ends the deploy with an error and prints the stacks' state.
|
8. Smoke tests. A failure ends the deploy with an error and prints the stacks' state.
|
||||||
- `/health` answers through Traefik with a valid certificate.
|
- `/health` answers through Traefik with a valid certificate.
|
||||||
- `/login/` leads to Keycloak with `client_id=petrolyst` and PKCE.
|
- `/login/` leads to Keycloak with `client_id=petrolyst` and PKCE.
|
||||||
- The dashboard redirects to Keycloak.
|
- The dashboard redirects to Keycloak.
|
||||||
- Trino reads `fuel`, and Superset reaches Trino.
|
- Superset's "Fuel" connection reads fuel-db as `fuel_ro`, and its transactions are read-only.
|
||||||
9. Retention:
|
9. Retention:
|
||||||
- 5 releases;
|
- 5 releases;
|
||||||
- backups for 30 days;
|
- backups for 30 days;
|
||||||
@@ -75,7 +77,6 @@ The manual deploy then checks both hosts from outside.
|
|||||||
| `/opt/petrolyst/incoming/<sha>-manual-<time>` | deploy | one upload; only `deploy.sh` turns it into a release |
|
| `/opt/petrolyst/incoming/<sha>-manual-<time>` | deploy | one upload; only `deploy.sh` turns it into a release |
|
||||||
| `/opt/petrolyst/releases/<sha>` | deploy, read-only | finalized release plus `ARTIFACTS`; the last 5 are kept |
|
| `/opt/petrolyst/releases/<sha>` | deploy, read-only | finalized release plus `ARTIFACTS`; the last 5 are kept |
|
||||||
| `/opt/petrolyst/secrets/{shared,traefik,fuel-monitoring}/` | root:deploy 0750, files 0444 | secrets, mounted as `/run/secrets/<name>` |
|
| `/opt/petrolyst/secrets/{shared,traefik,fuel-monitoring}/` | root:deploy 0750, files 0444 | secrets, mounted as `/run/secrets/<name>` |
|
||||||
| `/opt/petrolyst/env/fuel-monitoring.env` | root:deploy 0440 | `FUEL_DB_TRINO_PASSWORD` for Trino's `${ENV:…}` |
|
|
||||||
| `/opt/petrolyst/backups/` | deploy 0750 | Superset metadata dumps |
|
| `/opt/petrolyst/backups/` | deploy 0750 | Superset metadata dumps |
|
||||||
| `/opt/petrolyst/state/` | deploy | last deployed SHA and Superset image tag; `rollback-in-progress` during a migration rollback |
|
| `/opt/petrolyst/state/` | deploy | last deployed SHA and Superset image tag; `rollback-in-progress` during a migration rollback |
|
||||||
|
|
||||||
@@ -160,10 +161,10 @@ becomes the first Superset admin; there is no local admin account.
|
|||||||
| Re-deploy a kept release | `sudo -u deploy /opt/petrolyst/infra/deploy/deploy.sh <sha>` |
|
| Re-deploy a kept release | `sudo -u deploy /opt/petrolyst/infra/deploy/deploy.sh <sha>` |
|
||||||
| Roll back | see below |
|
| Roll back | see below |
|
||||||
| Traefik dashboard | https://traefik.petrolyst.com (role `traefik-admin`) |
|
| Traefik dashboard | https://traefik.petrolyst.com (role `traefik-admin`) |
|
||||||
| Trino shell | `docker exec -it fuel-monitoring-trino-1 trino` |
|
| fuel-db shell | `docker exec -it petrolyst-fuel-db-1 psql -U postgres -d fuel` |
|
||||||
| fuel-db from the Mac | `ssh -L 5433:127.0.0.1:5433 petrolyst`, then connect to `localhost:5433`, database `fuel` |
|
| fuel-db from the Mac | `ssh -L 5433:127.0.0.1:5433 petrolyst`, then connect to `localhost:5433`, database `fuel` |
|
||||||
| Logs | `docker compose -p fuel-monitoring logs -f superset` (or `-p traefik`) |
|
| Logs | `docker compose -p petrolyst logs -f superset` (or `-p traefik`) |
|
||||||
| Superset break-glass | `docker exec -it fuel-monitoring-superset-1 superset fab ...` |
|
| Superset break-glass | `docker exec -it petrolyst-superset-1 superset fab ...` |
|
||||||
|
|
||||||
**Manual deploy**, the only way to deploy (there is no CI runner). On the Mac, in this
|
**Manual deploy**, the only way to deploy (there is no CI runner). On the Mac, in this
|
||||||
repository:
|
repository:
|
||||||
@@ -227,12 +228,12 @@ sudo -u deploy /opt/petrolyst/infra/deploy/rollback.sh <sha>
|
|||||||
Afterwards, `git revert` the bad commit and push, so `main` matches. A plain revert without the
|
Afterwards, `git revert` the bad commit and push, so `main` matches. A plain revert without the
|
||||||
rollback script is stopped by the downgrade guard.
|
rollback script is stopped by the downgrade guard.
|
||||||
|
|
||||||
**Data loaders** (fuel-db). Every table in `public` must belong to `fuel_owner`. `trino_ro` can
|
**Data loaders** (fuel-db). Every table in `public` must belong to `fuel_owner`. `fuel_ro` can
|
||||||
read new tables only through `fuel_owner`'s default privileges, and PostgreSQL applies those
|
read new tables only through `fuel_owner`'s default privileges, and PostgreSQL applies those
|
||||||
only to objects `fuel_owner` creates itself. A table that a loader creates as its own role stays
|
only to objects `fuel_owner` creates itself. A table that a loader creates as its own role stays
|
||||||
unreadable for Trino. Create each loader role so that its sessions act as `fuel_owner`, and it
|
unreadable for Superset and the Petrolyst server. Create each loader role so that its sessions act as `fuel_owner`, and it
|
||||||
cannot create anything as itself. Run this in
|
cannot create anything as itself. Run this in
|
||||||
`docker exec -it fuel-monitoring-fuel-db-1 psql -U postgres -d fuel`:
|
`docker exec -it petrolyst-fuel-db-1 psql -U postgres -d fuel`:
|
||||||
|
|
||||||
```sql
|
```sql
|
||||||
CREATE ROLE <loader> LOGIN PASSWORD '<password>';
|
CREATE ROLE <loader> LOGIN PASSWORD '<password>';
|
||||||
@@ -240,9 +241,46 @@ GRANT fuel_owner TO <loader> WITH INHERIT FALSE, SET TRUE;
|
|||||||
ALTER ROLE <loader> SET role TO 'fuel_owner';
|
ALTER ROLE <loader> SET role TO 'fuel_owner';
|
||||||
```
|
```
|
||||||
|
|
||||||
Tables it creates, hypertables included, then belong to `fuel_owner`, and Trino can read them.
|
Tables it creates, hypertables included, then belong to `fuel_owner`, and `fuel_ro` can read them.
|
||||||
After `SET ROLE NONE`, it cannot create anything in `public`. To repair a table that another
|
After `SET ROLE NONE`, it cannot create anything in `public`. To repair a table that another
|
||||||
role created, run `ALTER TABLE <t> OWNER TO fuel_owner`, then `GRANT SELECT ON <t> TO trino_ro`.
|
role created, run `ALTER TABLE <t> OWNER TO fuel_owner`, then `GRANT SELECT ON <t> TO fuel_ro`.
|
||||||
|
|
||||||
|
**One-time steps after 29 Oct 2026**, at the first deploy that includes them. Two changes
|
||||||
|
meet there:
|
||||||
|
- Trino is removed: Superset reads fuel-db directly. The role `trino_ro` becomes `fuel_ro` and
|
||||||
|
keeps its password (a SCRAM password survives a rename); its secret file moves with it.
|
||||||
|
- The Compose project `fuel-monitoring` is renamed `petrolyst`. Containers are then called
|
||||||
|
`petrolyst-<service>-1`. The data volumes keep their names (`fuel-monitoring_superset_db_data`,
|
||||||
|
`fuel-monitoring_fuel_db_data`; pinned in `compose.yaml`), so no data moves. The old project's
|
||||||
|
containers must be gone first: two PostgreSQL servers on one data volume corrupt it.
|
||||||
|
Preflight refuses to deploy while any of them exists.
|
||||||
|
|
||||||
|
As root on the server, right before the deploy (Superset is down from step 3 until the deploy
|
||||||
|
finishes):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
S=/opt/petrolyst/secrets/fuel-monitoring
|
||||||
|
mv "$S/fuel_db_trino_password" "$S/fuel_db_ro_password" # 1
|
||||||
|
docker exec -i fuel-monitoring-fuel-db-1 psql -U postgres -d fuel -v ON_ERROR_STOP=1 \
|
||||||
|
-c 'ALTER ROLE trino_ro RENAME TO fuel_ro' \
|
||||||
|
-c 'ALTER ROLE fuel_ro SET default_transaction_read_only = on' # 2
|
||||||
|
docker rm -f $(docker ps -aq --filter label=com.docker.compose.project=fuel-monitoring) # 3
|
||||||
|
```
|
||||||
|
|
||||||
|
4. Deploy as usual ("Manual deploy"). Compose starts the `petrolyst` project on the same volumes
|
||||||
|
and no longer starts Trino. Compose warns once that the volumes belong to another project;
|
||||||
|
that is expected. `superset-init` renames the connection "Fuel (Trino)" to "Fuel" and points
|
||||||
|
it at fuel-db; the smoke test checks that it reads as `fuel_ro`, read-only.
|
||||||
|
5. Clean up: `rm /opt/petrolyst/env/fuel-monitoring.env`, `rmdir /opt/petrolyst/env`,
|
||||||
|
`docker image rm trinodb/trino:484`,
|
||||||
|
`docker network rm fuel-monitoring_backend fuel-monitoring_fuel-db-tunnel`.
|
||||||
|
|
||||||
|
**Rollback across the Trino removal** (not tested). `rollback.sh` stops for a release from before
|
||||||
|
it. To go back anyway: remove the `petrolyst` project's containers, reverse steps 1 and 2 above (rename `fuel_ro` back to `trino_ro` and
|
||||||
|
`ALTER ROLE trino_ro RESET default_transaction_read_only`, move the secret file back, write
|
||||||
|
`/opt/petrolyst/env/fuel-monitoring.env` with `FUEL_DB_TRINO_PASSWORD=<that secret>`), rename
|
||||||
|
the Superset connection "Fuel" back to "Fuel (Trino)" with `trino://superset@trino:8080/fuel`,
|
||||||
|
then run that release's own script, `/opt/petrolyst/releases/<sha>/deploy/rollback.sh <sha>`.
|
||||||
|
|
||||||
**Adding a project**: create `stacks/<name>/compose.yaml`, attach the public service to the
|
**Adding a project**: create `stacks/<name>/compose.yaml`, attach the public service to the
|
||||||
external network `proxy`, and set `traefik.*` labels with a unique router name and
|
external network `proxy`, and set `traefik.*` labels with a unique router name and
|
||||||
@@ -261,6 +299,8 @@ its own gate in its stack:
|
|||||||
|
|
||||||
## Verified locally (2026-10-23)
|
## Verified locally (2026-10-23)
|
||||||
|
|
||||||
|
Before Trino's removal (29 Oct 2026): its lines below are history.
|
||||||
|
|
||||||
An end-to-end run ran the real `deploy.sh` and `rollback.sh` in a Linux container against
|
An end-to-end run ran the real `deploy.sh` and `rollback.sh` in a Linux container against
|
||||||
Docker Desktop, with a scratch `/opt/petrolyst`. Production differences in that run:
|
Docker Desktop, with a scratch `/opt/petrolyst`. Production differences in that run:
|
||||||
- a test CA certificate stood in for Let's Encrypt;
|
- a test CA certificate stood in for Let's Encrypt;
|
||||||
|
|||||||
+1
-1
@@ -44,7 +44,7 @@ resolve_artifacts() {
|
|||||||
fi
|
fi
|
||||||
chmod -R a-w "$release"
|
chmod -R a-w "$release"
|
||||||
load_artifacts "$release"
|
load_artifacts "$release"
|
||||||
log "artifacts: superset $SUPERSET_IMAGE_TAG, trino config $TRINO_CONFIG_HASH, traefik config $TRAEFIK_CONFIG_HASH"
|
log "artifacts: superset $SUPERSET_IMAGE_TAG, traefik config $TRAEFIK_CONFIG_HASH"
|
||||||
}
|
}
|
||||||
|
|
||||||
# prepare_fuel_monitoring <release>: before anything live changes, build the
|
# prepare_fuel_monitoring <release>: before anything live changes, build the
|
||||||
|
|||||||
+30
-10
@@ -45,8 +45,7 @@ REQUIRED_FILES=(
|
|||||||
secrets/fuel-monitoring/superset_secret_key
|
secrets/fuel-monitoring/superset_secret_key
|
||||||
secrets/fuel-monitoring/superset_db_password
|
secrets/fuel-monitoring/superset_db_password
|
||||||
secrets/fuel-monitoring/fuel_db_password
|
secrets/fuel-monitoring/fuel_db_password
|
||||||
secrets/fuel-monitoring/fuel_db_trino_password
|
secrets/fuel-monitoring/fuel_db_ro_password
|
||||||
env/fuel-monitoring.env
|
|
||||||
)
|
)
|
||||||
|
|
||||||
log() { printf '%s %s\n' "$(date -u +%H:%M:%S)" "$*" >&2; }
|
log() { printf '%s %s\n' "$(date -u +%H:%M:%S)" "$*" >&2; }
|
||||||
@@ -83,7 +82,6 @@ hash_inputs() {
|
|||||||
|
|
||||||
compute_artifacts() {
|
compute_artifacts() {
|
||||||
printf 'SUPERSET_IMAGE_TAG=%s\n' "$(hash_inputs "$1" "${SUPERSET_INPUTS[@]}")"
|
printf 'SUPERSET_IMAGE_TAG=%s\n' "$(hash_inputs "$1" "${SUPERSET_INPUTS[@]}")"
|
||||||
printf 'TRINO_CONFIG_HASH=%s\n' "$(hash_inputs "$1" stacks/fuel-monitoring/trino/etc)"
|
|
||||||
# Only the static config: dynamic/ is watched and applies without a restart.
|
# Only the static config: dynamic/ is watched and applies without a restart.
|
||||||
printf 'TRAEFIK_CONFIG_HASH=%s\n' "$(hash_inputs "$1" traefik/static)"
|
printf 'TRAEFIK_CONFIG_HASH=%s\n' "$(hash_inputs "$1" traefik/static)"
|
||||||
}
|
}
|
||||||
@@ -92,16 +90,21 @@ compute_artifacts() {
|
|||||||
load_artifacts() {
|
load_artifacts() {
|
||||||
local file=$1/ARTIFACTS line key value
|
local file=$1/ARTIFACTS line key value
|
||||||
[ -s "$file" ] || die "$file is missing"
|
[ -s "$file" ] || die "$file is missing"
|
||||||
unset SUPERSET_IMAGE_TAG TRINO_CONFIG_HASH TRAEFIK_CONFIG_HASH
|
unset SUPERSET_IMAGE_TAG TRAEFIK_CONFIG_HASH
|
||||||
while IFS= read -r line; do
|
while IFS= read -r line; do
|
||||||
key=${line%%=*}
|
key=${line%%=*}
|
||||||
value=${line#*=}
|
value=${line#*=}
|
||||||
[[ $key =~ ^(SUPERSET_IMAGE_TAG|TRINO_CONFIG_HASH|TRAEFIK_CONFIG_HASH)$ && $value =~ ^[0-9a-f]{16}$ ]] ||
|
# A release from before 29 Oct 2026 still runs Trino, which logs in as
|
||||||
|
# `trino_ro`, since renamed `fuel_ro`: it cannot start (README "Rollback
|
||||||
|
# across the Trino removal").
|
||||||
|
[ "$key" != TRINO_CONFIG_HASH ] ||
|
||||||
|
die "$file is from before Trino was removed; see README \"Rollback across the Trino removal\""
|
||||||
|
[[ $key =~ ^(SUPERSET_IMAGE_TAG|TRAEFIK_CONFIG_HASH)$ && $value =~ ^[0-9a-f]{16}$ ]] ||
|
||||||
die "unexpected line in $file: $line"
|
die "unexpected line in $file: $line"
|
||||||
printf -v "$key" '%s' "$value"
|
printf -v "$key" '%s' "$value"
|
||||||
export "${key?}"
|
export "${key?}"
|
||||||
done <"$file"
|
done <"$file"
|
||||||
[ -n "${SUPERSET_IMAGE_TAG:-}" ] && [ -n "${TRINO_CONFIG_HASH:-}" ] && [ -n "${TRAEFIK_CONFIG_HASH:-}" ] ||
|
[ -n "${SUPERSET_IMAGE_TAG:-}" ] && [ -n "${TRAEFIK_CONFIG_HASH:-}" ] ||
|
||||||
die "$file is incomplete"
|
die "$file is incomplete"
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -137,6 +140,10 @@ preflight() {
|
|||||||
[ -z "$bad" ] || die "petrolyst-insights/ holds customization only; found: $bad"
|
[ -z "$bad" ] || die "petrolyst-insights/ holds customization only; found: $bad"
|
||||||
compose "$release" traefik config -q
|
compose "$release" traefik config -q
|
||||||
while IFS= read -r dir; do compose "$release" "$dir" config -q; done < <(stack_dirs "$release")
|
while IFS= read -r dir; do compose "$release" "$dir" config -q; done < <(stack_dirs "$release")
|
||||||
|
# The stack was the Compose project "fuel-monitoring" until 29 Oct 2026; "petrolyst" uses the
|
||||||
|
# same data volumes, so both must never run at once (README "One-time steps after 29 Oct 2026").
|
||||||
|
[ -z "$("${DOCKER[@]}" ps -aq --filter label=com.docker.compose.project=fuel-monitoring)" ] ||
|
||||||
|
die "containers of the old project fuel-monitoring exist; remove them first (README \"One-time steps after 29 Oct 2026\")"
|
||||||
}
|
}
|
||||||
|
|
||||||
# promote <release> <sha>: make the release the live tree. --chmod applies to
|
# promote <release> <sha>: make the release the live tree. --chmod applies to
|
||||||
@@ -255,10 +262,23 @@ smoke_tests() {
|
|||||||
[ "$code" = 302 ] && [[ $location == "$KEYCLOAK_REALM_URL/"* || $location == *"/oauth2/start"* ]] ||
|
[ "$code" = 302 ] && [[ $location == "$KEYCLOAK_REALM_URL/"* || $location == *"/oauth2/start"* ]] ||
|
||||||
die "https://$TRAEFIK_HOST/ should redirect to Keycloak, got $code $location"
|
die "https://$TRAEFIK_HOST/ should redirect to Keycloak, got $code $location"
|
||||||
|
|
||||||
fm exec -T trino trino --execute 'SELECT count(*) FROM fuel.information_schema.schemata' >/dev/null ||
|
# Superset's own connection "Fuel" (fuel_ro, password from /run/secrets) reads fuel-db and
|
||||||
die "Trino cannot read the fuel catalog"
|
# cannot write to it.
|
||||||
fm exec -T superset curl -fsS --max-time 10 http://trino:8080/v1/info >/dev/null ||
|
fm exec -T superset /app/.venv/bin/python - <<'PY' || die "Superset cannot read fuel-db as fuel_ro, or fuel_ro can write"
|
||||||
die "Superset cannot reach Trino"
|
import sys
|
||||||
|
|
||||||
|
from superset.app import create_app
|
||||||
|
|
||||||
|
with create_app().app_context():
|
||||||
|
from superset import db
|
||||||
|
from superset.models.core import Database
|
||||||
|
|
||||||
|
fuel = db.session.query(Database).filter_by(database_name="Fuel").one()
|
||||||
|
with fuel.get_sqla_engine() as engine, engine.connect() as conn:
|
||||||
|
user, read_only = conn.exec_driver_sql(
|
||||||
|
"SELECT current_user, current_setting('transaction_read_only')").one()
|
||||||
|
sys.exit(0 if (user, read_only) == ("fuel_ro", "on") else 1)
|
||||||
|
PY
|
||||||
log "smoke tests passed"
|
log "smoke tests passed"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -40,7 +40,7 @@ if [ "$(cat /etc/docker/daemon.json 2>/dev/null)" != "$daemon_json" ]; then
|
|||||||
fi
|
fi
|
||||||
systemctl enable --now docker
|
systemctl enable --now docker
|
||||||
|
|
||||||
echo "== swap (4 GB) for Trino, Superset and two Postgres in 7.8 GB"
|
echo "== swap (4 GB) for Superset and two Postgres in 7.8 GB"
|
||||||
if ! swapon --show=NAME --noheadings | grep -qx /swapfile; then
|
if ! swapon --show=NAME --noheadings | grep -qx /swapfile; then
|
||||||
[ -e /swapfile ] || fallocate -l 4G /swapfile
|
[ -e /swapfile ] || fallocate -l 4G /swapfile
|
||||||
chmod 600 /swapfile
|
chmod 600 /swapfile
|
||||||
|
|||||||
+3
-15
@@ -1,5 +1,5 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# Server secrets and env files, run as root after bootstrap.sh.
|
# Server secrets, run as root after bootstrap.sh.
|
||||||
# secrets-init.sh generate every missing secret; never overwrites one
|
# secrets-init.sh generate every missing secret; never overwrites one
|
||||||
# secrets-init.sh keycloak-secret store the client secret of Keycloak client
|
# secrets-init.sh keycloak-secret store the client secret of Keycloak client
|
||||||
# "petrolyst" (Clients > petrolyst > Credentials),
|
# "petrolyst" (Clients > petrolyst > Credentials),
|
||||||
@@ -17,7 +17,7 @@ getent group deploy >/dev/null || die "group deploy is missing: run host/bootstr
|
|||||||
ROOT=/opt/petrolyst
|
ROOT=/opt/petrolyst
|
||||||
SECRETS=$ROOT/secrets
|
SECRETS=$ROOT/secrets
|
||||||
umask 077
|
umask 077
|
||||||
install -d -o root -g deploy -m 0750 "$SECRETS" "$SECRETS/shared" "$SECRETS/traefik" "$SECRETS/fuel-monitoring" "$ROOT/env"
|
install -d -o root -g deploy -m 0750 "$SECRETS" "$SECRETS/shared" "$SECRETS/traefik" "$SECRETS/fuel-monitoring"
|
||||||
|
|
||||||
# write_secret <path> <value>: no trailing newline (oauth2-proxy reads the file
|
# write_secret <path> <value>: no trailing newline (oauth2-proxy reads the file
|
||||||
# byte for byte), replaced atomically.
|
# byte for byte), replaced atomically.
|
||||||
@@ -62,22 +62,10 @@ esac
|
|||||||
generate "$SECRETS/fuel-monitoring/superset_secret_key" 32
|
generate "$SECRETS/fuel-monitoring/superset_secret_key" 32
|
||||||
generate "$SECRETS/fuel-monitoring/superset_db_password" 32
|
generate "$SECRETS/fuel-monitoring/superset_db_password" 32
|
||||||
generate "$SECRETS/fuel-monitoring/fuel_db_password" 32
|
generate "$SECRETS/fuel-monitoring/fuel_db_password" 32
|
||||||
generate "$SECRETS/fuel-monitoring/fuel_db_trino_password" 32
|
generate "$SECRETS/fuel-monitoring/fuel_db_ro_password" 32
|
||||||
# 32 hex characters = a 32-byte cookie key, as oauth2-proxy requires.
|
# 32 hex characters = a 32-byte cookie key, as oauth2-proxy requires.
|
||||||
generate "$SECRETS/traefik/oauth2_proxy_cookie_secret" 16
|
generate "$SECRETS/traefik/oauth2_proxy_cookie_secret" 16
|
||||||
|
|
||||||
# Trino reads the catalog password from its environment (${ENV:...}); keep the
|
|
||||||
# env file in step with the secret file that fuel-db's init script uses.
|
|
||||||
env_file=$ROOT/env/fuel-monitoring.env
|
|
||||||
line="FUEL_DB_TRINO_PASSWORD=$(cat "$SECRETS/fuel-monitoring/fuel_db_trino_password")"
|
|
||||||
if ! grep -qxF "$line" "$env_file" 2>/dev/null; then
|
|
||||||
printf '%s\n' "$line" >"$env_file.tmp"
|
|
||||||
chown root:deploy "$env_file.tmp"
|
|
||||||
chmod 0440 "$env_file.tmp"
|
|
||||||
mv "$env_file.tmp" "$env_file"
|
|
||||||
echo "wrote $env_file"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [ ! -s "$SECRETS/shared/keycloak_petrolyst_client_secret" ]; then
|
if [ ! -s "$SECRETS/shared/keycloak_petrolyst_client_secret" ]; then
|
||||||
echo
|
echo
|
||||||
echo "Still missing: the Keycloak client secret. Copy it from Keycloak"
|
echo "Still missing: the Keycloak client secret. Copy it from Keycloak"
|
||||||
|
|||||||
@@ -11,11 +11,11 @@ image, so changes here do not rebuild or restart Superset.
|
|||||||
|
|
||||||
Build and import:
|
Build and import:
|
||||||
|
|
||||||
docker cp atoma/import_bundle.py fuel-monitoring-superset-1:/tmp/
|
docker cp atoma/import_bundle.py petrolyst-superset-1:/tmp/
|
||||||
docker exec fuel-monitoring-superset-1 /app/.venv/bin/python /tmp/import_bundle.py --show-database
|
docker exec petrolyst-superset-1 /app/.venv/bin/python /tmp/import_bundle.py --show-database
|
||||||
python3 atoma/build_bundle.py --database-uuid <printed uuid> --out atoma-dashboards.zip
|
python3 atoma/build_bundle.py --database-uuid <printed uuid> --out atoma-dashboards.zip
|
||||||
docker cp atoma-dashboards.zip fuel-monitoring-superset-1:/tmp/
|
docker cp atoma-dashboards.zip petrolyst-superset-1:/tmp/
|
||||||
docker exec fuel-monitoring-superset-1 /app/.venv/bin/python /tmp/import_bundle.py /tmp/atoma-dashboards.zip --user <username>
|
docker exec petrolyst-superset-1 /app/.venv/bin/python /tmp/import_bundle.py /tmp/atoma-dashboards.zip --user <username>
|
||||||
|
|
||||||
Exit status 2 means a dataset's columns differ from its view: regenerate `atoma/columns.tsv`
|
Exit status 2 means a dataset's columns differ from its view: regenerate `atoma/columns.tsv`
|
||||||
(command in its header), rebuild and import again.
|
(command in its header), rebuild and import again.
|
||||||
@@ -34,12 +34,12 @@ image, so changes here do not rebuild or restart Superset.
|
|||||||
|
|
||||||
docker exec fmdb-scratch-superset /app/.venv/bin/python /tmp/import_bundle.py /tmp/atoma-dashboards.zip \
|
docker exec fmdb-scratch-superset /app/.venv/bin/python /tmp/import_bundle.py /tmp/atoma-dashboards.zip \
|
||||||
--user admin --embedded-out /tmp/embedded-dashboards.json --embed-domain http://127.0.0.1:8098
|
--user admin --embedded-out /tmp/embedded-dashboards.json --embed-domain http://127.0.0.1:8098
|
||||||
docker cp fmdb-scratch-superset:/tmp/embedded-dashboards.json <Petrolist>/.local/fmdb-scratch/
|
docker cp fmdb-scratch-superset:/tmp/embedded-dashboards.json <Petrolyst>/.local/fmdb-scratch/
|
||||||
|
|
||||||
Export from a running instance (inside the container, as the server's deploy user):
|
Export from a running instance (inside the container, as the server's deploy user):
|
||||||
|
|
||||||
docker exec fuel-monitoring-superset-1 superset export-dashboards -f /tmp/dashboards.zip
|
docker exec petrolyst-superset-1 superset export-dashboards -f /tmp/dashboards.zip
|
||||||
docker cp fuel-monitoring-superset-1:/tmp/dashboards.zip .
|
docker cp petrolyst-superset-1:/tmp/dashboards.zip .
|
||||||
|
|
||||||
Import is the reverse with `superset import-dashboards -p <zip> -u <username>`. The
|
Import is the reverse with `superset import-dashboards -p <zip> -u <username>`. The
|
||||||
username must be an existing Superset user, which here means someone who has signed in
|
username must be an existing Superset user, which here means someone who has signed in
|
||||||
|
|||||||
@@ -15,8 +15,9 @@ them in Superset with the filter column.
|
|||||||
- Nothing is imported into the live Superset without the owner's go-ahead (phase P3).
|
- Nothing is imported into the live Superset without the owner's go-ahead (phase P3).
|
||||||
|
|
||||||
**Datasets.** The `public.v_*` views of
|
**Datasets.** The `public.v_*` views of
|
||||||
`stacks/fuel-monitoring/fuel-db/migrations/0003_bi_views.sql`, read through Trino as
|
`stacks/fuel-monitoring/fuel-db/migrations/0003_bi_views.sql`, read from fuel-db directly (database
|
||||||
`fuel.public`.
|
`fuel`, schema `public`) as the read-only role `fuel_ro`. *Until 29 Oct 2026 Superset read them
|
||||||
|
through Trino as `fuel.public`; Trino was removed then.*
|
||||||
|
|
||||||
**Design and rationale** live in the fuel-monitoring corpus, in
|
**Design and rationale** live in the fuel-monitoring corpus, in
|
||||||
`.context/plans/2026-10-24-fuel-db-baseline-dashboard.md`:
|
`.context/plans/2026-10-24-fuel-db-baseline-dashboard.md`:
|
||||||
@@ -223,7 +224,7 @@ fleet map.
|
|||||||
|
|
||||||
## 5. Datasets and filter mapping
|
## 5. Datasets and filter mapping
|
||||||
|
|
||||||
Physical datasets, one per `public.v_*` view on Trino `fuel.public`, plus one virtual dataset
|
Physical datasets, one per `public.v_*` view in fuel-db's `public` schema, plus one virtual dataset
|
||||||
*(P2)*. `coverage_detail` adds two columns to `v_coverage`:
|
*(P2)*. `coverage_detail` adds two columns to `v_coverage`:
|
||||||
- `lane_name`, the lane without its sort number, which read as a count;
|
- `lane_name`, the lane without its sort number, which read as a count;
|
||||||
- `amount`, a stock interval's delivered litres (from its opening visit up to its closing visit,
|
- `amount`, a stock interval's delivered litres (from its opening visit up to its closing visit,
|
||||||
@@ -280,8 +281,8 @@ output:
|
|||||||
- fixed native-filter ids (`NATIVE_FILTER-site`, `NATIVE_FILTER-period`) and tab ids
|
- fixed native-filter ids (`NATIVE_FILTER-site`, `NATIVE_FILTER-period`) and tab ids
|
||||||
(`TAB-fleet`, `TAB-site`, `TAB-summary`, …);
|
(`TAB-fleet`, `TAB-site`, `TAB-summary`, …);
|
||||||
- `label_colors` for the four statuses;
|
- `label_colors` for the four statuses;
|
||||||
- every view column, from `columns.tsv` (a snapshot of Trino's `information_schema`);
|
- every view column, from `columns.tsv` (a snapshot of PostgreSQL's catalog);
|
||||||
- the existing "Fuel (Trino)" database UUID, which differs per instance:
|
- the existing "Fuel" database UUID, which differs per instance:
|
||||||
`import_bundle.py --show-database` prints it.
|
`import_bundle.py --show-database` prints it.
|
||||||
- Before writing, it checks that every column a chart or filter uses exists in its dataset,
|
- Before writing, it checks that every column a chart or filter uses exists in its dataset,
|
||||||
that every chart in a Site filter's scope has `site_label`, that the fleet map's iframe
|
that every chart in a Site filter's scope has `site_label`, that the fleet map's iframe
|
||||||
|
|||||||
@@ -5,7 +5,7 @@ Implements SPEC.md in this folder: the two dashboards the Petrolyst app embeds,
|
|||||||
(atoma-overview) and "ATOMA · Site" (atoma-site, the four detail tabs), one dataset per
|
(atoma-overview) and "ATOMA · Site" (atoma-site, the four detail tabs), one dataset per
|
||||||
public.v_* view and every chart. Nothing in the bundle names a site.
|
public.v_* view and every chart. Nothing in the bundle names a site.
|
||||||
|
|
||||||
build_bundle.py --database-uuid <uuid of "Fuel (Trino)"> [--out atoma-dashboards.zip]
|
build_bundle.py --database-uuid <uuid of "Fuel"> [--out atoma-dashboards.zip]
|
||||||
|
|
||||||
The database uuid differs per Superset instance; `import_bundle.py --show-database`
|
The database uuid differs per Superset instance; `import_bundle.py --show-database`
|
||||||
prints it. Output is deterministic: the same inputs give a byte-identical zip,
|
prints it. Output is deterministic: the same inputs give a byte-identical zip,
|
||||||
@@ -37,8 +37,10 @@ import yaml
|
|||||||
HERE = Path(__file__).resolve().parent
|
HERE = Path(__file__).resolve().parent
|
||||||
VERSION = "1.0.0"
|
VERSION = "1.0.0"
|
||||||
NAMESPACE = uuid.uuid5(uuid.NAMESPACE_URL, "https://petrolyst.com/superset/atoma-dashboards")
|
NAMESPACE = uuid.uuid5(uuid.NAMESPACE_URL, "https://petrolyst.com/superset/atoma-dashboards")
|
||||||
DATABASE_NAME = "Fuel (Trino)"
|
DATABASE_NAME = "Fuel"
|
||||||
DATABASE_URI = "trino://superset@trino:8080/fuel" # as superset-init registers it
|
# As superset-init registers it: fuel-db directly, as the read-only role fuel_ro. The password
|
||||||
|
# comes from a secret file at connect time (superset_config.py), never from the bundle.
|
||||||
|
DATABASE_URI = "postgresql+psycopg2://fuel_ro@fuel-db:5432/fuel"
|
||||||
CATALOG, SCHEMA = "fuel", "public"
|
CATALOG, SCHEMA = "fuel", "public"
|
||||||
|
|
||||||
SITE_FILTER = "NATIVE_FILTER-site"
|
SITE_FILTER = "NATIVE_FILTER-site"
|
||||||
@@ -69,7 +71,7 @@ MAIN_DTTM = {
|
|||||||
"v_electricity_cost": "bill_month",
|
"v_electricity_cost": "bill_month",
|
||||||
}
|
}
|
||||||
|
|
||||||
# Calculated columns: (name, Trino SQL, type, label). The Petrolyst server's site_status
|
# Calculated columns: (name, PostgreSQL SQL, type, label). The Petrolyst server's site_status
|
||||||
# (petrolyst-platform server/) shows the same two texts.
|
# (petrolyst-platform server/) shows the same two texts.
|
||||||
CALCULATED = {
|
CALCULATED = {
|
||||||
"v_site": [
|
"v_site": [
|
||||||
@@ -93,7 +95,7 @@ CALCULATED = {
|
|||||||
],
|
],
|
||||||
}
|
}
|
||||||
|
|
||||||
# Virtual datasets: name -> (view it extends, extra output columns, Trino SQL).
|
# Virtual datasets: name -> (view it extends, extra output columns, PostgreSQL SQL).
|
||||||
# coverage_detail feeds S3. The view's lane names start with a sort number ("1 Stock
|
# coverage_detail feeds S3. The view's lane names start with a sort number ("1 Stock
|
||||||
# intervals"), which reads as a count on the chart, so it adds lane_name without the number
|
# intervals"), which reads as a count on the chart, so it adds lane_name without the number
|
||||||
# (the chart still sorts by lane). Its amount column is the tooltip: the litres delivered in a
|
# (the chart still sorts by lane). Its amount column is the tooltip: the litres delivered in a
|
||||||
@@ -106,7 +108,7 @@ CALCULATED = {
|
|||||||
# day cannot be told apart. Everywhere else the date window is exact: on a single-visit day the
|
# day cannot be told apart. Everywhere else the date window is exact: on a single-visit day the
|
||||||
# opening visit's delivery is included and the closing visit's excluded.
|
# opening visit's delivery is included and the closing visit's excluded.
|
||||||
VIRTUAL = {
|
VIRTUAL = {
|
||||||
"coverage_detail": ("v_coverage", [("lane_name", "varchar(60)"), ("amount", "varchar(120)")], """\
|
"coverage_detail": ("v_coverage", [("lane_name", "character varying(60)"), ("amount", "character varying(120)")], """\
|
||||||
SELECT c.site_uid, c.site_code, c.site_name, c.site_label, c.lane, c.status,
|
SELECT c.site_uid, c.site_code, c.site_name, c.site_label, c.lane, c.status,
|
||||||
c.start_on, c.end_on, c.start_ts, c.end_ts,
|
c.start_on, c.end_on, c.start_ts, c.end_ts,
|
||||||
regexp_replace(c.lane, '^[0-9]+ ', '') AS lane_name,
|
regexp_replace(c.lane, '^[0-9]+ ', '') AS lane_name,
|
||||||
@@ -114,19 +116,19 @@ SELECT c.site_uid, c.site_code, c.site_name, c.site_label, c.lane, c.status,
|
|||||||
CASE WHEN f.ambiguous THEN 'not attributed: a delivery on a day with several visits (see Deliveries)'
|
CASE WHEN f.ambiguous THEN 'not attributed: a delivery on a day with several visits (see Deliveries)'
|
||||||
WHEN f.filled_l IS NULL AND coalesce(f.n_unknown, 0) = 0 THEN 'no delivery'
|
WHEN f.filled_l IS NULL AND coalesce(f.n_unknown, 0) = 0 THEN 'no delivery'
|
||||||
WHEN f.filled_l IS NULL THEN 'filled, quantity not recorded'
|
WHEN f.filled_l IS NULL THEN 'filled, quantity not recorded'
|
||||||
ELSE format('%,.0f L filled', CAST(f.filled_l AS double)) END
|
ELSE to_char(round(f.filled_l), 'FM999,999,999,990') || ' L filled' END
|
||||||
|| CASE WHEN f.n_unknown > 0 AND f.filled_l IS NOT NULL AND NOT f.ambiguous
|
|| CASE WHEN f.n_unknown > 0 AND f.filled_l IS NOT NULL AND NOT f.ambiguous
|
||||||
THEN ', plus ' || CAST(f.n_unknown AS varchar) || ' of unknown quantity' ELSE '' END
|
THEN ', plus ' || CAST(f.n_unknown AS varchar) || ' of unknown quantity' ELSE '' END
|
||||||
WHEN c.lane = '4 Electricity bills' THEN
|
WHEN c.lane = '4 Electricity bills' THEN
|
||||||
CASE WHEN b.n_bills > 1 THEN 'several bills start this day (see Electricity bills)'
|
CASE WHEN b.n_bills > 1 THEN 'several bills start this day (see Electricity bills)'
|
||||||
WHEN b.disputed THEN 'bill disputed, amount withheld'
|
WHEN b.disputed THEN 'bill disputed, amount withheld'
|
||||||
WHEN b.amount IS NULL THEN 'bill amount not recorded'
|
WHEN b.amount IS NULL THEN 'bill amount not recorded'
|
||||||
ELSE format('%,.0f AFN billed (inferred)', CAST(b.amount AS double)) END
|
ELSE to_char(round(b.amount), 'FM999,999,999,990') || ' AFN billed (inferred)' END
|
||||||
END AS amount
|
END AS amount
|
||||||
FROM public.v_coverage c
|
FROM public.v_coverage c
|
||||||
LEFT JOIN (SELECT s.site_uid, s.start_on, s.end_on,
|
LEFT JOIN (SELECT s.site_uid, s.start_on, s.end_on,
|
||||||
sum(CASE WHEN d.delivery_date < s.end_on THEN d.delivered_l END) AS filled_l,
|
sum(CASE WHEN d.delivery_date < s.end_on THEN d.delivered_l END) AS filled_l,
|
||||||
count_if(d.delivery_date < s.end_on AND d.delivered_l IS NULL) AS n_unknown,
|
count(*) FILTER (WHERE d.delivery_date < s.end_on AND d.delivered_l IS NULL) AS n_unknown,
|
||||||
bool_or(d.delivery_date IN (s.start_on, s.end_on) AND m.site_uid IS NOT NULL) AS ambiguous
|
bool_or(d.delivery_date IN (s.start_on, s.end_on) AND m.site_uid IS NOT NULL) AS ambiguous
|
||||||
FROM (SELECT DISTINCT site_uid, start_on, end_on FROM public.v_coverage
|
FROM (SELECT DISTINCT site_uid, start_on, end_on FROM public.v_coverage
|
||||||
WHERE lane = '1 Stock intervals') s
|
WHERE lane = '1 Stock intervals') s
|
||||||
@@ -216,8 +218,8 @@ def read_columns(path: Path) -> dict[str, list[tuple[str, str]]]:
|
|||||||
return views
|
return views
|
||||||
|
|
||||||
|
|
||||||
def superset_type(trino_type: str) -> str:
|
def superset_type(data_type: str) -> str:
|
||||||
return trino_type.upper().replace(",", ", ")
|
return data_type.upper().replace(",", ", ")
|
||||||
|
|
||||||
|
|
||||||
def dataset_yaml(view: str, columns: list[tuple[str, str]], database_uuid: str) -> dict[str, Any]:
|
def dataset_yaml(view: str, columns: list[tuple[str, str]], database_uuid: str) -> dict[str, Any]:
|
||||||
@@ -968,7 +970,7 @@ def build(database_uuid: str, columns_path: Path) -> dict[str, Any]:
|
|||||||
views = read_columns(columns_path)
|
views = read_columns(columns_path)
|
||||||
files: dict[str, Any] = {
|
files: dict[str, Any] = {
|
||||||
"metadata.yaml": {"version": VERSION, "type": "Dashboard", "timestamp": "2026-10-24T00:00:00+00:00"},
|
"metadata.yaml": {"version": VERSION, "type": "Dashboard", "timestamp": "2026-10-24T00:00:00+00:00"},
|
||||||
"databases/Fuel_Trino.yaml": {
|
"databases/Fuel.yaml": {
|
||||||
"database_name": DATABASE_NAME,
|
"database_name": DATABASE_NAME,
|
||||||
"sqlalchemy_uri": DATABASE_URI,
|
"sqlalchemy_uri": DATABASE_URI,
|
||||||
"cache_timeout": None,
|
"cache_timeout": None,
|
||||||
@@ -988,7 +990,7 @@ def build(database_uuid: str, columns_path: Path) -> dict[str, Any]:
|
|||||||
for name, (base, extra, _) in VIRTUAL.items():
|
for name, (base, extra, _) in VIRTUAL.items():
|
||||||
views[name] = views[base] + extra
|
views[name] = views[base] + extra
|
||||||
for view in sorted(views):
|
for view in sorted(views):
|
||||||
files[f"datasets/Fuel_Trino/{view}.yaml"] = dataset_yaml(view, views[view], database_uuid)
|
files[f"datasets/Fuel/{view}.yaml"] = dataset_yaml(view, views[view], database_uuid)
|
||||||
|
|
||||||
# PD-12: the Petrolyst app embeds the Overview and one site dashboard, whose four detail
|
# PD-12: the Petrolyst app embeds the Overview and one site dashboard, whose four detail
|
||||||
# tabs are Superset's own (owner, 29 Oct). The app shows the site header and chooses site
|
# tabs are Superset's own (owner, 29 Oct). The app shows the site header and chooses site
|
||||||
@@ -1083,7 +1085,7 @@ def write_zip(files: dict[str, Any], out: Path) -> None:
|
|||||||
def main(argv: list[str] | None = None) -> int:
|
def main(argv: list[str] | None = None) -> int:
|
||||||
parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
|
parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
|
||||||
parser.add_argument("--database-uuid", required=True, type=uuid.UUID,
|
parser.add_argument("--database-uuid", required=True, type=uuid.UUID,
|
||||||
help='uuid of the "Fuel (Trino)" database in the target Superset')
|
help='uuid of the "Fuel" database in the target Superset')
|
||||||
parser.add_argument("--columns", type=Path, default=HERE / "columns.tsv")
|
parser.add_argument("--columns", type=Path, default=HERE / "columns.tsv")
|
||||||
parser.add_argument("--out", type=Path, default=Path("atoma-dashboards.zip"))
|
parser.add_argument("--out", type=Path, default=Path("atoma-dashboards.zip"))
|
||||||
args = parser.parse_args(argv)
|
args = parser.parse_args(argv)
|
||||||
|
|||||||
@@ -1,309 +1,310 @@
|
|||||||
# Columns of the public.v_* views (migrations 0001-0003), as Trino reports them.
|
# Columns of the public.v_* views (migrations 0001-0003), as PostgreSQL reports them.
|
||||||
# build_bundle.py lists every one in its dataset; import_bundle.py reports drift.
|
# build_bundle.py lists every one in its dataset; import_bundle.py reports drift.
|
||||||
# Regenerate after a migration that changes a view:
|
# Regenerate after a migration that changes a view (psql -d fuel -AtF $'\t' -c "..."):
|
||||||
# trino --output-format TSV --execute "select table_name, column_name, data_type
|
# select c.relname, a.attname, format_type(a.atttypid, a.atttypmod)
|
||||||
# from fuel.information_schema.columns where table_schema = 'public'
|
# from pg_attribute a join pg_class c on c.oid = a.attrelid join pg_namespace n on n.oid = c.relnamespace
|
||||||
# and table_name like 'v\_%' escape '\' order by table_name, ordinal_position"
|
# where n.nspname = 'public' and c.relkind = 'v' and c.relname like 'v\_%' and a.attnum > 0
|
||||||
v_assumption site_uid varchar(12)
|
# and not a.attisdropped order by c.relname, a.attnum
|
||||||
v_assumption site_code varchar(20)
|
v_assumption site_uid character varying(12)
|
||||||
v_assumption site_name varchar(120)
|
v_assumption site_code character varying(20)
|
||||||
v_assumption site_label varchar(170)
|
v_assumption site_name character varying(120)
|
||||||
v_assumption code varchar(60)
|
v_assumption site_label character varying(170)
|
||||||
v_assumption topic varchar(30)
|
v_assumption code character varying(60)
|
||||||
v_assumption asset_uid varchar(40)
|
v_assumption topic character varying(30)
|
||||||
v_assumption value decimal(10,3)
|
v_assumption asset_uid character varying(40)
|
||||||
v_assumption value_low decimal(10,3)
|
v_assumption value numeric(10,3)
|
||||||
v_assumption value_high decimal(10,3)
|
v_assumption value_low numeric(10,3)
|
||||||
v_assumption unit varchar(20)
|
v_assumption value_high numeric(10,3)
|
||||||
|
v_assumption unit character varying(20)
|
||||||
v_assumption applies_from date
|
v_assumption applies_from date
|
||||||
v_assumption applies_to date
|
v_assumption applies_to date
|
||||||
v_assumption basis varchar(20)
|
v_assumption basis character varying(20)
|
||||||
v_assumption derived_from_fuel boolean
|
v_assumption derived_from_fuel boolean
|
||||||
v_assumption status varchar(12)
|
v_assumption status character varying(12)
|
||||||
v_assumption accepted_on date
|
v_assumption accepted_on date
|
||||||
v_assumption decision_ref varchar(12)
|
v_assumption decision_ref character varying(12)
|
||||||
v_assumption rationale varchar(400)
|
v_assumption rationale character varying(400)
|
||||||
v_assumption contradicted_by varchar(200)
|
v_assumption contradicted_by character varying(200)
|
||||||
v_assumption members varchar(400)
|
v_assumption members character varying(400)
|
||||||
v_baseline_input site_uid varchar(12)
|
v_baseline_input site_uid character varying(12)
|
||||||
v_baseline_input site_code varchar(20)
|
v_baseline_input site_code character varying(20)
|
||||||
v_baseline_input site_name varchar(120)
|
v_baseline_input site_name character varying(120)
|
||||||
v_baseline_input site_label varchar(170)
|
v_baseline_input site_label character varying(170)
|
||||||
v_baseline_input baseline_code varchar(60)
|
v_baseline_input baseline_code character varying(60)
|
||||||
v_baseline_input baseline_status varchar(12)
|
v_baseline_input baseline_status character varying(12)
|
||||||
v_baseline_input interval_label varchar(30)
|
v_baseline_input interval_label character varying(30)
|
||||||
v_baseline_input start_on date
|
v_baseline_input start_on date
|
||||||
v_baseline_input end_on date
|
v_baseline_input end_on date
|
||||||
v_baseline_input included boolean
|
v_baseline_input included boolean
|
||||||
v_baseline_input exclusion_reason varchar(300)
|
v_baseline_input exclusion_reason character varying(300)
|
||||||
v_baseline_input fuel_l decimal(10,1)
|
v_baseline_input fuel_l numeric(10,1)
|
||||||
v_baseline_input runtime_h decimal(10,1)
|
v_baseline_input runtime_h numeric(10,1)
|
||||||
v_baseline_input days integer
|
v_baseline_input days integer
|
||||||
v_baseline_input eligibility_at_computation varchar(12)
|
v_baseline_input eligibility_at_computation character varying(12)
|
||||||
v_baseline_input evidence_grade_at_computation varchar(2)
|
v_baseline_input evidence_grade_at_computation character varying(2)
|
||||||
v_baseline_input dependencies_changed boolean
|
v_baseline_input dependencies_changed boolean
|
||||||
v_baseline_version site_uid varchar(12)
|
v_baseline_version site_uid character varying(12)
|
||||||
v_baseline_version site_code varchar(20)
|
v_baseline_version site_code character varying(20)
|
||||||
v_baseline_version site_name varchar(120)
|
v_baseline_version site_name character varying(120)
|
||||||
v_baseline_version site_label varchar(170)
|
v_baseline_version site_label character varying(170)
|
||||||
v_baseline_version code varchar(60)
|
v_baseline_version code character varying(60)
|
||||||
v_baseline_version metric varchar(40)
|
v_baseline_version metric character varying(40)
|
||||||
v_baseline_version status varchar(12)
|
v_baseline_version status character varying(12)
|
||||||
v_baseline_version population varchar(200)
|
v_baseline_version population character varying(200)
|
||||||
v_baseline_version value decimal(8,3)
|
v_baseline_version value numeric(8,3)
|
||||||
v_baseline_version range_low decimal(8,3)
|
v_baseline_version range_low numeric(8,3)
|
||||||
v_baseline_version range_high decimal(8,3)
|
v_baseline_version range_high numeric(8,3)
|
||||||
v_baseline_version unit varchar(8)
|
v_baseline_version unit character varying(8)
|
||||||
v_baseline_version window_start date
|
v_baseline_version window_start date
|
||||||
v_baseline_version window_end date
|
v_baseline_version window_end date
|
||||||
v_baseline_version fuel_l decimal(10,1)
|
v_baseline_version fuel_l numeric(10,1)
|
||||||
v_baseline_version runtime_h decimal(10,1)
|
v_baseline_version runtime_h numeric(10,1)
|
||||||
v_baseline_version matched_days integer
|
v_baseline_version matched_days integer
|
||||||
v_baseline_version policy_version varchar(10)
|
v_baseline_version policy_version character varying(10)
|
||||||
v_baseline_version meets_policy boolean
|
v_baseline_version meets_policy boolean
|
||||||
v_baseline_version policy_check varchar(300)
|
v_baseline_version policy_check character varying(300)
|
||||||
v_baseline_version evidence_grade varchar(2)
|
v_baseline_version evidence_grade character varying(2)
|
||||||
v_baseline_version conditions varchar(200)
|
v_baseline_version conditions character varying(200)
|
||||||
v_baseline_version season_note varchar(200)
|
v_baseline_version season_note character varying(200)
|
||||||
v_baseline_version approved_on date
|
v_baseline_version approved_on date
|
||||||
v_baseline_version approved_by_role varchar(40)
|
v_baseline_version approved_by_role character varying(40)
|
||||||
v_baseline_version decision_ref varchar(12)
|
v_baseline_version decision_ref character varying(12)
|
||||||
v_baseline_version inputs_unchanged boolean
|
v_baseline_version inputs_unchanged boolean
|
||||||
v_baseline_version computed_at timestamp(0)
|
v_baseline_version computed_at timestamp(0) without time zone
|
||||||
v_coverage site_uid varchar(12)
|
v_coverage site_uid character varying(12)
|
||||||
v_coverage site_code varchar(20)
|
v_coverage site_code character varying(20)
|
||||||
v_coverage site_name varchar(120)
|
v_coverage site_name character varying(120)
|
||||||
v_coverage site_label varchar(170)
|
v_coverage site_label character varying(170)
|
||||||
v_coverage lane varchar(60)
|
v_coverage lane character varying(60)
|
||||||
v_coverage status varchar(40)
|
v_coverage status character varying(40)
|
||||||
v_coverage start_on date
|
v_coverage start_on date
|
||||||
v_coverage end_on date
|
v_coverage end_on date
|
||||||
v_coverage start_ts timestamp(0)
|
v_coverage start_ts timestamp(0) without time zone
|
||||||
v_coverage end_ts timestamp(0)
|
v_coverage end_ts timestamp(0) without time zone
|
||||||
v_data_quality site_uid varchar(12)
|
v_data_quality site_uid character varying(12)
|
||||||
v_data_quality site_code varchar(20)
|
v_data_quality site_code character varying(20)
|
||||||
v_data_quality site_name varchar(120)
|
v_data_quality site_name character varying(120)
|
||||||
v_data_quality site_label varchar(170)
|
v_data_quality site_label character varying(170)
|
||||||
v_data_quality check_order integer
|
v_data_quality check_order integer
|
||||||
v_data_quality check_code varchar(40)
|
v_data_quality check_code character varying(40)
|
||||||
v_data_quality label varchar(80)
|
v_data_quality label character varying(80)
|
||||||
v_data_quality status varchar(10)
|
v_data_quality status character varying(10)
|
||||||
v_data_quality detail varchar(300)
|
v_data_quality detail character varying(300)
|
||||||
v_delivery site_uid varchar(12)
|
v_delivery site_uid character varying(12)
|
||||||
v_delivery site_code varchar(20)
|
v_delivery site_code character varying(20)
|
||||||
v_delivery site_name varchar(120)
|
v_delivery site_name character varying(120)
|
||||||
v_delivery site_label varchar(170)
|
v_delivery site_label character varying(170)
|
||||||
v_delivery event_key varchar(80)
|
v_delivery event_key character varying(80)
|
||||||
v_delivery delivery_date date
|
v_delivery delivery_date date
|
||||||
v_delivery seq_in_day integer
|
v_delivery seq_in_day integer
|
||||||
v_delivery delivered_l decimal(8,1)
|
v_delivery delivered_l numeric(8,1)
|
||||||
v_delivery unit_price decimal(8,2)
|
v_delivery unit_price numeric(8,2)
|
||||||
v_delivery currency varchar(8)
|
v_delivery currency character varying(8)
|
||||||
v_delivery source_ref varchar(60)
|
v_delivery source_ref character varying(60)
|
||||||
v_delivery n_sources integer
|
v_delivery n_sources integer
|
||||||
v_delivery sources_disagree boolean
|
v_delivery sources_disagree boolean
|
||||||
v_delivery reviewed boolean
|
v_delivery reviewed boolean
|
||||||
v_electricity_cost site_uid varchar(12)
|
v_electricity_cost site_uid character varying(12)
|
||||||
v_electricity_cost site_code varchar(20)
|
v_electricity_cost site_code character varying(20)
|
||||||
v_electricity_cost site_name varchar(120)
|
v_electricity_cost site_name character varying(120)
|
||||||
v_electricity_cost site_label varchar(170)
|
v_electricity_cost site_label character varying(170)
|
||||||
v_electricity_cost bill_month date
|
v_electricity_cost bill_month date
|
||||||
v_electricity_cost period_start date
|
v_electricity_cost period_start date
|
||||||
v_electricity_cost period_end date
|
v_electricity_cost period_end date
|
||||||
v_electricity_cost period_known boolean
|
v_electricity_cost period_known boolean
|
||||||
v_electricity_cost amount decimal(12,2)
|
v_electricity_cost amount numeric(12,2)
|
||||||
v_electricity_cost currency varchar(8)
|
v_electricity_cost currency character varying(8)
|
||||||
v_electricity_cost currency_basis varchar(10)
|
v_electricity_cost currency_basis character varying(10)
|
||||||
v_electricity_cost kwh decimal(10,1)
|
v_electricity_cost kwh numeric(10,1)
|
||||||
v_electricity_cost source_ref varchar(60)
|
v_electricity_cost source_ref character varying(60)
|
||||||
v_electricity_cost claim_status varchar(12)
|
v_electricity_cost claim_status character varying(12)
|
||||||
v_evidence site_uid varchar(12)
|
v_evidence site_uid character varying(12)
|
||||||
v_evidence site_code varchar(20)
|
v_evidence site_code character varying(20)
|
||||||
v_evidence site_name varchar(120)
|
v_evidence site_name character varying(120)
|
||||||
v_evidence site_label varchar(170)
|
v_evidence site_label character varying(170)
|
||||||
v_evidence document_key varchar(16)
|
v_evidence document_key character varying(16)
|
||||||
v_evidence title varchar(200)
|
v_evidence title character varying(200)
|
||||||
v_evidence kind varchar(20)
|
v_evidence kind character varying(20)
|
||||||
v_evidence issuer varchar(80)
|
v_evidence issuer character varying(80)
|
||||||
v_evidence document_date date
|
v_evidence document_date date
|
||||||
v_evidence used_for varchar(200)
|
v_evidence used_for character varying(200)
|
||||||
v_evidence contains_personal_data boolean
|
v_evidence contains_personal_data boolean
|
||||||
v_hour_meter_evidence site_uid varchar(12)
|
v_hour_meter_evidence site_uid character varying(12)
|
||||||
v_hour_meter_evidence site_code varchar(20)
|
v_hour_meter_evidence site_code character varying(20)
|
||||||
v_hour_meter_evidence site_name varchar(120)
|
v_hour_meter_evidence site_name character varying(120)
|
||||||
v_hour_meter_evidence site_label varchar(170)
|
v_hour_meter_evidence site_label character varying(170)
|
||||||
v_hour_meter_evidence meter_label varchar(40)
|
v_hour_meter_evidence meter_label character varying(40)
|
||||||
v_hour_meter_evidence mapped_set varchar(60)
|
v_hour_meter_evidence mapped_set character varying(60)
|
||||||
v_hour_meter_evidence reading_date date
|
v_hour_meter_evidence reading_date date
|
||||||
v_hour_meter_evidence date_basis varchar(30)
|
v_hour_meter_evidence date_basis character varying(30)
|
||||||
v_hour_meter_evidence hours decimal(9,1)
|
v_hour_meter_evidence hours numeric(9,1)
|
||||||
v_hour_meter_evidence reading_state varchar(16)
|
v_hour_meter_evidence reading_state character varying(16)
|
||||||
v_hour_meter_evidence source_field varchar(16)
|
v_hour_meter_evidence source_field character varying(16)
|
||||||
v_hour_meter_evidence epoch_no integer
|
v_hour_meter_evidence epoch_no integer
|
||||||
v_hour_meter_evidence claim_status varchar(12)
|
v_hour_meter_evidence claim_status character varying(12)
|
||||||
v_imported_calculation site_uid varchar(12)
|
v_imported_calculation site_uid character varying(12)
|
||||||
v_imported_calculation site_code varchar(20)
|
v_imported_calculation site_code character varying(20)
|
||||||
v_imported_calculation site_name varchar(120)
|
v_imported_calculation site_name character varying(120)
|
||||||
v_imported_calculation site_label varchar(170)
|
v_imported_calculation site_label character varying(170)
|
||||||
v_imported_calculation cell_ref varchar(40)
|
v_imported_calculation cell_ref character varying(40)
|
||||||
v_imported_calculation source_label varchar(80)
|
v_imported_calculation source_label character varying(80)
|
||||||
v_imported_calculation formula varchar(200)
|
v_imported_calculation formula character varying(200)
|
||||||
v_imported_calculation cached_value decimal(14,6)
|
v_imported_calculation cached_value numeric(14,6)
|
||||||
v_imported_calculation unit_as_labelled varchar(30)
|
v_imported_calculation unit_as_labelled character varying(30)
|
||||||
v_imported_calculation our_reading varchar(300)
|
v_imported_calculation our_reading character varying(300)
|
||||||
v_imported_calculation baseline_use varchar(12)
|
v_imported_calculation baseline_use character varying(12)
|
||||||
v_imported_calculation exclusion_reason varchar(300)
|
v_imported_calculation exclusion_reason character varying(300)
|
||||||
v_imported_calculation document_key varchar(16)
|
v_imported_calculation document_key character varying(16)
|
||||||
v_imported_calculation claim_status varchar(12)
|
v_imported_calculation claim_status character varying(12)
|
||||||
v_interval site_uid varchar(12)
|
v_interval site_uid character varying(12)
|
||||||
v_interval site_code varchar(20)
|
v_interval site_code character varying(20)
|
||||||
v_interval site_name varchar(120)
|
v_interval site_name character varying(120)
|
||||||
v_interval site_label varchar(170)
|
v_interval site_label character varying(170)
|
||||||
v_interval interval_label varchar(30)
|
v_interval interval_label character varying(30)
|
||||||
v_interval start_on date
|
v_interval start_on date
|
||||||
v_interval end_on date
|
v_interval end_on date
|
||||||
v_interval days integer
|
v_interval days integer
|
||||||
v_interval opening_l decimal(9,1)
|
v_interval opening_l numeric(9,1)
|
||||||
v_interval opening_basis varchar(8)
|
v_interval opening_basis character varying(8)
|
||||||
v_interval deliveries_l decimal(9,1)
|
v_interval deliveries_l numeric(9,1)
|
||||||
v_interval withdrawals_l decimal(9,1)
|
v_interval withdrawals_l numeric(9,1)
|
||||||
v_interval closing_l decimal(9,1)
|
v_interval closing_l numeric(9,1)
|
||||||
v_interval closing_basis varchar(8)
|
v_interval closing_basis character varying(8)
|
||||||
v_interval fuel_used_l decimal(9,1)
|
v_interval fuel_used_l numeric(9,1)
|
||||||
v_interval fuel_used_unc_l decimal(9,1)
|
v_interval fuel_used_unc_l numeric(9,1)
|
||||||
v_interval fuel_l_per_day decimal(8,1)
|
v_interval fuel_l_per_day numeric(8,1)
|
||||||
v_interval stock_status varchar(40)
|
v_interval stock_status character varying(40)
|
||||||
v_interval runtime_h decimal(9,1)
|
v_interval runtime_h numeric(9,1)
|
||||||
v_interval runtime_h_per_day decimal(6,2)
|
v_interval runtime_h_per_day numeric(6,2)
|
||||||
v_interval n_meters integer
|
v_interval n_meters integer
|
||||||
v_interval metric varchar(40)
|
v_interval metric character varying(40)
|
||||||
v_interval population_key varchar(200)
|
v_interval population_key character varying(200)
|
||||||
v_interval l_per_h decimal(8,3)
|
v_interval l_per_h numeric(8,3)
|
||||||
v_interval l_per_h_low decimal(8,3)
|
v_interval l_per_h_low numeric(8,3)
|
||||||
v_interval l_per_h_high decimal(8,3)
|
v_interval l_per_h_high numeric(8,3)
|
||||||
v_interval eligibility varchar(12)
|
v_interval eligibility character varying(12)
|
||||||
v_interval exclusion_reasons varchar(600)
|
v_interval exclusion_reasons character varying(600)
|
||||||
v_interval evidence_grade varchar(2)
|
v_interval evidence_grade character varying(2)
|
||||||
v_interval conditions varchar(200)
|
v_interval conditions character varying(200)
|
||||||
v_interval in_current_baseline boolean
|
v_interval in_current_baseline boolean
|
||||||
v_interval baseline_comparable boolean
|
v_interval baseline_comparable boolean
|
||||||
v_interval current_baseline_l_per_h decimal(8,3)
|
v_interval current_baseline_l_per_h numeric(8,3)
|
||||||
v_kpi_measures site_uid varchar(12)
|
v_kpi_measures site_uid character varying(12)
|
||||||
v_kpi_measures site_code varchar(20)
|
v_kpi_measures site_code character varying(20)
|
||||||
v_kpi_measures site_name varchar(120)
|
v_kpi_measures site_name character varying(120)
|
||||||
v_kpi_measures site_label varchar(170)
|
v_kpi_measures site_label character varying(170)
|
||||||
v_kpi_measures measure_order integer
|
v_kpi_measures measure_order integer
|
||||||
v_kpi_measures measure varchar(60)
|
v_kpi_measures measure character varying(60)
|
||||||
v_kpi_measures value decimal(12,2)
|
v_kpi_measures value numeric(12,2)
|
||||||
v_kpi_measures unit varchar(12)
|
v_kpi_measures unit character varying(12)
|
||||||
v_kpi_measures value_kind varchar(16)
|
v_kpi_measures value_kind character varying(16)
|
||||||
v_kpi_measures window_start date
|
v_kpi_measures window_start date
|
||||||
v_kpi_measures window_end date
|
v_kpi_measures window_end date
|
||||||
v_kpi_measures note varchar(200)
|
v_kpi_measures note character varying(200)
|
||||||
v_modelled_lph site_uid varchar(12)
|
v_modelled_lph site_uid character varying(12)
|
||||||
v_modelled_lph site_code varchar(20)
|
v_modelled_lph site_code character varying(20)
|
||||||
v_modelled_lph site_name varchar(120)
|
v_modelled_lph site_name character varying(120)
|
||||||
v_modelled_lph site_label varchar(170)
|
v_modelled_lph site_label character varying(170)
|
||||||
v_modelled_lph assumption_code varchar(60)
|
v_modelled_lph assumption_code character varying(60)
|
||||||
v_modelled_lph status varchar(12)
|
v_modelled_lph status character varying(12)
|
||||||
v_modelled_lph basis varchar(20)
|
v_modelled_lph basis character varying(20)
|
||||||
v_modelled_lph h_per_day decimal(5,2)
|
v_modelled_lph h_per_day numeric(5,2)
|
||||||
v_modelled_lph h_per_day_low decimal(5,2)
|
v_modelled_lph h_per_day_low numeric(5,2)
|
||||||
v_modelled_lph h_per_day_high decimal(5,2)
|
v_modelled_lph h_per_day_high numeric(5,2)
|
||||||
v_modelled_lph window_start date
|
v_modelled_lph window_start date
|
||||||
v_modelled_lph window_end date
|
v_modelled_lph window_end date
|
||||||
v_modelled_lph matched_days integer
|
v_modelled_lph matched_days integer
|
||||||
v_modelled_lph l_per_day decimal(7,1)
|
v_modelled_lph l_per_day numeric(7,1)
|
||||||
v_modelled_lph modelled_l_per_h decimal(6,2)
|
v_modelled_lph modelled_l_per_h numeric(6,2)
|
||||||
v_modelled_lph modelled_l_per_h_low decimal(6,2)
|
v_modelled_lph modelled_l_per_h_low numeric(6,2)
|
||||||
v_modelled_lph modelled_l_per_h_high decimal(6,2)
|
v_modelled_lph modelled_l_per_h_high numeric(6,2)
|
||||||
v_modelled_lph value_kind varchar(70)
|
v_modelled_lph value_kind character varying(70)
|
||||||
v_reconciliation site_uid varchar(12)
|
v_reconciliation site_uid character varying(12)
|
||||||
v_reconciliation site_code varchar(20)
|
v_reconciliation site_code character varying(20)
|
||||||
v_reconciliation site_name varchar(120)
|
v_reconciliation site_name character varying(120)
|
||||||
v_reconciliation site_label varchar(170)
|
v_reconciliation site_label character varying(170)
|
||||||
v_reconciliation interval_label varchar(30)
|
v_reconciliation interval_label character varying(30)
|
||||||
v_reconciliation start_on date
|
v_reconciliation start_on date
|
||||||
v_reconciliation end_on date
|
v_reconciliation end_on date
|
||||||
v_reconciliation stock_status varchar(40)
|
v_reconciliation stock_status character varying(40)
|
||||||
v_reconciliation endpoint_times_known boolean
|
v_reconciliation endpoint_times_known boolean
|
||||||
v_reconciliation asset_uid varchar(40)
|
v_reconciliation asset_uid character varying(40)
|
||||||
v_reconciliation placed_throughout boolean
|
v_reconciliation placed_throughout boolean
|
||||||
v_reconciliation stock_balance_l decimal(9,1)
|
v_reconciliation stock_balance_l numeric(9,1)
|
||||||
v_reconciliation engine_l decimal(9,1)
|
v_reconciliation engine_l numeric(9,1)
|
||||||
v_reconciliation covered_h decimal(9,1)
|
v_reconciliation covered_h numeric(9,1)
|
||||||
v_reconciliation interval_h decimal(9,1)
|
v_reconciliation interval_h numeric(9,1)
|
||||||
v_reconciliation engine_all_generators_l decimal(9,1)
|
v_reconciliation engine_all_generators_l numeric(9,1)
|
||||||
v_reconciliation all_generators_metered boolean
|
v_reconciliation all_generators_metered boolean
|
||||||
v_reconciliation difference_l decimal(9,1)
|
v_reconciliation difference_l numeric(9,1)
|
||||||
v_reconciliation difference_pct decimal(7,1)
|
v_reconciliation difference_pct numeric(7,1)
|
||||||
v_runtime site_uid varchar(12)
|
v_runtime site_uid character varying(12)
|
||||||
v_runtime site_code varchar(20)
|
v_runtime site_code character varying(20)
|
||||||
v_runtime site_name varchar(120)
|
v_runtime site_name character varying(120)
|
||||||
v_runtime site_label varchar(170)
|
v_runtime site_label character varying(170)
|
||||||
v_runtime meter_label varchar(40)
|
v_runtime meter_label character varying(40)
|
||||||
v_runtime mapped_set varchar(60)
|
v_runtime mapped_set character varying(60)
|
||||||
v_runtime interval_label varchar(80)
|
v_runtime interval_label character varying(80)
|
||||||
v_runtime start_on date
|
v_runtime start_on date
|
||||||
v_runtime end_on date
|
v_runtime end_on date
|
||||||
v_runtime days integer
|
v_runtime days integer
|
||||||
v_runtime runtime_h decimal(9,1)
|
v_runtime runtime_h numeric(9,1)
|
||||||
v_runtime h_per_day decimal(6,2)
|
v_runtime h_per_day numeric(6,2)
|
||||||
v_runtime pair_valid boolean
|
v_runtime pair_valid boolean
|
||||||
v_runtime same_epoch boolean
|
v_runtime same_epoch boolean
|
||||||
v_runtime monotonic boolean
|
v_runtime monotonic boolean
|
||||||
v_runtime over_24h_per_day boolean
|
v_runtime over_24h_per_day boolean
|
||||||
v_runtime reviewed boolean
|
v_runtime reviewed boolean
|
||||||
v_site site_uid varchar(12)
|
v_site site_uid character varying(12)
|
||||||
v_site site_code varchar(20)
|
v_site site_code character varying(20)
|
||||||
v_site site_name varchar(120)
|
v_site site_name character varying(120)
|
||||||
v_site site_label varchar(170)
|
v_site site_label character varying(170)
|
||||||
v_site zone varchar(40)
|
v_site zone character varying(40)
|
||||||
v_site province varchar(40)
|
v_site province character varying(40)
|
||||||
v_site district varchar(60)
|
v_site district character varying(60)
|
||||||
v_site pilot_status varchar(12)
|
v_site pilot_status character varying(12)
|
||||||
v_site latitude decimal(9,6)
|
v_site latitude numeric(9,6)
|
||||||
v_site longitude decimal(9,6)
|
v_site longitude numeric(9,6)
|
||||||
v_site location_status varchar(20)
|
v_site location_status character varying(20)
|
||||||
v_site location_basis varchar(20)
|
v_site location_basis character varying(20)
|
||||||
v_site location_precision_m integer
|
v_site location_precision_m integer
|
||||||
v_site baseline_status varchar(24)
|
v_site baseline_status character varying(24)
|
||||||
v_site baseline_status_label varchar(40)
|
v_site baseline_status_label character varying(40)
|
||||||
v_site baseline_validity_note varchar(60)
|
v_site baseline_validity_note character varying(60)
|
||||||
v_site lph_estimate decimal(6,2)
|
v_site lph_estimate numeric(6,2)
|
||||||
v_site lph_method varchar(60)
|
v_site lph_method character varying(60)
|
||||||
v_site lph_window_start date
|
v_site lph_window_start date
|
||||||
v_site lph_window_end date
|
v_site lph_window_end date
|
||||||
v_site baseline_conditions varchar(200)
|
v_site baseline_conditions character varying(200)
|
||||||
v_site baseline_decision_ref varchar(12)
|
v_site baseline_decision_ref character varying(12)
|
||||||
v_site fuel_l_per_calendar_day decimal(7,1)
|
v_site fuel_l_per_calendar_day numeric(7,1)
|
||||||
v_site latest_record_on date
|
v_site latest_record_on date
|
||||||
v_spec_reference site_uid varchar(12)
|
v_spec_reference site_uid character varying(12)
|
||||||
v_spec_reference site_code varchar(20)
|
v_spec_reference site_code character varying(20)
|
||||||
v_spec_reference site_name varchar(120)
|
v_spec_reference site_name character varying(120)
|
||||||
v_spec_reference site_label varchar(170)
|
v_spec_reference site_label character varying(170)
|
||||||
v_spec_reference slot varchar(20)
|
v_spec_reference slot character varying(20)
|
||||||
v_spec_reference asset_uid varchar(40)
|
v_spec_reference asset_uid character varying(40)
|
||||||
v_spec_reference asset_kind varchar(10)
|
v_spec_reference asset_kind character varying(10)
|
||||||
v_spec_reference make_model varchar(120)
|
v_spec_reference make_model character varying(120)
|
||||||
v_spec_reference attribute varchar(40)
|
v_spec_reference attribute character varying(40)
|
||||||
v_spec_reference load_pct decimal(5,1)
|
v_spec_reference load_pct numeric(5,1)
|
||||||
v_spec_reference value decimal(12,3)
|
v_spec_reference value numeric(12,3)
|
||||||
v_spec_reference value_text varchar(120)
|
v_spec_reference value_text character varying(120)
|
||||||
v_spec_reference unit varchar(20)
|
v_spec_reference unit character varying(20)
|
||||||
v_spec_reference basis varchar(30)
|
v_spec_reference basis character varying(30)
|
||||||
v_spec_reference applicability varchar(12)
|
v_spec_reference applicability character varying(12)
|
||||||
v_spec_reference applicability_note varchar(300)
|
v_spec_reference applicability_note character varying(300)
|
||||||
v_spec_reference value_kind varchar(30)
|
v_spec_reference value_kind character varying(30)
|
||||||
v_spec_reference claim_status varchar(12)
|
v_spec_reference claim_status character varying(12)
|
||||||
v_stock_reading site_uid varchar(12)
|
v_stock_reading site_uid character varying(12)
|
||||||
v_stock_reading site_code varchar(20)
|
v_stock_reading site_code character varying(20)
|
||||||
v_stock_reading site_name varchar(120)
|
v_stock_reading site_name character varying(120)
|
||||||
v_stock_reading site_label varchar(170)
|
v_stock_reading site_label character varying(170)
|
||||||
v_stock_reading observed_on date
|
v_stock_reading observed_on date
|
||||||
v_stock_reading seq_in_day integer
|
v_stock_reading seq_in_day integer
|
||||||
v_stock_reading reading_type varchar(20)
|
v_stock_reading reading_type character varying(20)
|
||||||
v_stock_reading litres decimal(8,1)
|
v_stock_reading litres numeric(8,1)
|
||||||
v_stock_reading value_kind varchar(12)
|
v_stock_reading value_kind character varying(12)
|
||||||
v_stock_reading spread_l decimal(8,1)
|
v_stock_reading spread_l numeric(8,1)
|
||||||
v_stock_reading reviewed boolean
|
v_stock_reading reviewed boolean
|
||||||
|
Can't render this file because it contains an unexpected character in line 4 and column 41.
|
@@ -15,7 +15,7 @@ the bundle are dropped; afterwards each dataset's columns are compared with the
|
|||||||
live view and any drift is reported (exit status 2).
|
live view and any drift is reported (exit status 2).
|
||||||
|
|
||||||
The database is never created or changed: the bundle's database uuid must
|
The database is never created or changed: the bundle's database uuid must
|
||||||
already exist under the name "Fuel (Trino)", otherwise nothing is imported.
|
already exist under the name "Fuel", otherwise nothing is imported.
|
||||||
|
|
||||||
The dashboards (atoma-overview, atoma-site) get no role: admins see them in Superset, and
|
The dashboards (atoma-overview, atoma-site) get no role: admins see them in Superset, and
|
||||||
the Petrolyst app shows them with guest passes (PD-12). With --embedded-out FILE
|
the Petrolyst app shows them with guest passes (PD-12). With --embedded-out FILE
|
||||||
@@ -35,7 +35,7 @@ import zipfile
|
|||||||
|
|
||||||
import yaml
|
import yaml
|
||||||
|
|
||||||
DATABASE_NAME = "Fuel (Trino)"
|
DATABASE_NAME = "Fuel"
|
||||||
EMBEDDED_PREFIX = "atoma-"
|
EMBEDDED_PREFIX = "atoma-"
|
||||||
STEPS = [("datasets", "SqlaTable"), ("charts", "Slice"), ("dashboards", "Dashboard")]
|
STEPS = [("datasets", "SqlaTable"), ("charts", "Slice"), ("dashboards", "Dashboard")]
|
||||||
|
|
||||||
|
|||||||
@@ -45,6 +45,21 @@ SQLALCHEMY_DATABASE_URI = URL.create(
|
|||||||
database="superset",
|
database="superset",
|
||||||
).render_as_string(hide_password=False)
|
).render_as_string(hide_password=False)
|
||||||
|
|
||||||
|
# --- The data: fuel-db ----------------------------------------------------------
|
||||||
|
# Superset reads fuel-db directly as the read-only role fuel_ro (no Trino since 29 Oct 2026);
|
||||||
|
# superset-init registers that connection, "Fuel", without a password. Passwords of data
|
||||||
|
# connections come from secret files at connect time, so none is kept in Superset's own
|
||||||
|
# database. A connection added in the UI gets no password this way.
|
||||||
|
_DATABASE_PASSWORD_SECRETS = {"fuel_ro": "fuel_db_ro_password"}
|
||||||
|
|
||||||
|
|
||||||
|
def _database_password(url: URL) -> str | None:
|
||||||
|
secret = _DATABASE_PASSWORD_SECRETS.get(url.username or "")
|
||||||
|
return _read_secret(secret) if secret else None
|
||||||
|
|
||||||
|
|
||||||
|
SQLALCHEMY_CUSTOM_PASSWORD_STORE = _database_password
|
||||||
|
|
||||||
# --- Sign-in through Keycloak (client "petrolyst") ----------------------------
|
# --- Sign-in through Keycloak (client "petrolyst") ----------------------------
|
||||||
# Users and roles are managed in Keycloak; there is no local admin account.
|
# Users and roles are managed in Keycloak; there is no local admin account.
|
||||||
# Break-glass access: `superset fab ...` inside the container on the server.
|
# Break-glass access: `superset fab ...` inside the container on the server.
|
||||||
|
|||||||
@@ -1,9 +1,9 @@
|
|||||||
# Fuel-monitoring analytics: Superset on atoma.petrolyst.com, reading the
|
# Fuel-monitoring analytics: Superset on atoma.petrolyst.com, reading the
|
||||||
# operational store (fuel-db) through Trino. Run only through deploy/deploy.sh
|
# operational store (fuel-db) directly as the read-only role fuel_ro (Trino was
|
||||||
# or deploy/rollback.sh: they export SUPERSET_IMAGE_TAG and TRINO_CONFIG_HASH
|
# removed on 29 Oct 2026). Run only through deploy/deploy.sh or
|
||||||
# from the release's ARTIFACTS file, and the :? guards below stop Compose from
|
# deploy/rollback.sh: they export SUPERSET_IMAGE_TAG from the release's
|
||||||
# running with either unset.
|
# ARTIFACTS file, and the :? guard below stops Compose from running without it.
|
||||||
name: fuel-monitoring
|
name: petrolyst
|
||||||
|
|
||||||
x-superset-env: &superset-env
|
x-superset-env: &superset-env
|
||||||
KEYCLOAK_REALM_URL: ${KEYCLOAK_REALM_URL:-https://iam.thedigitalbridge.io/realms/thedigitalbridge}
|
KEYCLOAK_REALM_URL: ${KEYCLOAK_REALM_URL:-https://iam.thedigitalbridge.io/realms/thedigitalbridge}
|
||||||
@@ -15,6 +15,8 @@ x-superset: &superset
|
|||||||
- superset_secret_key
|
- superset_secret_key
|
||||||
- superset_db_password
|
- superset_db_password
|
||||||
- keycloak_petrolyst_client_secret
|
- keycloak_petrolyst_client_secret
|
||||||
|
# fuel_ro's password, read at connect time (superset_config.py)
|
||||||
|
- fuel_db_ro_password
|
||||||
|
|
||||||
services:
|
services:
|
||||||
superset:
|
superset:
|
||||||
@@ -100,25 +102,6 @@ services:
|
|||||||
timeout: 5s
|
timeout: 5s
|
||||||
retries: 10
|
retries: 10
|
||||||
|
|
||||||
trino:
|
|
||||||
image: trinodb/trino:484
|
|
||||||
env_file:
|
|
||||||
- ${PETROLYST_ROOT:-/opt/petrolyst}/env/fuel-monitoring.env
|
|
||||||
volumes:
|
|
||||||
- ./trino/etc:/etc/trino:ro
|
|
||||||
labels:
|
|
||||||
# Compose cannot see content changes in a bind mount; this label changes
|
|
||||||
# with trino/etc, so exactly a config change recreates the container.
|
|
||||||
petrolyst.config-hash: ${TRINO_CONFIG_HASH:?TRINO_CONFIG_HASH unset - run through deploy/deploy.sh}
|
|
||||||
depends_on:
|
|
||||||
fuel-db:
|
|
||||||
condition: service_healthy
|
|
||||||
networks:
|
|
||||||
- backend
|
|
||||||
restart: unless-stopped
|
|
||||||
mem_limit: 3g
|
|
||||||
# The image's own healthcheck (/usr/lib/trino/bin/health-check) applies.
|
|
||||||
|
|
||||||
fuel-db:
|
fuel-db:
|
||||||
image: timescale/timescaledb:2.31.0-pg17
|
image: timescale/timescaledb:2.31.0-pg17
|
||||||
environment:
|
environment:
|
||||||
@@ -128,7 +111,7 @@ services:
|
|||||||
TS_TUNE_NUM_CPUS: "2"
|
TS_TUNE_NUM_CPUS: "2"
|
||||||
secrets:
|
secrets:
|
||||||
- fuel_db_password
|
- fuel_db_password
|
||||||
- fuel_db_trino_password
|
- fuel_db_ro_password
|
||||||
volumes:
|
volumes:
|
||||||
- fuel_db_data:/var/lib/postgresql/data
|
- fuel_db_data:/var/lib/postgresql/data
|
||||||
# A single file, not the folder: the image's own TimescaleDB init scripts
|
# A single file, not the folder: the image's own TimescaleDB init scripts
|
||||||
@@ -157,9 +140,14 @@ networks:
|
|||||||
# so fuel-db's loopback port works.
|
# so fuel-db's loopback port works.
|
||||||
fuel-db-tunnel: {}
|
fuel-db-tunnel: {}
|
||||||
|
|
||||||
|
# Created under the project's first name, fuel-monitoring; named explicitly so the
|
||||||
|
# data stayed when the project became petrolyst (29 Oct 2026). Compose warns once
|
||||||
|
# that they belong to another project; that is expected.
|
||||||
volumes:
|
volumes:
|
||||||
superset_db_data:
|
superset_db_data:
|
||||||
|
name: fuel-monitoring_superset_db_data
|
||||||
fuel_db_data:
|
fuel_db_data:
|
||||||
|
name: fuel-monitoring_fuel_db_data
|
||||||
|
|
||||||
secrets:
|
secrets:
|
||||||
superset_secret_key:
|
superset_secret_key:
|
||||||
@@ -168,7 +156,7 @@ secrets:
|
|||||||
file: ${PETROLYST_ROOT:-/opt/petrolyst}/secrets/fuel-monitoring/superset_db_password
|
file: ${PETROLYST_ROOT:-/opt/petrolyst}/secrets/fuel-monitoring/superset_db_password
|
||||||
fuel_db_password:
|
fuel_db_password:
|
||||||
file: ${PETROLYST_ROOT:-/opt/petrolyst}/secrets/fuel-monitoring/fuel_db_password
|
file: ${PETROLYST_ROOT:-/opt/petrolyst}/secrets/fuel-monitoring/fuel_db_password
|
||||||
fuel_db_trino_password:
|
fuel_db_ro_password:
|
||||||
file: ${PETROLYST_ROOT:-/opt/petrolyst}/secrets/fuel-monitoring/fuel_db_trino_password
|
file: ${PETROLYST_ROOT:-/opt/petrolyst}/secrets/fuel-monitoring/fuel_db_ro_password
|
||||||
keycloak_petrolyst_client_secret:
|
keycloak_petrolyst_client_secret:
|
||||||
file: ${PETROLYST_ROOT:-/opt/petrolyst}/secrets/shared/keycloak_petrolyst_client_secret
|
file: ${PETROLYST_ROOT:-/opt/petrolyst}/secrets/shared/keycloak_petrolyst_client_secret
|
||||||
@@ -5,21 +5,24 @@
|
|||||||
# Loaders log in as their own role but act as fuel_owner (README
|
# Loaders log in as their own role but act as fuel_owner (README
|
||||||
# "Data loaders"): the default privileges below cover only what
|
# "Data loaders"): the default privileges below cover only what
|
||||||
# fuel_owner itself creates, not what its members create.
|
# fuel_owner itself creates, not what its members create.
|
||||||
# trino_ro what Trino connects as: read-only, also on tables created later.
|
# fuel_ro what Superset and the Petrolyst server read as: SELECT on public,
|
||||||
|
# also on tables created later. Read-only by default as well: a
|
||||||
|
# second lock behind the grants.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
trino_password=$(cat /run/secrets/fuel_db_trino_password)
|
ro_password=$(cat /run/secrets/fuel_db_ro_password)
|
||||||
|
|
||||||
psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" \
|
psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" \
|
||||||
-v trino_password="$trino_password" <<'SQL'
|
-v ro_password="$ro_password" <<'SQL'
|
||||||
CREATE ROLE fuel_owner NOLOGIN;
|
CREATE ROLE fuel_owner NOLOGIN;
|
||||||
ALTER DATABASE fuel OWNER TO fuel_owner;
|
ALTER DATABASE fuel OWNER TO fuel_owner;
|
||||||
REVOKE CREATE ON SCHEMA public FROM PUBLIC;
|
REVOKE CREATE ON SCHEMA public FROM PUBLIC;
|
||||||
|
|
||||||
CREATE ROLE trino_ro LOGIN PASSWORD :'trino_password';
|
CREATE ROLE fuel_ro LOGIN PASSWORD :'ro_password';
|
||||||
GRANT CONNECT ON DATABASE fuel TO trino_ro;
|
ALTER ROLE fuel_ro SET default_transaction_read_only = on;
|
||||||
GRANT USAGE ON SCHEMA public TO trino_ro;
|
GRANT CONNECT ON DATABASE fuel TO fuel_ro;
|
||||||
GRANT SELECT ON ALL TABLES IN SCHEMA public TO trino_ro;
|
GRANT USAGE ON SCHEMA public TO fuel_ro;
|
||||||
|
GRANT SELECT ON ALL TABLES IN SCHEMA public TO fuel_ro;
|
||||||
ALTER DEFAULT PRIVILEGES FOR ROLE fuel_owner IN SCHEMA public
|
ALTER DEFAULT PRIVILEGES FOR ROLE fuel_owner IN SCHEMA public
|
||||||
GRANT SELECT ON TABLES TO trino_ro;
|
GRANT SELECT ON TABLES TO fuel_ro;
|
||||||
SQL
|
SQL
|
||||||
@@ -1,4 +1,4 @@
|
|||||||
-- 0002_fm_derivations.sql: internal derivations in schema `fm` (not visible to Trino).
|
-- 0002_fm_derivations.sql: internal derivations in schema `fm` (not readable by fuel_ro).
|
||||||
-- DRAFT, not applied. Applied after 0001 by a loader acting as fuel_owner (README in this folder).
|
-- DRAFT, not applied. Applied after 0001 by a loader acting as fuel_owner (README in this folder).
|
||||||
-- Design: fuel-monitoring .context/plans/2026-10-24-fuel-db-baseline-dashboard.md, §4 and §5.1.
|
-- Design: fuel-monitoring .context/plans/2026-10-24-fuel-db-baseline-dashboard.md, §4 and §5.1.
|
||||||
BEGIN;
|
BEGIN;
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
-- 0003_bi_views.sql: the views Superset reads through Trino (schema `public`), plus
|
-- 0003_bi_views.sql: the views Superset reads (schema `public`), plus
|
||||||
-- fm.v_site_baseline_status. Owned by fuel_owner, so trino_ro can read them through the existing
|
-- fm.v_site_baseline_status. Owned by fuel_owner, so fuel_ro can read them through the existing
|
||||||
-- default privileges; they run with the owner's rights, so Trino never reads `fm` (it lists fm's
|
-- default privileges; they run with the owner's rights, so fuel_ro never reads `fm` (the catalog
|
||||||
-- table names, as it lists every schema's, but any query on them is refused).
|
-- shows fm's table names, as it shows every schema's, but any query on them is refused).
|
||||||
-- DRAFT, not applied. Applied after 0002 by a loader acting as fuel_owner (README in this folder).
|
-- DRAFT, not applied. Applied after 0002 by a loader acting as fuel_owner (README in this folder).
|
||||||
-- Design: fuel-monitoring .context/plans/2026-10-24-fuel-db-baseline-dashboard.md, §5.2 and §6.
|
-- Design: fuel-monitoring .context/plans/2026-10-24-fuel-db-baseline-dashboard.md, §5.2 and §6.
|
||||||
BEGIN;
|
BEGIN;
|
||||||
@@ -97,7 +97,7 @@ LEFT JOIN pp USING (site_uid) LEFT JOIN fm.v_site_baseline_status bs USING (site
|
|||||||
LEFT JOIN cal USING (site_uid) LEFT JOIN latest USING (site_uid);
|
LEFT JOIN cal USING (site_uid) LEFT JOIN latest USING (site_uid);
|
||||||
|
|
||||||
-- Every view below joins public.v_site for site_uid, site_code, site_name and site_label, so one
|
-- Every view below joins public.v_site for site_uid, site_code, site_name and site_label, so one
|
||||||
-- Site filter drives every dataset, and casts to Trino-safe types (varchar(n), integer,
|
-- Site filter drives every dataset, and casts to plain types (varchar(n), integer,
|
||||||
-- numeric(p,s), date, timestamp(0), boolean). Live = asserted, reviewed or disputed.
|
-- numeric(p,s), date, timestamp(0), boolean). Live = asserted, reviewed or disputed.
|
||||||
|
|
||||||
-- S2, R3: one row per site and measure; the rows stay when a value is not established.
|
-- S2, R3: one row per site and measure; the rows stay when a value is not established.
|
||||||
|
|||||||
@@ -10,7 +10,7 @@ The design and its rationale live in the fuel-monitoring corpus:
|
|||||||
|
|
||||||
## Files
|
## Files
|
||||||
|
|
||||||
| File | Creates | Visible to Trino |
|
| File | Creates | Readable by `fuel_ro` (Superset, Petrolyst server) |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `0001_fm_core.sql` | schema `fm`, `btree_gist` in it, 25 tables, constraints, trigger-backed rules 1–13 | no |
|
| `0001_fm_core.sql` | schema `fm`, `btree_gist` in it, 25 tables, constraints, trigger-backed rules 1–13 | no |
|
||||||
| `0002_fm_derivations.sql` | internal views in `fm`: claims → visits → stock intervals → L/h and L/day intervals, dependency hashes, baseline validity | no |
|
| `0002_fm_derivations.sql` | internal views in `fm`: claims → visits → stock intervals → L/h and L/day intervals, dependency hashes, baseline validity | no |
|
||||||
@@ -21,7 +21,7 @@ Each file is one transaction. Apply them in order.
|
|||||||
## Applying (on the server)
|
## Applying (on the server)
|
||||||
|
|
||||||
1. Create a loader role as in the repository README, "Data loaders". Its sessions act as
|
1. Create a loader role as in the repository README, "Data loaders". Its sessions act as
|
||||||
`fuel_owner`, so every object belongs to `fuel_owner`. `trino_ro` can then read the `public`
|
`fuel_owner`, so every object belongs to `fuel_owner`. `fuel_ro` can then read the `public`
|
||||||
views through the existing default privileges, with no role or grant changes.
|
views through the existing default privileges, with no role or grant changes.
|
||||||
2. Open the tunnel and apply the three files as the loader:
|
2. Open the tunnel and apply the three files as the loader:
|
||||||
|
|
||||||
@@ -32,13 +32,14 @@ Each file is one transaction. Apply them in order.
|
|||||||
done
|
done
|
||||||
```
|
```
|
||||||
|
|
||||||
3. Check through Trino (`docker exec -it fuel-monitoring-trino-1 trino`):
|
3. Check as `fuel_ro`
|
||||||
- `SHOW TABLES FROM fuel.public` lists the 18 `v_*` views;
|
(`docker exec -it petrolyst-fuel-db-1 psql -U postgres -d fuel`, then `SET ROLE fuel_ro;`):
|
||||||
- `SELECT count(*) FROM fuel.fm.site` fails with "permission denied for schema fm".
|
- `\dv public.*` lists the 18 `v_*` views;
|
||||||
|
- `SELECT count(*) FROM fm.site;` fails with "permission denied for schema fm".
|
||||||
|
|
||||||
The `public` views are owned by `fuel_owner` and run with its rights, so Trino never needs, and
|
The `public` views are owned by `fuel_owner` and run with its rights, so `fuel_ro` never needs,
|
||||||
never gets, read access to `fm`. Trino's metadata still lists fm's table and column names, as it
|
and never gets, read access to `fm`. PostgreSQL's catalog still shows every schema's table and
|
||||||
lists every schema in fuel-db (TimescaleDB's internal ones included); names only, no rows.
|
column names to any role, so Superset's SQL Lab can list fm's names; names only, no rows.
|
||||||
|
|
||||||
## Data
|
## Data
|
||||||
|
|
||||||
@@ -91,6 +92,8 @@ it, except baseline versions and their approvals, which live only in fuel-db: ex
|
|||||||
|
|
||||||
## Verified locally (2026-10-24, scratch only)
|
## Verified locally (2026-10-24, scratch only)
|
||||||
|
|
||||||
|
With Trino, before its removal on 29 Oct 2026; `trino_ro` is now `fuel_ro`.
|
||||||
|
|
||||||
Setup:
|
Setup:
|
||||||
- `timescale/timescaledb:2.31.0-pg17`, with this repository's `01-roles.sh`;
|
- `timescale/timescaledb:2.31.0-pg17`, with this repository's `01-roles.sh`;
|
||||||
- a loader created by the README recipe;
|
- a loader created by the README recipe;
|
||||||
|
|||||||
@@ -6,10 +6,9 @@
|
|||||||
FROM apache/superset:6.1.0@sha256:16b50bbef6648912a79e3293d418fefa743ce555d36c5af6b85d859119ed7f88
|
FROM apache/superset:6.1.0@sha256:16b50bbef6648912a79e3293d418fefa743ce555d36c5af6b85d859119ed7f88
|
||||||
|
|
||||||
USER root
|
USER root
|
||||||
# The 6.1.0 image ships neither an OAuth client nor the Postgres and Trino
|
# The 6.1.0 image ships neither an OAuth client nor the Postgres driver.
|
||||||
# drivers.
|
|
||||||
RUN uv pip install --no-cache --python /app/.venv/bin/python \
|
RUN uv pip install --no-cache --python /app/.venv/bin/python \
|
||||||
authlib==1.9.1 psycopg2-binary==2.9.13 "trino[sqlalchemy]==0.340.0"
|
authlib==1.9.1 psycopg2-binary==2.9.13
|
||||||
|
|
||||||
COPY --from=insights --chown=superset:superset superset_config.py petrolyst_security.py /app/pythonpath/
|
COPY --from=insights --chown=superset:superset superset_config.py petrolyst_security.py /app/pythonpath/
|
||||||
COPY --from=insights --chown=superset:superset roles/ /app/petrolyst-roles/
|
COPY --from=insights --chown=superset:superset roles/ /app/petrolyst-roles/
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# superset-init: migrate the metadata database, sync roles and permissions, and
|
# superset-init: migrate the metadata database, sync roles and permissions, and
|
||||||
# register the Trino connection. It is copied into the Superset image, so a
|
# register the connection to fuel-db. It is copied into the Superset image, so a
|
||||||
# change here changes the image tag; deploy/deploy.sh runs it on a fresh install
|
# change here changes the image tag; deploy/deploy.sh runs it on a fresh install
|
||||||
# and whenever that tag changes. Every step is idempotent. Unlike the
|
# and whenever that tag changes. Every step is idempotent. Unlike the
|
||||||
# upstream docker-init.sh it never creates a local admin: users come from
|
# upstream docker-init.sh it never creates a local admin: users come from
|
||||||
@@ -9,6 +9,25 @@ set -euo pipefail
|
|||||||
|
|
||||||
superset db upgrade
|
superset db upgrade
|
||||||
superset init
|
superset init
|
||||||
|
|
||||||
|
# Trino was removed on 29 Oct 2026. An instance still holding its connection
|
||||||
|
# "Fuel (Trino)" gets it renamed, so its datasets and dashboards keep their ids;
|
||||||
|
# the next step points it at fuel-db. Drop this once no instance has it.
|
||||||
|
/app/.venv/bin/python - <<'PY'
|
||||||
|
from superset.app import create_app
|
||||||
|
|
||||||
|
with create_app().app_context():
|
||||||
|
from superset import db
|
||||||
|
from superset.models.core import Database
|
||||||
|
|
||||||
|
old = db.session.query(Database).filter_by(database_name="Fuel (Trino)").one_or_none()
|
||||||
|
if old and not db.session.query(Database).filter_by(database_name="Fuel").count():
|
||||||
|
old.database_name = "Fuel"
|
||||||
|
db.session.commit()
|
||||||
|
print('renamed the connection "Fuel (Trino)" to "Fuel"')
|
||||||
|
PY
|
||||||
|
|
||||||
|
# No password here: superset_config.py reads it from /run/secrets at connect time.
|
||||||
superset set-database-uri \
|
superset set-database-uri \
|
||||||
--database_name "Fuel (Trino)" \
|
--database_name "Fuel" \
|
||||||
--uri "trino://superset@trino:8080/fuel"
|
--uri "postgresql+psycopg2://fuel_ro@fuel-db:5432/fuel"
|
||||||
@@ -1,3 +0,0 @@
|
|||||||
# Analytics never writes through Trino. fuel-db's trino_ro role is read-only as
|
|
||||||
# well; this is the second lock.
|
|
||||||
access-control.name=read-only
|
|
||||||
@@ -1,5 +0,0 @@
|
|||||||
# The fuel-monitoring operational store (service fuel-db), read-only.
|
|
||||||
connector.name=postgresql
|
|
||||||
connection-url=jdbc:postgresql://fuel-db:5432/fuel
|
|
||||||
connection-user=trino_ro
|
|
||||||
connection-password=${ENV:FUEL_DB_TRINO_PASSWORD}
|
|
||||||
@@ -1,14 +0,0 @@
|
|||||||
# Single node: coordinator and worker in one process. Reachable only on the
|
|
||||||
# backend network (Superset); no port is published and Traefik has no route.
|
|
||||||
coordinator=true
|
|
||||||
node-scheduler.include-coordinator=true
|
|
||||||
http-server.http.port=8080
|
|
||||||
discovery.uri=http://localhost:8080
|
|
||||||
catalog.management=static
|
|
||||||
|
|
||||||
# Memory bounds. The container is capped at 3g (compose.yaml) and jvm.config
|
|
||||||
# gives the heap 75 % of it (~2.25 GB); 1 GB for queries plus 512 MB headroom
|
|
||||||
# leaves room for the rest of the heap.
|
|
||||||
query.max-memory=1GB
|
|
||||||
query.max-memory-per-node=1GB
|
|
||||||
memory.heap-headroom-per-node=512MB
|
|
||||||
@@ -1,14 +0,0 @@
|
|||||||
-server
|
|
||||||
-agentpath:/usr/lib/trino/bin/libjvmkill.so
|
|
||||||
-XX:InitialRAMPercentage=75
|
|
||||||
-XX:MaxRAMPercentage=75
|
|
||||||
-XX:G1HeapRegionSize=32M
|
|
||||||
-XX:+ExplicitGCInvokesConcurrent
|
|
||||||
-XX:+HeapDumpOnOutOfMemoryError
|
|
||||||
-XX:+ExitOnOutOfMemoryError
|
|
||||||
-XX:-OmitStackTraceInFastThrow
|
|
||||||
-XX:ReservedCodeCacheSize=256M
|
|
||||||
-XX:PerMethodRecompilationCutoff=10000
|
|
||||||
-XX:PerBytecodeRecompilationCutoff=10000
|
|
||||||
-Djdk.attach.allowAttachSelf=true
|
|
||||||
-Djdk.nio.maxCachedBufferSize=2000000
|
|
||||||
@@ -1 +0,0 @@
|
|||||||
io.trino=INFO
|
|
||||||
@@ -1,3 +0,0 @@
|
|||||||
node.environment=petrolyst
|
|
||||||
node.data-dir=/data/trino
|
|
||||||
node.id=${ENV:HOSTNAME}
|
|
||||||
Reference in new issue
Block a user