Trino out of the code; commit messages name bands and rules

The rename step in init.sh and the guard in deploy/lib.sh are gone. README's
procedure is now "Removing the old project"; it keeps the old names until it
has run on the server. Migrations stay here (open point 8), and their README
says so. AGENTS.md adds the commit-message rule. Not deployed: live still runs
Trino.

Bands: 4 (Superset), persistence (docs). Rules: 2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Nasim SadatandClaude Opus 5.5 committed 2026-10-30 13:50:57 +01:00
1 parent 8405706ed8
commit e4806f5799
9 files changed
+55 -67

No files matched your search

+2
View File
@@ -15,6 +15,8 @@ keep the rules there; a change against a rule or an open conflict goes to the ow
- `keycloak/`: the `petrolyst` client import. - `keycloak/`: the `petrolyst` client import.
## Rules ## Rules
- **Commit messages** end their body with the bands and rules touched, for example
`Bands: 4 (server). Rules: 2, 3.`, or `Bands: none (docs).` for text only (architecture rule 1).
- **Deploys:** only by README "Manual deploy", only when the owner asks. There is no CI. Commit on - **Deploys:** only by README "Manual deploy", only when the owner asks. There is no CI. Commit on
`main` first; never push or deploy unprompted. `main` first; never push or deploy unprompted.
- **No secrets** in the repository; `deploy/deploy.sh` refuses if one appears. - **No secrets** in the repository; `deploy/deploy.sh` refuses if one appears.
+38 -31
View File
@@ -6,7 +6,7 @@ Everything that runs on the petrolyst server (109.199.127.84, Ubuntu 24.04):
https://traefik.petrolyst.com, behind Keycloak. https://traefik.petrolyst.com, behind Keycloak.
- **fuel-monitoring** (ATOMA): Superset at https://atoma.petrolyst.com, reading the operational - **fuel-monitoring** (ATOMA): Superset at https://atoma.petrolyst.com, reading the operational
store `fuel-db` (Postgres 17 with TimescaleDB) directly, as the read-only role `fuel_ro` store `fuel-db` (Postgres 17 with TimescaleDB) directly, as the read-only role `fuel_ro`
(Trino was removed on 29 Oct 2026). Sign-in goes through Keycloak (no Trino since 29 Oct 2026). Sign-in goes through Keycloak
client `petrolyst`, and the UI carries the Petrolyst brand. client `petrolyst`, and the UI carries the Petrolyst brand.
**Deploys are manual.** A person deploys a clean commit of `main` from the Mac ("Manual deploy" **Deploys are manual.** A person deploys a clean commit of `main` from the Mac ("Manual deploy"
@@ -245,18 +245,14 @@ Tables it creates, hypertables included, then belong to `fuel_owner`, and `fuel_
After `SET ROLE NONE`, it cannot create anything in `public`. To repair a table that another After `SET ROLE NONE`, it cannot create anything in `public`. To repair a table that another
role created, run `ALTER TABLE <t> OWNER TO fuel_owner`, then `GRANT SELECT ON <t> TO fuel_ro`. role created, run `ALTER TABLE <t> OWNER TO fuel_owner`, then `GRANT SELECT ON <t> TO fuel_ro`.
**One-time steps after 29 Oct 2026**, at the first deploy that includes them. Two changes **Removing the old project**, once, right before the next deploy; delete this paragraph when it
meet there: is done. The server still runs the Compose project `fuel-monitoring`, with Trino. Afterwards it runs
- Trino is removed: Superset reads fuel-db directly. The role `trino_ro` becomes `fuel_ro` and `petrolyst`, without it: containers are `petrolyst-<service>-1`, and the data volumes keep their
keeps its password (a SCRAM password survives a rename); its secret file moves with it. names (pinned in `compose.yaml`), so no data moves. The old containers must be gone first: two
- The Compose project `fuel-monitoring` is renamed `petrolyst`. Containers are then called PostgreSQL servers on one data volume corrupt it, and preflight refuses to deploy while any exists.
`petrolyst-<service>-1`. The data volumes keep their names (`fuel-monitoring_superset_db_data`,
`fuel-monitoring_fuel_db_data`; pinned in `compose.yaml`), so no data moves. The old project's
containers must be gone first: two PostgreSQL servers on one data volume corrupt it.
Preflight refuses to deploy while any of them exists.
As root on the server, right before the deploy (Superset is down from step 3 until the deploy As root on the server (Superset is down from step 3 until the deploy finishes). The role keeps its
finishes): password, because a SCRAM password survives a rename:
```bash ```bash
S=/opt/petrolyst/secrets/fuel-monitoring S=/opt/petrolyst/secrets/fuel-monitoring
@@ -267,20 +263,35 @@ docker exec -i fuel-monitoring-fuel-db-1 psql -U postgres -d fuel -v ON_ERROR_ST
docker rm -f $(docker ps -aq --filter label=com.docker.compose.project=fuel-monitoring) # 3 docker rm -f $(docker ps -aq --filter label=com.docker.compose.project=fuel-monitoring) # 3
``` ```
4. Deploy as usual ("Manual deploy"). Compose starts the `petrolyst` project on the same volumes 4. Deploy as usual ("Manual deploy"). Compose warns once that the volumes belong to another project;
and no longer starts Trino. Compose warns once that the volumes belong to another project; that is expected. `superset-init` registers the connection "Fuel"; the smoke test checks that it
that is expected. `superset-init` renames the connection "Fuel (Trino)" to "Fuel" and points reads as `fuel_ro`, read-only.
it at fuel-db; the smoke test checks that it reads as `fuel_ro`, read-only. 5. Delete the old connection. It refuses if a dataset still uses it; then stop and look:
5. Clean up: `rm /opt/petrolyst/env/fuel-monitoring.env`, `rmdir /opt/petrolyst/env`,
```bash
docker exec -i petrolyst-superset-1 /app/.venv/bin/python - <<'PY'
from superset.app import create_app
with create_app().app_context():
from superset import db
from superset.connectors.sqla.models import SqlaTable
from superset.models.core import Database
old = db.session.query(Database).filter_by(database_name="Fuel (Trino)").one_or_none()
if old:
used = db.session.query(SqlaTable).filter_by(database_id=old.id).count()
if used:
raise SystemExit(f"{used} datasets still use the old connection")
db.session.delete(old)
db.session.commit()
print("old connection deleted")
PY
```
6. Clean up: `rm /opt/petrolyst/env/fuel-monitoring.env`, `rmdir /opt/petrolyst/env`,
`docker image rm trinodb/trino:484`, `docker image rm trinodb/trino:484`,
`docker network rm fuel-monitoring_backend fuel-monitoring_fuel-db-tunnel`. `docker network rm fuel-monitoring_backend fuel-monitoring_fuel-db-tunnel`.
7. Releases from before this deploy can no longer run, as their role is gone, and `rollback.sh`
**Rollback across the Trino removal** (not tested). `rollback.sh` stops for a release from before stops on them. Remove them from `/opt/petrolyst/releases/`.
it. To go back anyway: remove the `petrolyst` project's containers, reverse steps 1 and 2 above (rename `fuel_ro` back to `trino_ro` and
`ALTER ROLE trino_ro RESET default_transaction_read_only`, move the secret file back, write
`/opt/petrolyst/env/fuel-monitoring.env` with `FUEL_DB_TRINO_PASSWORD=<that secret>`), rename
the Superset connection "Fuel" back to "Fuel (Trino)" with `trino://superset@trino:8080/fuel`,
then run that release's own script, `/opt/petrolyst/releases/<sha>/deploy/rollback.sh <sha>`.
**Adding a project**: create `stacks/<name>/compose.yaml`, attach the public service to the **Adding a project**: create `stacks/<name>/compose.yaml`, attach the public service to the
external network `proxy`, and set `traefik.*` labels with a unique router name and external network `proxy`, and set `traefik.*` labels with a unique router name and
@@ -299,8 +310,6 @@ its own gate in its stack:
## Verified locally (2026-10-23) ## Verified locally (2026-10-23)
Before Trino's removal (29 Oct 2026): its lines below are history.
An end-to-end run ran the real `deploy.sh` and `rollback.sh` in a Linux container against An end-to-end run ran the real `deploy.sh` and `rollback.sh` in a Linux container against
Docker Desktop, with a scratch `/opt/petrolyst`. Production differences in that run: Docker Desktop, with a scratch `/opt/petrolyst`. Production differences in that run:
- a test CA certificate stood in for Let's Encrypt; - a test CA certificate stood in for Let's Encrypt;
@@ -310,7 +319,6 @@ Docker Desktop, with a scratch `/opt/petrolyst`. Production differences in that
Results: Results:
- The fresh install took the init path. - The fresh install took the init path.
- An identical re-run recreated nothing and left `ARTIFACTS` byte-identical. - An identical re-run recreated nothing and left `ARTIFACTS` byte-identical.
- A Trino-only change recreated only `trino`.
- A Superset change stopped, backed up, migrated and recreated only `superset`. - A Superset change stopped, backed up, migrated and recreated only `superset`.
- A change to `superset/init.sh` alone did the same, and the changed script ran. - A change to `superset/init.sh` alone did the same, and the changed script ran.
- A same-SHA upload with different content was refused. - A same-SHA upload with different content was refused.
@@ -327,13 +335,12 @@ Results:
- Retention pruned read-only releases. - Retention pruned read-only releases.
- Retention removed a queued upload that a plain `rsync -a` had given an old checkout's - Retention removed a queued upload that a plain `rsync -a` had given an old checkout's
directory times. It kept one uploaded with `--omit-dir-times`. directory times. It kept one uploaded with `--omit-dir-times`.
- In `fuel-db`, a loader that created a table as its own role left it unreadable for `trino_ro`. - In `fuel-db`, a loader that created a table as its own role left it unreadable for the read-only
With the "Data loaders" recipe, tables (hypertables too) belonged to `fuel_owner` and Trino role. With the "Data loaders" recipe, tables (hypertables too) belonged to `fuel_owner` and that
could read them. After `SET ROLE NONE`, the loader could not create anything. role could read them. After `SET ROLE NONE`, the loader could not create anything.
- Smoke tests passed, including the redirects to the real Keycloak with `client_id=petrolyst` - Smoke tests passed, including the redirects to the real Keycloak with `client_id=petrolyst`
and PKCE S256. and PKCE S256.
- TLS 1.1 and unknown SNI were refused, and HTTP redirected to HTTPS. - TLS 1.1 and unknown SNI were refused, and HTTP redirected to HTTPS.
- Trino and `trino_ro` both refused writes.
Not verified locally: Not verified locally:
- Let's Encrypt issuance; - Let's Encrypt issuance;
+1 -6
View File
@@ -94,11 +94,6 @@ load_artifacts() {
while IFS= read -r line; do while IFS= read -r line; do
key=${line%%=*} key=${line%%=*}
value=${line#*=} value=${line#*=}
# A release from before 29 Oct 2026 still runs Trino, which logs in as
# `trino_ro`, since renamed `fuel_ro`: it cannot start (README "Rollback
# across the Trino removal").
[ "$key" != TRINO_CONFIG_HASH ] ||
die "$file is from before Trino was removed; see README \"Rollback across the Trino removal\""
[[ $key =~ ^(SUPERSET_IMAGE_TAG|TRAEFIK_CONFIG_HASH)$ && $value =~ ^[0-9a-f]{16}$ ]] || [[ $key =~ ^(SUPERSET_IMAGE_TAG|TRAEFIK_CONFIG_HASH)$ && $value =~ ^[0-9a-f]{16}$ ]] ||
die "unexpected line in $file: $line" die "unexpected line in $file: $line"
printf -v "$key" '%s' "$value" printf -v "$key" '%s' "$value"
@@ -143,7 +138,7 @@ preflight() {
# The stack was the Compose project "fuel-monitoring" until 29 Oct 2026; "petrolyst" uses the # The stack was the Compose project "fuel-monitoring" until 29 Oct 2026; "petrolyst" uses the
# same data volumes, so both must never run at once (README "One-time steps after 29 Oct 2026"). # same data volumes, so both must never run at once (README "One-time steps after 29 Oct 2026").
[ -z "$("${DOCKER[@]}" ps -aq --filter label=com.docker.compose.project=fuel-monitoring)" ] || [ -z "$("${DOCKER[@]}" ps -aq --filter label=com.docker.compose.project=fuel-monitoring)" ] ||
die "containers of the old project fuel-monitoring exist; remove them first (README \"One-time steps after 29 Oct 2026\")" die "containers of the old project fuel-monitoring exist; remove them first (README \"Removing the old project\")"
} }
# promote <release> <sha>: make the release the live tree. --chmod applies to # promote <release> <sha>: make the release the live tree. --chmod applies to
+1 -1
View File
@@ -16,7 +16,7 @@ and where it runs lives in `stacks/`, `traefik/`, `deploy/` and `host/`.
| `superset_config.py` | Keycloak sign-in, role mapping, proxy and cookie settings, Redis caches, theme and chart palettes | `/app/pythonpath/` | | `superset_config.py` | Keycloak sign-in, role mapping, proxy and cookie settings, Redis caches, theme and chart palettes | `/app/pythonpath/` |
| `petrolyst_security.py` | `PetrolystSecurityManager`: user info and roles from the Keycloak ID token, `/login/` straight to Keycloak, `/logout/` through Keycloak | `/app/pythonpath/` | | `petrolyst_security.py` | `PetrolystSecurityManager`: user info and roles from the Keycloak ID token, `/login/` straight to Keycloak, `/logout/` through Keycloak | `/app/pythonpath/` |
| `roles/` | `atoma_guest.py`: the role "ATOMA Guest" for the Petrolyst app's embedded dashboards, with exactly its permissions; run by hand (see "Embedding") | `/app/petrolyst-roles/` | | `roles/` | `atoma_guest.py`: the role "ATOMA Guest" for the Petrolyst app's embedded dashboards, with exactly its permissions; run by hand (see "Embedding") | `/app/petrolyst-roles/` |
| `assets/branding/` | logos (full colour and white): the wordmark alone for the navbar, and the stacked logo; favicons, app icons; ATOMA's logo, which the Petrolyst app shows in its map's corner (`PETROLYST_OPERATOR_LOGO`) | `/app/superset/static/assets/branding/`, served at `/static/assets/branding/…` | | `assets/branding/` | logos (full colour and white): the wordmark alone for the navbar, and the stacked logo; favicons, app icons; ATOMA's logo, which the Petrolyst app shows in its map's corner (the server reads this file itself: `PETROLYST_OPERATOR_LOGO_FILE`) | `/app/superset/static/assets/branding/`, served at `/static/assets/branding/…` |
| `patches/` | runtime patches, `fix_*.py` with `--verify` (see its README); none at present | `/app/patches/petrolyst/`, applied during the build | | `patches/` | runtime patches, `fix_*.py` with `--verify` (see its README); none at present | `/app/patches/petrolyst/`, applied during the build |
| `dashboards/` | exported dashboards and import scripts (see its README) | not in the image | | `dashboards/` | exported dashboards and import scripts (see its README) | not in the image |
+3 -4
View File
@@ -16,8 +16,7 @@ them in Superset with the filter column.
**Datasets.** The `public.v_*` views of **Datasets.** The `public.v_*` views of
`stacks/fuel-monitoring/fuel-db/migrations/0003_bi_views.sql`, read from fuel-db directly (database `stacks/fuel-monitoring/fuel-db/migrations/0003_bi_views.sql`, read from fuel-db directly (database
`fuel`, schema `public`) as the read-only role `fuel_ro`. *Until 29 Oct 2026 Superset read them `fuel`, schema `public`) as the read-only role `fuel_ro`.
through Trino as `fuel.public`; Trino was removed then.*
**Design and rationale** live in the fuel-monitoring corpus, in **Design and rationale** live in the fuel-monitoring corpus, in
`.context/plans/2026-10-24-fuel-db-baseline-dashboard.md`: `.context/plans/2026-10-24-fuel-db-baseline-dashboard.md`:
@@ -259,7 +258,7 @@ across methods or generator sets. The site's L/h figure itself comes only from a
(petrolyst-platform `views/src/shared/deepLink.ts`, retired) from its iframe address: the (petrolyst-platform `views/src/shared/deepLink.ts`, retired) from its iframe address: the
dashboard path, the Site filter's id and the tab. The Petrolyst app uses the same rison shape for dashboard path, the Site filter's id and the tab. The Petrolyst app uses the same rison shape for
the Site filter's preset (`app/src/app/core/site-filter.ts`). The rison shape is the one the sites the Site filter's preset (`app/src/app/core/site-filter.ts`). The rison shape is the one the sites
list used until 28 Oct, confirmed in P2, and the view's tests compared it with that list's Trino list used until 28 Oct, confirmed in P2, and the view's tests compared it with that list's
output: output:
``` ```
@@ -409,7 +408,7 @@ Run in P2 against a local Superset, and repeated as a smoke test in P3.
## 8. P2 results (local, 24 Oct 2026) ## 8. P2 results (local, 24 Oct 2026)
Local Superset 6.1.0 image built from this repository, Trino 484, fuel-db with migrations Local Superset 6.1.0 image built from this repository, fuel-db with migrations
0001–0003 and the corpus data loaded; browser checks in headless Chrome. 0001–0003 and the corpus data loaded; browser checks in headless Chrome.
| Check | Result | | Check | Result |
+1 -1
View File
@@ -46,7 +46,7 @@ SQLALCHEMY_DATABASE_URI = URL.create(
).render_as_string(hide_password=False) ).render_as_string(hide_password=False)
# --- The data: fuel-db ---------------------------------------------------------- # --- The data: fuel-db ----------------------------------------------------------
# Superset reads fuel-db directly as the read-only role fuel_ro (no Trino since 29 Oct 2026); # Superset reads fuel-db directly as the read-only role fuel_ro;
# superset-init registers that connection, "Fuel", without a password. Passwords of data # superset-init registers that connection, "Fuel", without a password. Passwords of data
# connections come from secret files at connect time, so none is kept in Superset's own # connections come from secret files at connect time, so none is kept in Superset's own
# database. A connection added in the UI gets no password this way. # database. A connection added in the UI gets no password this way.
+1 -2
View File
@@ -1,6 +1,5 @@
# Fuel-monitoring analytics: Superset on atoma.petrolyst.com, reading the # Fuel-monitoring analytics: Superset on atoma.petrolyst.com, reading the
# operational store (fuel-db) directly as the read-only role fuel_ro (Trino was # operational store (fuel-db) directly as the read-only role fuel_ro. Run only through deploy/deploy.sh or
# removed on 29 Oct 2026). Run only through deploy/deploy.sh or
# deploy/rollback.sh: they export SUPERSET_IMAGE_TAG from the release's # deploy/rollback.sh: they export SUPERSET_IMAGE_TAG from the release's
# ARTIFACTS file, and the :? guard below stops Compose from running without it. # ARTIFACTS file, and the :? guard below stops Compose from running without it.
name: petrolyst name: petrolyst
@@ -18,6 +18,12 @@ The design and its rationale live in the fuel-monitoring corpus:
Each file is one transaction. Apply them in order. Each file is one transaction. Apply them in order.
**Who depends on the views.** Superset's dashboards list every column (`petrolyst-insights/dashboards/atoma/columns.tsv`;
the import reports drift). The Petrolyst server reads `public.v_site`; its integration test
(petrolyst-platform `server/tests/test_integration.py`) names the columns and types it needs. After
a change to `0003_bi_views.sql`, run both checks. The migrations stay in this repository (open
point 8, closed 30 Oct 2026).
## Applying (on the server) ## Applying (on the server)
1. Create a loader role as in the repository README, "Data loaders". Its sessions act as 1. Create a loader role as in the repository README, "Data loaders". Its sessions act as
@@ -92,12 +98,11 @@ it, except baseline versions and their approvals, which live only in fuel-db: ex
## Verified locally (2026-10-24, scratch only) ## Verified locally (2026-10-24, scratch only)
With Trino, before its removal on 29 Oct 2026; `trino_ro` is now `fuel_ro`. The reads below ran through Trino, removed since on 29 Oct 2026, as today's `fuel_ro`.
Setup: Setup:
- `timescale/timescaledb:2.31.0-pg17`, with this repository's `01-roles.sh`; - `timescale/timescaledb:2.31.0-pg17`, with this repository's `01-roles.sh`;
- a loader created by the README recipe; - a loader created by the README recipe;
- `trinodb/trino:484`, with this repository's `trino/etc` connected as `trino_ro`.
Results: Results:
- **Apply:** all three files applied as the loader, under the loader's default `search_path`. - **Apply:** all three files applied as the loader, under the loader's default `search_path`.
@@ -117,9 +122,7 @@ Results:
the version's validity, corrections under one `event_key`, fixed meter and epoch owners, and the version's validity, corrections under one `event_key`, fixed meter and epoch owners, and
population keys that name the generators a coverage assumption covers; population keys that name the generators a coverage assumption covers;
- every refusal listed above. - every refusal listed above.
- **Trino:** - **Read-only role:**
- all 303 columns of the 18 views have identical names, order and types in PostgreSQL and
Trino;
- `fm` is unreadable: a query is refused, though its names are listed (see Applying); - `fm` is unreadable: a query is refused, though its names are listed (see Applying);
- writes are refused. - writes are refused.
- **Not run on the server:** none of this has been run against the live fuel-db. - **Not run on the server:** none of this has been run against the live fuel-db.
-17
View File
@@ -10,23 +10,6 @@ set -euo pipefail
superset db upgrade superset db upgrade
superset init superset init
# Trino was removed on 29 Oct 2026. An instance still holding its connection
# "Fuel (Trino)" gets it renamed, so its datasets and dashboards keep their ids;
# the next step points it at fuel-db. Drop this once no instance has it.
/app/.venv/bin/python - <<'PY'
from superset.app import create_app
with create_app().app_context():
from superset import db
from superset.models.core import Database
old = db.session.query(Database).filter_by(database_name="Fuel (Trino)").one_or_none()
if old and not db.session.query(Database).filter_by(database_name="Fuel").count():
old.database_name = "Fuel"
db.session.commit()
print('renamed the connection "Fuel (Trino)" to "Fuel"')
PY
# No password here: superset_config.py reads it from /run/secrets at connect time. # No password here: superset_config.py reads it from /run/secrets at connect time.
superset set-database-uri \ superset set-database-uri \
--database_name "Fuel" \ --database_name "Fuel" \