Trino out of the code; commit messages name bands and rules

The rename step in init.sh and the guard in deploy/lib.sh are gone. README's
procedure is now "Removing the old project"; it keeps the old names until it
has run on the server. Migrations stay here (open point 8), and their README
says so. AGENTS.md adds the commit-message rule. Not deployed: live still runs
Trino.

Bands: 4 (Superset), persistence (docs). Rules: 2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Nasim SadatandClaude Opus 5.5 committed 2026-10-30 13:50:57 +01:00
1 parent 8405706ed8
commit e4806f5799
9 files changed
+55 -67

No files matched your search

+2
View File
@@ -15,6 +15,8 @@ keep the rules there; a change against a rule or an open conflict goes to the ow
- `keycloak/`: the `petrolyst` client import.
## Rules
- **Commit messages** end their body with the bands and rules touched, for example
`Bands: 4 (server). Rules: 2, 3.`, or `Bands: none (docs).` for text only (architecture rule 1).
- **Deploys:** only by README "Manual deploy", only when the owner asks. There is no CI. Commit on
`main` first; never push or deploy unprompted.
- **No secrets** in the repository; `deploy/deploy.sh` refuses if one appears.
+38 -31
View File
@@ -6,7 +6,7 @@ Everything that runs on the petrolyst server (109.199.127.84, Ubuntu 24.04):
https://traefik.petrolyst.com, behind Keycloak.
- **fuel-monitoring** (ATOMA): Superset at https://atoma.petrolyst.com, reading the operational
store `fuel-db` (Postgres 17 with TimescaleDB) directly, as the read-only role `fuel_ro`
(Trino was removed on 29 Oct 2026). Sign-in goes through Keycloak
(no Trino since 29 Oct 2026). Sign-in goes through Keycloak
client `petrolyst`, and the UI carries the Petrolyst brand.
**Deploys are manual.** A person deploys a clean commit of `main` from the Mac ("Manual deploy"
@@ -245,18 +245,14 @@ Tables it creates, hypertables included, then belong to `fuel_owner`, and `fuel_
After `SET ROLE NONE`, it cannot create anything in `public`. To repair a table that another
role created, run `ALTER TABLE <t> OWNER TO fuel_owner`, then `GRANT SELECT ON <t> TO fuel_ro`.
**One-time steps after 29 Oct 2026**, at the first deploy that includes them. Two changes
meet there:
- Trino is removed: Superset reads fuel-db directly. The role `trino_ro` becomes `fuel_ro` and
keeps its password (a SCRAM password survives a rename); its secret file moves with it.
- The Compose project `fuel-monitoring` is renamed `petrolyst`. Containers are then called
`petrolyst-<service>-1`. The data volumes keep their names (`fuel-monitoring_superset_db_data`,
`fuel-monitoring_fuel_db_data`; pinned in `compose.yaml`), so no data moves. The old project's
containers must be gone first: two PostgreSQL servers on one data volume corrupt it.
Preflight refuses to deploy while any of them exists.
**Removing the old project**, once, right before the next deploy; delete this paragraph when it
is done. The server still runs the Compose project `fuel-monitoring`, with Trino. Afterwards it runs
`petrolyst`, without it: containers are `petrolyst-<service>-1`, and the data volumes keep their
names (pinned in `compose.yaml`), so no data moves. The old containers must be gone first: two
PostgreSQL servers on one data volume corrupt it, and preflight refuses to deploy while any exists.
As root on the server, right before the deploy (Superset is down from step 3 until the deploy
finishes):
As root on the server (Superset is down from step 3 until the deploy finishes). The role keeps its
password, because a SCRAM password survives a rename:
```bash
S=/opt/petrolyst/secrets/fuel-monitoring
@@ -267,20 +263,35 @@ docker exec -i fuel-monitoring-fuel-db-1 psql -U postgres -d fuel -v ON_ERROR_ST
docker rm -f $(docker ps -aq --filter label=com.docker.compose.project=fuel-monitoring) # 3
```
4. Deploy as usual ("Manual deploy"). Compose starts the `petrolyst` project on the same volumes
and no longer starts Trino. Compose warns once that the volumes belong to another project;
that is expected. `superset-init` renames the connection "Fuel (Trino)" to "Fuel" and points
it at fuel-db; the smoke test checks that it reads as `fuel_ro`, read-only.
5. Clean up: `rm /opt/petrolyst/env/fuel-monitoring.env`, `rmdir /opt/petrolyst/env`,
4. Deploy as usual ("Manual deploy"). Compose warns once that the volumes belong to another project;
that is expected. `superset-init` registers the connection "Fuel"; the smoke test checks that it
reads as `fuel_ro`, read-only.
5. Delete the old connection. It refuses if a dataset still uses it; then stop and look:
```bash
docker exec -i petrolyst-superset-1 /app/.venv/bin/python - <<'PY'
from superset.app import create_app
with create_app().app_context():
from superset import db
from superset.connectors.sqla.models import SqlaTable
from superset.models.core import Database
old = db.session.query(Database).filter_by(database_name="Fuel (Trino)").one_or_none()
if old:
used = db.session.query(SqlaTable).filter_by(database_id=old.id).count()
if used:
raise SystemExit(f"{used} datasets still use the old connection")
db.session.delete(old)
db.session.commit()
print("old connection deleted")
PY
```
6. Clean up: `rm /opt/petrolyst/env/fuel-monitoring.env`, `rmdir /opt/petrolyst/env`,
`docker image rm trinodb/trino:484`,
`docker network rm fuel-monitoring_backend fuel-monitoring_fuel-db-tunnel`.
**Rollback across the Trino removal** (not tested). `rollback.sh` stops for a release from before
it. To go back anyway: remove the `petrolyst` project's containers, reverse steps 1 and 2 above (rename `fuel_ro` back to `trino_ro` and
`ALTER ROLE trino_ro RESET default_transaction_read_only`, move the secret file back, write
`/opt/petrolyst/env/fuel-monitoring.env` with `FUEL_DB_TRINO_PASSWORD=<that secret>`), rename
the Superset connection "Fuel" back to "Fuel (Trino)" with `trino://superset@trino:8080/fuel`,
then run that release's own script, `/opt/petrolyst/releases/<sha>/deploy/rollback.sh <sha>`.
7. Releases from before this deploy can no longer run, as their role is gone, and `rollback.sh`
stops on them. Remove them from `/opt/petrolyst/releases/`.
**Adding a project**: create `stacks/<name>/compose.yaml`, attach the public service to the
external network `proxy`, and set `traefik.*` labels with a unique router name and
@@ -299,8 +310,6 @@ its own gate in its stack:
## Verified locally (2026-10-23)
Before Trino's removal (29 Oct 2026): its lines below are history.
An end-to-end run ran the real `deploy.sh` and `rollback.sh` in a Linux container against
Docker Desktop, with a scratch `/opt/petrolyst`. Production differences in that run:
- a test CA certificate stood in for Let's Encrypt;
@@ -310,7 +319,6 @@ Docker Desktop, with a scratch `/opt/petrolyst`. Production differences in that
Results:
- The fresh install took the init path.
- An identical re-run recreated nothing and left `ARTIFACTS` byte-identical.
- A Trino-only change recreated only `trino`.
- A Superset change stopped, backed up, migrated and recreated only `superset`.
- A change to `superset/init.sh` alone did the same, and the changed script ran.
- A same-SHA upload with different content was refused.
@@ -327,13 +335,12 @@ Results:
- Retention pruned read-only releases.
- Retention removed a queued upload that a plain `rsync -a` had given an old checkout's
directory times. It kept one uploaded with `--omit-dir-times`.
- In `fuel-db`, a loader that created a table as its own role left it unreadable for `trino_ro`.
With the "Data loaders" recipe, tables (hypertables too) belonged to `fuel_owner` and Trino
could read them. After `SET ROLE NONE`, the loader could not create anything.
- In `fuel-db`, a loader that created a table as its own role left it unreadable for the read-only
role. With the "Data loaders" recipe, tables (hypertables too) belonged to `fuel_owner` and that
role could read them. After `SET ROLE NONE`, the loader could not create anything.
- Smoke tests passed, including the redirects to the real Keycloak with `client_id=petrolyst`
and PKCE S256.
- TLS 1.1 and unknown SNI were refused, and HTTP redirected to HTTPS.
- Trino and `trino_ro` both refused writes.
Not verified locally:
- Let's Encrypt issuance;
+1 -6
View File
@@ -94,11 +94,6 @@ load_artifacts() {
while IFS= read -r line; do
key=${line%%=*}
value=${line#*=}
# A release from before 29 Oct 2026 still runs Trino, which logs in as
# `trino_ro`, since renamed `fuel_ro`: it cannot start (README "Rollback
# across the Trino removal").
[ "$key" != TRINO_CONFIG_HASH ] ||
die "$file is from before Trino was removed; see README \"Rollback across the Trino removal\""
[[ $key =~ ^(SUPERSET_IMAGE_TAG|TRAEFIK_CONFIG_HASH)$ && $value =~ ^[0-9a-f]{16}$ ]] ||
die "unexpected line in $file: $line"
printf -v "$key" '%s' "$value"
@@ -143,7 +138,7 @@ preflight() {
# The stack was the Compose project "fuel-monitoring" until 29 Oct 2026; "petrolyst" uses the
# same data volumes, so both must never run at once (README "One-time steps after 29 Oct 2026").
[ -z "$("${DOCKER[@]}" ps -aq --filter label=com.docker.compose.project=fuel-monitoring)" ] ||
die "containers of the old project fuel-monitoring exist; remove them first (README \"One-time steps after 29 Oct 2026\")"
die "containers of the old project fuel-monitoring exist; remove them first (README \"Removing the old project\")"
}
# promote <release> <sha>: make the release the live tree. --chmod applies to
+1 -1
View File
@@ -16,7 +16,7 @@ and where it runs lives in `stacks/`, `traefik/`, `deploy/` and `host/`.
| `superset_config.py` | Keycloak sign-in, role mapping, proxy and cookie settings, Redis caches, theme and chart palettes | `/app/pythonpath/` |
| `petrolyst_security.py` | `PetrolystSecurityManager`: user info and roles from the Keycloak ID token, `/login/` straight to Keycloak, `/logout/` through Keycloak | `/app/pythonpath/` |
| `roles/` | `atoma_guest.py`: the role "ATOMA Guest" for the Petrolyst app's embedded dashboards, with exactly its permissions; run by hand (see "Embedding") | `/app/petrolyst-roles/` |
| `assets/branding/` | logos (full colour and white): the wordmark alone for the navbar, and the stacked logo; favicons, app icons; ATOMA's logo, which the Petrolyst app shows in its map's corner (`PETROLYST_OPERATOR_LOGO`) | `/app/superset/static/assets/branding/`, served at `/static/assets/branding/…` |
| `assets/branding/` | logos (full colour and white): the wordmark alone for the navbar, and the stacked logo; favicons, app icons; ATOMA's logo, which the Petrolyst app shows in its map's corner (the server reads this file itself: `PETROLYST_OPERATOR_LOGO_FILE`) | `/app/superset/static/assets/branding/`, served at `/static/assets/branding/…` |
| `patches/` | runtime patches, `fix_*.py` with `--verify` (see its README); none at present | `/app/patches/petrolyst/`, applied during the build |
| `dashboards/` | exported dashboards and import scripts (see its README) | not in the image |
+3 -4
View File
@@ -16,8 +16,7 @@ them in Superset with the filter column.
**Datasets.** The `public.v_*` views of
`stacks/fuel-monitoring/fuel-db/migrations/0003_bi_views.sql`, read from fuel-db directly (database
`fuel`, schema `public`) as the read-only role `fuel_ro`. *Until 29 Oct 2026 Superset read them
through Trino as `fuel.public`; Trino was removed then.*
`fuel`, schema `public`) as the read-only role `fuel_ro`.
**Design and rationale** live in the fuel-monitoring corpus, in
`.context/plans/2026-10-24-fuel-db-baseline-dashboard.md`:
@@ -259,7 +258,7 @@ across methods or generator sets. The site's L/h figure itself comes only from a
(petrolyst-platform `views/src/shared/deepLink.ts`, retired) from its iframe address: the
dashboard path, the Site filter's id and the tab. The Petrolyst app uses the same rison shape for
the Site filter's preset (`app/src/app/core/site-filter.ts`). The rison shape is the one the sites
list used until 28 Oct, confirmed in P2, and the view's tests compared it with that list's Trino
list used until 28 Oct, confirmed in P2, and the view's tests compared it with that list's
output:
```
@@ -409,7 +408,7 @@ Run in P2 against a local Superset, and repeated as a smoke test in P3.
## 8. P2 results (local, 24 Oct 2026)
Local Superset 6.1.0 image built from this repository, Trino 484, fuel-db with migrations
Local Superset 6.1.0 image built from this repository, fuel-db with migrations
0001–0003 and the corpus data loaded; browser checks in headless Chrome.
| Check | Result |
+1 -1
View File
@@ -46,7 +46,7 @@ SQLALCHEMY_DATABASE_URI = URL.create(
).render_as_string(hide_password=False)
# --- The data: fuel-db ----------------------------------------------------------
# Superset reads fuel-db directly as the read-only role fuel_ro (no Trino since 29 Oct 2026);
# Superset reads fuel-db directly as the read-only role fuel_ro;
# superset-init registers that connection, "Fuel", without a password. Passwords of data
# connections come from secret files at connect time, so none is kept in Superset's own
# database. A connection added in the UI gets no password this way.
+1 -2
View File
@@ -1,6 +1,5 @@
# Fuel-monitoring analytics: Superset on atoma.petrolyst.com, reading the
# operational store (fuel-db) directly as the read-only role fuel_ro (Trino was
# removed on 29 Oct 2026). Run only through deploy/deploy.sh or
# operational store (fuel-db) directly as the read-only role fuel_ro. Run only through deploy/deploy.sh or
# deploy/rollback.sh: they export SUPERSET_IMAGE_TAG from the release's
# ARTIFACTS file, and the :? guard below stops Compose from running without it.
name: petrolyst
@@ -18,6 +18,12 @@ The design and its rationale live in the fuel-monitoring corpus:
Each file is one transaction. Apply them in order.
**Who depends on the views.** Superset's dashboards list every column (`petrolyst-insights/dashboards/atoma/columns.tsv`;
the import reports drift). The Petrolyst server reads `public.v_site`; its integration test
(petrolyst-platform `server/tests/test_integration.py`) names the columns and types it needs. After
a change to `0003_bi_views.sql`, run both checks. The migrations stay in this repository (open
point 8, closed 30 Oct 2026).
## Applying (on the server)
1. Create a loader role as in the repository README, "Data loaders". Its sessions act as
@@ -92,12 +98,11 @@ it, except baseline versions and their approvals, which live only in fuel-db: ex
## Verified locally (2026-10-24, scratch only)
With Trino, before its removal on 29 Oct 2026; `trino_ro` is now `fuel_ro`.
The reads below ran through Trino, removed since on 29 Oct 2026, as today's `fuel_ro`.
Setup:
- `timescale/timescaledb:2.31.0-pg17`, with this repository's `01-roles.sh`;
- a loader created by the README recipe;
- `trinodb/trino:484`, with this repository's `trino/etc` connected as `trino_ro`.
Results:
- **Apply:** all three files applied as the loader, under the loader's default `search_path`.
@@ -117,9 +122,7 @@ Results:
the version's validity, corrections under one `event_key`, fixed meter and epoch owners, and
population keys that name the generators a coverage assumption covers;
- every refusal listed above.
- **Trino:**
- all 303 columns of the 18 views have identical names, order and types in PostgreSQL and
Trino;
- **Read-only role:**
- `fm` is unreadable: a query is refused, though its names are listed (see Applying);
- writes are refused.
- **Not run on the server:** none of this has been run against the live fuel-db.
-17
View File
@@ -10,23 +10,6 @@ set -euo pipefail
superset db upgrade
superset init
# Trino was removed on 29 Oct 2026. An instance still holding its connection
# "Fuel (Trino)" gets it renamed, so its datasets and dashboards keep their ids;
# the next step points it at fuel-db. Drop this once no instance has it.
/app/.venv/bin/python - <<'PY'
from superset.app import create_app
with create_app().app_context():
from superset import db
from superset.models.core import Database
old = db.session.query(Database).filter_by(database_name="Fuel (Trino)").one_or_none()
if old and not db.session.query(Database).filter_by(database_name="Fuel").count():
old.database_name = "Fuel"
db.session.commit()
print('renamed the connection "Fuel (Trino)" to "Fuel"')
PY
# No password here: superset_config.py reads it from /run/secrets at connect time.
superset set-database-uri \
--database_name "Fuel" \