diff --git a/AGENTS.md b/AGENTS.md index d2d6756..54392cb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -15,6 +15,8 @@ keep the rules there; a change against a rule or an open conflict goes to the ow - `keycloak/`: the `petrolyst` client import. ## Rules +- **Commit messages** end their body with the bands and rules touched, for example + `Bands: 4 (server). Rules: 2, 3.`, or `Bands: none (docs).` for text only (architecture rule 1). - **Deploys:** only by README "Manual deploy", only when the owner asks. There is no CI. Commit on `main` first; never push or deploy unprompted. - **No secrets** in the repository; `deploy/deploy.sh` refuses if one appears. diff --git a/README.md b/README.md index adddcf4..a282b0d 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ Everything that runs on the petrolyst server (109.199.127.84, Ubuntu 24.04): https://traefik.petrolyst.com, behind Keycloak. - **fuel-monitoring** (ATOMA): Superset at https://atoma.petrolyst.com, reading the operational store `fuel-db` (Postgres 17 with TimescaleDB) directly, as the read-only role `fuel_ro` - (Trino was removed on 29 Oct 2026). Sign-in goes through Keycloak + (no Trino since 29 Oct 2026). Sign-in goes through Keycloak client `petrolyst`, and the UI carries the Petrolyst brand. **Deploys are manual.** A person deploys a clean commit of `main` from the Mac ("Manual deploy" @@ -245,18 +245,14 @@ Tables it creates, hypertables included, then belong to `fuel_owner`, and `fuel_ After `SET ROLE NONE`, it cannot create anything in `public`. To repair a table that another role created, run `ALTER TABLE OWNER TO fuel_owner`, then `GRANT SELECT ON TO fuel_ro`. -**One-time steps after 29 Oct 2026**, at the first deploy that includes them. Two changes -meet there: -- Trino is removed: Superset reads fuel-db directly. The role `trino_ro` becomes `fuel_ro` and - keeps its password (a SCRAM password survives a rename); its secret file moves with it. -- The Compose project `fuel-monitoring` is renamed `petrolyst`. Containers are then called - `petrolyst--1`. The data volumes keep their names (`fuel-monitoring_superset_db_data`, - `fuel-monitoring_fuel_db_data`; pinned in `compose.yaml`), so no data moves. The old project's - containers must be gone first: two PostgreSQL servers on one data volume corrupt it. - Preflight refuses to deploy while any of them exists. +**Removing the old project**, once, right before the next deploy; delete this paragraph when it +is done. The server still runs the Compose project `fuel-monitoring`, with Trino. Afterwards it runs +`petrolyst`, without it: containers are `petrolyst--1`, and the data volumes keep their +names (pinned in `compose.yaml`), so no data moves. The old containers must be gone first: two +PostgreSQL servers on one data volume corrupt it, and preflight refuses to deploy while any exists. -As root on the server, right before the deploy (Superset is down from step 3 until the deploy -finishes): +As root on the server (Superset is down from step 3 until the deploy finishes). The role keeps its +password, because a SCRAM password survives a rename: ```bash S=/opt/petrolyst/secrets/fuel-monitoring @@ -267,20 +263,35 @@ docker exec -i fuel-monitoring-fuel-db-1 psql -U postgres -d fuel -v ON_ERROR_ST docker rm -f $(docker ps -aq --filter label=com.docker.compose.project=fuel-monitoring) # 3 ``` -4. Deploy as usual ("Manual deploy"). Compose starts the `petrolyst` project on the same volumes - and no longer starts Trino. Compose warns once that the volumes belong to another project; - that is expected. `superset-init` renames the connection "Fuel (Trino)" to "Fuel" and points - it at fuel-db; the smoke test checks that it reads as `fuel_ro`, read-only. -5. Clean up: `rm /opt/petrolyst/env/fuel-monitoring.env`, `rmdir /opt/petrolyst/env`, +4. Deploy as usual ("Manual deploy"). Compose warns once that the volumes belong to another project; + that is expected. `superset-init` registers the connection "Fuel"; the smoke test checks that it + reads as `fuel_ro`, read-only. +5. Delete the old connection. It refuses if a dataset still uses it; then stop and look: + + ```bash + docker exec -i petrolyst-superset-1 /app/.venv/bin/python - <<'PY' + from superset.app import create_app + + with create_app().app_context(): + from superset import db + from superset.connectors.sqla.models import SqlaTable + from superset.models.core import Database + + old = db.session.query(Database).filter_by(database_name="Fuel (Trino)").one_or_none() + if old: + used = db.session.query(SqlaTable).filter_by(database_id=old.id).count() + if used: + raise SystemExit(f"{used} datasets still use the old connection") + db.session.delete(old) + db.session.commit() + print("old connection deleted") + PY + ``` +6. Clean up: `rm /opt/petrolyst/env/fuel-monitoring.env`, `rmdir /opt/petrolyst/env`, `docker image rm trinodb/trino:484`, `docker network rm fuel-monitoring_backend fuel-monitoring_fuel-db-tunnel`. - -**Rollback across the Trino removal** (not tested). `rollback.sh` stops for a release from before -it. To go back anyway: remove the `petrolyst` project's containers, reverse steps 1 and 2 above (rename `fuel_ro` back to `trino_ro` and -`ALTER ROLE trino_ro RESET default_transaction_read_only`, move the secret file back, write -`/opt/petrolyst/env/fuel-monitoring.env` with `FUEL_DB_TRINO_PASSWORD=`), rename -the Superset connection "Fuel" back to "Fuel (Trino)" with `trino://superset@trino:8080/fuel`, -then run that release's own script, `/opt/petrolyst/releases//deploy/rollback.sh `. +7. Releases from before this deploy can no longer run, as their role is gone, and `rollback.sh` + stops on them. Remove them from `/opt/petrolyst/releases/`. **Adding a project**: create `stacks//compose.yaml`, attach the public service to the external network `proxy`, and set `traefik.*` labels with a unique router name and @@ -299,8 +310,6 @@ its own gate in its stack: ## Verified locally (2026-10-23) -Before Trino's removal (29 Oct 2026): its lines below are history. - An end-to-end run ran the real `deploy.sh` and `rollback.sh` in a Linux container against Docker Desktop, with a scratch `/opt/petrolyst`. Production differences in that run: - a test CA certificate stood in for Let's Encrypt; @@ -310,7 +319,6 @@ Docker Desktop, with a scratch `/opt/petrolyst`. Production differences in that Results: - The fresh install took the init path. - An identical re-run recreated nothing and left `ARTIFACTS` byte-identical. -- A Trino-only change recreated only `trino`. - A Superset change stopped, backed up, migrated and recreated only `superset`. - A change to `superset/init.sh` alone did the same, and the changed script ran. - A same-SHA upload with different content was refused. @@ -327,13 +335,12 @@ Results: - Retention pruned read-only releases. - Retention removed a queued upload that a plain `rsync -a` had given an old checkout's directory times. It kept one uploaded with `--omit-dir-times`. -- In `fuel-db`, a loader that created a table as its own role left it unreadable for `trino_ro`. - With the "Data loaders" recipe, tables (hypertables too) belonged to `fuel_owner` and Trino - could read them. After `SET ROLE NONE`, the loader could not create anything. +- In `fuel-db`, a loader that created a table as its own role left it unreadable for the read-only + role. With the "Data loaders" recipe, tables (hypertables too) belonged to `fuel_owner` and that + role could read them. After `SET ROLE NONE`, the loader could not create anything. - Smoke tests passed, including the redirects to the real Keycloak with `client_id=petrolyst` and PKCE S256. - TLS 1.1 and unknown SNI were refused, and HTTP redirected to HTTPS. -- Trino and `trino_ro` both refused writes. Not verified locally: - Let's Encrypt issuance; diff --git a/deploy/lib.sh b/deploy/lib.sh index b7544f5..82d0a2c 100644 --- a/deploy/lib.sh +++ b/deploy/lib.sh @@ -94,11 +94,6 @@ load_artifacts() { while IFS= read -r line; do key=${line%%=*} value=${line#*=} - # A release from before 29 Oct 2026 still runs Trino, which logs in as - # `trino_ro`, since renamed `fuel_ro`: it cannot start (README "Rollback - # across the Trino removal"). - [ "$key" != TRINO_CONFIG_HASH ] || - die "$file is from before Trino was removed; see README \"Rollback across the Trino removal\"" [[ $key =~ ^(SUPERSET_IMAGE_TAG|TRAEFIK_CONFIG_HASH)$ && $value =~ ^[0-9a-f]{16}$ ]] || die "unexpected line in $file: $line" printf -v "$key" '%s' "$value" @@ -143,7 +138,7 @@ preflight() { # The stack was the Compose project "fuel-monitoring" until 29 Oct 2026; "petrolyst" uses the # same data volumes, so both must never run at once (README "One-time steps after 29 Oct 2026"). [ -z "$("${DOCKER[@]}" ps -aq --filter label=com.docker.compose.project=fuel-monitoring)" ] || - die "containers of the old project fuel-monitoring exist; remove them first (README \"One-time steps after 29 Oct 2026\")" + die "containers of the old project fuel-monitoring exist; remove them first (README \"Removing the old project\")" } # promote : make the release the live tree. --chmod applies to diff --git a/petrolyst-insights/README.md b/petrolyst-insights/README.md index 0f364c4..5b18a57 100644 --- a/petrolyst-insights/README.md +++ b/petrolyst-insights/README.md @@ -16,7 +16,7 @@ and where it runs lives in `stacks/`, `traefik/`, `deploy/` and `host/`. | `superset_config.py` | Keycloak sign-in, role mapping, proxy and cookie settings, Redis caches, theme and chart palettes | `/app/pythonpath/` | | `petrolyst_security.py` | `PetrolystSecurityManager`: user info and roles from the Keycloak ID token, `/login/` straight to Keycloak, `/logout/` through Keycloak | `/app/pythonpath/` | | `roles/` | `atoma_guest.py`: the role "ATOMA Guest" for the Petrolyst app's embedded dashboards, with exactly its permissions; run by hand (see "Embedding") | `/app/petrolyst-roles/` | -| `assets/branding/` | logos (full colour and white): the wordmark alone for the navbar, and the stacked logo; favicons, app icons; ATOMA's logo, which the Petrolyst app shows in its map's corner (`PETROLYST_OPERATOR_LOGO`) | `/app/superset/static/assets/branding/`, served at `/static/assets/branding/…` | +| `assets/branding/` | logos (full colour and white): the wordmark alone for the navbar, and the stacked logo; favicons, app icons; ATOMA's logo, which the Petrolyst app shows in its map's corner (the server reads this file itself: `PETROLYST_OPERATOR_LOGO_FILE`) | `/app/superset/static/assets/branding/`, served at `/static/assets/branding/…` | | `patches/` | runtime patches, `fix_*.py` with `--verify` (see its README); none at present | `/app/patches/petrolyst/`, applied during the build | | `dashboards/` | exported dashboards and import scripts (see its README) | not in the image | diff --git a/petrolyst-insights/dashboards/atoma/SPEC.md b/petrolyst-insights/dashboards/atoma/SPEC.md index 2f95a03..46054a8 100644 --- a/petrolyst-insights/dashboards/atoma/SPEC.md +++ b/petrolyst-insights/dashboards/atoma/SPEC.md @@ -16,8 +16,7 @@ them in Superset with the filter column. **Datasets.** The `public.v_*` views of `stacks/fuel-monitoring/fuel-db/migrations/0003_bi_views.sql`, read from fuel-db directly (database -`fuel`, schema `public`) as the read-only role `fuel_ro`. *Until 29 Oct 2026 Superset read them -through Trino as `fuel.public`; Trino was removed then.* +`fuel`, schema `public`) as the read-only role `fuel_ro`. **Design and rationale** live in the fuel-monitoring corpus, in `.context/plans/2026-10-24-fuel-db-baseline-dashboard.md`: @@ -259,7 +258,7 @@ across methods or generator sets. The site's L/h figure itself comes only from a (petrolyst-platform `views/src/shared/deepLink.ts`, retired) from its iframe address: the dashboard path, the Site filter's id and the tab. The Petrolyst app uses the same rison shape for the Site filter's preset (`app/src/app/core/site-filter.ts`). The rison shape is the one the sites -list used until 28 Oct, confirmed in P2, and the view's tests compared it with that list's Trino +list used until 28 Oct, confirmed in P2, and the view's tests compared it with that list's output: ``` @@ -409,7 +408,7 @@ Run in P2 against a local Superset, and repeated as a smoke test in P3. ## 8. P2 results (local, 24 Oct 2026) -Local Superset 6.1.0 image built from this repository, Trino 484, fuel-db with migrations +Local Superset 6.1.0 image built from this repository, fuel-db with migrations 0001–0003 and the corpus data loaded; browser checks in headless Chrome. | Check | Result | diff --git a/petrolyst-insights/superset_config.py b/petrolyst-insights/superset_config.py index 45588eb..3ce6aef 100644 --- a/petrolyst-insights/superset_config.py +++ b/petrolyst-insights/superset_config.py @@ -46,7 +46,7 @@ SQLALCHEMY_DATABASE_URI = URL.create( ).render_as_string(hide_password=False) # --- The data: fuel-db ---------------------------------------------------------- -# Superset reads fuel-db directly as the read-only role fuel_ro (no Trino since 29 Oct 2026); +# Superset reads fuel-db directly as the read-only role fuel_ro; # superset-init registers that connection, "Fuel", without a password. Passwords of data # connections come from secret files at connect time, so none is kept in Superset's own # database. A connection added in the UI gets no password this way. diff --git a/stacks/fuel-monitoring/compose.yaml b/stacks/fuel-monitoring/compose.yaml index 6b6a410..0539bff 100644 --- a/stacks/fuel-monitoring/compose.yaml +++ b/stacks/fuel-monitoring/compose.yaml @@ -1,6 +1,5 @@ # Fuel-monitoring analytics: Superset on atoma.petrolyst.com, reading the -# operational store (fuel-db) directly as the read-only role fuel_ro (Trino was -# removed on 29 Oct 2026). Run only through deploy/deploy.sh or +# operational store (fuel-db) directly as the read-only role fuel_ro. Run only through deploy/deploy.sh or # deploy/rollback.sh: they export SUPERSET_IMAGE_TAG from the release's # ARTIFACTS file, and the :? guard below stops Compose from running without it. name: petrolyst diff --git a/stacks/fuel-monitoring/fuel-db/migrations/README.md b/stacks/fuel-monitoring/fuel-db/migrations/README.md index 8e695be..41985e2 100644 --- a/stacks/fuel-monitoring/fuel-db/migrations/README.md +++ b/stacks/fuel-monitoring/fuel-db/migrations/README.md @@ -18,6 +18,12 @@ The design and its rationale live in the fuel-monitoring corpus: Each file is one transaction. Apply them in order. +**Who depends on the views.** Superset's dashboards list every column (`petrolyst-insights/dashboards/atoma/columns.tsv`; +the import reports drift). The Petrolyst server reads `public.v_site`; its integration test +(petrolyst-platform `server/tests/test_integration.py`) names the columns and types it needs. After +a change to `0003_bi_views.sql`, run both checks. The migrations stay in this repository (open +point 8, closed 30 Oct 2026). + ## Applying (on the server) 1. Create a loader role as in the repository README, "Data loaders". Its sessions act as @@ -92,12 +98,11 @@ it, except baseline versions and their approvals, which live only in fuel-db: ex ## Verified locally (2026-10-24, scratch only) -With Trino, before its removal on 29 Oct 2026; `trino_ro` is now `fuel_ro`. +The reads below ran through Trino, removed since on 29 Oct 2026, as today's `fuel_ro`. Setup: - `timescale/timescaledb:2.31.0-pg17`, with this repository's `01-roles.sh`; - a loader created by the README recipe; -- `trinodb/trino:484`, with this repository's `trino/etc` connected as `trino_ro`. Results: - **Apply:** all three files applied as the loader, under the loader's default `search_path`. @@ -117,9 +122,7 @@ Results: the version's validity, corrections under one `event_key`, fixed meter and epoch owners, and population keys that name the generators a coverage assumption covers; - every refusal listed above. -- **Trino:** - - all 303 columns of the 18 views have identical names, order and types in PostgreSQL and - Trino; +- **Read-only role:** - `fm` is unreadable: a query is refused, though its names are listed (see Applying); - writes are refused. - **Not run on the server:** none of this has been run against the live fuel-db. diff --git a/stacks/fuel-monitoring/superset/init.sh b/stacks/fuel-monitoring/superset/init.sh index a4401d7..db8d92d 100755 --- a/stacks/fuel-monitoring/superset/init.sh +++ b/stacks/fuel-monitoring/superset/init.sh @@ -10,23 +10,6 @@ set -euo pipefail superset db upgrade superset init -# Trino was removed on 29 Oct 2026. An instance still holding its connection -# "Fuel (Trino)" gets it renamed, so its datasets and dashboards keep their ids; -# the next step points it at fuel-db. Drop this once no instance has it. -/app/.venv/bin/python - <<'PY' -from superset.app import create_app - -with create_app().app_context(): - from superset import db - from superset.models.core import Database - - old = db.session.query(Database).filter_by(database_name="Fuel (Trino)").one_or_none() - if old and not db.session.query(Database).filter_by(database_name="Fuel").count(): - old.database_name = "Fuel" - db.session.commit() - print('renamed the connection "Fuel (Trino)" to "Fuel"') -PY - # No password here: superset_config.py reads it from /run/secrets at connect time. superset set-database-uri \ --database_name "Fuel" \