Trino out of the code; commit messages name bands and rules
The rename step in init.sh and the guard in deploy/lib.sh are gone. README's procedure is now "Removing the old project"; it keeps the old names until it has run on the server. Migrations stay here (open point 8), and their README says so. AGENTS.md adds the commit-message rule. Not deployed: live still runs Trino. Bands: 4 (Superset), persistence (docs). Rules: 2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
8405706ed8
commit
e4806f5799
9 files changed
+55
-67
No files matched your search
@@ -15,6 +15,8 @@ keep the rules there; a change against a rule or an open conflict goes to the ow
|
||||
- `keycloak/`: the `petrolyst` client import.
|
||||
|
||||
## Rules
|
||||
- **Commit messages** end their body with the bands and rules touched, for example
|
||||
`Bands: 4 (server). Rules: 2, 3.`, or `Bands: none (docs).` for text only (architecture rule 1).
|
||||
- **Deploys:** only by README "Manual deploy", only when the owner asks. There is no CI. Commit on
|
||||
`main` first; never push or deploy unprompted.
|
||||
- **No secrets** in the repository; `deploy/deploy.sh` refuses if one appears.
|
||||
|
||||
@@ -6,7 +6,7 @@ Everything that runs on the petrolyst server (109.199.127.84, Ubuntu 24.04):
|
||||
https://traefik.petrolyst.com, behind Keycloak.
|
||||
- **fuel-monitoring** (ATOMA): Superset at https://atoma.petrolyst.com, reading the operational
|
||||
store `fuel-db` (Postgres 17 with TimescaleDB) directly, as the read-only role `fuel_ro`
|
||||
(Trino was removed on 29 Oct 2026). Sign-in goes through Keycloak
|
||||
(no Trino since 29 Oct 2026). Sign-in goes through Keycloak
|
||||
client `petrolyst`, and the UI carries the Petrolyst brand.
|
||||
|
||||
**Deploys are manual.** A person deploys a clean commit of `main` from the Mac ("Manual deploy"
|
||||
@@ -245,18 +245,14 @@ Tables it creates, hypertables included, then belong to `fuel_owner`, and `fuel_
|
||||
After `SET ROLE NONE`, it cannot create anything in `public`. To repair a table that another
|
||||
role created, run `ALTER TABLE <t> OWNER TO fuel_owner`, then `GRANT SELECT ON <t> TO fuel_ro`.
|
||||
|
||||
**One-time steps after 29 Oct 2026**, at the first deploy that includes them. Two changes
|
||||
meet there:
|
||||
- Trino is removed: Superset reads fuel-db directly. The role `trino_ro` becomes `fuel_ro` and
|
||||
keeps its password (a SCRAM password survives a rename); its secret file moves with it.
|
||||
- The Compose project `fuel-monitoring` is renamed `petrolyst`. Containers are then called
|
||||
`petrolyst-<service>-1`. The data volumes keep their names (`fuel-monitoring_superset_db_data`,
|
||||
`fuel-monitoring_fuel_db_data`; pinned in `compose.yaml`), so no data moves. The old project's
|
||||
containers must be gone first: two PostgreSQL servers on one data volume corrupt it.
|
||||
Preflight refuses to deploy while any of them exists.
|
||||
**Removing the old project**, once, right before the next deploy; delete this paragraph when it
|
||||
is done. The server still runs the Compose project `fuel-monitoring`, with Trino. Afterwards it runs
|
||||
`petrolyst`, without it: containers are `petrolyst-<service>-1`, and the data volumes keep their
|
||||
names (pinned in `compose.yaml`), so no data moves. The old containers must be gone first: two
|
||||
PostgreSQL servers on one data volume corrupt it, and preflight refuses to deploy while any exists.
|
||||
|
||||
As root on the server, right before the deploy (Superset is down from step 3 until the deploy
|
||||
finishes):
|
||||
As root on the server (Superset is down from step 3 until the deploy finishes). The role keeps its
|
||||
password, because a SCRAM password survives a rename:
|
||||
|
||||
```bash
|
||||
S=/opt/petrolyst/secrets/fuel-monitoring
|
||||
@@ -267,20 +263,35 @@ docker exec -i fuel-monitoring-fuel-db-1 psql -U postgres -d fuel -v ON_ERROR_ST
|
||||
docker rm -f $(docker ps -aq --filter label=com.docker.compose.project=fuel-monitoring) # 3
|
||||
```
|
||||
|
||||
4. Deploy as usual ("Manual deploy"). Compose starts the `petrolyst` project on the same volumes
|
||||
and no longer starts Trino. Compose warns once that the volumes belong to another project;
|
||||
that is expected. `superset-init` renames the connection "Fuel (Trino)" to "Fuel" and points
|
||||
it at fuel-db; the smoke test checks that it reads as `fuel_ro`, read-only.
|
||||
5. Clean up: `rm /opt/petrolyst/env/fuel-monitoring.env`, `rmdir /opt/petrolyst/env`,
|
||||
4. Deploy as usual ("Manual deploy"). Compose warns once that the volumes belong to another project;
|
||||
that is expected. `superset-init` registers the connection "Fuel"; the smoke test checks that it
|
||||
reads as `fuel_ro`, read-only.
|
||||
5. Delete the old connection. It refuses if a dataset still uses it; then stop and look:
|
||||
|
||||
```bash
|
||||
docker exec -i petrolyst-superset-1 /app/.venv/bin/python - <<'PY'
|
||||
from superset.app import create_app
|
||||
|
||||
with create_app().app_context():
|
||||
from superset import db
|
||||
from superset.connectors.sqla.models import SqlaTable
|
||||
from superset.models.core import Database
|
||||
|
||||
old = db.session.query(Database).filter_by(database_name="Fuel (Trino)").one_or_none()
|
||||
if old:
|
||||
used = db.session.query(SqlaTable).filter_by(database_id=old.id).count()
|
||||
if used:
|
||||
raise SystemExit(f"{used} datasets still use the old connection")
|
||||
db.session.delete(old)
|
||||
db.session.commit()
|
||||
print("old connection deleted")
|
||||
PY
|
||||
```
|
||||
6. Clean up: `rm /opt/petrolyst/env/fuel-monitoring.env`, `rmdir /opt/petrolyst/env`,
|
||||
`docker image rm trinodb/trino:484`,
|
||||
`docker network rm fuel-monitoring_backend fuel-monitoring_fuel-db-tunnel`.
|
||||
|
||||
**Rollback across the Trino removal** (not tested). `rollback.sh` stops for a release from before
|
||||
it. To go back anyway: remove the `petrolyst` project's containers, reverse steps 1 and 2 above (rename `fuel_ro` back to `trino_ro` and
|
||||
`ALTER ROLE trino_ro RESET default_transaction_read_only`, move the secret file back, write
|
||||
`/opt/petrolyst/env/fuel-monitoring.env` with `FUEL_DB_TRINO_PASSWORD=<that secret>`), rename
|
||||
the Superset connection "Fuel" back to "Fuel (Trino)" with `trino://superset@trino:8080/fuel`,
|
||||
then run that release's own script, `/opt/petrolyst/releases/<sha>/deploy/rollback.sh <sha>`.
|
||||
7. Releases from before this deploy can no longer run, as their role is gone, and `rollback.sh`
|
||||
stops on them. Remove them from `/opt/petrolyst/releases/`.
|
||||
|
||||
**Adding a project**: create `stacks/<name>/compose.yaml`, attach the public service to the
|
||||
external network `proxy`, and set `traefik.*` labels with a unique router name and
|
||||
@@ -299,8 +310,6 @@ its own gate in its stack:
|
||||
|
||||
## Verified locally (2026-10-23)
|
||||
|
||||
Before Trino's removal (29 Oct 2026): its lines below are history.
|
||||
|
||||
An end-to-end run ran the real `deploy.sh` and `rollback.sh` in a Linux container against
|
||||
Docker Desktop, with a scratch `/opt/petrolyst`. Production differences in that run:
|
||||
- a test CA certificate stood in for Let's Encrypt;
|
||||
@@ -310,7 +319,6 @@ Docker Desktop, with a scratch `/opt/petrolyst`. Production differences in that
|
||||
Results:
|
||||
- The fresh install took the init path.
|
||||
- An identical re-run recreated nothing and left `ARTIFACTS` byte-identical.
|
||||
- A Trino-only change recreated only `trino`.
|
||||
- A Superset change stopped, backed up, migrated and recreated only `superset`.
|
||||
- A change to `superset/init.sh` alone did the same, and the changed script ran.
|
||||
- A same-SHA upload with different content was refused.
|
||||
@@ -327,13 +335,12 @@ Results:
|
||||
- Retention pruned read-only releases.
|
||||
- Retention removed a queued upload that a plain `rsync -a` had given an old checkout's
|
||||
directory times. It kept one uploaded with `--omit-dir-times`.
|
||||
- In `fuel-db`, a loader that created a table as its own role left it unreadable for `trino_ro`.
|
||||
With the "Data loaders" recipe, tables (hypertables too) belonged to `fuel_owner` and Trino
|
||||
could read them. After `SET ROLE NONE`, the loader could not create anything.
|
||||
- In `fuel-db`, a loader that created a table as its own role left it unreadable for the read-only
|
||||
role. With the "Data loaders" recipe, tables (hypertables too) belonged to `fuel_owner` and that
|
||||
role could read them. After `SET ROLE NONE`, the loader could not create anything.
|
||||
- Smoke tests passed, including the redirects to the real Keycloak with `client_id=petrolyst`
|
||||
and PKCE S256.
|
||||
- TLS 1.1 and unknown SNI were refused, and HTTP redirected to HTTPS.
|
||||
- Trino and `trino_ro` both refused writes.
|
||||
|
||||
Not verified locally:
|
||||
- Let's Encrypt issuance;
|
||||
|
||||
+1
-6
@@ -94,11 +94,6 @@ load_artifacts() {
|
||||
while IFS= read -r line; do
|
||||
key=${line%%=*}
|
||||
value=${line#*=}
|
||||
# A release from before 29 Oct 2026 still runs Trino, which logs in as
|
||||
# `trino_ro`, since renamed `fuel_ro`: it cannot start (README "Rollback
|
||||
# across the Trino removal").
|
||||
[ "$key" != TRINO_CONFIG_HASH ] ||
|
||||
die "$file is from before Trino was removed; see README \"Rollback across the Trino removal\""
|
||||
[[ $key =~ ^(SUPERSET_IMAGE_TAG|TRAEFIK_CONFIG_HASH)$ && $value =~ ^[0-9a-f]{16}$ ]] ||
|
||||
die "unexpected line in $file: $line"
|
||||
printf -v "$key" '%s' "$value"
|
||||
@@ -143,7 +138,7 @@ preflight() {
|
||||
# The stack was the Compose project "fuel-monitoring" until 29 Oct 2026; "petrolyst" uses the
|
||||
# same data volumes, so both must never run at once (README "One-time steps after 29 Oct 2026").
|
||||
[ -z "$("${DOCKER[@]}" ps -aq --filter label=com.docker.compose.project=fuel-monitoring)" ] ||
|
||||
die "containers of the old project fuel-monitoring exist; remove them first (README \"One-time steps after 29 Oct 2026\")"
|
||||
die "containers of the old project fuel-monitoring exist; remove them first (README \"Removing the old project\")"
|
||||
}
|
||||
|
||||
# promote <release> <sha>: make the release the live tree. --chmod applies to
|
||||
|
||||
@@ -16,7 +16,7 @@ and where it runs lives in `stacks/`, `traefik/`, `deploy/` and `host/`.
|
||||
| `superset_config.py` | Keycloak sign-in, role mapping, proxy and cookie settings, Redis caches, theme and chart palettes | `/app/pythonpath/` |
|
||||
| `petrolyst_security.py` | `PetrolystSecurityManager`: user info and roles from the Keycloak ID token, `/login/` straight to Keycloak, `/logout/` through Keycloak | `/app/pythonpath/` |
|
||||
| `roles/` | `atoma_guest.py`: the role "ATOMA Guest" for the Petrolyst app's embedded dashboards, with exactly its permissions; run by hand (see "Embedding") | `/app/petrolyst-roles/` |
|
||||
| `assets/branding/` | logos (full colour and white): the wordmark alone for the navbar, and the stacked logo; favicons, app icons; ATOMA's logo, which the Petrolyst app shows in its map's corner (`PETROLYST_OPERATOR_LOGO`) | `/app/superset/static/assets/branding/`, served at `/static/assets/branding/…` |
|
||||
| `assets/branding/` | logos (full colour and white): the wordmark alone for the navbar, and the stacked logo; favicons, app icons; ATOMA's logo, which the Petrolyst app shows in its map's corner (the server reads this file itself: `PETROLYST_OPERATOR_LOGO_FILE`) | `/app/superset/static/assets/branding/`, served at `/static/assets/branding/…` |
|
||||
| `patches/` | runtime patches, `fix_*.py` with `--verify` (see its README); none at present | `/app/patches/petrolyst/`, applied during the build |
|
||||
| `dashboards/` | exported dashboards and import scripts (see its README) | not in the image |
|
||||
|
||||
|
||||
@@ -16,8 +16,7 @@ them in Superset with the filter column.
|
||||
|
||||
**Datasets.** The `public.v_*` views of
|
||||
`stacks/fuel-monitoring/fuel-db/migrations/0003_bi_views.sql`, read from fuel-db directly (database
|
||||
`fuel`, schema `public`) as the read-only role `fuel_ro`. *Until 29 Oct 2026 Superset read them
|
||||
through Trino as `fuel.public`; Trino was removed then.*
|
||||
`fuel`, schema `public`) as the read-only role `fuel_ro`.
|
||||
|
||||
**Design and rationale** live in the fuel-monitoring corpus, in
|
||||
`.context/plans/2026-10-24-fuel-db-baseline-dashboard.md`:
|
||||
@@ -259,7 +258,7 @@ across methods or generator sets. The site's L/h figure itself comes only from a
|
||||
(petrolyst-platform `views/src/shared/deepLink.ts`, retired) from its iframe address: the
|
||||
dashboard path, the Site filter's id and the tab. The Petrolyst app uses the same rison shape for
|
||||
the Site filter's preset (`app/src/app/core/site-filter.ts`). The rison shape is the one the sites
|
||||
list used until 28 Oct, confirmed in P2, and the view's tests compared it with that list's Trino
|
||||
list used until 28 Oct, confirmed in P2, and the view's tests compared it with that list's
|
||||
output:
|
||||
|
||||
```
|
||||
@@ -409,7 +408,7 @@ Run in P2 against a local Superset, and repeated as a smoke test in P3.
|
||||
|
||||
## 8. P2 results (local, 24 Oct 2026)
|
||||
|
||||
Local Superset 6.1.0 image built from this repository, Trino 484, fuel-db with migrations
|
||||
Local Superset 6.1.0 image built from this repository, fuel-db with migrations
|
||||
0001–0003 and the corpus data loaded; browser checks in headless Chrome.
|
||||
|
||||
| Check | Result |
|
||||
|
||||
@@ -46,7 +46,7 @@ SQLALCHEMY_DATABASE_URI = URL.create(
|
||||
).render_as_string(hide_password=False)
|
||||
|
||||
# --- The data: fuel-db ----------------------------------------------------------
|
||||
# Superset reads fuel-db directly as the read-only role fuel_ro (no Trino since 29 Oct 2026);
|
||||
# Superset reads fuel-db directly as the read-only role fuel_ro;
|
||||
# superset-init registers that connection, "Fuel", without a password. Passwords of data
|
||||
# connections come from secret files at connect time, so none is kept in Superset's own
|
||||
# database. A connection added in the UI gets no password this way.
|
||||
|
||||
@@ -1,6 +1,5 @@
|
||||
# Fuel-monitoring analytics: Superset on atoma.petrolyst.com, reading the
|
||||
# operational store (fuel-db) directly as the read-only role fuel_ro (Trino was
|
||||
# removed on 29 Oct 2026). Run only through deploy/deploy.sh or
|
||||
# operational store (fuel-db) directly as the read-only role fuel_ro. Run only through deploy/deploy.sh or
|
||||
# deploy/rollback.sh: they export SUPERSET_IMAGE_TAG from the release's
|
||||
# ARTIFACTS file, and the :? guard below stops Compose from running without it.
|
||||
name: petrolyst
|
||||
|
||||
@@ -18,6 +18,12 @@ The design and its rationale live in the fuel-monitoring corpus:
|
||||
|
||||
Each file is one transaction. Apply them in order.
|
||||
|
||||
**Who depends on the views.** Superset's dashboards list every column (`petrolyst-insights/dashboards/atoma/columns.tsv`;
|
||||
the import reports drift). The Petrolyst server reads `public.v_site`; its integration test
|
||||
(petrolyst-platform `server/tests/test_integration.py`) names the columns and types it needs. After
|
||||
a change to `0003_bi_views.sql`, run both checks. The migrations stay in this repository (open
|
||||
point 8, closed 30 Oct 2026).
|
||||
|
||||
## Applying (on the server)
|
||||
|
||||
1. Create a loader role as in the repository README, "Data loaders". Its sessions act as
|
||||
@@ -92,12 +98,11 @@ it, except baseline versions and their approvals, which live only in fuel-db: ex
|
||||
|
||||
## Verified locally (2026-10-24, scratch only)
|
||||
|
||||
With Trino, before its removal on 29 Oct 2026; `trino_ro` is now `fuel_ro`.
|
||||
The reads below ran through Trino, removed since on 29 Oct 2026, as today's `fuel_ro`.
|
||||
|
||||
Setup:
|
||||
- `timescale/timescaledb:2.31.0-pg17`, with this repository's `01-roles.sh`;
|
||||
- a loader created by the README recipe;
|
||||
- `trinodb/trino:484`, with this repository's `trino/etc` connected as `trino_ro`.
|
||||
|
||||
Results:
|
||||
- **Apply:** all three files applied as the loader, under the loader's default `search_path`.
|
||||
@@ -117,9 +122,7 @@ Results:
|
||||
the version's validity, corrections under one `event_key`, fixed meter and epoch owners, and
|
||||
population keys that name the generators a coverage assumption covers;
|
||||
- every refusal listed above.
|
||||
- **Trino:**
|
||||
- all 303 columns of the 18 views have identical names, order and types in PostgreSQL and
|
||||
Trino;
|
||||
- **Read-only role:**
|
||||
- `fm` is unreadable: a query is refused, though its names are listed (see Applying);
|
||||
- writes are refused.
|
||||
- **Not run on the server:** none of this has been run against the live fuel-db.
|
||||
@@ -10,23 +10,6 @@ set -euo pipefail
|
||||
superset db upgrade
|
||||
superset init
|
||||
|
||||
# Trino was removed on 29 Oct 2026. An instance still holding its connection
|
||||
# "Fuel (Trino)" gets it renamed, so its datasets and dashboards keep their ids;
|
||||
# the next step points it at fuel-db. Drop this once no instance has it.
|
||||
/app/.venv/bin/python - <<'PY'
|
||||
from superset.app import create_app
|
||||
|
||||
with create_app().app_context():
|
||||
from superset import db
|
||||
from superset.models.core import Database
|
||||
|
||||
old = db.session.query(Database).filter_by(database_name="Fuel (Trino)").one_or_none()
|
||||
if old and not db.session.query(Database).filter_by(database_name="Fuel").count():
|
||||
old.database_name = "Fuel"
|
||||
db.session.commit()
|
||||
print('renamed the connection "Fuel (Trino)" to "Fuel"')
|
||||
PY
|
||||
|
||||
# No password here: superset_config.py reads it from /run/secrets at connect time.
|
||||
superset set-database-uri \
|
||||
--database_name "Fuel" \
|
||||
|
||||
Reference in new issue
Block a user