petrolyst-infra/README.md

petrolyst-infra

Everything that runs on the petrolyst server (109.199.127.84, Ubuntu 24.04):

• Traefik at the edge for every project on the server. Its dashboard is at https://traefik.petrolyst.com, behind Keycloak.
• fuel-monitoring (ATOMA): Superset at https://atoma.petrolyst.com, reading the operational store fuel-db (Postgres 17 with TimescaleDB) directly, as the read-only role fuel_ro (no Trino since 29 Oct 2026). Sign-in goes through Keycloak client petrolyst, and the UI carries the Petrolyst brand.

Deploys are manual. A person deploys a clean commit of main from the Mac ("Manual deploy" under Operating). There is no CI runner, and a push to Gitea deploys nothing.

Customization and deployment are separate, as with media-insights and media-intelligence-docker:

Folder	Holds
petrolyst-insights/	how Superset looks and behaves for Petrolyst: config, Keycloak login code, branding, patches, dashboards. See its README.
traefik/	edge proxy, TLS, the sso middleware (oauth2-proxy) that guards the dashboard
stacks/fuel-monitoring/	Compose stack, Superset Dockerfile and init, fuel-db roles and migrations
deploy/	deploy.sh (run on the server by a manual deploy), rollback.sh (manual), lib.sh (shared)
host/	one-time server setup: bootstrap.sh, secrets-init.sh
keycloak/	the clients petrolyst and petrolyst-app for Keycloak's partial import

No credential is in this repository. Secrets exist only on the server, under /opt/petrolyst/secrets/. A deploy needs only root SSH access to the server (ssh petrolyst).

How a deploy works

1. A person runs "Manual deploy" (Operating, below) from a clean checkout of main: the scripts are linted, git archive sends exactly that commit to a fresh /opt/petrolyst/incoming/<sha>-manual-<time>/, and deploy/deploy.sh <sha> <that dir> runs there as user deploy (root hands over with sudo -u deploy).
2. deploy.sh takes the host lock /opt/petrolyst/deploy.lock, which it shares with rollback.sh, and keeps it until it exits.
3. It finalizes the upload. A new commit is moved to releases/<sha>, and the release is made read-only. A re-run of a deployed commit must match it byte for byte; a mismatch aborts.
4. It resolves the artifacts and writes them to releases/<sha>/ARTIFACTS:
   • SUPERSET_IMAGE_TAG is a hash of exactly the Superset image's build inputs. They include superset/init.sh, so changing that file runs superset-init again.
   • TRAEFIK_CONFIG_HASH is a hash of Traefik's bind-mounted static config.
   • Compose recreates a container only when its value changes.
5. Preflight, before anything live changes:
   • every secret file exists;
   • petrolyst-insights/ holds no deployment files;
   • docker compose config passes for every stack;
   • the Superset image is built if it is new;
   • the downgrade guard refuses an image older than the metadata database's schema.
6. It promotes the release to /opt/petrolyst/infra, then starts Traefik, then each stack under stacks/.
7. For Superset, it reads the metadata database's state:
   • An empty database is a fresh install: it runs superset-init.
   • If the image changed, or the database is not at the image's migration head: it stops superset, writes backups/superset-meta_<old tag>_<revision>_<time>.dump, then runs superset-init (db upgrade, init, registers "Fuel": fuel-db as fuel_ro, its password read from /run/secrets at connect time, never stored in Superset).
   • If the image is unchanged and the schema is at its head, it does none of this.
8. Smoke tests. A failure ends the deploy with an error and prints the stacks' state.
   • /health answers through Traefik with a valid certificate.
   • /login/ leads to Keycloak with client_id=petrolyst and PKCE.
   • The dashboard redirects to Keycloak.
   • Superset's "Fuel" connection reads fuel-db as fuel_ro, and its transactions are read-only.
9. Retention:
   • 5 releases;
   • backups for 30 days;
   • the 3 newest Superset images, plus any image that a kept release or backup names;
   • uploads older than a day.

The manual deploy then checks both hosts from outside.

Server paths

Path	Owner	What
/opt/petrolyst/infra	deploy	live tree (the promoted release)
/opt/petrolyst/incoming/<sha>-manual-<time>	deploy	one upload; only deploy.sh turns it into a release
/opt/petrolyst/releases/<sha>	deploy, read-only	finalized release plus ARTIFACTS; the last 5 are kept
/opt/petrolyst/secrets/{shared,traefik,fuel-monitoring}/	root:deploy 0750, files 0444	secrets, mounted as /run/secrets/<name>
/opt/petrolyst/backups/	deploy 0750	Superset metadata dumps
/opt/petrolyst/state/	deploy	last deployed SHA and Superset image tag; rollback-in-progress during a migration rollback

First-time setup

1. Gitea

Create the private repository Petrolyst/petrolyst-infra and push this repository's main to it. Gitea only keeps the history: deploys are manual, so it needs no runner, no Actions and no deploy secrets.

2. Keycloak (realm petrolyst on iam.thedigitalbridge.io)

1. Realm settings → Action → Partial import: choose keycloak/petrolyst-realm-import.json, and "Skip" if a resource exists. This creates:

   • the confidential client petrolyst, for Superset and Traefik: standard flow only, PKCE S256, redirect URIs for both hosts;
   • its client roles superset-admin, superset-editor, superset-viewer and traefik-admin;
   • a mapper that puts those roles into the claim petrolyst_roles;
   • the confidential client petrolyst-app, for the Petrolyst app, with its role reader and two mappers: petrolyst_roles, and the person's groups with their full path in petrolyst_groups, both in the ID token. Both clients' addresses are on app.petrolyst.com, where everything moves (PD-16, PD-21); atoma.petrolyst.com is going away and is not in the import.

2. Clients → petrolyst → Credentials: copy the client secret (step 3 stores it on the server). Realm settings: registration off; brute-force detection on; a password policy of at least 12 characters, because a password is the only lock for customers' users; SSO session idle 8 hours, max 10 hours, because the app's session is renewed through Keycloak about hourly and the default idle limit of 30 minutes would ask for a password each time.

3. MFA, which Exhibit B requires. A copy of the built-in browser flow is not enough, for two reasons: its Cookie step accepts an existing password-only session from another client of the realm, and its conditional OTP subflow skips users who have not set up OTP. Instead:

   • Authentication → Create flow petrolyst-mfa (basic flow).
   • Add exactly two executions, both Required, in this order: Username Password Form, then OTP Form. Add no Cookie, no Identity Provider Redirector and no conditional subflow.
   • Clients → petrolyst → Advanced → Authentication flow overrides → Browser flow: petrolyst-mfa.

   Result: every sign-in to Superset or Traefik asks for password and OTP, and a user without OTP enrols first. There is no silent SSO between them. Client petrolyst-app keeps the realm's own flow: customers' users sign in with a password only, the owner's choice "for now" (PD-21).

4. A customer is a top-level group. Groups → Create group, named as the customer's keycloak_group in fuel-db (without the slash), then Role mapping → Assign role → filter by clients → petrolyst-app reader. Adding a person to the group is then the only step, and they see that customer in the app and no other. Exactly one customer per person: two give "No access". They need no role on client petrolyst, and Superset refuses them. A person in no customer's group sees "No access"; that includes our own staff, who join a customer's group to see it in the app. Administrators: Users → (user) → Role mapping → Assign role → filter by clients → petrolyst.

5. Users of the Traefik dashboard need Email verified switched on, because oauth2-proxy refuses unverified addresses.

6. Branding (optional): the sign-in and sign-out pages, theme petrolyst in keycloak/themes/, for the whole realm, both clients.

   • Build on the Mac: keycloak/themes/build.sh <folder with Satoshi-Variable.woff2>, normally petrolyst-platform/web/src/fonts/satoshi. The font goes only into dist/petrolyst-theme.jar, never into Git, because its licence forbids distributing it: so build it yourself and do not hand the file to outsiders.
   • Install on Keycloak's server, either way. No restart: put the jar's theme/petrolyst/ folder into Keycloak's themes/ folder, readable by Keycloak's user. Keycloak listed and used it without a restart on a test 26.4 with theme caching on; not tried on the real one. With a restart, which drops sign-in for every realm for a moment: the jar into providers/, and kc.sh build if it runs an optimised build.
   • Realm petrolyst → Realm settings → Themes → Login theme → petrolyst → Save. Other realms keep their look. To undo, set it back.
   • keycloak/themes/check-brand.sh says whether the theme's copies of the brand tokens still match Petrolyst/design/.

3. Server (as root, ssh petrolyst)

The server needs only the two host scripts; it never gets Gitea credentials.

# on the Mac, in this repository
scp host/bootstrap.sh host/secrets-init.sh petrolyst:/root/
ssh petrolyst
# on the server, as root
/root/bootstrap.sh
/root/secrets-init.sh                   # generates every secret; reports the missing Keycloak one
/root/secrets-init.sh keycloak-secret   # paste the secret from Keycloak step 2

bootstrap.sh does the following:

• installs Docker from Docker's apt repository;
• adds 4 GB swap;
• turns on ufw for 22, 80 and 443 only;
• turns off SSH password login and root password login;
• creates the deploy user, which runs deploy.sh. It has no SSH key and is reached only through sudo -u deploy from root; a re-run removes the CI key an earlier version installed;
• creates /opt/petrolyst and the Docker network proxy.

Before closing your root session, check from a second terminal that ssh petrolyst still works.

4. First deploy

Use "Manual deploy" below. The first run builds the Superset image and takes the fresh-install path. Let's Encrypt needs about a minute for the two certificates; the smoke tests wait up to two minutes.

Then sign in at https://atoma.petrolyst.com with a user who has superset-admin. That person becomes the first Superset admin; there is no local admin account.

Operating

Task	How
Deploy	"Manual deploy" below, from a clean main
Re-deploy a kept release	sudo -u deploy /opt/petrolyst/infra/deploy/deploy.sh <sha>. Not across a change of Keycloak realm: see "Moving to realm petrolyst"
Roll back	see below
Traefik dashboard	https://traefik.petrolyst.com (role traefik-admin)
fuel-db shell	docker exec -it petrolyst-fuel-db-1 psql -U postgres -d fuel
fuel-db from the Mac	ssh -L 5433:127.0.0.1:5433 petrolyst, then connect to localhost:5433, database fuel
Logs	docker compose -p petrolyst logs -f superset (or -p traefik)
Superset break-glass	docker exec -it petrolyst-superset-1 superset fab ...

Manual deploy, the only way to deploy (there is no CI runner). On the Mac, in this repository:

1. Commit on main, so that git status is clean. The live commit is in /opt/petrolyst/state/deployed-sha (ssh petrolyst cat /opt/petrolyst/state/deployed-sha).

2. Lint the shell scripts. Warnings and errors must be zero; the info notes on the deliberate test && test || die guards in deploy/lib.sh (SC2015) are filtered out:

   docker run --rm -v "$PWD:/mnt" -w /mnt koalaman/shellcheck:v0.10.0 -S warning -x \
     host/*.sh deploy/*.sh stacks/*/superset/init.sh stacks/*/fuel-db/init/*.sh
   

3. Deploy:

   sha=$(git rev-parse HEAD)
   dir=/opt/petrolyst/incoming/$sha-manual-$(date -u +%s)
   git archive --format=tar "$sha" |
     ssh petrolyst "install -d -o deploy -g deploy -m 0755 $dir && tar -x --no-same-owner -C $dir && chown -R deploy:deploy $dir"
   ssh petrolyst "sudo -u deploy $dir/deploy/deploy.sh $sha $dir"
   

4. Check both hosts from outside:

   test "$(curl -fsS --max-time 20 https://atoma.petrolyst.com/health)" = OK && echo atoma ok
   curl -fsS --max-time 20 -o /dev/null -w '%{http_code}\n' https://traefik.petrolyst.com/   # 302
   

5. Push main to Gitea, so that the repository matches what is live.

git archive sends exactly the commit, never uncommitted or ignored files. The server runs deploy.sh under the host lock it shares with rollback.sh, and records the live commit in /opt/petrolyst/state/deployed-sha.

Rollback (deploy/rollback.sh, manual, under the same lock as deploys):

ls /opt/petrolyst/releases                         # kept releases
sudo -u deploy /opt/petrolyst/infra/deploy/rollback.sh <sha>

• If the target image's migration head equals the database's revision, this is a code-only rollback: promote and restart.

• Otherwise it is a migration rollback:

  1. stop superset;
  2. drop and recreate the superset metadata database;
  3. restore the backup made for the target image, aborting on any error;
  4. check the restored revision;
  5. start the retained older image.

  It refuses if that backup or image is gone. fuel-db is never touched.

  Before step 1 it writes state/rollback-in-progress, and it removes that file only after the smoke tests pass. While the file exists, deploy.sh refuses to run. If a rollback is interrupted, run the same command again. It also restores when the failed attempt left the database empty or had already dropped it.

Afterwards, git revert the bad commit and push, so main matches. A plain revert without the rollback script is stopped by the downgrade guard.

Data loaders (fuel-db). Every table in public must belong to fuel_owner. fuel_ro can read new tables only through fuel_owner's default privileges, and PostgreSQL applies those only to objects fuel_owner creates itself. A table that a loader creates as its own role stays unreadable for Superset and the Petrolyst server. Create each loader role so that its sessions act as fuel_owner, and it cannot create anything as itself. Run this in docker exec -it petrolyst-fuel-db-1 psql -U postgres -d fuel:

CREATE ROLE <loader> LOGIN PASSWORD '<password>';
GRANT fuel_owner TO <loader> WITH INHERIT FALSE, SET TRUE;
ALTER ROLE <loader> SET role TO 'fuel_owner';

Tables it creates, hypertables included, then belong to fuel_owner, and fuel_ro can read them. After SET ROLE NONE, it cannot create anything in public. To repair a table that another role created, run ALTER TABLE <t> OWNER TO fuel_owner, then GRANT SELECT ON <t> TO fuel_ro.

Removing the old project, once, right before the next deploy; delete this paragraph when it is done. The server still runs the Compose project fuel-monitoring, with Trino. Afterwards it runs petrolyst, without it: containers are petrolyst-<service>-1, and the data volumes keep their names (pinned in compose.yaml), so no data moves. The old containers must be gone first: two PostgreSQL servers on one data volume corrupt it, and preflight refuses to deploy while any exists.

As root on the server (Superset is down from step 3 until the deploy finishes). The role keeps its password, because a SCRAM password survives a rename:

S=/opt/petrolyst/secrets/fuel-monitoring
mv "$S/fuel_db_trino_password" "$S/fuel_db_ro_password"                       # 1
docker exec -i fuel-monitoring-fuel-db-1 psql -U postgres -d fuel -v ON_ERROR_STOP=1 \
  -c 'ALTER ROLE trino_ro RENAME TO fuel_ro' \
  -c 'ALTER ROLE fuel_ro SET default_transaction_read_only = on'               # 2
docker rm -f $(docker ps -aq --filter label=com.docker.compose.project=fuel-monitoring)   # 3

4. Deploy as usual ("Manual deploy"). Compose warns once that the volumes belong to another project; that is expected. superset-init registers the connection "Fuel"; the smoke test checks that it reads as fuel_ro, read-only.

5. Delete the old connection. It refuses if a dataset still uses it; then stop and look:

   docker exec -i petrolyst-superset-1 /app/.venv/bin/python - <<'PY'
   from superset.app import create_app
   
   with create_app().app_context():
       from superset import db
       from superset.connectors.sqla.models import SqlaTable
       from superset.models.core import Database
   
       old = db.session.query(Database).filter_by(database_name="Fuel (Trino)").one_or_none()
       if old:
           used = db.session.query(SqlaTable).filter_by(database_id=old.id).count()
           if used:
               raise SystemExit(f"{used} datasets still use the old connection")
           db.session.delete(old)
           db.session.commit()
           print("old connection deleted")
   PY
   

6. Clean up: rm /opt/petrolyst/env/fuel-monitoring.env, rmdir /opt/petrolyst/env, docker image rm trinodb/trino:484, docker network rm fuel-monitoring_backend fuel-monitoring_fuel-db-tunnel.

7. Releases from before this deploy can no longer run, as their role is gone, and rollback.sh stops on them. Remove them from /opt/petrolyst/releases/.

Moving to realm petrolyst, one time. A deploy of its own, after the deploy that removes Trino: releases from before that one can no longer run, so a combined deploy leaves nothing to roll back to.

1. The realm is ready (Keycloak, above) and an admin has signed in on client petrolyst there. While Superset still answers at atoma.petrolyst.com, add that address's redirect URI to the client by hand, or nobody can sign in after step 3; remove it when the domain goes.
2. As root: host/secrets-init.sh keycloak-secret. The value goes into the new file shared/keycloak_petrolyst_realm_client_secret, so nothing changes for anyone yet. Leave the old file shared/keycloak_petrolyst_client_secret alone.
3. Deploy by "Manual deploy" only. That one step moves Superset and Traefik together.
4. Prove it by hand: an admin signs in to Superset and to the Traefik dashboard. The smoke tests check redirects only, and pass with a wrong secret.
5. To go back: rollback.sh <previous sha> returns to the old realm with the old secret. Afterwards go forward again by "Manual deploy" only: "Re-deploy a kept release" runs the old lib.sh, whose realm address would override the new release's.
6. Not before 7 days of working sign-ins, and on the owner's word: retire client petrolyst in realm thedigitalbridge and delete the old secret file. From then on there is no way back.

Adding a project: create stacks/<name>/compose.yaml, attach the public service to the external network proxy, and set traefik.* labels with a unique router name and tls.certresolver: le. Add the DNS record first. deploy.sh picks the new folder up automatically.

sso@docker works only for traefik.petrolyst.com. Its cookie, its allowed return hosts and its role (traefik-admin) are all tied to the dashboard. An app without a login of its own needs its own gate in its stack:

• an oauth2-proxy service with its own cookie secret;
• a router for the app host's /oauth2/ path and a forwardAuth middleware pointing at that service, as in traefik/compose.yaml;
• --redirect-url, --cookie-domain and --whitelist-domain set to the app's host;
• an --allowed-role with a new client role in Keycloak;
• the app's /oauth2/callback added to the petrolyst client's redirect URIs.

Verified locally (2026-10-23)

An end-to-end run ran the real deploy.sh and rollback.sh in a Linux container against Docker Desktop, with a scratch /opt/petrolyst. Production differences in that run:

• a test CA certificate stood in for Let's Encrypt;
• the ACME server in the test copy was unreachable on purpose;
• a dummy Keycloak client secret was used.

Results:

• The fresh install took the init path.
• An identical re-run recreated nothing and left ARTIFACTS byte-identical.
• A Superset change stopped, backed up, migrated and recreated only superset.
• A change to superset/init.sh alone did the same, and the changed script ran.
• A same-SHA upload with different content was refused.
• An older image against a newer schema was refused before promotion.
• A migration rollback dropped a table added by a "newer migration" and restored the revision.
• A code-only rollback brought back the previous image.
• A re-deploy of an unchanged image, against a schema one migration behind its head, migrated the schema back to the head.
• A migration rollback whose restore failed after the drop left rollback-in-progress and an empty database. deploy.sh then refused to run. Running the rollback again restored the database and removed the file. It did the same after a simulated stop between dropping and recreating the database. Without the file, a missing database stops the rollback with an error.
• The host lock blocked a concurrent run.
• Retention pruned read-only releases.
• Retention removed a queued upload that a plain rsync -a had given an old checkout's directory times. It kept one uploaded with --omit-dir-times.
• In fuel-db, a loader that created a table as its own role left it unreadable for the read-only role. With the "Data loaders" recipe, tables (hypertables too) belonged to fuel_owner and that role could read them. After SET ROLE NONE, the loader could not create anything.
• Smoke tests passed, including the redirects to the real Keycloak with client_id=petrolyst and PKCE S256.
• TLS 1.1 and unknown SNI were refused, and HTTP redirected to HTTPS.

Not verified locally:

• Let's Encrypt issuance;
• a real Keycloak sign-in, OTP and logout (they need the client from setup step 2);
• the theme as rendered in a browser.

Open items

• Exhibit B: encryption at rest; confirm the VPS is in a German data centre.
• Nightly pg_dump of both databases with an offsite copy. Today only the pre-migration Superset dumps exist.
• Certificate expiry monitoring. There is no ACME e-mail, so nothing warns if renewal fails between deploys.
• Celery worker and beat, if Superset alerts or reports are needed.
• A Petrolyst login theme for Keycloak itself, which lives on the Keycloak host.