petrolyst-infra/README.md

petrolyst-infra

Everything that runs on the petrolyst server (109.199.127.84, Ubuntu 24.04):

• Traefik at the edge for every project on the server. Its dashboard is at https://traefik.petrolyst.com, behind Keycloak.
• fuel-monitoring (ATOMA): Superset at https://atoma.petrolyst.com, reading the operational store fuel-db (Postgres 17 with TimescaleDB) through Trino. Sign-in goes through Keycloak client petrolyst, and the UI carries the Petrolyst brand.

Deploys are manual. A person deploys a clean commit of main from the Mac ("Manual deploy" under Operating). There is no CI runner, and a push to Gitea deploys nothing.

Customization and deployment are separate, as with media-insights and media-intelligence-docker:

Folder	Holds
petrolyst-insights/	how Superset looks and behaves for Petrolyst: config, Keycloak login code, branding, patches, dashboards. See its README.
traefik/	edge proxy, TLS, the sso middleware (oauth2-proxy) that guards the dashboard
stacks/fuel-monitoring/	Compose stack, Superset Dockerfile, Trino config and catalog, fuel-db roles
deploy/	deploy.sh (run on the server by a manual deploy), rollback.sh (manual), lib.sh (shared)
host/	one-time server setup: bootstrap.sh, secrets-init.sh
keycloak/	the petrolyst client for Keycloak's partial import

No credential is in this repository. Secrets exist only on the server, under /opt/petrolyst/secrets/. A deploy needs only root SSH access to the server (ssh petrolyst).

How a deploy works

1. A person runs "Manual deploy" (Operating, below) from a clean checkout of main: the scripts are linted, git archive sends exactly that commit to a fresh /opt/petrolyst/incoming/<sha>-manual-<time>/, and deploy/deploy.sh <sha> <that dir> runs there as user deploy (root hands over with sudo -u deploy).
2. deploy.sh takes the host lock /opt/petrolyst/deploy.lock, which it shares with rollback.sh, and keeps it until it exits.
3. It finalizes the upload. A new commit is moved to releases/<sha>, and the release is made read-only. A re-run of a deployed commit must match it byte for byte; a mismatch aborts.
4. It resolves the artifacts and writes them to releases/<sha>/ARTIFACTS:
   • SUPERSET_IMAGE_TAG is a hash of exactly the Superset image's build inputs. They include superset/init.sh, so changing that file runs superset-init again.
   • TRINO_CONFIG_HASH and TRAEFIK_CONFIG_HASH are hashes of the bind-mounted config.
   • Compose recreates a container only when its value changes.
5. Preflight, before anything live changes:
   • every secret and env file exists;
   • petrolyst-insights/ holds no deployment files;
   • docker compose config passes for every stack;
   • the Superset image is built if it is new;
   • the downgrade guard refuses an image older than the metadata database's schema.
6. It promotes the release to /opt/petrolyst/infra, then starts Traefik, then each stack under stacks/.
7. For Superset, it reads the metadata database's state:
   • An empty database is a fresh install: it runs superset-init.
   • If the image changed, or the database is not at the image's migration head: it stops superset, writes backups/superset-meta_<old tag>_<revision>_<time>.dump, then runs superset-init (db upgrade, init, registers "Fuel (Trino)").
   • If the image is unchanged and the schema is at its head, it does none of this.
8. Smoke tests. A failure ends the deploy with an error and prints the stacks' state.
   • /health answers through Traefik with a valid certificate.
   • /login/ leads to Keycloak with client_id=petrolyst and PKCE.
   • The dashboard redirects to Keycloak.
   • Trino reads fuel, and Superset reaches Trino.
9. Retention:
   • 5 releases;
   • backups for 30 days;
   • the 3 newest Superset images, plus any image that a kept release or backup names;
   • uploads older than a day.

The manual deploy then checks both hosts from outside.

Server paths

Path	Owner	What
/opt/petrolyst/infra	deploy	live tree (the promoted release)
/opt/petrolyst/incoming/<sha>-manual-<time>	deploy	one upload; only deploy.sh turns it into a release
/opt/petrolyst/releases/<sha>	deploy, read-only	finalized release plus ARTIFACTS; the last 5 are kept
/opt/petrolyst/secrets/{shared,traefik,fuel-monitoring}/	root:deploy 0750, files 0444	secrets, mounted as /run/secrets/<name>
/opt/petrolyst/env/fuel-monitoring.env	root:deploy 0440	FUEL_DB_TRINO_PASSWORD for Trino's ${ENV:…}
/opt/petrolyst/backups/	deploy 0750	Superset metadata dumps
/opt/petrolyst/state/	deploy	last deployed SHA and Superset image tag; rollback-in-progress during a migration rollback

First-time setup

1. Gitea

Create the private repository Petrolyst/petrolyst-infra and push this repository's main to it. Gitea only keeps the history: deploys are manual, so it needs no runner, no Actions and no deploy secrets.

2. Keycloak (realm thedigitalbridge on iam.thedigitalbridge.io)

1. Realm settings → Action → Partial import: choose keycloak/petrolyst-partial-import.json, and "Skip" if a resource exists. This creates:

   • the confidential client petrolyst: standard flow only, PKCE S256, redirect URIs for both hosts;
   • its client roles superset-admin, superset-editor, superset-viewer, atoma-reader and traefik-admin;
   • a mapper that puts those roles into the claim petrolyst_roles.

2. Clients → petrolyst → Credentials: copy the client secret (step 3 stores it on the server).

3. MFA, which Exhibit B requires. A copy of the built-in browser flow is not enough, for two reasons: its Cookie step accepts an existing password-only session from another client of the realm, and its conditional OTP subflow skips users who have not set up OTP. Instead:

   • Authentication → Create flow petrolyst-mfa (basic flow).
   • Add exactly two executions, both Required, in this order: Username Password Form, then OTP Form. Add no Cookie, no Identity Provider Redirector and no conditional subflow.
   • Clients → petrolyst → Advanced → Authentication flow overrides → Browser flow: petrolyst-mfa.

   Result: every sign-in to a Petrolyst app asks for password and OTP, and a user without OTP enrols first. Other clients of the realm are unaffected. There is no silent SSO between Superset and the dashboard.

4. Give people roles: Users → (user) → Role mapping → Assign role → filter by clients → petrolyst. Group mappings work too. ATOMA's users get atoma-reader and nothing else. They then see only the ATOMA dashboard: no menus, nothing to edit, and Settings offers only Logout (Superset role "ATOMA Reader", petrolyst-insights/roles/atoma_reader.py). A realm that already has the client gets the new role by hand (Clients → petrolyst → Roles → Create role) or by importing again with "Skip".

5. Users of the Traefik dashboard need Email verified switched on, because oauth2-proxy refuses unverified addresses.

3. Server (as root, ssh petrolyst)

The server needs only the two host scripts; it never gets Gitea credentials.

# on the Mac, in this repository
scp host/bootstrap.sh host/secrets-init.sh petrolyst:/root/
ssh petrolyst
# on the server, as root
/root/bootstrap.sh
/root/secrets-init.sh                   # generates every secret; reports the missing Keycloak one
/root/secrets-init.sh keycloak-secret   # paste the secret from Keycloak step 2

bootstrap.sh does the following:

• installs Docker from Docker's apt repository;
• adds 4 GB swap;
• turns on ufw for 22, 80 and 443 only;
• turns off SSH password login and root password login;
• creates the deploy user, which runs deploy.sh. It has no SSH key and is reached only through sudo -u deploy from root; a re-run removes the CI key an earlier version installed;
• creates /opt/petrolyst and the Docker network proxy.

Before closing your root session, check from a second terminal that ssh petrolyst still works.

4. First deploy

Use "Manual deploy" below. The first run builds the Superset image and takes the fresh-install path. Let's Encrypt needs about a minute for the two certificates; the smoke tests wait up to two minutes.

Then sign in at https://atoma.petrolyst.com with a user who has superset-admin. That person becomes the first Superset admin; there is no local admin account.

Operating

Task	How
Deploy	"Manual deploy" below, from a clean main
Re-deploy a kept release	sudo -u deploy /opt/petrolyst/infra/deploy/deploy.sh <sha>
Roll back	see below
Traefik dashboard	https://traefik.petrolyst.com (role traefik-admin)
Trino shell	docker exec -it fuel-monitoring-trino-1 trino
fuel-db from the Mac	ssh -L 5433:127.0.0.1:5433 petrolyst, then connect to localhost:5433, database fuel
Logs	docker compose -p fuel-monitoring logs -f superset (or -p traefik)
Superset break-glass	docker exec -it fuel-monitoring-superset-1 superset fab ...
Reset the role "ATOMA Reader" (after an edit in the UI)	docker exec fuel-monitoring-superset-1 /app/.venv/bin/python /app/petrolyst-roles/atoma_reader.py; a deploy does it whenever the Superset image changes

Manual deploy, the only way to deploy (there is no CI runner). On the Mac, in this repository:

1. Commit on main, so that git status is clean. The live commit is in /opt/petrolyst/state/deployed-sha (ssh petrolyst cat /opt/petrolyst/state/deployed-sha).

2. Lint the shell scripts. Warnings and errors must be zero; the info notes on the deliberate test && test || die guards in deploy/lib.sh (SC2015) are filtered out:

   docker run --rm -v "$PWD:/mnt" -w /mnt koalaman/shellcheck:v0.10.0 -S warning -x \
     host/*.sh deploy/*.sh stacks/*/superset/init.sh stacks/*/fuel-db/init/*.sh
   

3. Deploy:

   sha=$(git rev-parse HEAD)
   dir=/opt/petrolyst/incoming/$sha-manual-$(date -u +%s)
   git archive --format=tar "$sha" |
     ssh petrolyst "install -d -o deploy -g deploy -m 0755 $dir && tar -x --no-same-owner -C $dir && chown -R deploy:deploy $dir"
   ssh petrolyst "sudo -u deploy $dir/deploy/deploy.sh $sha $dir"
   

4. Check both hosts from outside:

   test "$(curl -fsS --max-time 20 https://atoma.petrolyst.com/health)" = OK && echo atoma ok
   curl -fsS --max-time 20 -o /dev/null -w '%{http_code}\n' https://traefik.petrolyst.com/   # 302
   

5. Push main to Gitea, so that the repository matches what is live.

git archive sends exactly the commit, never uncommitted or ignored files. The server runs deploy.sh under the host lock it shares with rollback.sh, and records the live commit in /opt/petrolyst/state/deployed-sha.

Rollback (deploy/rollback.sh, manual, under the same lock as deploys):

ls /opt/petrolyst/releases                         # kept releases
sudo -u deploy /opt/petrolyst/infra/deploy/rollback.sh <sha>

• If the target image's migration head equals the database's revision, this is a code-only rollback: promote and restart.

• Otherwise it is a migration rollback:

  1. stop superset;
  2. drop and recreate the superset metadata database;
  3. restore the backup made for the target image, aborting on any error;
  4. check the restored revision;
  5. start the retained older image.

  It refuses if that backup or image is gone. fuel-db is never touched.

  Before step 1 it writes state/rollback-in-progress, and it removes that file only after the smoke tests pass. While the file exists, deploy.sh refuses to run. If a rollback is interrupted, run the same command again. It also restores when the failed attempt left the database empty or had already dropped it.

Afterwards, git revert the bad commit and push, so main matches. A plain revert without the rollback script is stopped by the downgrade guard.

Data loaders (fuel-db). Every table in public must belong to fuel_owner. trino_ro can read new tables only through fuel_owner's default privileges, and PostgreSQL applies those only to objects fuel_owner creates itself. A table that a loader creates as its own role stays unreadable for Trino. Create each loader role so that its sessions act as fuel_owner, and it cannot create anything as itself. Run this in docker exec -it fuel-monitoring-fuel-db-1 psql -U postgres -d fuel:

CREATE ROLE <loader> LOGIN PASSWORD '<password>';
GRANT fuel_owner TO <loader> WITH INHERIT FALSE, SET TRUE;
ALTER ROLE <loader> SET role TO 'fuel_owner';

Tables it creates, hypertables included, then belong to fuel_owner, and Trino can read them. After SET ROLE NONE, it cannot create anything in public. To repair a table that another role created, run ALTER TABLE <t> OWNER TO fuel_owner, then GRANT SELECT ON <t> TO trino_ro.

Adding a project: create stacks/<name>/compose.yaml, attach the public service to the external network proxy, and set traefik.* labels with a unique router name and tls.certresolver: le. Add the DNS record first. deploy.sh picks the new folder up automatically.

sso@docker works only for traefik.petrolyst.com. Its cookie, its allowed return hosts and its role (traefik-admin) are all tied to the dashboard. An app without a login of its own needs its own gate in its stack:

• an oauth2-proxy service with its own cookie secret;
• a router for the app host's /oauth2/ path and a forwardAuth middleware pointing at that service, as in traefik/compose.yaml;
• --redirect-url, --cookie-domain and --whitelist-domain set to the app's host;
• an --allowed-role with a new client role in Keycloak;
• the app's /oauth2/callback added to the petrolyst client's redirect URIs.

Verified locally (2026-10-23)

An end-to-end run ran the real deploy.sh and rollback.sh in a Linux container against Docker Desktop, with a scratch /opt/petrolyst. Production differences in that run:

• a test CA certificate stood in for Let's Encrypt;
• the ACME server in the test copy was unreachable on purpose;
• a dummy Keycloak client secret was used.

Results:

• The fresh install took the init path.
• An identical re-run recreated nothing and left ARTIFACTS byte-identical.
• A Trino-only change recreated only trino.
• A Superset change stopped, backed up, migrated and recreated only superset.
• A change to superset/init.sh alone did the same, and the changed script ran.
• A same-SHA upload with different content was refused.
• An older image against a newer schema was refused before promotion.
• A migration rollback dropped a table added by a "newer migration" and restored the revision.
• A code-only rollback brought back the previous image.
• A re-deploy of an unchanged image, against a schema one migration behind its head, migrated the schema back to the head.
• A migration rollback whose restore failed after the drop left rollback-in-progress and an empty database. deploy.sh then refused to run. Running the rollback again restored the database and removed the file. It did the same after a simulated stop between dropping and recreating the database. Without the file, a missing database stops the rollback with an error.
• The host lock blocked a concurrent run.
• Retention pruned read-only releases.
• Retention removed a queued upload that a plain rsync -a had given an old checkout's directory times. It kept one uploaded with --omit-dir-times.
• In fuel-db, a loader that created a table as its own role left it unreadable for trino_ro. With the "Data loaders" recipe, tables (hypertables too) belonged to fuel_owner and Trino could read them. After SET ROLE NONE, the loader could not create anything.
• Smoke tests passed, including the redirects to the real Keycloak with client_id=petrolyst and PKCE S256.
• TLS 1.1 and unknown SNI were refused, and HTTP redirected to HTTPS.
• Trino and trino_ro both refused writes.

Not verified locally:

• Let's Encrypt issuance;
• a real Keycloak sign-in, OTP and logout (they need the client from setup step 2);
• the theme as rendered in a browser.

Open items

• Exhibit B: encryption at rest; confirm the VPS is in a German data centre.
• Nightly pg_dump of both databases with an offsite copy. Today only the pre-migration Superset dumps exist.
• Certificate expiry monitoring. There is no ACME e-mail, so nothing warns if renewal fails between deploys.
• Celery worker and beat, if Superset alerts or reports are needed.
• A Petrolyst login theme for Keycloak itself, which lives on the Keycloak host.