petrolyst-infra

Everything that runs on the petrolyst server (109.199.127.84, Ubuntu 24.04), deployed on every push to main:

• Traefik at the edge for every project on the server. Its dashboard is at https://traefik.petrolyst.com, behind Keycloak.
• fuel-monitoring (ATOMA): Superset at https://atoma.petrolyst.com, reading the operational store fuel-db (Postgres 17 with TimescaleDB) through Trino. Sign-in goes through Keycloak client petrolyst, and the UI carries the Petrolyst brand.

Customization and deployment are separate, as with media-insights and media-intelligence-docker:

Folder	Holds
petrolyst-insights/	how Superset looks and behaves for Petrolyst: config, Keycloak login code, branding, patches, dashboards. See its README.
traefik/	edge proxy, TLS, the sso middleware (oauth2-proxy) that guards the dashboard
stacks/fuel-monitoring/	Compose stack, Superset Dockerfile, Trino config and catalog, fuel-db roles
deploy/	deploy.sh (run by CI), rollback.sh (manual), lib.sh (shared)
host/	one-time server setup: bootstrap.sh, secrets-init.sh
keycloak/	the petrolyst client for Keycloak's partial import
.gitea/workflows/deploy.yml	the pipeline

No credential is in this repository. Secrets exist only on the server, under /opt/petrolyst/secrets/; CI holds only the SSH deploy key.

How a deploy works

1. A push to main (or a manual run) starts .gitea/workflows/deploy.yml on the act_runner. It lints the scripts, rsyncs the checkout to a fresh /opt/petrolyst/incoming/<sha>-<run>/, and calls deploy/deploy.sh <sha> <that dir> over SSH as user deploy.
2. deploy.sh takes the host lock /opt/petrolyst/deploy.lock, which it shares with rollback.sh, and keeps it until it exits.
3. It finalizes the upload. A new commit is moved to releases/<sha>, and the release is made read-only. A re-run of a deployed commit must match it byte for byte; a mismatch aborts.
4. It resolves the artifacts and writes them to releases/<sha>/ARTIFACTS:
   • SUPERSET_IMAGE_TAG is a hash of exactly the Superset image's build inputs. They include superset/init.sh, so changing that file runs superset-init again.
   • TRINO_CONFIG_HASH and TRAEFIK_CONFIG_HASH are hashes of the bind-mounted config.
   • Compose recreates a container only when its value changes.
5. Preflight, before anything live changes:
   • every secret and env file exists;
   • petrolyst-insights/ holds no deployment files;
   • docker compose config passes for every stack;
   • the Superset image is built if it is new;
   • the downgrade guard refuses an image older than the metadata database's schema.
6. It promotes the release to /opt/petrolyst/infra, then starts Traefik, then each stack under stacks/.
7. For Superset, it reads the metadata database's state:
   • An empty database is a fresh install: it runs superset-init.
   • If the image changed, or the database is not at the image's migration head: it stops superset, writes backups/superset-meta_<old tag>_<revision>_<time>.dump, then runs superset-init (db upgrade, init, registers "Fuel (Trino)").
   • If the image is unchanged and the schema is at its head, it does none of this.
8. Smoke tests. A failure turns CI red and prints the stacks' state.
   • /health answers through Traefik with a valid certificate.
   • /login/ leads to Keycloak with client_id=petrolyst and PKCE.
   • The dashboard redirects to Keycloak.
   • Trino reads fuel, and Superset reaches Trino.
9. Retention:
   • 5 releases;
   • backups for 30 days;
   • the 3 newest Superset images, plus any image that a kept release or backup names;
   • uploads older than a day.

The runner then checks both hosts from outside.

Server paths

Path	Owner	What
/opt/petrolyst/infra	deploy	live tree (the promoted release)
/opt/petrolyst/incoming/<sha>-<run>	deploy	one CI upload; only deploy.sh turns it into a release
/opt/petrolyst/releases/<sha>	deploy, read-only	finalized release plus ARTIFACTS; the last 5 are kept
/opt/petrolyst/secrets/{shared,traefik,fuel-monitoring}/	root:deploy 0750, files 0444	secrets, mounted as /run/secrets/<name>
/opt/petrolyst/env/fuel-monitoring.env	root:deploy 0440	FUEL_DB_TRINO_PASSWORD for Trino's ${ENV:…}
/opt/petrolyst/backups/	deploy 0750	Superset metadata dumps
/opt/petrolyst/state/	deploy	last deployed SHA and Superset image tag; rollback-in-progress during a migration rollback

First-time setup

1. Gitea

1. Create the private repository Petrolyst/petrolyst-infra and push this repository's main to it. Enable Actions under Settings → Repository → Units.
2. Runner. Check Site/Org Administration → Actions → Runners. If none is registered, run Gitea Runner as the container docker.io/gitea/runner:5.2.2-dind (privileged, with /data and /var/lib/docker volumes), for example on the Gitea host (164.68.121.117):
   • register it as an organization runner for Petrolyst: take the token from Settings → Actions → Runners → Create new Runner;
   • set GITEA_RUNNER_LABELS=ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latest;
   • note its egress IPv4 (curl -4 -s https://ifconfig.me).
3. Generate a deploy key on your Mac, used only for this: ssh-keygen -t ed25519 -N '' -C petrolyst-ci -f petrolyst-ci
4. Add repository secrets (Settings → Actions → Secrets):
   • DEPLOY_SSH_KEY: contents of petrolyst-ci (the private key)
   • DEPLOY_HOST: 109.199.127.84
   • DEPLOY_KNOWN_HOSTS: output of ssh-keyscan -t ed25519 109.199.127.84. Compare its fingerprint (ssh-keygen -lf <file>) with ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub run on the server.

2. Keycloak (realm thedigitalbridge on iam.thedigitalbridge.io)

1. Realm settings → Action → Partial import: choose keycloak/petrolyst-partial-import.json, and "Skip" if a resource exists. This creates:

   • the confidential client petrolyst: standard flow only, PKCE S256, redirect URIs for both hosts;
   • its client roles superset-admin, superset-editor, superset-viewer and traefik-admin;
   • a mapper that puts those roles into the claim petrolyst_roles.

2. Clients → petrolyst → Credentials: copy the client secret (step 3 stores it on the server).

3. MFA, which Exhibit B requires. A copy of the built-in browser flow is not enough, for two reasons: its Cookie step accepts an existing password-only session from another client of the realm, and its conditional OTP subflow skips users who have not set up OTP. Instead:

   • Authentication → Create flow petrolyst-mfa (basic flow).
   • Add exactly two executions, both Required, in this order: Username Password Form, then OTP Form. Add no Cookie, no Identity Provider Redirector and no conditional subflow.
   • Clients → petrolyst → Advanced → Authentication flow overrides → Browser flow: petrolyst-mfa.

   Result: every sign-in to a Petrolyst app asks for password and OTP, and a user without OTP enrols first. Other clients of the realm are unaffected. There is no silent SSO between Superset and the dashboard.

4. Give people roles: Users → (user) → Role mapping → Assign role → filter by clients → petrolyst. Group mappings work too.

5. Users of the Traefik dashboard need Email verified switched on, because oauth2-proxy refuses unverified addresses.

3. Server (as root, ssh petrolyst)

The server needs only the two host scripts; it never gets Gitea credentials.

# on the Mac, in this repository
scp host/bootstrap.sh host/secrets-init.sh petrolyst-ci.pub petrolyst:/root/
ssh petrolyst
# on the server, as root
/root/bootstrap.sh --ci-key /root/petrolyst-ci.pub --runner-ip <runner egress IP>
/root/secrets-init.sh                   # generates every secret; reports the missing Keycloak one
/root/secrets-init.sh keycloak-secret   # paste the secret from Keycloak step 2

No runner yet? Pass --runner-ip 127.0.0.1. That installs the CI key but leaves it unusable from outside. Deploy manually until a runner exists (see "Manual deploy" below). When the runner is running, run bootstrap.sh again with its egress IP.

bootstrap.sh does the following:

• installs Docker from Docker's apt repository;
• adds 4 GB swap;
• turns on ufw for 22, 80 and 443 only;
• turns off SSH password login and root password login;
• creates the deploy user, whose key works only from the runner IP;
• creates /opt/petrolyst and the Docker network proxy.

Before closing your root session, check from a second terminal that ssh petrolyst still works.

4. First deploy

Run the workflow manually (Actions → deploy → Run workflow) or push to main. Without a runner, use "Manual deploy" below. The first run builds the Superset image and takes the fresh-install path. Let's Encrypt needs about a minute for the two certificates; the smoke tests wait up to two minutes.

Then sign in at https://atoma.petrolyst.com with a user who has superset-admin. That person becomes the first Superset admin; there is no local admin account.

Operating

Task	How
Deploy	push to main (without a runner: "Manual deploy" below)
Re-deploy a kept release	sudo -u deploy /opt/petrolyst/infra/deploy/deploy.sh <sha>
Roll back	see below
Traefik dashboard	https://traefik.petrolyst.com (role traefik-admin)
Trino shell	docker exec -it fuel-monitoring-trino-1 trino
fuel-db from the Mac	ssh -L 5433:127.0.0.1:5433 petrolyst, then connect to localhost:5433, database fuel
Logs	docker compose -p fuel-monitoring logs -f superset (or -p traefik)
Superset break-glass	docker exec -it fuel-monitoring-superset-1 superset fab ...

Manual deploy (no runner): commit first, so that git status is clean. Then run this on the Mac, in this repository:

sha=$(git rev-parse HEAD)
dir=/opt/petrolyst/incoming/$sha-manual-$(date -u +%s)
git archive --format=tar "$sha" |
  ssh petrolyst "install -d -o deploy -g deploy -m 0755 $dir && tar -x --no-same-owner -C $dir && chown -R deploy:deploy $dir"
ssh petrolyst "sudo -u deploy $dir/deploy/deploy.sh $sha $dir"

git archive sends exactly the commit, never uncommitted or ignored files such as the deploy key. The server then runs the same deploy.sh that CI runs, under the same lock. A later push queues a CI run that waits for a runner. Cancel stale waiting runs under Actions before you register one.

Rollback (deploy/rollback.sh, manual, under the same lock as deploys):

ls /opt/petrolyst/releases                         # kept releases
sudo -u deploy /opt/petrolyst/infra/deploy/rollback.sh <sha>

• If the target image's migration head equals the database's revision, this is a code-only rollback: promote and restart.

• Otherwise it is a migration rollback:

  1. stop superset;
  2. drop and recreate the superset metadata database;
  3. restore the backup made for the target image, aborting on any error;
  4. check the restored revision;
  5. start the retained older image.

  It refuses if that backup or image is gone. fuel-db is never touched.

  Before step 1 it writes state/rollback-in-progress, and it removes that file only after the smoke tests pass. While the file exists, deploy.sh refuses to run. If a rollback is interrupted, run the same command again. It also restores when the failed attempt left the database empty or had already dropped it.

Afterwards, git revert the bad commit and push, so main matches. A plain revert without the rollback script is stopped by the downgrade guard.

Data loaders (fuel-db). Every table in public must belong to fuel_owner. trino_ro can read new tables only through fuel_owner's default privileges, and PostgreSQL applies those only to objects fuel_owner creates itself. A table that a loader creates as its own role stays unreadable for Trino. Create each loader role so that its sessions act as fuel_owner, and it cannot create anything as itself. Run this in docker exec -it fuel-monitoring-fuel-db-1 psql -U postgres -d fuel:

CREATE ROLE <loader> LOGIN PASSWORD '<password>';
GRANT fuel_owner TO <loader> WITH INHERIT FALSE, SET TRUE;
ALTER ROLE <loader> SET role TO 'fuel_owner';

Tables it creates, hypertables included, then belong to fuel_owner, and Trino can read them. After SET ROLE NONE, it cannot create anything in public. To repair a table that another role created, run ALTER TABLE <t> OWNER TO fuel_owner, then GRANT SELECT ON <t> TO trino_ro.

Adding a project: create stacks/<name>/compose.yaml, attach the public service to the external network proxy, and set traefik.* labels with a unique router name and tls.certresolver: le. Add the DNS record first. deploy.sh picks the new folder up automatically.

sso@docker works only for traefik.petrolyst.com. Its cookie, its allowed return hosts and its role (traefik-admin) are all tied to the dashboard. An app without a login of its own needs its own gate in its stack:

• an oauth2-proxy service with its own cookie secret;
• a router for the app host's /oauth2/ path and a forwardAuth middleware pointing at that service, as in traefik/compose.yaml;
• --redirect-url, --cookie-domain and --whitelist-domain set to the app's host;
• an --allowed-role with a new client role in Keycloak;
• the app's /oauth2/callback added to the petrolyst client's redirect URIs.

Verified locally (2026-10-23)

An end-to-end run ran the real deploy.sh and rollback.sh in a Linux container against Docker Desktop, with a scratch /opt/petrolyst. Production differences in that run:

• a test CA certificate stood in for Let's Encrypt;
• the ACME server in the test copy was unreachable on purpose;
• a dummy Keycloak client secret was used.

Results:

• The fresh install took the init path.
• An identical re-run recreated nothing and left ARTIFACTS byte-identical.
• A Trino-only change recreated only trino.
• A Superset change stopped, backed up, migrated and recreated only superset.
• A change to superset/init.sh alone did the same, and the changed script ran.
• A same-SHA upload with different content was refused.
• An older image against a newer schema was refused before promotion.
• A migration rollback dropped a table added by a "newer migration" and restored the revision.
• A code-only rollback brought back the previous image.
• A re-deploy of an unchanged image, against a schema one migration behind its head, migrated the schema back to the head.
• A migration rollback whose restore failed after the drop left rollback-in-progress and an empty database. deploy.sh then refused to run. Running the rollback again restored the database and removed the file. It did the same after a simulated stop between dropping and recreating the database. Without the file, a missing database stops the rollback with an error.
• The host lock blocked a concurrent run.
• Retention pruned read-only releases.
• Retention removed a queued upload that a plain rsync -a had given an old checkout's directory times. It kept one uploaded with --omit-dir-times.
• In fuel-db, a loader that created a table as its own role left it unreadable for trino_ro. With the "Data loaders" recipe, tables (hypertables too) belonged to fuel_owner and Trino could read them. After SET ROLE NONE, the loader could not create anything.
• Smoke tests passed, including the redirects to the real Keycloak with client_id=petrolyst and PKCE S256.
• TLS 1.1 and unknown SNI were refused, and HTTP redirected to HTTPS.
• Trino and trino_ro both refused writes.

Not verified locally:

• Let's Encrypt issuance;
• a real Keycloak sign-in, OTP and logout (they need the client from setup step 2);
• the theme as rendered in a browser;
• the Gitea runner.

Open items

• Exhibit B: encryption at rest; confirm the VPS is in a German data centre.
• Nightly pg_dump of both databases with an offsite copy. Today only the pre-migration Superset dumps exist.
• Certificate expiry monitoring. There is no ACME e-mail, so nothing warns if renewal fails between deploys.
• Celery worker and beat, if Superset alerts or reports are needed.
• A Petrolyst login theme for Keycloak itself, which lives on the Keycloak host.