petrolyst-infra/README.md

# petrolyst-infra

Everything that runs on the petrolyst server (109.199.127.84, Ubuntu 24.04), deployed on every
push to `main`:

- **Traefik** at the edge for every project on the server. Its dashboard is at
  https://traefik.petrolyst.com, behind Keycloak.
- **fuel-monitoring** (ATOMA): Superset at https://atoma.petrolyst.com, reading the operational
  store `fuel-db` (Postgres 17 with TimescaleDB) through Trino. Sign-in goes through Keycloak
  client `petrolyst`, and the UI carries the Petrolyst brand.

Customization and deployment are separate, as with media-insights and media-intelligence-docker:

| Folder | Holds |
|---|---|
| `petrolyst-insights/` | how Superset looks and behaves for Petrolyst: config, Keycloak login code, branding, patches, dashboards. See its README. |
| `traefik/` | edge proxy, TLS, the `sso` middleware (oauth2-proxy) that guards the dashboard |
| `stacks/fuel-monitoring/` | Compose stack, Superset Dockerfile, Trino config and catalog, fuel-db roles |
| `deploy/` | `deploy.sh` (run by CI), `rollback.sh` (manual), `lib.sh` (shared) |
| `host/` | one-time server setup: `bootstrap.sh`, `secrets-init.sh` |
| `keycloak/` | the `petrolyst` client for Keycloak's partial import |
| `.gitea/workflows/deploy.yml` | the pipeline |

No credential is in this repository. Secrets exist only on the server, under
`/opt/petrolyst/secrets/`; CI holds only the SSH deploy key.

## How a deploy works

1. A push to `main` (or a manual run) starts `.gitea/workflows/deploy.yml` on the act_runner. It
   lints the scripts, rsyncs the checkout to a fresh `/opt/petrolyst/incoming/<sha>-<run>/`, and
   calls `deploy/deploy.sh <sha> <that dir>` over SSH as user `deploy`.
2. `deploy.sh` takes the host lock `/opt/petrolyst/deploy.lock`, which it shares with
   `rollback.sh`, and keeps it until it exits.
3. It finalizes the upload. A new commit is moved to `releases/<sha>`, and the release is made
   read-only. A re-run of a deployed commit must match it byte for byte; a mismatch aborts.
4. It resolves the artifacts and writes them to `releases/<sha>/ARTIFACTS`:
   - `SUPERSET_IMAGE_TAG` is a hash of exactly the Superset image's build inputs. They include
     `superset/init.sh`, so changing that file runs `superset-init` again.
   - `TRINO_CONFIG_HASH` and `TRAEFIK_CONFIG_HASH` are hashes of the bind-mounted config.
   - Compose recreates a container only when its value changes.
5. Preflight, before anything live changes:
   - every secret and env file exists;
   - `petrolyst-insights/` holds no deployment files;
   - `docker compose config` passes for every stack;
   - the Superset image is built if it is new;
   - the **downgrade guard** refuses an image older than the metadata database's schema.
6. It promotes the release to `/opt/petrolyst/infra`, then starts Traefik, then each stack under
   `stacks/`.
7. For Superset, it reads the metadata database's state:
   - An empty database is a fresh install: it runs `superset-init`.
   - If the image changed, or the database is not at the image's migration head: it stops
     `superset`, writes `backups/superset-meta_<old tag>_<revision>_<time>.dump`, then runs
     `superset-init` (`db upgrade`, `init`, registers "Fuel (Trino)").
   - If the image is unchanged and the schema is at its head, it does none of this.
8. Smoke tests. A failure turns CI red and prints the stacks' state.
   - `/health` answers through Traefik with a valid certificate.
   - `/login/` leads to Keycloak with `client_id=petrolyst` and PKCE.
   - The dashboard redirects to Keycloak.
   - Trino reads `fuel`, and Superset reaches Trino.
9. Retention:
   - 5 releases;
   - backups for 30 days;
   - the 3 newest Superset images, plus any image that a kept release or backup names;
   - uploads older than a day.

The runner then checks both hosts from outside.

## Server paths

| Path | Owner | What |
|---|---|---|
| `/opt/petrolyst/infra` | deploy | live tree (the promoted release) |
| `/opt/petrolyst/incoming/<sha>-<run>` | deploy | one CI upload; only `deploy.sh` turns it into a release |
| `/opt/petrolyst/releases/<sha>` | deploy, read-only | finalized release plus `ARTIFACTS`; the last 5 are kept |
| `/opt/petrolyst/secrets/{shared,traefik,fuel-monitoring}/` | root:deploy 0750, files 0444 | secrets, mounted as `/run/secrets/<name>` |
| `/opt/petrolyst/env/fuel-monitoring.env` | root:deploy 0440 | `FUEL_DB_TRINO_PASSWORD` for Trino's `${ENV:…}` |
| `/opt/petrolyst/backups/` | deploy 0750 | Superset metadata dumps |
| `/opt/petrolyst/state/` | deploy | last deployed SHA and Superset image tag; `rollback-in-progress` during a migration rollback |

## First-time setup

### 1. Gitea

1. Create the private repository `Petrolyst/petrolyst-infra` and push this repository's
   `main` to it. Enable Actions under Settings → Repository → Units.
2. Runner. Check Site/Org Administration → Actions → Runners. If none is registered, run
   [Gitea Runner](https://docs.gitea.com/runner/installation/docker/) as the container
   `docker.io/gitea/runner:5.2.2-dind` (privileged, with `/data` and `/var/lib/docker` volumes),
   for example on the Gitea host (164.68.121.117):
   - register it as an **organization runner** for `Petrolyst`: take the token from
     Settings → Actions → Runners → Create new Runner;
   - set `GITEA_RUNNER_LABELS=ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latest`;
   - note its egress IPv4 (`curl -4 -s https://ifconfig.me`).
3. Generate a deploy key on your Mac, used only for this:
   `ssh-keygen -t ed25519 -N '' -C petrolyst-ci -f petrolyst-ci`
4. Add repository secrets (Settings → Actions → Secrets):
   - `DEPLOY_SSH_KEY`: contents of `petrolyst-ci` (the private key)
   - `DEPLOY_HOST`: `109.199.127.84`
   - `DEPLOY_KNOWN_HOSTS`: output of `ssh-keyscan -t ed25519 109.199.127.84`. Compare its
     fingerprint (`ssh-keygen -lf <file>`) with `ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub`
     run on the server.

### 2. Keycloak (realm `thedigitalbridge` on iam.thedigitalbridge.io)

1. Realm settings → Action → **Partial import**: choose `keycloak/petrolyst-partial-import.json`,
   and "Skip" if a resource exists. This creates:
   - the confidential client `petrolyst`: standard flow only, PKCE S256, redirect URIs for both
     hosts;
   - its client roles `superset-admin`, `superset-editor`, `superset-viewer` and `traefik-admin`;
   - a mapper that puts those roles into the claim `petrolyst_roles`.
2. Clients → petrolyst → Credentials: copy the client secret (step 3 stores it on the server).
3. **MFA**, which Exhibit B requires. A copy of the built-in `browser` flow is not enough, for two
   reasons: its Cookie step accepts an existing password-only session from another client of the
   realm, and its conditional OTP subflow skips users who have not set up OTP. Instead:
   - Authentication → Create flow `petrolyst-mfa` (basic flow).
   - Add exactly two executions, both **Required**, in this order: `Username Password Form`,
     then `OTP Form`. Add no Cookie, no Identity Provider Redirector and no conditional subflow.
   - Clients → petrolyst → Advanced → Authentication flow overrides → Browser flow:
     `petrolyst-mfa`.

   Result: every sign-in to a Petrolyst app asks for password and OTP, and a user without OTP
   enrols first. Other clients of the realm are unaffected. There is no silent SSO between
   Superset and the dashboard.
4. Give people roles: Users → (user) → Role mapping → Assign role → filter by clients →
   `petrolyst`. Group mappings work too.
5. Users of the Traefik dashboard need **Email verified** switched on, because oauth2-proxy
   refuses unverified addresses.

### 3. Server (as root, `ssh petrolyst`)

The server needs only the two host scripts; it never gets Gitea credentials.

```bash
# on the Mac, in this repository
scp host/bootstrap.sh host/secrets-init.sh petrolyst-ci.pub petrolyst:/root/
ssh petrolyst
# on the server, as root
/root/bootstrap.sh --ci-key /root/petrolyst-ci.pub --runner-ip <runner egress IP>
/root/secrets-init.sh                   # generates every secret; reports the missing Keycloak one
/root/secrets-init.sh keycloak-secret   # paste the secret from Keycloak step 2
```

No runner yet? Pass `--runner-ip 127.0.0.1`. That installs the CI key but leaves it unusable from
outside. Deploy manually until a runner exists (see "Manual deploy" below). When the runner is
running, run `bootstrap.sh` again with its egress IP.

`bootstrap.sh` does the following:
- installs Docker from Docker's apt repository;
- adds 4 GB swap;
- turns on ufw for 22, 80 and 443 only;
- turns off SSH password login and root password login;
- creates the `deploy` user, whose key works only from the runner IP;
- creates `/opt/petrolyst` and the Docker network `proxy`.

**Before closing your root session**, check from a second terminal that `ssh petrolyst` still
works.

### 4. First deploy

Run the workflow manually (Actions → deploy → Run workflow) or push to `main`. Without a runner,
use "Manual deploy" below. The first run builds the Superset image and takes the fresh-install
path. Let's Encrypt needs about a minute
for the two certificates; the smoke tests wait up to two minutes.

Then sign in at https://atoma.petrolyst.com with a user who has `superset-admin`. That person
becomes the first Superset admin; there is no local admin account.

## Operating

| Task | How |
|---|---|
| Deploy | push to `main` (without a runner: "Manual deploy" below) |
| Re-deploy a kept release | `sudo -u deploy /opt/petrolyst/infra/deploy/deploy.sh <sha>` |
| Roll back | see below |
| Traefik dashboard | https://traefik.petrolyst.com (role `traefik-admin`) |
| Trino shell | `docker exec -it fuel-monitoring-trino-1 trino` |
| fuel-db from the Mac | `ssh -L 5433:127.0.0.1:5433 petrolyst`, then connect to `localhost:5433`, database `fuel` |
| Logs | `docker compose -p fuel-monitoring logs -f superset` (or `-p traefik`) |
| Superset break-glass | `docker exec -it fuel-monitoring-superset-1 superset fab ...` |

**Manual deploy** (no runner): commit first, so that `git status` is clean. Then run this on the
Mac, in this repository:

```bash
sha=$(git rev-parse HEAD)
dir=/opt/petrolyst/incoming/$sha-manual-$(date -u +%s)
git archive --format=tar "$sha" |
  ssh petrolyst "install -d -o deploy -g deploy -m 0755 $dir && tar -x --no-same-owner -C $dir && chown -R deploy:deploy $dir"
ssh petrolyst "sudo -u deploy $dir/deploy/deploy.sh $sha $dir"
```

`git archive` sends exactly the commit, never uncommitted or ignored files such as the deploy
key. The server then runs the same `deploy.sh` that CI runs, under the same lock. A later push
queues a CI run that waits for a runner. Cancel stale waiting runs under Actions before you
register one.

**Rollback** (`deploy/rollback.sh`, manual, under the same lock as deploys):

```bash
ls /opt/petrolyst/releases                         # kept releases
sudo -u deploy /opt/petrolyst/infra/deploy/rollback.sh <sha>
```

- If the target image's migration head equals the database's revision, this is a **code-only**
  rollback: promote and restart.
- Otherwise it is a **migration rollback**:
  1. stop `superset`;
  2. drop and recreate the `superset` metadata database;
  3. restore the backup made for the target image, aborting on any error;
  4. check the restored revision;
  5. start the retained older image.

  It refuses if that backup or image is gone. `fuel-db` is never touched.

  Before step 1 it writes `state/rollback-in-progress`, and it removes that file only after the
  smoke tests pass. While the file exists, `deploy.sh` refuses to run. If a rollback is
  interrupted, run the same command again. It also restores when the failed attempt left the
  database empty or had already dropped it.

Afterwards, `git revert` the bad commit and push, so `main` matches. A plain revert without the
rollback script is stopped by the downgrade guard.

**Data loaders** (fuel-db). Every table in `public` must belong to `fuel_owner`. `trino_ro` can
read new tables only through `fuel_owner`'s default privileges, and PostgreSQL applies those
only to objects `fuel_owner` creates itself. A table that a loader creates as its own role stays
unreadable for Trino. Create each loader role so that its sessions act as `fuel_owner`, and it
cannot create anything as itself. Run this in
`docker exec -it fuel-monitoring-fuel-db-1 psql -U postgres -d fuel`:

```sql
CREATE ROLE <loader> LOGIN PASSWORD '<password>';
GRANT fuel_owner TO <loader> WITH INHERIT FALSE, SET TRUE;
ALTER ROLE <loader> SET role TO 'fuel_owner';
```

Tables it creates, hypertables included, then belong to `fuel_owner`, and Trino can read them.
After `SET ROLE NONE`, it cannot create anything in `public`. To repair a table that another
role created, run `ALTER TABLE <t> OWNER TO fuel_owner`, then `GRANT SELECT ON <t> TO trino_ro`.

**Adding a project**: create `stacks/<name>/compose.yaml`, attach the public service to the
external network `proxy`, and set `traefik.*` labels with a unique router name and
`tls.certresolver: le`. Add the DNS record first. `deploy.sh` picks the new folder up
automatically.

`sso@docker` works only for `traefik.petrolyst.com`. Its cookie, its allowed return hosts and its
role (`traefik-admin`) are all tied to the dashboard. An app without a login of its own needs
its own gate in its stack:
- an oauth2-proxy service with its own cookie secret;
- a router for the app host's `/oauth2/` path and a forwardAuth middleware pointing at that
  service, as in `traefik/compose.yaml`;
- `--redirect-url`, `--cookie-domain` and `--whitelist-domain` set to the app's host;
- an `--allowed-role` with a new client role in Keycloak;
- the app's `/oauth2/callback` added to the `petrolyst` client's redirect URIs.

## Verified locally (2026-10-23)

An end-to-end run ran the real `deploy.sh` and `rollback.sh` in a Linux container against
Docker Desktop, with a scratch `/opt/petrolyst`. Production differences in that run:
- a test CA certificate stood in for Let's Encrypt;
- the ACME server in the test copy was unreachable on purpose;
- a dummy Keycloak client secret was used.

Results:
- The fresh install took the init path.
- An identical re-run recreated nothing and left `ARTIFACTS` byte-identical.
- A Trino-only change recreated only `trino`.
- A Superset change stopped, backed up, migrated and recreated only `superset`.
- A change to `superset/init.sh` alone did the same, and the changed script ran.
- A same-SHA upload with different content was refused.
- An older image against a newer schema was refused before promotion.
- A migration rollback dropped a table added by a "newer migration" and restored the revision.
- A code-only rollback brought back the previous image.
- A re-deploy of an unchanged image, against a schema one migration behind its head, migrated
  the schema back to the head.
- A migration rollback whose restore failed after the drop left `rollback-in-progress` and an
  empty database. `deploy.sh` then refused to run. Running the rollback again restored the
  database and removed the file. It did the same after a simulated stop between dropping and
  recreating the database. Without the file, a missing database stops the rollback with an error.
- The host lock blocked a concurrent run.
- Retention pruned read-only releases.
- Retention removed a queued upload that a plain `rsync -a` had given an old checkout's
  directory times. It kept one uploaded with `--omit-dir-times`.
- In `fuel-db`, a loader that created a table as its own role left it unreadable for `trino_ro`.
  With the "Data loaders" recipe, tables (hypertables too) belonged to `fuel_owner` and Trino
  could read them. After `SET ROLE NONE`, the loader could not create anything.
- Smoke tests passed, including the redirects to the real Keycloak with `client_id=petrolyst`
  and PKCE S256.
- TLS 1.1 and unknown SNI were refused, and HTTP redirected to HTTPS.
- Trino and `trino_ro` both refused writes.

Not verified locally:
- Let's Encrypt issuance;
- a real Keycloak sign-in, OTP and logout (they need the client from setup step 2);
- the theme as rendered in a browser;
- the Gitea runner.

## Open items

- Exhibit B: encryption at rest; confirm the VPS is in a German data centre.
- Nightly `pg_dump` of both databases with an offsite copy. Today only the pre-migration
  Superset dumps exist.
- Certificate expiry monitoring. There is no ACME e-mail, so nothing warns if renewal fails
  between deploys.
- Celery worker and beat, if Superset alerts or reports are needed.
- A Petrolyst login theme for Keycloak itself, which lives on the Keycloak host.