petrolyst-infra/README.md

# petrolyst-infra

Everything that runs on the petrolyst server (109.199.127.84, Ubuntu 24.04):

- **Traefik** at the edge for every project on the server. Its dashboard is at
  https://traefik.petrolyst.com, behind Keycloak.
- **fuel-monitoring** (ATOMA): Superset at https://atoma.petrolyst.com, reading the operational
  store `fuel-db` (Postgres 17 with TimescaleDB) directly, as the read-only role `fuel_ro`
  (Trino was removed on 29 Oct 2026). Sign-in goes through Keycloak
  client `petrolyst`, and the UI carries the Petrolyst brand.

**Deploys are manual.** A person deploys a clean commit of `main` from the Mac ("Manual deploy"
under Operating). There is no CI runner, and a push to Gitea deploys nothing.

Customization and deployment are separate, as with media-insights and media-intelligence-docker:

| Folder | Holds |
|---|---|
| `petrolyst-insights/` | how Superset looks and behaves for Petrolyst: config, Keycloak login code, branding, patches, dashboards. See its README. |
| `traefik/` | edge proxy, TLS, the `sso` middleware (oauth2-proxy) that guards the dashboard |
| `stacks/fuel-monitoring/` | Compose stack, Superset Dockerfile and init, fuel-db roles and migrations |
| `deploy/` | `deploy.sh` (run on the server by a manual deploy), `rollback.sh` (manual), `lib.sh` (shared) |
| `host/` | one-time server setup: `bootstrap.sh`, `secrets-init.sh` |
| `keycloak/` | the `petrolyst` client for Keycloak's partial import |

No credential is in this repository. Secrets exist only on the server, under
`/opt/petrolyst/secrets/`. A deploy needs only root SSH access to the server (`ssh petrolyst`).

## How a deploy works

1. A person runs "Manual deploy" (Operating, below) from a clean checkout of `main`: the scripts
   are linted, `git archive` sends exactly that commit to a fresh
   `/opt/petrolyst/incoming/<sha>-manual-<time>/`, and `deploy/deploy.sh <sha> <that dir>` runs
   there as user `deploy` (root hands over with `sudo -u deploy`).
2. `deploy.sh` takes the host lock `/opt/petrolyst/deploy.lock`, which it shares with
   `rollback.sh`, and keeps it until it exits.
3. It finalizes the upload. A new commit is moved to `releases/<sha>`, and the release is made
   read-only. A re-run of a deployed commit must match it byte for byte; a mismatch aborts.
4. It resolves the artifacts and writes them to `releases/<sha>/ARTIFACTS`:
   - `SUPERSET_IMAGE_TAG` is a hash of exactly the Superset image's build inputs. They include
     `superset/init.sh`, so changing that file runs `superset-init` again.
   - `TRAEFIK_CONFIG_HASH` is a hash of Traefik's bind-mounted static config.
   - Compose recreates a container only when its value changes.
5. Preflight, before anything live changes:
   - every secret file exists;
   - `petrolyst-insights/` holds no deployment files;
   - `docker compose config` passes for every stack;
   - the Superset image is built if it is new;
   - the **downgrade guard** refuses an image older than the metadata database's schema.
6. It promotes the release to `/opt/petrolyst/infra`, then starts Traefik, then each stack under
   `stacks/`.
7. For Superset, it reads the metadata database's state:
   - An empty database is a fresh install: it runs `superset-init`.
   - If the image changed, or the database is not at the image's migration head: it stops
     `superset`, writes `backups/superset-meta_<old tag>_<revision>_<time>.dump`, then runs
     `superset-init` (`db upgrade`, `init`, registers "Fuel": fuel-db as `fuel_ro`, its password
     read from `/run/secrets` at connect time, never stored in Superset).
   - If the image is unchanged and the schema is at its head, it does none of this.
8. Smoke tests. A failure ends the deploy with an error and prints the stacks' state.
   - `/health` answers through Traefik with a valid certificate.
   - `/login/` leads to Keycloak with `client_id=petrolyst` and PKCE.
   - The dashboard redirects to Keycloak.
   - Superset's "Fuel" connection reads fuel-db as `fuel_ro`, and its transactions are read-only.
9. Retention:
   - 5 releases;
   - backups for 30 days;
   - the 3 newest Superset images, plus any image that a kept release or backup names;
   - uploads older than a day.

The manual deploy then checks both hosts from outside.

## Server paths

| Path | Owner | What |
|---|---|---|
| `/opt/petrolyst/infra` | deploy | live tree (the promoted release) |
| `/opt/petrolyst/incoming/<sha>-manual-<time>` | deploy | one upload; only `deploy.sh` turns it into a release |
| `/opt/petrolyst/releases/<sha>` | deploy, read-only | finalized release plus `ARTIFACTS`; the last 5 are kept |
| `/opt/petrolyst/secrets/{shared,traefik,fuel-monitoring}/` | root:deploy 0750, files 0444 | secrets, mounted as `/run/secrets/<name>` |
| `/opt/petrolyst/backups/` | deploy 0750 | Superset metadata dumps |
| `/opt/petrolyst/state/` | deploy | last deployed SHA and Superset image tag; `rollback-in-progress` during a migration rollback |

## First-time setup

### 1. Gitea

Create the private repository `Petrolyst/petrolyst-infra` and push this repository's `main` to
it. Gitea only keeps the history: deploys are manual, so it needs no runner, no Actions and no
deploy secrets.

### 2. Keycloak (realm `thedigitalbridge` on iam.thedigitalbridge.io)

1. Realm settings → Action → **Partial import**: choose `keycloak/petrolyst-partial-import.json`,
   and "Skip" if a resource exists. This creates:
   - the confidential client `petrolyst`: standard flow only, PKCE S256, redirect URIs for both
     hosts;
   - its client roles `superset-admin`, `superset-editor`, `superset-viewer`, `atoma-reader`
     and `traefik-admin`;
   - a mapper that puts those roles into the claim `petrolyst_roles`.
2. Clients → petrolyst → Credentials: copy the client secret (step 3 stores it on the server).
3. **MFA**, which Exhibit B requires. A copy of the built-in `browser` flow is not enough, for two
   reasons: its Cookie step accepts an existing password-only session from another client of the
   realm, and its conditional OTP subflow skips users who have not set up OTP. Instead:
   - Authentication → Create flow `petrolyst-mfa` (basic flow).
   - Add exactly two executions, both **Required**, in this order: `Username Password Form`,
     then `OTP Form`. Add no Cookie, no Identity Provider Redirector and no conditional subflow.
   - Clients → petrolyst → Advanced → Authentication flow overrides → Browser flow:
     `petrolyst-mfa`.

   Result: every sign-in to a Petrolyst app asks for password and OTP, and a user without OTP
   enrols first. Other clients of the realm are unaffected. There is no silent SSO between
   Superset and the dashboard.
4. Give people roles: Users → (user) → Role mapping → Assign role → filter by clients →
   `petrolyst`. Group mappings work too. ATOMA's users get `atoma-reader` and nothing else:
   it opens the Petrolyst app (petrolyst-platform) and maps to no Superset role, so in
   Superset they see nothing. A realm that already has the client gets the role by hand
   (Clients → petrolyst → Roles → Create role) or by importing again with "Skip".
5. Users of the Traefik dashboard need **Email verified** switched on, because oauth2-proxy
   refuses unverified addresses.

### 3. Server (as root, `ssh petrolyst`)

The server needs only the two host scripts; it never gets Gitea credentials.

```bash
# on the Mac, in this repository
scp host/bootstrap.sh host/secrets-init.sh petrolyst:/root/
ssh petrolyst
# on the server, as root
/root/bootstrap.sh
/root/secrets-init.sh                   # generates every secret; reports the missing Keycloak one
/root/secrets-init.sh keycloak-secret   # paste the secret from Keycloak step 2
```

`bootstrap.sh` does the following:
- installs Docker from Docker's apt repository;
- adds 4 GB swap;
- turns on ufw for 22, 80 and 443 only;
- turns off SSH password login and root password login;
- creates the `deploy` user, which runs `deploy.sh`. It has no SSH key and is reached only
  through `sudo -u deploy` from root; a re-run removes the CI key an earlier version installed;
- creates `/opt/petrolyst` and the Docker network `proxy`.

**Before closing your root session**, check from a second terminal that `ssh petrolyst` still
works.

### 4. First deploy

Use "Manual deploy" below. The first run builds the Superset image and takes the fresh-install
path. Let's Encrypt needs about a minute
for the two certificates; the smoke tests wait up to two minutes.

Then sign in at https://atoma.petrolyst.com with a user who has `superset-admin`. That person
becomes the first Superset admin; there is no local admin account.

## Operating

| Task | How |
|---|---|
| Deploy | "Manual deploy" below, from a clean `main` |
| Re-deploy a kept release | `sudo -u deploy /opt/petrolyst/infra/deploy/deploy.sh <sha>` |
| Roll back | see below |
| Traefik dashboard | https://traefik.petrolyst.com (role `traefik-admin`) |
| fuel-db shell | `docker exec -it petrolyst-fuel-db-1 psql -U postgres -d fuel` |
| fuel-db from the Mac | `ssh -L 5433:127.0.0.1:5433 petrolyst`, then connect to `localhost:5433`, database `fuel` |
| Logs | `docker compose -p petrolyst logs -f superset` (or `-p traefik`) |
| Superset break-glass | `docker exec -it petrolyst-superset-1 superset fab ...` |

**Manual deploy**, the only way to deploy (there is no CI runner). On the Mac, in this
repository:

1. Commit on `main`, so that `git status` is clean. The live commit is in
   `/opt/petrolyst/state/deployed-sha` (`ssh petrolyst cat /opt/petrolyst/state/deployed-sha`).
2. Lint the shell scripts. Warnings and errors must be zero; the info notes on the deliberate
   `test && test || die` guards in `deploy/lib.sh` (SC2015) are filtered out:

   ```bash
   docker run --rm -v "$PWD:/mnt" -w /mnt koalaman/shellcheck:v0.10.0 -S warning -x \
     host/*.sh deploy/*.sh stacks/*/superset/init.sh stacks/*/fuel-db/init/*.sh
   ```

3. Deploy:

   ```bash
   sha=$(git rev-parse HEAD)
   dir=/opt/petrolyst/incoming/$sha-manual-$(date -u +%s)
   git archive --format=tar "$sha" |
     ssh petrolyst "install -d -o deploy -g deploy -m 0755 $dir && tar -x --no-same-owner -C $dir && chown -R deploy:deploy $dir"
   ssh petrolyst "sudo -u deploy $dir/deploy/deploy.sh $sha $dir"
   ```

4. Check both hosts from outside:

   ```bash
   test "$(curl -fsS --max-time 20 https://atoma.petrolyst.com/health)" = OK && echo atoma ok
   curl -fsS --max-time 20 -o /dev/null -w '%{http_code}\n' https://traefik.petrolyst.com/   # 302
   ```

5. Push `main` to Gitea, so that the repository matches what is live.

`git archive` sends exactly the commit, never uncommitted or ignored files. The server runs
`deploy.sh` under the host lock it shares with `rollback.sh`, and records the live commit in
`/opt/petrolyst/state/deployed-sha`.

**Rollback** (`deploy/rollback.sh`, manual, under the same lock as deploys):

```bash
ls /opt/petrolyst/releases                         # kept releases
sudo -u deploy /opt/petrolyst/infra/deploy/rollback.sh <sha>
```

- If the target image's migration head equals the database's revision, this is a **code-only**
  rollback: promote and restart.
- Otherwise it is a **migration rollback**:
  1. stop `superset`;
  2. drop and recreate the `superset` metadata database;
  3. restore the backup made for the target image, aborting on any error;
  4. check the restored revision;
  5. start the retained older image.

  It refuses if that backup or image is gone. `fuel-db` is never touched.

  Before step 1 it writes `state/rollback-in-progress`, and it removes that file only after the
  smoke tests pass. While the file exists, `deploy.sh` refuses to run. If a rollback is
  interrupted, run the same command again. It also restores when the failed attempt left the
  database empty or had already dropped it.

Afterwards, `git revert` the bad commit and push, so `main` matches. A plain revert without the
rollback script is stopped by the downgrade guard.

**Data loaders** (fuel-db). Every table in `public` must belong to `fuel_owner`. `fuel_ro` can
read new tables only through `fuel_owner`'s default privileges, and PostgreSQL applies those
only to objects `fuel_owner` creates itself. A table that a loader creates as its own role stays
unreadable for Superset and the Petrolyst server. Create each loader role so that its sessions act as `fuel_owner`, and it
cannot create anything as itself. Run this in
`docker exec -it petrolyst-fuel-db-1 psql -U postgres -d fuel`:

```sql
CREATE ROLE <loader> LOGIN PASSWORD '<password>';
GRANT fuel_owner TO <loader> WITH INHERIT FALSE, SET TRUE;
ALTER ROLE <loader> SET role TO 'fuel_owner';
```

Tables it creates, hypertables included, then belong to `fuel_owner`, and `fuel_ro` can read them.
After `SET ROLE NONE`, it cannot create anything in `public`. To repair a table that another
role created, run `ALTER TABLE <t> OWNER TO fuel_owner`, then `GRANT SELECT ON <t> TO fuel_ro`.

**One-time steps after 29 Oct 2026**, at the first deploy that includes them. Two changes
meet there:
- Trino is removed: Superset reads fuel-db directly. The role `trino_ro` becomes `fuel_ro` and
  keeps its password (a SCRAM password survives a rename); its secret file moves with it.
- The Compose project `fuel-monitoring` is renamed `petrolyst`. Containers are then called
  `petrolyst-<service>-1`. The data volumes keep their names (`fuel-monitoring_superset_db_data`,
  `fuel-monitoring_fuel_db_data`; pinned in `compose.yaml`), so no data moves. The old project's
  containers must be gone first: two PostgreSQL servers on one data volume corrupt it.
  Preflight refuses to deploy while any of them exists.

As root on the server, right before the deploy (Superset is down from step 3 until the deploy
finishes):

```bash
S=/opt/petrolyst/secrets/fuel-monitoring
mv "$S/fuel_db_trino_password" "$S/fuel_db_ro_password"                       # 1
docker exec -i fuel-monitoring-fuel-db-1 psql -U postgres -d fuel -v ON_ERROR_STOP=1 \
  -c 'ALTER ROLE trino_ro RENAME TO fuel_ro' \
  -c 'ALTER ROLE fuel_ro SET default_transaction_read_only = on'               # 2
docker rm -f $(docker ps -aq --filter label=com.docker.compose.project=fuel-monitoring)   # 3
```

4. Deploy as usual ("Manual deploy"). Compose starts the `petrolyst` project on the same volumes
   and no longer starts Trino. Compose warns once that the volumes belong to another project;
   that is expected. `superset-init` renames the connection "Fuel (Trino)" to "Fuel" and points
   it at fuel-db; the smoke test checks that it reads as `fuel_ro`, read-only.
5. Clean up: `rm /opt/petrolyst/env/fuel-monitoring.env`, `rmdir /opt/petrolyst/env`,
   `docker image rm trinodb/trino:484`,
   `docker network rm fuel-monitoring_backend fuel-monitoring_fuel-db-tunnel`.

**Rollback across the Trino removal** (not tested). `rollback.sh` stops for a release from before
it. To go back anyway: remove the `petrolyst` project's containers, reverse steps 1 and 2 above (rename `fuel_ro` back to `trino_ro` and
`ALTER ROLE trino_ro RESET default_transaction_read_only`, move the secret file back, write
`/opt/petrolyst/env/fuel-monitoring.env` with `FUEL_DB_TRINO_PASSWORD=<that secret>`), rename
the Superset connection "Fuel" back to "Fuel (Trino)" with `trino://superset@trino:8080/fuel`,
then run that release's own script, `/opt/petrolyst/releases/<sha>/deploy/rollback.sh <sha>`.

**Adding a project**: create `stacks/<name>/compose.yaml`, attach the public service to the
external network `proxy`, and set `traefik.*` labels with a unique router name and
`tls.certresolver: le`. Add the DNS record first. `deploy.sh` picks the new folder up
automatically.

`sso@docker` works only for `traefik.petrolyst.com`. Its cookie, its allowed return hosts and its
role (`traefik-admin`) are all tied to the dashboard. An app without a login of its own needs
its own gate in its stack:
- an oauth2-proxy service with its own cookie secret;
- a router for the app host's `/oauth2/` path and a forwardAuth middleware pointing at that
  service, as in `traefik/compose.yaml`;
- `--redirect-url`, `--cookie-domain` and `--whitelist-domain` set to the app's host;
- an `--allowed-role` with a new client role in Keycloak;
- the app's `/oauth2/callback` added to the `petrolyst` client's redirect URIs.

## Verified locally (2026-10-23)

Before Trino's removal (29 Oct 2026): its lines below are history.

An end-to-end run ran the real `deploy.sh` and `rollback.sh` in a Linux container against
Docker Desktop, with a scratch `/opt/petrolyst`. Production differences in that run:
- a test CA certificate stood in for Let's Encrypt;
- the ACME server in the test copy was unreachable on purpose;
- a dummy Keycloak client secret was used.

Results:
- The fresh install took the init path.
- An identical re-run recreated nothing and left `ARTIFACTS` byte-identical.
- A Trino-only change recreated only `trino`.
- A Superset change stopped, backed up, migrated and recreated only `superset`.
- A change to `superset/init.sh` alone did the same, and the changed script ran.
- A same-SHA upload with different content was refused.
- An older image against a newer schema was refused before promotion.
- A migration rollback dropped a table added by a "newer migration" and restored the revision.
- A code-only rollback brought back the previous image.
- A re-deploy of an unchanged image, against a schema one migration behind its head, migrated
  the schema back to the head.
- A migration rollback whose restore failed after the drop left `rollback-in-progress` and an
  empty database. `deploy.sh` then refused to run. Running the rollback again restored the
  database and removed the file. It did the same after a simulated stop between dropping and
  recreating the database. Without the file, a missing database stops the rollback with an error.
- The host lock blocked a concurrent run.
- Retention pruned read-only releases.
- Retention removed a queued upload that a plain `rsync -a` had given an old checkout's
  directory times. It kept one uploaded with `--omit-dir-times`.
- In `fuel-db`, a loader that created a table as its own role left it unreadable for `trino_ro`.
  With the "Data loaders" recipe, tables (hypertables too) belonged to `fuel_owner` and Trino
  could read them. After `SET ROLE NONE`, the loader could not create anything.
- Smoke tests passed, including the redirects to the real Keycloak with `client_id=petrolyst`
  and PKCE S256.
- TLS 1.1 and unknown SNI were refused, and HTTP redirected to HTTPS.
- Trino and `trino_ro` both refused writes.

Not verified locally:
- Let's Encrypt issuance;
- a real Keycloak sign-in, OTP and logout (they need the client from setup step 2);
- the theme as rendered in a browser.

## Open items

- Exhibit B: encryption at rest; confirm the VPS is in a German data centre.
- Nightly `pg_dump` of both databases with an offsite copy. Today only the pre-migration
  Superset dumps exist.
- Certificate expiry monitoring. There is no ACME e-mail, so nothing warns if renewal fails
  between deploys.
- Celery worker and beat, if Superset alerts or reports are needed.
- A Petrolyst login theme for Keycloak itself, which lives on the Keycloak host.