petrolyst-infra/README.md

# petrolyst-infra

Everything that runs on the petrolyst server (109.199.127.84, Ubuntu 24.04):

- **Traefik** at the edge for every project on the server. Its dashboard is at
  https://traefik.petrolyst.com, behind Keycloak.
- **fuel-monitoring** (ATOMA): Superset at https://atoma.petrolyst.com, reading the operational
  store `fuel-db` (Postgres 17 with TimescaleDB) directly, as the read-only role `fuel_ro`
  (no Trino since 29 Oct 2026). Sign-in goes through Keycloak
  client `petrolyst`, and the UI carries the Petrolyst brand.

**Deploys are manual.** A person deploys a clean commit of `main` from the Mac ("Manual deploy"
under Operating). There is no CI runner, and a push to Gitea deploys nothing.

Customization and deployment are separate, as with media-insights and media-intelligence-docker:

| Folder | Holds |
|---|---|
| `petrolyst-insights/` | how Superset looks and behaves for Petrolyst: config, Keycloak login code, branding, patches, dashboards. See its README. |
| `traefik/` | edge proxy, TLS, the `sso` middleware (oauth2-proxy) that guards the dashboard |
| `stacks/fuel-monitoring/` | Compose stack, Superset Dockerfile and init, fuel-db roles and migrations |
| `deploy/` | `deploy.sh` (run on the server by a manual deploy), `rollback.sh` (manual), `lib.sh` (shared) |
| `host/` | one-time server setup: `bootstrap.sh`, `secrets-init.sh` |
| `keycloak/` | the `petrolyst` client for Keycloak's partial import |

No credential is in this repository. Secrets exist only on the server, under
`/opt/petrolyst/secrets/`. A deploy needs only root SSH access to the server (`ssh petrolyst`).

## How a deploy works

1. A person runs "Manual deploy" (Operating, below) from a clean checkout of `main`: the scripts
   are linted, `git archive` sends exactly that commit to a fresh
   `/opt/petrolyst/incoming/<sha>-manual-<time>/`, and `deploy/deploy.sh <sha> <that dir>` runs
   there as user `deploy` (root hands over with `sudo -u deploy`).
2. `deploy.sh` takes the host lock `/opt/petrolyst/deploy.lock`, which it shares with
   `rollback.sh`, and keeps it until it exits.
3. It finalizes the upload. A new commit is moved to `releases/<sha>`, and the release is made
   read-only. A re-run of a deployed commit must match it byte for byte; a mismatch aborts.
4. It resolves the artifacts and writes them to `releases/<sha>/ARTIFACTS`:
   - `SUPERSET_IMAGE_TAG` is a hash of exactly the Superset image's build inputs. They include
     `superset/init.sh`, so changing that file runs `superset-init` again.
   - `TRAEFIK_CONFIG_HASH` is a hash of Traefik's bind-mounted static config.
   - Compose recreates a container only when its value changes.
5. Preflight, before anything live changes:
   - every secret file exists;
   - `petrolyst-insights/` holds no deployment files;
   - `docker compose config` passes for every stack;
   - the Superset image is built if it is new;
   - the **downgrade guard** refuses an image older than the metadata database's schema.
6. It promotes the release to `/opt/petrolyst/infra`, then starts Traefik, then each stack under
   `stacks/`.
7. For Superset, it reads the metadata database's state:
   - An empty database is a fresh install: it runs `superset-init`.
   - If the image changed, or the database is not at the image's migration head: it stops
     `superset`, writes `backups/superset-meta_<old tag>_<revision>_<time>.dump`, then runs
     `superset-init` (`db upgrade`, `init`, registers "Fuel": fuel-db as `fuel_ro`, its password
     read from `/run/secrets` at connect time, never stored in Superset).
   - If the image is unchanged and the schema is at its head, it does none of this.
8. Smoke tests. A failure ends the deploy with an error and prints the stacks' state.
   - `/health` answers through Traefik with a valid certificate.
   - `/login/` leads to Keycloak with `client_id=petrolyst` and PKCE.
   - The dashboard redirects to Keycloak.
   - Superset's "Fuel" connection reads fuel-db as `fuel_ro`, and its transactions are read-only.
9. Retention:
   - 5 releases;
   - backups for 30 days;
   - the 3 newest Superset images, plus any image that a kept release or backup names;
   - uploads older than a day.

The manual deploy then checks both hosts from outside.

## Server paths

| Path | Owner | What |
|---|---|---|
| `/opt/petrolyst/infra` | deploy | live tree (the promoted release) |
| `/opt/petrolyst/incoming/<sha>-manual-<time>` | deploy | one upload; only `deploy.sh` turns it into a release |
| `/opt/petrolyst/releases/<sha>` | deploy, read-only | finalized release plus `ARTIFACTS`; the last 5 are kept |
| `/opt/petrolyst/secrets/{shared,traefik,fuel-monitoring}/` | root:deploy 0750, files 0444 | secrets, mounted as `/run/secrets/<name>` |
| `/opt/petrolyst/backups/` | deploy 0750 | Superset metadata dumps |
| `/opt/petrolyst/state/` | deploy | last deployed SHA and Superset image tag; `rollback-in-progress` during a migration rollback |

## First-time setup

### 1. Gitea

Create the private repository `Petrolyst/petrolyst-infra` and push this repository's `main` to
it. Gitea only keeps the history: deploys are manual, so it needs no runner, no Actions and no
deploy secrets.

### 2. Keycloak (realm `thedigitalbridge` on iam.thedigitalbridge.io)

1. Realm settings → Action → **Partial import**: choose `keycloak/petrolyst-partial-import.json`,
   and "Skip" if a resource exists. This creates:
   - the confidential client `petrolyst`: standard flow only, PKCE S256, redirect URIs for both
     hosts;
   - its client roles `superset-admin`, `superset-editor`, `superset-viewer`, `atoma-reader`
     and `traefik-admin`;
   - a mapper that puts those roles into the claim `petrolyst_roles`.
2. Clients → petrolyst → Credentials: copy the client secret (step 3 stores it on the server).
3. **MFA**, which Exhibit B requires. A copy of the built-in `browser` flow is not enough, for two
   reasons: its Cookie step accepts an existing password-only session from another client of the
   realm, and its conditional OTP subflow skips users who have not set up OTP. Instead:
   - Authentication → Create flow `petrolyst-mfa` (basic flow).
   - Add exactly two executions, both **Required**, in this order: `Username Password Form`,
     then `OTP Form`. Add no Cookie, no Identity Provider Redirector and no conditional subflow.
   - Clients → petrolyst → Advanced → Authentication flow overrides → Browser flow:
     `petrolyst-mfa`.

   Result: every sign-in to a Petrolyst app asks for password and OTP, and a user without OTP
   enrols first. Other clients of the realm are unaffected. There is no silent SSO between
   Superset and the dashboard.
4. Give people roles: Users → (user) → Role mapping → Assign role → filter by clients →
   `petrolyst`. Group mappings work too. ATOMA's users get `atoma-reader` and nothing else:
   it opens the Petrolyst app (petrolyst-platform) and maps to no Superset role, so in
   Superset they see nothing. A realm that already has the client gets the role by hand
   (Clients → petrolyst → Roles → Create role) or by importing again with "Skip".
5. Users of the Traefik dashboard need **Email verified** switched on, because oauth2-proxy
   refuses unverified addresses.

### 3. Server (as root, `ssh petrolyst`)

The server needs only the two host scripts; it never gets Gitea credentials.

```bash
# on the Mac, in this repository
scp host/bootstrap.sh host/secrets-init.sh petrolyst:/root/
ssh petrolyst
# on the server, as root
/root/bootstrap.sh
/root/secrets-init.sh                   # generates every secret; reports the missing Keycloak one
/root/secrets-init.sh keycloak-secret   # paste the secret from Keycloak step 2
```

`bootstrap.sh` does the following:
- installs Docker from Docker's apt repository;
- adds 4 GB swap;
- turns on ufw for 22, 80 and 443 only;
- turns off SSH password login and root password login;
- creates the `deploy` user, which runs `deploy.sh`. It has no SSH key and is reached only
  through `sudo -u deploy` from root; a re-run removes the CI key an earlier version installed;
- creates `/opt/petrolyst` and the Docker network `proxy`.

**Before closing your root session**, check from a second terminal that `ssh petrolyst` still
works.

### 4. First deploy

Use "Manual deploy" below. The first run builds the Superset image and takes the fresh-install
path. Let's Encrypt needs about a minute
for the two certificates; the smoke tests wait up to two minutes.

Then sign in at https://atoma.petrolyst.com with a user who has `superset-admin`. That person
becomes the first Superset admin; there is no local admin account.

## Operating

| Task | How |
|---|---|
| Deploy | "Manual deploy" below, from a clean `main` |
| Re-deploy a kept release | `sudo -u deploy /opt/petrolyst/infra/deploy/deploy.sh <sha>` |
| Roll back | see below |
| Traefik dashboard | https://traefik.petrolyst.com (role `traefik-admin`) |
| fuel-db shell | `docker exec -it petrolyst-fuel-db-1 psql -U postgres -d fuel` |
| fuel-db from the Mac | `ssh -L 5433:127.0.0.1:5433 petrolyst`, then connect to `localhost:5433`, database `fuel` |
| Logs | `docker compose -p petrolyst logs -f superset` (or `-p traefik`) |
| Superset break-glass | `docker exec -it petrolyst-superset-1 superset fab ...` |

**Manual deploy**, the only way to deploy (there is no CI runner). On the Mac, in this
repository:

1. Commit on `main`, so that `git status` is clean. The live commit is in
   `/opt/petrolyst/state/deployed-sha` (`ssh petrolyst cat /opt/petrolyst/state/deployed-sha`).
2. Lint the shell scripts. Warnings and errors must be zero; the info notes on the deliberate
   `test && test || die` guards in `deploy/lib.sh` (SC2015) are filtered out:

   ```bash
   docker run --rm -v "$PWD:/mnt" -w /mnt koalaman/shellcheck:v0.10.0 -S warning -x \
     host/*.sh deploy/*.sh stacks/*/superset/init.sh stacks/*/fuel-db/init/*.sh
   ```

3. Deploy:

   ```bash
   sha=$(git rev-parse HEAD)
   dir=/opt/petrolyst/incoming/$sha-manual-$(date -u +%s)
   git archive --format=tar "$sha" |
     ssh petrolyst "install -d -o deploy -g deploy -m 0755 $dir && tar -x --no-same-owner -C $dir && chown -R deploy:deploy $dir"
   ssh petrolyst "sudo -u deploy $dir/deploy/deploy.sh $sha $dir"
   ```

4. Check both hosts from outside:

   ```bash
   test "$(curl -fsS --max-time 20 https://atoma.petrolyst.com/health)" = OK && echo atoma ok
   curl -fsS --max-time 20 -o /dev/null -w '%{http_code}\n' https://traefik.petrolyst.com/   # 302
   ```

5. Push `main` to Gitea, so that the repository matches what is live.

`git archive` sends exactly the commit, never uncommitted or ignored files. The server runs
`deploy.sh` under the host lock it shares with `rollback.sh`, and records the live commit in
`/opt/petrolyst/state/deployed-sha`.

**Rollback** (`deploy/rollback.sh`, manual, under the same lock as deploys):

```bash
ls /opt/petrolyst/releases                         # kept releases
sudo -u deploy /opt/petrolyst/infra/deploy/rollback.sh <sha>
```

- If the target image's migration head equals the database's revision, this is a **code-only**
  rollback: promote and restart.
- Otherwise it is a **migration rollback**:
  1. stop `superset`;
  2. drop and recreate the `superset` metadata database;
  3. restore the backup made for the target image, aborting on any error;
  4. check the restored revision;
  5. start the retained older image.

  It refuses if that backup or image is gone. `fuel-db` is never touched.

  Before step 1 it writes `state/rollback-in-progress`, and it removes that file only after the
  smoke tests pass. While the file exists, `deploy.sh` refuses to run. If a rollback is
  interrupted, run the same command again. It also restores when the failed attempt left the
  database empty or had already dropped it.

Afterwards, `git revert` the bad commit and push, so `main` matches. A plain revert without the
rollback script is stopped by the downgrade guard.

**Data loaders** (fuel-db). Every table in `public` must belong to `fuel_owner`. `fuel_ro` can
read new tables only through `fuel_owner`'s default privileges, and PostgreSQL applies those
only to objects `fuel_owner` creates itself. A table that a loader creates as its own role stays
unreadable for Superset and the Petrolyst server. Create each loader role so that its sessions act as `fuel_owner`, and it
cannot create anything as itself. Run this in
`docker exec -it petrolyst-fuel-db-1 psql -U postgres -d fuel`:

```sql
CREATE ROLE <loader> LOGIN PASSWORD '<password>';
GRANT fuel_owner TO <loader> WITH INHERIT FALSE, SET TRUE;
ALTER ROLE <loader> SET role TO 'fuel_owner';
```

Tables it creates, hypertables included, then belong to `fuel_owner`, and `fuel_ro` can read them.
After `SET ROLE NONE`, it cannot create anything in `public`. To repair a table that another
role created, run `ALTER TABLE <t> OWNER TO fuel_owner`, then `GRANT SELECT ON <t> TO fuel_ro`.

**Removing the old project**, once, right before the next deploy; delete this paragraph when it
is done. The server still runs the Compose project `fuel-monitoring`, with Trino. Afterwards it runs
`petrolyst`, without it: containers are `petrolyst-<service>-1`, and the data volumes keep their
names (pinned in `compose.yaml`), so no data moves. The old containers must be gone first: two
PostgreSQL servers on one data volume corrupt it, and preflight refuses to deploy while any exists.

As root on the server (Superset is down from step 3 until the deploy finishes). The role keeps its
password, because a SCRAM password survives a rename:

```bash
S=/opt/petrolyst/secrets/fuel-monitoring
mv "$S/fuel_db_trino_password" "$S/fuel_db_ro_password"                       # 1
docker exec -i fuel-monitoring-fuel-db-1 psql -U postgres -d fuel -v ON_ERROR_STOP=1 \
  -c 'ALTER ROLE trino_ro RENAME TO fuel_ro' \
  -c 'ALTER ROLE fuel_ro SET default_transaction_read_only = on'               # 2
docker rm -f $(docker ps -aq --filter label=com.docker.compose.project=fuel-monitoring)   # 3
```

4. Deploy as usual ("Manual deploy"). Compose warns once that the volumes belong to another project;
   that is expected. `superset-init` registers the connection "Fuel"; the smoke test checks that it
   reads as `fuel_ro`, read-only.
5. Delete the old connection. It refuses if a dataset still uses it; then stop and look:

   ```bash
   docker exec -i petrolyst-superset-1 /app/.venv/bin/python - <<'PY'
   from superset.app import create_app

   with create_app().app_context():
       from superset import db
       from superset.connectors.sqla.models import SqlaTable
       from superset.models.core import Database

       old = db.session.query(Database).filter_by(database_name="Fuel (Trino)").one_or_none()
       if old:
           used = db.session.query(SqlaTable).filter_by(database_id=old.id).count()
           if used:
               raise SystemExit(f"{used} datasets still use the old connection")
           db.session.delete(old)
           db.session.commit()
           print("old connection deleted")
   PY
   ```
6. Clean up: `rm /opt/petrolyst/env/fuel-monitoring.env`, `rmdir /opt/petrolyst/env`,
   `docker image rm trinodb/trino:484`,
   `docker network rm fuel-monitoring_backend fuel-monitoring_fuel-db-tunnel`.
7. Releases from before this deploy can no longer run, as their role is gone, and `rollback.sh`
   stops on them. Remove them from `/opt/petrolyst/releases/`.

**Adding a project**: create `stacks/<name>/compose.yaml`, attach the public service to the
external network `proxy`, and set `traefik.*` labels with a unique router name and
`tls.certresolver: le`. Add the DNS record first. `deploy.sh` picks the new folder up
automatically.

`sso@docker` works only for `traefik.petrolyst.com`. Its cookie, its allowed return hosts and its
role (`traefik-admin`) are all tied to the dashboard. An app without a login of its own needs
its own gate in its stack:
- an oauth2-proxy service with its own cookie secret;
- a router for the app host's `/oauth2/` path and a forwardAuth middleware pointing at that
  service, as in `traefik/compose.yaml`;
- `--redirect-url`, `--cookie-domain` and `--whitelist-domain` set to the app's host;
- an `--allowed-role` with a new client role in Keycloak;
- the app's `/oauth2/callback` added to the `petrolyst` client's redirect URIs.

## Verified locally (2026-10-23)

An end-to-end run ran the real `deploy.sh` and `rollback.sh` in a Linux container against
Docker Desktop, with a scratch `/opt/petrolyst`. Production differences in that run:
- a test CA certificate stood in for Let's Encrypt;
- the ACME server in the test copy was unreachable on purpose;
- a dummy Keycloak client secret was used.

Results:
- The fresh install took the init path.
- An identical re-run recreated nothing and left `ARTIFACTS` byte-identical.
- A Superset change stopped, backed up, migrated and recreated only `superset`.
- A change to `superset/init.sh` alone did the same, and the changed script ran.
- A same-SHA upload with different content was refused.
- An older image against a newer schema was refused before promotion.
- A migration rollback dropped a table added by a "newer migration" and restored the revision.
- A code-only rollback brought back the previous image.
- A re-deploy of an unchanged image, against a schema one migration behind its head, migrated
  the schema back to the head.
- A migration rollback whose restore failed after the drop left `rollback-in-progress` and an
  empty database. `deploy.sh` then refused to run. Running the rollback again restored the
  database and removed the file. It did the same after a simulated stop between dropping and
  recreating the database. Without the file, a missing database stops the rollback with an error.
- The host lock blocked a concurrent run.
- Retention pruned read-only releases.
- Retention removed a queued upload that a plain `rsync -a` had given an old checkout's
  directory times. It kept one uploaded with `--omit-dir-times`.
- In `fuel-db`, a loader that created a table as its own role left it unreadable for the read-only
  role. With the "Data loaders" recipe, tables (hypertables too) belonged to `fuel_owner` and that
  role could read them. After `SET ROLE NONE`, the loader could not create anything.
- Smoke tests passed, including the redirects to the real Keycloak with `client_id=petrolyst`
  and PKCE S256.
- TLS 1.1 and unknown SNI were refused, and HTTP redirected to HTTPS.

Not verified locally:
- Let's Encrypt issuance;
- a real Keycloak sign-in, OTP and logout (they need the client from setup step 2);
- the theme as rendered in a browser.

## Open items

- Exhibit B: encryption at rest; confirm the VPS is in a German data centre.
- Nightly `pg_dump` of both databases with an offsite copy. Today only the pre-migration
  Superset dumps exist.
- Certificate expiry monitoring. There is no ACME e-mail, so nothing warns if renewal fails
  between deploys.
- Celery worker and beat, if Superset alerts or reports are needed.
- A Petrolyst login theme for Keycloak itself, which lives on the Keycloak host.