From d3800ac9dd763e9e53880df2c815a1546e02eab7 Mon Sep 17 00:00:00 2001 From: Nasim Sadat Date: Sat, 24 Oct 2026 00:33:11 +0200 Subject: [PATCH] Petrolyst server infrastructure: Traefik, Trino and Superset Traefik as the server-wide edge (TLS via Let's Encrypt, dashboard on traefik.petrolyst.com behind oauth2-proxy and Keycloak). The fuel-monitoring stack: Superset on atoma.petrolyst.com with Keycloak login and Petrolyst branding, Trino reading the TimescaleDB fuel-db as catalog "fuel". Customization lives in petrolyst-insights/, deployment in the other folders. Deploys run server-side (deploy/deploy.sh) from a Gitea workflow or manually; releases are immutable, Superset metadata is backed up before migrations, and deploy/rollback.sh restores it. Co-Authored-By: Claude Opus 5.5 --- .gitea/workflows/deploy.yml | 57 ++++ .gitignore | 11 + README.md | 305 ++++++++++++++++++ deploy/deploy.sh | 163 ++++++++++ deploy/lib.sh | 271 ++++++++++++++++ deploy/rollback.sh | 91 ++++++ host/bootstrap.sh | 105 ++++++ host/secrets-init.sh | 86 +++++ keycloak/petrolyst-partial-import.json | 56 ++++ petrolyst-insights/README.md | 73 +++++ .../assets/branding/favicon.ico | Bin 0 -> 441 bytes .../assets/branding/favicon.svg | 2 + .../assets/branding/petrolyst_icon_180.png | Bin 0 -> 4691 bytes .../assets/branding/petrolyst_icon_192.png | Bin 0 -> 5000 bytes ...yst_logo_standard_fullcolor_no_tagline.svg | 35 ++ ...trolyst_logo_standard_white_no_tagline.svg | 17 + petrolyst-insights/dashboards/README.md | 14 + petrolyst-insights/patches/README.md | 16 + petrolyst-insights/petrolyst_security.py | 92 ++++++ petrolyst-insights/superset_config.py | 252 +++++++++++++++ stacks/fuel-monitoring/compose.yaml | 174 ++++++++++ .../fuel-monitoring/fuel-db/init/01-roles.sh | 25 ++ stacks/fuel-monitoring/superset/Dockerfile | 28 ++ stacks/fuel-monitoring/superset/init.sh | 14 + .../trino/etc/access-control.properties | 3 + .../trino/etc/catalog/fuel.properties | 5 + .../trino/etc/config.properties | 14 + stacks/fuel-monitoring/trino/etc/jvm.config | 14 + .../fuel-monitoring/trino/etc/log.properties | 1 + .../fuel-monitoring/trino/etc/node.properties | 3 + traefik/compose.yaml | 117 +++++++ traefik/dynamic/middlewares.yml | 10 + traefik/dynamic/tls.yml | 7 + traefik/static/traefik.yml | 56 ++++ 34 files changed, 2117 insertions(+) create mode 100644 .gitea/workflows/deploy.yml create mode 100644 .gitignore create mode 100644 README.md create mode 100755 deploy/deploy.sh create mode 100644 deploy/lib.sh create mode 100755 deploy/rollback.sh create mode 100755 host/bootstrap.sh create mode 100755 host/secrets-init.sh create mode 100644 keycloak/petrolyst-partial-import.json create mode 100644 petrolyst-insights/README.md create mode 100644 petrolyst-insights/assets/branding/favicon.ico create mode 100644 petrolyst-insights/assets/branding/favicon.svg create mode 100644 petrolyst-insights/assets/branding/petrolyst_icon_180.png create mode 100644 petrolyst-insights/assets/branding/petrolyst_icon_192.png create mode 100644 petrolyst-insights/assets/branding/petrolyst_logo_standard_fullcolor_no_tagline.svg create mode 100644 petrolyst-insights/assets/branding/petrolyst_logo_standard_white_no_tagline.svg create mode 100644 petrolyst-insights/dashboards/README.md create mode 100644 petrolyst-insights/patches/README.md create mode 100644 petrolyst-insights/petrolyst_security.py create mode 100644 petrolyst-insights/superset_config.py create mode 100644 stacks/fuel-monitoring/compose.yaml create mode 100755 stacks/fuel-monitoring/fuel-db/init/01-roles.sh create mode 100644 stacks/fuel-monitoring/superset/Dockerfile create mode 100755 stacks/fuel-monitoring/superset/init.sh create mode 100644 stacks/fuel-monitoring/trino/etc/access-control.properties create mode 100644 stacks/fuel-monitoring/trino/etc/catalog/fuel.properties create mode 100644 stacks/fuel-monitoring/trino/etc/config.properties create mode 100644 stacks/fuel-monitoring/trino/etc/jvm.config create mode 100644 stacks/fuel-monitoring/trino/etc/log.properties create mode 100644 stacks/fuel-monitoring/trino/etc/node.properties create mode 100644 traefik/compose.yaml create mode 100644 traefik/dynamic/middlewares.yml create mode 100644 traefik/dynamic/tls.yml create mode 100644 traefik/static/traefik.yml diff --git a/.gitea/workflows/deploy.yml b/.gitea/workflows/deploy.yml new file mode 100644 index 0000000..c2a8468 --- /dev/null +++ b/.gitea/workflows/deploy.yml @@ -0,0 +1,57 @@ +# Every push to main deploys everything: Traefik first, then each stack under +# stacks/. The runner only uploads; the server validates, deploys and +# smoke-tests (deploy/deploy.sh), so this job needs no Docker of its own. +name: deploy + +on: + push: + branches: [main] + workflow_dispatch: + +# Serializes CI runs; deploy.sh's host lock also covers manual rollbacks. +concurrency: + group: deploy-petrolyst + cancel-in-progress: false + +jobs: + deploy: + runs-on: ubuntu-latest + timeout-minutes: 45 + env: + DEPLOY_HOST: ${{ secrets.DEPLOY_HOST }} + steps: + - uses: actions/checkout@v4 + + - name: Lint shell scripts + run: | + if ! command -v shellcheck >/dev/null; then + SUDO=$(command -v sudo || true) + $SUDO apt-get update -qq && $SUDO apt-get install -y -qq shellcheck + fi + shellcheck -x host/*.sh deploy/*.sh stacks/*/superset/init.sh stacks/*/fuel-db/init/*.sh + + - name: Install the deploy key + env: + DEPLOY_SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }} + DEPLOY_KNOWN_HOSTS: ${{ secrets.DEPLOY_KNOWN_HOSTS }} + run: | + install -d -m 0700 "$HOME/.ssh" + printf '%s\n' "$DEPLOY_SSH_KEY" > "$HOME/.ssh/deploy_key" + chmod 0600 "$HOME/.ssh/deploy_key" + printf '%s\n' "$DEPLOY_KNOWN_HOSTS" > "$HOME/.ssh/known_hosts" + command -v rsync >/dev/null || { SUDO=$(command -v sudo || true); $SUDO apt-get install -y -qq rsync; } + + - name: Upload and deploy + run: | + ssh_cmd="ssh -i $HOME/.ssh/deploy_key -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes -o UserKnownHostsFile=$HOME/.ssh/known_hosts" + # A fresh directory per run; only deploy.sh, under the host lock, + # turns it into releases/. --omit-dir-times: the upload's age + # must be the server's, because retention prunes uploads by mtime. + incoming="/opt/petrolyst/incoming/${GITHUB_SHA}-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT:-1}-$(date -u +%s)" + rsync -az --omit-dir-times --exclude .git -e "$ssh_cmd" ./ "deploy@${DEPLOY_HOST}:${incoming}/" + $ssh_cmd "deploy@${DEPLOY_HOST}" "${incoming}/deploy/deploy.sh" "$GITHUB_SHA" "$incoming" + + - name: Check from outside + run: | + test "$(curl -fsS --max-time 20 https://atoma.petrolyst.com/health)" = OK + curl -fsS --max-time 20 -o /dev/null -w '%{http_code}\n' https://traefik.petrolyst.com/ | grep -qx 302 diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..afd37f3 --- /dev/null +++ b/.gitignore @@ -0,0 +1,11 @@ +# Secrets and server env files live only on the server (/opt/petrolyst/). +*.env +.env* +secrets/ +# CI deploy key, generated here during first-time setup (README, Gitea step 3) +petrolyst-ci +petrolyst-ci.pub +# Local test runs +.scratch/ +.DS_Store +__pycache__/ diff --git a/README.md b/README.md new file mode 100644 index 0000000..14d8d0b --- /dev/null +++ b/README.md @@ -0,0 +1,305 @@ +# petrolyst-infra + +Everything that runs on the petrolyst server (109.199.127.84, Ubuntu 24.04), deployed on every +push to `main`: + +- **Traefik** at the edge for every project on the server. Its dashboard is at + https://traefik.petrolyst.com, behind Keycloak. +- **fuel-monitoring** (ATOMA): Superset at https://atoma.petrolyst.com, reading the operational + store `fuel-db` (Postgres 17 with TimescaleDB) through Trino. Sign-in goes through Keycloak + client `petrolyst`, and the UI carries the Petrolyst brand. + +Customization and deployment are separate, as with media-insights and media-intelligence-docker: + +| Folder | Holds | +|---|---| +| `petrolyst-insights/` | how Superset looks and behaves for Petrolyst: config, Keycloak login code, branding, patches, dashboards. See its README. | +| `traefik/` | edge proxy, TLS, the `sso` middleware (oauth2-proxy) that guards the dashboard | +| `stacks/fuel-monitoring/` | Compose stack, Superset Dockerfile, Trino config and catalog, fuel-db roles | +| `deploy/` | `deploy.sh` (run by CI), `rollback.sh` (manual), `lib.sh` (shared) | +| `host/` | one-time server setup: `bootstrap.sh`, `secrets-init.sh` | +| `keycloak/` | the `petrolyst` client for Keycloak's partial import | +| `.gitea/workflows/deploy.yml` | the pipeline | + +No credential is in this repository. Secrets exist only on the server, under +`/opt/petrolyst/secrets/`; CI holds only the SSH deploy key. + +## How a deploy works + +1. A push to `main` (or a manual run) starts `.gitea/workflows/deploy.yml` on the act_runner. It + lints the scripts, rsyncs the checkout to a fresh `/opt/petrolyst/incoming/-/`, and + calls `deploy/deploy.sh ` over SSH as user `deploy`. +2. `deploy.sh` takes the host lock `/opt/petrolyst/deploy.lock`, which it shares with + `rollback.sh`, and keeps it until it exits. +3. It finalizes the upload. A new commit is moved to `releases/`, and the release is made + read-only. A re-run of a deployed commit must match it byte for byte; a mismatch aborts. +4. It resolves the artifacts and writes them to `releases//ARTIFACTS`: + - `SUPERSET_IMAGE_TAG` is a hash of exactly the Superset image's build inputs. They include + `superset/init.sh`, so changing that file runs `superset-init` again. + - `TRINO_CONFIG_HASH` and `TRAEFIK_CONFIG_HASH` are hashes of the bind-mounted config. + - Compose recreates a container only when its value changes. +5. Preflight, before anything live changes: + - every secret and env file exists; + - `petrolyst-insights/` holds no deployment files; + - `docker compose config` passes for every stack; + - the Superset image is built if it is new; + - the **downgrade guard** refuses an image older than the metadata database's schema. +6. It promotes the release to `/opt/petrolyst/infra`, then starts Traefik, then each stack under + `stacks/`. +7. For Superset, it reads the metadata database's state: + - An empty database is a fresh install: it runs `superset-init`. + - If the image changed, or the database is not at the image's migration head: it stops + `superset`, writes `backups/superset-meta___