petrolyst-infra/petrolyst-insights/dashboards/atoma/SPEC.md

# ATOMA dashboards: Sites Map and Site Detail

**Status: proposed, not built.**
- `build_bundle.py` (phase P2) will generate both dashboards, every dataset and every chart from
  this specification, as one Superset import bundle.
- Nothing is imported into the live Superset without the owner's go-ahead (phase P3).

**Datasets.** The `public.v_*` views of
`stacks/fuel-monitoring/fuel-db/migrations/0003_bi_views.sql`, read through Trino as
`fuel.public`.

**Design and rationale** live in the fuel-monitoring corpus, in
`.context/plans/2026-10-24-fuel-db-baseline-dashboard.md`:
- its §6 is this specification;
- the site-specific facts left out here (positions, distances, pilot sites) are there too;
- Q-numbers refer to its §7, and phases P0–P5 to its §8.

Two dashboards replace the earlier tab-per-site design; nothing in them is specific to a site.

## 1. Capabilities of the installed Superset 6.1.0

Source: read-only inspection of `config.py`, `superset_config.py` and the frontend bundle on
24 Oct 2026. Items marked *(local test)* are confirmed in P2 before the design relies on them.

| Requirement | What 6.1.0 provides | Choice |
|---|---|---|
| Map chart | `deck_scatter` (deck.gl Scatterplot): a modern plugin with the InteractiveChart behaviour. Also: deck_multi, deck_screengrid / hex / grid, country_map, world_map, cartodiagram, legacy `mapbox` | **deck.gl Scatterplot** |
| Basemap | `DECKGL_BASE_MAP` unset → OSM raster (`tile.openstreetmap.org`), OSM Swiss topo, Mapbox styles. `MAPBOX_API_KEY` is empty, so the Mapbox styles fail. CSP `connect-src` already allows the OSM host; Traefik sets no CSP | set `DECKGL_BASE_MAP` to **OSM only**; "© OpenStreetMap contributors" in a Markdown under the map |
| OSM terms | no key, no cost; tile usage policy: light use, attribution, no bulk download, a valid Referer (ours sends the origin) | fine for an internal user base; viewers' browsers reveal the viewed area (tile coordinates) and their IP to OSMF, not our data. Self-hosted tiles later if needed |
| Tooltip | "Tooltip contents" (drag columns) plus "Customize tooltips template" (Handlebars). `js_tooltip` exists but `ENABLE_JAVASCRIPT_CONTROLS` is False | contents and template; **no JavaScript** |
| Marker colour | "Categorical color" by a dimension, from the colour scheme; a fixed colour per label through the dashboard's `label_colors` *(local test)* | colour by `baseline_status_label`. Fallback: `deck_multi` with one fixed-colour scatter layer per status |
| Marker size | fixed radius, with minimum = maximum pixel radius | uniform |
| Viewport | `autozoom` fits the filtered points | on |
| Clustering | none in deck.gl; only the legacy `mapbox` chart (`clustering_radius`, needs a Mapbox key) | **no clustering** (§3) |
| Marker click | emits a **cross-filter** on the latitude and longitude columns: the exact point, or a box around the click that shrinks as you zoom in. It applies to other charts on the same dashboard | cross-filters the Sites list |
| Navigation | **no chart-to-dashboard navigation.** A dashboard reads `?native_filters=<rison>` into preselected native filter values (also `native_filters_key`). Table cells render sanitized HTML (`allow_render_html`) | an "Open site detail" link column in the Sites list |
| Native filters | Select (single or multi, required, searchable, "first value by default"), Time range; scoping per chart and tab; cross-filter scoping per chart | §4, §5 |

No custom frontend and no patch are needed.

## 2. Navigation flow

```
ATOMA — Sites  (landing, slug atoma-sites)
  filters: Site search · Zone · Province · Pilot status (default: active) · Baseline status
  M1 map: markers fitted to the filtered sites ──hover──► tooltip
          └─click marker──► cross-filter: M3 Sites list shows that site (or the few under the click)
  M3 Sites list ─────────────"Open site detail"──► ATOMA — Site Detail ?native_filters=(site preset)
  M4/M5 Sites without an accepted location ──────"Open site detail"──► same
ATOMA — Site Detail  (slug atoma-site-detail)
  filters: Site (exactly one, searchable) · Viewing period
  tabs: Summary · Fuel & Deliveries · Runtime & Electricity · Baseline & Evidence
  header link "← Sites map"
```

## 3. Sites Map (landing dashboard)

```
[Filter bar: Site search · Zone · Province · Pilot status · Baseline status]
[M1 Map (deck.gl Scatterplot) ................................ w8][M2 Status key + attribution w4]
[M3 Sites list (table with link column) ..................................................... w12]
[M4 Sites without accepted location: count w3][M5 Sites without accepted location (table) .. w9]
```

| # | Question | Type | Dataset | Content | Missing or insufficient |
|---|---|---|---|---|---|
| M1 | Where are the sites, and what is each one's baseline status? | deck.gl Scatterplot; lat/lon from `latitude`, `longitude`; uniform size; colour = `baseline_status_label`; autozoom | v_site | tooltip: `site_code`, `site_name`, `lph_estimate` ("not available" when NULL), `lph_method`, `baseline_status_label`, `baseline_validity_note` (when set), `latest_record_on` | sites without an accepted location have NULL lat/lon and are not plotted; M4/M5 show them |
| M2 | What do the colours mean? | Markdown (static) | — | the status definitions below; "Colours show evidence status, not performance"; OSM attribution | — |
| M3 | Which sites match, and how do I open one? | Table; search box; server pagination | v_site | code, name, province, pilot status, baseline status, L/h (+ method), L/day, latest record, location status, `detail_link` | marker click narrows it; "not available" shown as text |
| M4 | How many sites lack an accepted location? | Big Number Total, COUNT(*) with saved filter `location_status <> 'accepted'` | v_site | count; follows the dashboard filters | 0 is shown as 0 |
| M5 | Which sites are they? | Table; search box | v_site, same saved filter | code, name, province, `location_status` (awaiting acceptance · missing), baseline status, `detail_link` | — |

**Filters** (all on `v_site`, scoped to M1, M3, M4 and M5):
- Site search: Select on `site_label`, multi, searchable. The label holds code, name and uid, so
  code or name both match.
- Zone, Province: Select.
- Pilot status: default `active`.
- Baseline status: Select on `baseline_status_label`.

The cross-filter from M1 is scoped **to M3 only**. M4 and M5 hold the sites without coordinates and
must not be emptied by it.

**Baseline status and marker colours.** Colours are neutral categorical hues, with no red or green
and no reference to L/h:

| Status | Definition (`fm.v_site_baseline_status`) | Colour |
|---|---|---|
| Approved baseline | an approved `baseline_version` with a decision reference. The **only** state that says "approved" | teal #009688 |
| Eligible – not yet approved | no approval; the latest proposed KPI version has `meets_policy` (BP-1: ≥ 2 eligible G1 inputs, ≥ 30 matched days, no conditions, one population and metric) | blue #3987e5 |
| Provisional | no approval; a proposed version below BP-1, **or** usable intervals with no version yet (no estimate shown, method "no baseline version proposed") | amber #c98500 |
| Insufficient evidence | no usable interval. **Every pilot site while no runtime evidence exists** | slate #64748B |

**Nearby sites.**
- Sites a few kilometres apart separate at zoom ≥ 11 (≈ 60 px per 2–4 km at zoom 12) and coincide
  at country zoom (≈ 2 km per pixel at zoom 6). The pilot's own distances are in the design doc.
- Handling:
  - autozoom to the filtered sites (the default "active" filter opens on the pilot area);
  - small uniform markers (min = max radius, about 7 px) with an outline;
  - a click near overlapping markers selects every site in the click box, and M3 lists them;
  - no clustering.
- With hundreds of sites, a `deck_multi` density layer (screengrid) for low zoom is a later option.

## 4. Site Detail (one dashboard, any site)

**Filters**
- **Site** (`NATIVE_FILTER-site`):
  - Select on `v_site.site_label`, single value, **required**, searchable, no default;
  - scope: every chart on every tab;
  - every detail dataset carries `site_label` from `v_site`, so this one filter drives all of
    them *(local test)*.
- **Viewing period** (`NATIVE_FILTER-period`):
  - Time range, default no filter;
  - scope: the time-series charts on Fuel & Deliveries and Runtime & Electricity only;
  - Summary and Baseline & Evidence are outside its scope, so a viewing range never changes a
    baseline's fixed window.

**Exactly one site, and never network totals:**
1. A required, single-value filter *(local test: with the value cleared, no chart queries
   unfiltered data)*.
2. The header (D0) and KPI (S1) are Handlebars templates that render a site only when exactly one
   row arrives. Otherwise they show "Select one site".
3. No Big Number or aggregate tile for site figures. They are computed per site in `v_site`,
   never aggregated by Superset.
4. Fallback, if (1) fails its test: "Select first filter value by default", so the Site filter
   is never empty.

```
[D0 Site header: code + name (large) · province · pilot status · baseline status · "← Sites map"]
Summary              [S1 L/h KPI card w6][S2 Supporting measures w6]
                     [S3 Evidence coverage timeline (Gantt) w12]
Fuel & Deliveries    [F1 Deliveries w6][F2 Recorded stock w6]
                     [F3 Fuel per calendar day by interval w12]
                     [F4 Interval table w12]
Runtime & Electricity[R1 Runtime per day by meter interval w6][R2 Hour-meter evidence w6]
                     [R3 Grid energy, kWh w6][R4 Electricity cost, AFN w6]
Baseline & Evidence  [B1 Baseline versions w6][B2 Inputs of each version w6]
                     [B3 All intervals: eligibility and exclusions w12]
                     [B4 Interval L/h against baseline w12]
                     [B5 Assumptions w6][B6 Missing data and quality checks w6]
                     [B7 Evidence references w6][B8 Workbook calculations, assessed w6]
                     [B9 Stock against engine-measured fuel w6][B10 Equipment specifications w6]
                     [B11 Modelled L/h under an assumed runtime w12]
```

| # | Question it answers | Type | Dataset | Dims / measures (units) | Grain | When data are missing or insufficient |
|---|---|---|---|---|---|---|
| D0 | Which site am I looking at? | Handlebars | v_site | code and name (large), province, pilot status, baseline status, location status, link "← Sites map" | site | "Select one site" unless exactly one row |
| S1 | What is the site's L/h, by which method, over which window, how strong? | Handlebars | v_site | `lph_estimate`, `lph_method`, `baseline_status_label`, `baseline_validity_note`, window, decision; L/day whole record | site | Keeps its position. **In v1 every site shows "Insufficient runtime evidence: no hour-meter readings"**, plus L/day labelled "per calendar day, not per running hour". "Eligible" and "Provisional" carry "not approved" |
| S2 | What supports it? | Table | v_kpi_measures | measure, value, unit, `value_kind` (recorded · calculated · assumed · not established), window. Two L/day rows: the whole record (moves with new records) and the L/day reference version (fixed window, its status and any input drift in the note) | site × measure | Rows stay and show "not established" |
| S3 | Which periods are covered by which evidence, and where are the gaps? | Gantt | v_coverage | lane; start/end; colour = status | span | Empty lanes are visible; nothing is interpolated |
| F1 | When and how much was delivered? | Bar (time axis, **day** grain) | v_delivery (one row per delivery event) | `delivery_date`; **SUM(`delivered_l`)**, titled "Litres delivered per day (all deliveries that day)"; tooltip COUNT(deliveries) | day | Bars only on delivery dates; subtitle "deliveries are not consumption"; F4 lists the individual deliveries |
| F2 | What stock was recorded? | Scatter (no lines) | v_stock_reading | `observed_on`; litres; series = reading type × `value_kind` | observation | Points only; nothing between readings |
| F3 | How much fuel was used per calendar day in each observed interval? | Bar (categorical x, ordered by start) | v_interval | `interval_label`; MAX(`fuel_l_per_day`) (L/d); colour = `stock_status` | stock interval | Excluded intervals greyed, with the reason in the tooltip; no monthly spreading |
| F4 | Interval detail | Table | v_interval | dates, opening, deliveries, closing, used, L/d, status | interval | Shows the reasons |
| R1 | How many hours did each generator run per day? | Bar (categorical) | v_runtime | interval × meter; MAX(h/day). h/day is NULL for an invalid pair (`pair_valid` false: two epochs, a decreasing timer or over 24 h/day); such pairs show no bar and are listed with their flags in the tooltip | meter interval | **v1: empty**, with "No hour-meter readings" in the description |
| R2 | What runtime evidence exists? | Table | v_hour_meter_evidence | meter, mapped set, date, hours, state, epoch | reading | **v1: no rows**, "No hour-meter readings" |
| R3 | How much grid energy did the site receive? | Handlebars → bar once kWh exist | v_kpi_measures (kWh row) | kWh per bill month | month | "No kWh records; bills are cost only and are not converted to fuel or hours" |
| R4 | What did grid electricity cost? | Bar (month) + table | v_electricity_cost | bill month; amount (AFN, inferred) | month | Own chart, never on a fuel axis. A disputed bill is listed without amount or kWh |
| B1 | Which baseline versions exist, and who approved them? | Table | v_baseline_version | code, status, value, window, grade, decision | version | "No baseline defined" |
| B2 | What went into each version? | Table | v_baseline_input | interval, included, reason, fuel, runtime, drift | version × interval | empty until a version exists |
| B3 | Which intervals are usable, and why not? | Table | v_interval | eligibility, exclusion reasons, grade, conditions, in current baseline | interval | v1: every interval excluded, "no timer reading at both ends" (and "a generator has neither a counter nor an idle assumption") |
| B4 | What L/h did each interval give, against the approved baseline? | Mixed: bars + line | v_interval | x `interval_label`; A: MAX(`l_per_h`) where eligibility ≠ excluded; B: MAX(`current_baseline_l_per_h`) (approved value only, and only beside intervals of the same metric and population, `baseline_comparable`) | interval | v1: no bars; description points to S1 |
| B5 | Which assumptions apply? | Table | v_assumption | code, topic, value and range, basis, status, rationale | assumption | "None recorded" |
| B6 | What is missing or inconsistent? | Table | v_data_quality | check, status, detail | check | Lists the gaps: no runtime, placeholder rows, no kWh, no transfer records, location awaiting acceptance |
| B7 | Which documents do the figures rest on? | Table | v_evidence | title, kind, issuer, date, used for | document | — |
| B8 | What did the operator workbook calculate, and is it usable? | Table | v_imported_calculation | label, formula, value, our reading, use, reason, `claim_status` | cell | Shows "excluded" with the reason for "actual L/h" and "CP L/h". A disputed transcription shows its status, without formula or value |
| B9 | Does stock depletion match engine-measured fuel? | Table | v_reconciliation | stock L, engine L, `covered_h` of `interval_h`, difference L and %. The window runs from the opening visit's departure to the closing visit's arrival. The difference is shown only for an `ok` stock interval with both visit times recorded, in which every generator is placed at the site throughout and covered for the whole window | interval × generator | "No flow-meter data yet"; partial coverage, unknown visit times or a mid-interval placement show the hours covered and no difference |
| B10 | What does the equipment specification say? | Table | v_spec_reference | attribute, value, unit, basis, applicability, `claim_status` | spec | Titled "Manufacturer and nameplate data, not site performance". A disputed specification shows its status, without value |
| B11 | What would L/h be under the stated runtime? | Table | v_modelled_lph | assumption code, status, basis; h/day with low–high; window, `matched_days`; L/day; modelled L/h with low–high; `value_kind` | site × assumption | Titled "Modelled from an assumed runtime, not measured". "No runtime assumption recorded" |

Modelled L/h (B11) is in v1 (Q9, decided 24 Oct 2026). It pools only the usable intervals inside
the runtime assumption's dates, and shows that window and its days. It never reaches S1, the map or
the M3 Sites list.

## 5. Datasets and filter mapping

Physical datasets, one per `public.v_*` view on Trino `fuel.public`. Every dataset has the same
four site columns from `v_site`: `site_uid`, `site_code`, `site_name`, `site_label`.

| Dataset | Grain | Used by | Native filters that apply |
|---|---|---|---|
| v_site | site | M1–M5, D0, S1 | landing: all five filters, plus the M1 cross-filter (on M3); detail: Site |
| v_kpi_measures | site × measure | S2, R3 | Site |
| v_coverage | site × lane × span | S3 | Site |
| v_delivery | delivery | F1 | Site, Viewing period |
| v_stock_reading | observation | F2 | Site, Viewing period |
| v_interval | stock interval | F3, F4, B3, B4 | Site; Viewing period on F3 and F4 only |
| v_runtime | meter interval | R1 | Site, Viewing period |
| v_hour_meter_evidence | reading | R2 | Site, Viewing period |
| v_electricity_cost | bill month | R4 | Site, Viewing period |
| v_baseline_version, v_baseline_input | version (× interval) | B1, B2 | Site |
| v_assumption, v_data_quality, v_evidence, v_imported_calculation, v_reconciliation, v_spec_reference, v_modelled_lph | per view | B5–B11 | Site |

**Metrics.** Saved metrics enforce pooling, with the same eligibility filter on numerator and
denominator, e.g. `pooled_l_per_h = SUM(CASE WHEN eligibility <> 'excluded' THEN fuel_used_l END)
/ NULLIF(SUM(CASE WHEN eligibility <> 'excluded' THEN runtime_h END), 0)`. A chart that uses it
is grouped or filtered by `metric` and `population_key` (both in `v_interval`), so it never pools
across methods or generator sets. The site's L/h figure itself comes only from a baseline version
(`v_site`), never from this metric. There is no AVG metric on ratio columns.

**Link column.** `detail_link` is a calculated column on the `v_site` dataset, defined in the
bundle in Trino SQL. The native filter id stays inside Superset's own configuration:

```sql
'<a href="/superset/dashboard/atoma-site-detail/?native_filters='
  || url_encode('(NATIVE_FILTER-site:(id:NATIVE_FILTER-site,'
       || 'extraFormData:(filters:!((col:site_label,op:IN,val:!(' || q || ')))),'
       || 'filterState:(value:!(' || q || ')),ownState:()))')
  || '">Open site detail</a>'
-- q = '''' || replace(replace(site_label, '!', '!!'), '''', '!''') || ''''   -- rison string quoting
```

- The exact rison state shape is a *(local test)*.
- Fallback: the link opens Site Detail with the Site filter expanded (`expand_filters=1`) and the
  site's label shown beside the link, to pick by search.

## 6. Build as code, scale, access

**Build**
- `petrolyst-insights/dashboards/atoma/build_bundle.py` writes one import bundle containing both
  dashboards, every dataset and every chart:
  - stable UUIDs (uuid5), so a re-import updates instead of duplicating;
  - fixed native-filter ids (`NATIVE_FILTER-site`, `-period`, `-search`, …);
  - `label_colors` for the four statuses;
  - the existing "Fuel (Trino)" database UUID, read from an export first.
- Import: `superset import-dashboards`.

**Adding a site**
- Import its `site.yaml` and its records. Nothing in either dashboard names a site.
- New sites appear in M1 (once their location is accepted) or in M5, in M3, and in the Site filter.

**Hundreds of sites**
- `v_site` has one row per site, and the scatter layer handles thousands of points.
- The Select filters load the distinct labels; M3 and M5 paginate on the server.

**Superset configuration change** (petrolyst-insights; image rebuild in P3):
`DECKGL_BASE_MAP = [["tile://https://tile.openstreetmap.org/{z}/{x}/{y}.png", "OpenStreetMap"]]`.
The CSP needs no change *(local test: no CSP violation in the browser console)*.

**Access, internal first.** Both dashboards for Admin and Alpha only. No Gamma role until the
ATOMA-facing edit.

## 7. Acceptance checks

Run in P2 against a local Superset, and repeated as a smoke test in P3.

1. **New site, no new objects.** Count dashboards, charts and datasets in the metadata database.
   Then import a synthetic fourth site with records and an accepted location, and a fifth with no
   location. The counts are unchanged; site 4 appears in M1, M3 and the Site filter; site 5
   appears in M5, and M4 goes up by 1.
2. **Site switching.** In Site Detail, select pilot site A, then site B. D0 shows the new code and
   name; every chart's query (View query, or chart-data API) carries `site_label = …<site B>…`; no
   chart keeps site A's rows.
3. **Clearing the selection.** With the Site filter cleared, no chart shows multi-site data; D0 and
   S1 show "Select one site". If not, switch to the §4 fallback and re-test.
4. **Unavailable L/h.** For each pilot site:
   - the tooltip and M3 say "not available" with the method "no runtime evidence";
   - S1 shows the insufficient-evidence state plus L/day;
   - B3 shows every interval excluded, with its reason;
   - B11 shows a modelled range labelled "not measured", and that figure appears nowhere else.
5. **Missing coordinates.** A site without an accepted location is not plotted, is listed in M5
   and counted in M4, and its link opens its detail.
6. **Filter propagation.**
   - On the landing page, each filter changes M1, M3, M4 and M5, and the default "active" shows
     exactly the active pilot sites.
   - A marker click filters M3 only.
   - In Site Detail, the Viewing period changes F1–F3 and R1, R2, R4, but not S1 or B1–B4.
7. **Detail links.** For each active pilot site, "Open site detail" in M3 opens Site Detail with
   exactly that site selected, and D0 matches. Same from M5.
8. **Colours.**
   - Fixtures in each status show the four colours.
   - "Eligible" never renders as "Approved" unless the fixture has an approved version with a
     decision reference.
   - Marker size is identical across sites.
9. **Basemap.** OSM tiles load, there are no CSP violations in the console, and the attribution
   is visible.