weatherfeeder/docs/roadmap/spc.md

# SPC Convective Outlook Support Roadmap

## Summary

Add `weatherfeeder` support for Storm Prediction Center convective outlooks as a new canonical outlook domain. The feature should poll SPC GeoJSON outlook products, enrich them with Day 1-3 print-page discussion text, compute whether the configured forecast point is inside each active outlook polygon, normalize the result into a provider-independent schema, and persist it through existing sinks.

This is a new domain, not an extension of `weather.alert.v1`. SPC outlooks describe probabilistic/categorical risk areas over a validity window; alerts describe active actionable hazard messages.

## Public Contract

Add schema constants:

- Raw schema: `raw.spc.convective_outlook.v1`
- Canonical schema: `weather.outlook.v1`

Add source driver:

- `spc_convective_outlook`

Add event kind:

- `outlook`

Add canonical model types:

- `model.WeatherOutlookRun`
- `model.WeatherOutlook`

Recommended canonical run fields:

- `locationId`, `locationName`
- `latitude`, `longitude`
- `asOf`
- `issuedAt`
- `outlooks`

Recommended canonical outlook fields:

- `id`
- `provider`
- `product`
- `day`
- `outlookType`
- `label`
- `labelText`
- `severityRank`
- `validFrom`
- `validTo`
- `issuedAt`
- `expiresAt`
- `forecaster`
- `headline`
- `summary`
- `discussion`
- `sourceUrl`
- `imageUrl`
- `containsLocation`
- `geometry`

Contract defaults:

- `product` should be `convective`.
- `outlookType` should be one of `categorical`, `tornado`, `hail`, `wind`.
- `day` should be `1`, `2`, or `3` for this first implementation.
- `containsLocation` is computed against configured forecast coordinates.
- `geometry` should preserve compact GeoJSON geometry for auditability and future API use.
- Day 4-8 outlooks are out of scope for this first implementation. They have different product semantics and should be added in a later schema-compatible extension only after Day 1-3 support is proven.

## Source Scope

The source should fetch a bundle of SPC products in one poll cycle and emit one raw event containing fetched GeoJSON products, fetched Day 1-3 print pages, optional RSS metadata, configured point metadata, and per-product fetch metadata. The source may decode minimal timestamps for event identity/effective time, but provider-to-canonical mapping belongs in the normalizer.

Poll these GeoJSON URLs:

- `https://www.spc.noaa.gov/products/outlook/day1otlk_cat.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day1otlk_torn.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day1otlk_hail.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day1otlk_wind.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day2otlk_cat.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day2otlk_torn.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day2otlk_hail.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day2otlk_wind.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day3otlk_cat.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day3otlk_torn.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day3otlk_hail.nolyr.geojson`
- `https://www.spc.noaa.gov/products/outlook/day3otlk_wind.nolyr.geojson`

Implementation note: the initial candidate list duplicated Day 3 wind and omitted Day 2 wind. Use the corrected 12-product matrix above.

Poll these print-page discussion URLs:

- `https://www.spc.noaa.gov/products/outlook/day1otlk_prt.html`
- `https://www.spc.noaa.gov/products/outlook/day2otlk_prt.html`
- `https://www.spc.noaa.gov/products/outlook/day3otlk_prt.html`

Print-page usage:

- Use print pages as the authoritative source for Day 1-3 outlook discussion text.
- Preserve source URL, page title/header where available, page-level `Updated:` timestamp where parseable, and the raw HTML or extracted `<pre>` product block in the raw bundle.
- Extract canonical `headline`, `summary`, `discussion`, and discussion metadata in the normalizer or shared SPC provider helper.
- Treat the GeoJSON files, not print pages, as authoritative for polygons and validity windows.
- Ignore page risk tables and image assets for v1 unless they are needed to support discussion parsing.

Optionally poll:

- `https://www.spc.noaa.gov/products/spcacrss.xml`

RSS usage:

- Use RSS as supplemental discovery or health metadata, not as the primary discussion or geometry source.
- Preserve item title, link, description text, pubDate, and guid where useful.
- Do not depend on RSS completeness for Day 1-3 discussion text. The RSS feed can omit current outlook products, including a current Day 2 discussion.
- Do not attempt to parse polygons from RSS HTML.

Polling cadence:

- Use a default cadence of `30m`, not daily/twice-daily. SPC current outlook files can update several times per day, and a 30-minute poll with HTTP caching is low cost and less likely to miss updates.
- Rely on ETag/Last-Modified handling from the HTTP source layer where available.
- Keep the cadence configurable via `every`.

Recommended config shape:

```yaml
- name: SPCConvectiveOutlookSTL
  mode: poll
  kinds: ["outlook"]
  driver: spc_convective_outlook
  every: 30m
  params:
    latitude: 38.6239
    longitude: -90.3571
    location_id: "stl"
    location_name: "St. Louis, MO"
    user_agent: "HomeOps (eric@maximumdirect.net)"
```

The source should own default SPC URLs, with optional params to override GeoJSON URLs, print-page URLs, and the RSS URL for tests or future provider changes.

## Implementation Stages

### Stage 1: Raw Source and Schema

- Add schema constants.
- Add `internal/sources/spc` package.
- Implement `spc_convective_outlook` as a poll source.
- Fetch all configured GeoJSON products and Day 1-3 print pages during a poll cycle.
- Fetch RSS only when configured or retained as an optional supplemental source.
- Emit one raw event with kind `outlook` and schema `raw.spc.convective_outlook.v1`.
- Use the latest valid `ISSUE_ISO`, print-page `Updated:` timestamp, RSS `lastBuildDate`, or fetch time for `effectiveAt`, in that order.
- Preserve partial fetch errors as source errors for the first implementation; do not emit incomplete outlook bundles unless a later explicit degraded-mode policy is added.
- Keep source tests deterministic with local test servers and fixtures; do not call live SPC services from unit tests.

Tests:

- source driver builds as a `PollSource`
- source advertises kind `outlook`
- source emits one raw bundled event
- source chooses expected effective time
- source uses configured location metadata
- source fails clearly on missing latitude/longitude
- source fails clearly when a configured GeoJSON or print-page fetch fails

### Stage 2: GeoJSON and Spatial Matching

- Add an internal geometry helper package, for example `internal/geo`.
- Support GeoJSON `Polygon` and `MultiPolygon`.
- Implement point-in-polygon with ring support:
  - first ring is the exterior
  - subsequent rings are holes
  - boundary counts as inside
  - GeoJSON coordinate order is `[longitude, latitude]`
- Use planar ray casting for this operational point-in-polygon check.

Tests:

- point inside polygon
- point outside polygon
- point on polygon boundary
- point inside a hole is outside
- point inside one multipolygon member is inside
- longitude/latitude order regression test

### Stage 3: Normalizer and Canonical Mapping

- Add `model.WeatherOutlookRun` and `model.WeatherOutlook`.
- Add `SPCConvectiveOutlookNormalizer`.
- Match only `raw.spc.convective_outlook.v1`.
- Decode the raw source bundle.
- Map each GeoJSON feature to one `WeatherOutlook`.
- Preserve SPC feature order within each product, then order products by day and type.
- Map properties:
  - `VALID_ISO` -> `validFrom`
  - `EXPIRE_ISO` -> `validTo` and `expiresAt`
  - `ISSUE_ISO` -> `issuedAt`
  - `FORECASTER` -> `forecaster`
  - `LABEL` -> `label`
  - `LABEL2` -> `labelText`
  - `DN` -> `severityRank`
- Derive:
  - `day` from product key or URL
  - `outlookType` from product key or URL
  - `id` from day, type, label, issuedAt, validFrom, and feature index
  - `containsLocation` from configured point and GeoJSON geometry
- Enrich narrative fields from the matching Day 1-3 print page:
  - parse the main product text from the `<pre>` block
  - strip embedded tags and scripts before text parsing
  - preserve line breaks in discussion text
  - parse page/title metadata, product title, issued line, valid line, correction markers such as `CORR 1`, forecaster, summary, and full discussion where practical
  - use print-page text for canonical `headline`, `summary`, and `discussion`
- Use RSS only as optional supplemental metadata when present; missing or incomplete RSS must not prevent GeoJSON plus print-page normalization.
- Set run `asOf` to the latest valid `issuedAt` across features, falling back to print-page `Updated:` timestamps, RSS `lastBuildDate`, then input event time.
- Set run `issuedAt` to the latest valid `issuedAt` across features.
- Set output event `effectiveAt` to run `asOf`.

Tests:

- normalizer routes only raw SPC schema
- categorical and probabilistic products map expected fields
- `containsLocation` is true for a known point inside a fixture polygon
- `containsLocation` is false outside
- print-page discussion text maps to canonical headline, summary, and discussion
- correction markers such as `CORR 1` are preserved where present
- missing optional RSS still permits GeoJSON and print-page normalization
- malformed required GeoJSON timestamps fail with useful context
- canonical JSON shape does not expose raw-provider-only bundle internals

### Stage 4: Postgres Sink

Add tables:

- `outlook_runs`
- `outlooks`

Suggested `outlook_runs` columns:

- event envelope columns
- `location_id`
- `location_name`
- `latitude`
- `longitude`
- `as_of`
- `issued_at`
- `outlook_count`

Suggested `outlooks` columns:

- `run_event_id`
- `outlook_index`
- `as_of`
- `product`
- `day`
- `outlook_type`
- `label`
- `label_text`
- `severity_rank`
- `valid_from`
- `valid_to`
- `issued_at`
- `expires_at`
- `forecaster`
- `headline`
- `summary`
- `discussion`
- `source_url`
- `image_url`
- `contains_location`
- `geometry_json`

Suggested indexes:

- `outlook_runs(location_id, as_of)`
- `outlooks(contains_location, valid_from, valid_to)`
- `outlooks(day, outlook_type, label)`
- `outlooks(valid_from, valid_to)`

Mapping rules:

- Store `geometry` as compact GeoJSON text in `geometry_json`.
- Require run `asOf`.
- Require outlook `validFrom`, `validTo`, `issuedAt`, `day`, `outlookType`, and `label`.
- Preserve all outlook polygons, not only polygons containing the configured point.

Tests:

- schema includes outlook tables and indexes
- mapper writes one run row plus one row per outlook
- mapper stores compact geometry JSON
- mapper rejects missing required run/outlook fields

### Stage 5: Config and Documentation

- Update sample config with `SPCConvectiveOutlookSTL`.
- Add `outlook` to route examples.
- Update current-behavior docs in the implementation change, not before behavior exists.
- Update `docs/config.md` with source params.
- Update `docs/integrations/events.md` with `weather.outlook.v1`.
- Update `docs/integrations/postgres.md` with outlook table contract.
- Update relevant internal docs under `docs/internal/` if source, normalizer, or sink behavior needs maintainer context.
- Update `README.md` only with brief implemented-provider capability and links to canonical docs.

Docs should clearly state:

- Print pages are used for Day 1-3 narrative/discussion text.
- RSS, if retained, is supplemental and not required for complete discussion coverage.
- GeoJSON is used for polygons, validity windows, and point matching.
- `containsLocation` is computed by weatherfeeder at ingestion time.
- Geometry is stored for downstream audit/display.
- Day 4-8 outlooks are not included in the initial implementation.

### Stage 6: Weatherapi Follow-Up

Do not include weatherapi changes in the first weatherfeeder implementation unless explicitly requested.

Likely future weatherapi endpoints:

- `GET /outlooks/convective`
- `GET /outlooks/convective/active`
- `GET /outlooks/convective/location`

Recommended API behavior:

- latest run by default
- active outlooks filtered by current time and `containsLocation=true`
- optional filter query params for `day`, `outlookType`, and `containsLocation`

## Verification Commands

Run focused tests:

```sh
go test ./internal/sources ./internal/normalizers/... ./internal/sinks/postgres ./model
```

Run full weatherfeeder tests:

```sh
go test ./...
```

## Acceptance Criteria

- A configured SPC source emits raw outlook bundles.
- Normalization produces `weather.outlook.v1` events.
- Each outlook indicates whether the configured forecast point is inside its polygon.
- All current Day 1-3 categorical/tornado/hail/wind products are represented.
- Day 1-3 print-page discussion text is preserved where available.
- RSS metadata is optional and supplemental if retained.
- Postgres sink persists outlook runs and outlook rows.
- Sample config and public docs describe the new kind, driver, schema, and storage contract.