PortWatch_ports_database
FeatureServer/0/queryPort catalog: provider portid, name, country, coordinates, LOCODE,
vessel-count profile, industry profile, and national maritime import and export shares.
01Source boundary and scope
Three public, unauthenticated ArcGIS FeatureServer layers. The collector requests all fields,
no geometry, in ascending ObjectId order with pages of at most 1,000 records. Field names define the
stored groups; this integration assigns and converts no units beyond the provider's published schema.
Daily_Ports_Data
FeatureServer/0/queryDaily port calls, imports and exports, split into container, dry bulk, general cargo, RoRo, tanker, cargo aggregate, and total fields.
Daily_Chokepoints_Data
FeatureServer/0/queryDaily chokepoint vessel counts and capacity, split into the same vessel classes plus cargo aggregate and total fields. Also serves the distinct identity query used to validate canal overrides.
In this PR collected
- Typed ArcGIS adapters derived from live payloads, with provider aliases preserved.
- Audited LOCODE and canal identity for all 50 monitored locations.
- Idempotent Mongo persistence on unique natural keys.
- Resumable monthly backfill with per-stream checkpoints.
- Read-only coverage statistics over the latest roster snapshot.
- Paused manual backfill and paused daily Airflow DAGs.
Excluded out of scope
- PortWatch simulator outputs - the request explicitly excluded simulations.
- Climate scenario datasets.
- Restricted raw AIS - this integration reads only the public estimate layers.
- Duplicate GDACS ingestion - the existing safety integration already owns that source. No second GDACS source or write path is part of this rollout.
- Projection into the materialized traffic metrics - unit-safe semantics for the existing eight
metrics are unapproved, so PortWatch values are not written into
data.global-port-traffic-signalsordata.global-port-traffic-latest.
02Collection chain
A run enters at the left, is typed and gated in the middle, and lands in five named Mongo collections at the right. Hover or focus a block to trace its whole upstream and downstream; click to pin, Escape to release.
A run enters at stage 01 and lands in five named Mongo collections at stage 05. Edge overlays are drawn at 1240px and wider; here the stages read in numbered order. Hover or focus a block to trace it; click to pin, Escape to release.
- Run: an entry point starts a collection run
- Fetch: provider or roster data moves
- Gate: a check that can refuse
- Write: an idempotent upsert
01 Entry
02 Public source
03 Boundary
04 Collection
05 Store · database data
03Checkpoint identity: roster-sensitive vs roster-independent
The two streams deliberately compute their checkpoint identity differently, because a change to the monitored-port roster means something for one stream and nothing for the other.
Daily ports roster-sensitive
- Identity
- Hashes stream, window start, window end, and the hash of the exact mapping roster.
- Roster change
- Produces new checkpoints and collects that roster's matched ports, rather than treating an older roster's completion as covering the new one.
- Query
- Restricted to the matched
portidset, so an unmatched or ambiguous LOCODE contributes nothing. - Coverage
- 48 mapped port entities.
Daily chokepoints roster-independent
- Identity
- Hashes stream, window start and window end only. The roster hash is omitted.
- Roster change
- Changes nothing. Completion is provider-wide and stays valid, so a roster edit cannot force a re-collection of chokepoint history.
- Query
- Unfiltered for the window; every provider chokepoint in range is collected.
- Coverage
- 28 chokepoint entities, of which the two monitored canals are a subset.
Both streams share the same natural key, (portid, date), and the same resume rule: a
checkpoint carries the schema fingerprint it was collected under, and a resumed window whose required provider
schema has drifted raises rather than silently mixing two shapes.
04Mongo contract
The runtime writes only these five collections, all in database data. These are also the
downstream and operator read paths.
| Collection | Natural key and indexes | Purpose |
|---|---|---|
| portwatch-ports | unique portid | Current provider catalog with provenance and schema fingerprint. |
| portwatch-daily-ports | unique compound (portid, date) | AIS-derived daily port estimates for exactly matched monitored ports. |
| portwatch-daily-chokepoints | unique compound (portid, date) | AIS-derived daily chokepoint estimates, independent of the monitored-port roster. |
| portwatch-port-mappings | unique mapping_key; compound (status, normalized_locode) | Exact LOCODE mapping outcomes and roster snapshot identity. |
| portwatch-backfill-checkpoints | unique checkpoint_id; compound (stream, window_start, window_end); compound (status, updated_at desc) | Cursor, counts, mapping snapshot, schema fingerprint and completion state. |
This table is wider than the screen. Scroll it sideways, or focus it and use the arrow keys.
There are no TTL indexes. Catalog, daily rows, mappings and checkpoints remain until a separately reviewed retention change. Documents carry provider, methodology, source dataset, source URL, provider object id, schema fingerprint, provenance, a content hash, and first- and last-seen timestamps.
05Exact identity and audited overrides
The collector reads distinct monitored LOCODE values, trims outer whitespace, uppercases, removes all remaining whitespace, and accepts only a five-character alphanumeric result. It then performs an exact normalized match against the catalog. Runtime never fuzzy-matches.
Every roster value gets exactly one outcome
- matched
- Exactly one catalog row carries the normalized LOCODE. Its
portidbecomes eligible for daily port collection. - unmatched
- The LOCODE is valid, but the catalog has no exact match.
- ambiguous
- More than one catalog row carries the exact LOCODE, so no provider port is selected. Candidates are still recorded.
- invalid
- The source value is null or does not normalize to five alphanumeric characters.
The complete outcome roster, including ambiguous candidates and override provenance, is written to
portwatch-port-mappings on a live run.
Normalization, in order
- A null source value is invalid immediately.
- Trim outer whitespace.
- Uppercase.
- Remove every remaining whitespace character, including any inside the value.
- Require exactly five characters, all alphanumeric. Anything else is invalid.
The same normalization is applied to catalog LOCODE values before comparison, so both
sides of the match are normalized identically. There is no similarity threshold and no fallback.
Audited provider identities 11 catalog identity mismatches or duplicates, plus 2 monitored canals
| Monitored LOCODE | Provider identity | Expected name | Validated against |
|---|---|---|---|
| CNNGB | port824 | Ningbo | catalog |
| CNNSA | port425 | Guangzhou (Nansha) | catalog |
| CNSHA | port1188 | Shanghai | catalog |
| CNTAO | port1069 | Qingdao Port | catalog |
| CNXMN | port1404 | Xiamen | catalog |
| EGPSE | port192 | Port Said | catalog |
| MAPTM | port1265 | Tangier-Mediterranean | catalog |
| MYPKG | port960 | Port Klang | catalog |
| SGSIN | port1201 | Singapore | catalog |
| USNWK | port815 | New York-New Jersey | catalog |
| VNHPH | port434 | Hai Phong | catalog |
| EGSUZ | chokepoint1 | Suez Canal | live distinct identities |
| PAPAN | chokepoint2 | Panama Canal | live distinct identities |
This table is wider than the screen. Scroll it sideways, or focus it and use the arrow keys.
An override resolves only when both the provider id and the expected name are present. If either drifts, the outcome falls through and no provider entity is selected - collection fails closed rather than binding a monitored location to the wrong entity. The two canal mappings are validated against chokepoint history, not daily-port history.
06Airflow entry points
Both DAGs are created paused, allow one active run, and push the same scraper metric grouping on success and failure.
imf-portwatch-backfill manual only
- Task
collect-imf-portwatch-history- Range
- Starts at
2019-01-01;end_datedefaults to the run's data interval end. The range is divided into calendar-month windows. - dry_run
- Defaults to true, rendered as a native Airflow boolean. A dry run still reads the roster and all three public layers so it can report exact mapping coverage and provider usage, and creates no indexes, catalog, mappings, daily rows or checkpoints.
- Rerun
- Skips completed windows with a matching checkpoint identity and resumes an incomplete page from its cursor.
imf-portwatch-daily 0 3 * * *
- Task
collect-imf-portwatch-daily- Window
- The prior 14 complete UTC days, no catchup.
- dry_run
- Fixed false. There is no parameterized dry-run switch on this DAG, which is one reason it ships paused.
- Refusal
- Returns
BackfillRequiredwith missing port and chokepoint window counts instead of silently creating partial history.
Keep imf-portwatch-backfill paused after promotion: it is a manual recovery and history
tool, not a schedule.
07Coverage, evidence and verification
Collection and backfill were completed by direct runtime execution, not by deployed Airflow DAG runs. Readings below are the final Mongo statistics after that rollout.
- 50 / 50
- Monitored LOCODEs resolved: 50 matched, 0 unmatched, 0 ambiguous, 0 invalid.
- 48 / 48
- Mapped ports with daily-port history. Both mapped canals have chokepoint history.
- 133,248
portwatch-daily-portsrows over 48 entities, 2019-01-01 through 2026-08-07, exactly 2,776 rows per entity.- 77,784
portwatch-daily-chokepointsrows over 28 entities, 2019-01-01 through 2026-08-09, exactly 2,778 rows per entity.- 92 / 275
- Monthly windows completed and provider requests used by the full live backfill.
- 211,032
- Records fetched by that backfill.
- 436
- Overlap records fetched and upserted by the daily shadow run.
- 0 / 0
- Duplicate
(portid, date)natural keys in the daily-port and daily-chokepoint streams.
Rollout gates, in the order they were exercised
- The read-only Airflow Mongo URI read all 50 monitored LOCODEs from the roster collection.
- A dry run for 2019-01-01 resolved 50 of 50 mappings and fetched 76 real provider rows - 48 daily-port and 28 chokepoint - with no Mongo writes.
- A bounded one-day live canary wrote 76 records. Repeating the exact canary used 0 provider requests and fetched 0 records, proving checkpoint idempotence.
- The full live backfill completed 92 windows with 275 provider requests and 211,032 fetched records.
- A daily shadow completed with 436 fetched and upserted overlap records.
- Final statistics confirmed complete mapping and history coverage with zero duplicate natural keys.
Verification recorded 2026-08-17
- 28 focused and existing port tests passed.
- Ruff clean over the package, both DAG modules and the new tests.
- basedpyright reported zero errors.
- No-excuse clean.
- GitGuardian pre-commit hooks passed.
- Public live fetches from all three ArcGIS layers succeeded.
- Checked-in fixtures were captured from those live endpoints: sample data pages hold two ascending records, while the identity fixtures retain all 11 audited port targets and all 28 chokepoints.
08Review findings and residual risk
Four findings raised in review and resolved on this head, and three risks that remain open at merge time. Severity is what the issue could have cost, not how hard it was to fix. Filter by status and by severity; a finding must satisfy both filters to stay visible.
Showing all 7 findings.
- Mechanism
- Airflow renders a templated parameter as a string by default. A
dry_runof false would have reached the callable as a non-empty string, which is truthy, so a run requested as a dry run would have created indexes and written catalog, mappings, daily rows and checkpoints. - Fix
- The backfill DAG renders templates as native objects, and the callable rejects anything that is not a real boolean with a dedicated parameter error before any provider or Mongo call. The failure is closed and immediate rather than a silent write.
- Status
- Resolved on this head and covered by the DAG contract test.
- Mechanism
- The two monitored canals are not in the port catalog, so they resolve through constant overrides to
chokepoint1andchokepoint2. A constant alone would keep resolving even if the provider reassigned or renamed those identities, silently attributing one canal's traffic to another. - Fix
- Canal overrides now resolve only when a live distinct
Daily_Chokepoints_Dataidentity request returns that exact provider id and the expected name. On any drift the outcome falls through and no provider entity is selected. - Status
- Resolved, with the distinct identity request asserted against a live-captured fixture of all 28 chokepoints.
- Mechanism
- Hand-written fixtures can agree with the code and disagree with the provider. An override suite built on stubs proves the mapping function is self-consistent, not that the audited identities exist.
- Fix
- Fixtures were captured on 2026-08-17 from the live public endpoints and checked in with the exact query that produced each one. The override fixture retains all 11 audited port targets and the identity fixture all 28 chokepoints, and the override tests validate against those payloads. A companion test asserts that an override whose provider identity is absent resolves to no provider port.
- Status
- Resolved. The fixture README records each source query so any one can be re-captured and compared.
- Mechanism
- Collection orchestration and Mongo persistence had each grown into a single oversized module mixing contract, write path, checkpoint handling and window iteration.
- Fix
- Runtime was split into
runtime.py,runtime_types.pyandwindow_collection.py; storage intostorage.py,storage_contract.py,storage_checkpoints.pyandstorage_writes.py. Every module in the package now sits under the 250 pure-LOC ceiling, the largest being the mapping module. - Status
- Resolved as a behaviour-preserving refactor in the head commit, with the focused suite re-run after the split.
- Detail
- Every rollout gate above was exercised through direct runtime calls. Nothing in that evidence proves either DAG is deployed, parsed by a scheduler, scheduled live, or unpaused. The integration is therefore operationally partial, not complete.
- Blast radius
- A parse or environment failure would surface only at deployment, and the daily DAG has no parameterized dry-run switch to soften a first live run.
- Exit
- Verify both DAGs appear paused, trigger exactly one manual DAG-backed shadow, compare its statistics with the baseline above, and only then approve unpausing the daily DAG. Leave the backfill DAG paused.
- Detail
- Unit-safe semantics for the existing eight materialized traffic metrics are unapproved, so no PortWatch value is written into
data.global-port-traffic-signalsordata.global-port-traffic-latest. This is a deliberate exclusion, not an oversight. - Consequence
- Consumers must read the
portwatch-*collections directly and interpret imports, exports, capacity and share values against the provider's own published methodology. - Exit
- A separate reviewed change once unit semantics are agreed.
- Detail
- There are no TTL indexes. Catalog, daily rows, mappings and checkpoints remain indefinitely; the two daily streams already hold 211,032 rows between them and grow by roughly 76 rows per day.
- Exit
- A separately reviewed retention change. Checkpoint history in particular is what makes a rerun cheap, so it should not be trimmed casually.
No finding matches both filters. Re-enable a status or a severity to see the rest.
09Change map
29 files, all additions. Grouped by the job each group does, not by directory.
Provider boundary
Make untyped ArcGIS JSON safe to store
Frozen models keep every provider field and alias verbatim while requiring the handful the
collector depends on. The client owns pagination, the half-open date predicate, the strictly-increasing
ObjectId assertion, and the required-field fingerprint that refuses a drifted or error-enveloped
payload.
- models.py+127
- collector.py+212
Identity
Decide what a monitored LOCODE means, exactly
Normalization, exact matching, the four outcome statuses, the audited port and canal overrides, the roster snapshot hash used by checkpoint identity, and the matched-portid projection that bounds the daily-port query.
- mapping.py+257
Windowed collection
Make a multi-year backfill resumable and bounded
Per-stream window collection with keyset pagination, running and completed checkpoints, request and record budgets, the schema-drift resume guard, and the readiness rule that stops the daily DAG from manufacturing partial history.
- window_collection.py+232
- runtime.py+170
- runtime_types.py+53
- readiness.py+13
Persistence
One write surface, unique natural keys
Collection names, index creation, the checkpoint dataclass and its identity rule, checkpoint load and save, and idempotent upserts that attach provenance, source dataset, source URL, schema fingerprint and a content hash to every document.
- storage_writes.py+170
- storage_checkpoints.py+106
- storage_contract.py+100
- storage.py+27
- mongo_types.py+26
- __init__.py+0
Read-only reporting
Prove coverage without a write path
A single read-only callable that reports the latest roster's four status counts, mapped ports and canals with and without history, row and entity coverage, minimum and maximum dates, and duplicate natural keys per stream. It is scoped to the latest roster snapshot so an older roster cannot inflate the reading.
- statistics.py+138
Airflow entry points
Schedule the runtime, paused, with metrics
Two DAG modules that construct the internal Mongo URI, push the same scraper metric grouping on success and failure, ship paused, and - on the backfill side - render parameters as native objects and reject a non-boolean dry-run flag.
- imf-portwatch-backfill.py+89
- imf-portwatch-daily.py+87
Live-payload tests
Lock the contract against real payloads
Contract, regression, storage, statistics, override and DAG suites covering half-open windows, keyset ordering, the HTTP-200 error envelope, additive-versus-required schema drift, roster-sensitive versus roster-independent checkpoint identity, natural-key uniqueness without TTL indexes, and the latest-roster statistics scope.
- test_imf_portwatch_regressions.py+195
- test_imf_portwatch_storage.py+123
- test_imf_portwatch_dags.py+98
- test_imf_portwatch_contracts.py+78
- test_imf_portwatch_statistics.py+70
- test_imf_portwatch_overrides.py+53
Live fixtures
Real provider responses, with their queries
Five captured payloads plus a README that records the exact public query behind each one, so any fixture can be re-captured and diffed against the provider.
- fixtures README.md+11
- ports_catalog.json+1
- daily_ports.json+1
- daily_chokepoints.json+1
- chokepoint_identities.json+1
- override_targets.json+1
Operations
One runbook for the whole integration
Source, mapping, Mongo, DAG and statistics contracts, the recorded rollout evidence, the exact verification commands, the remaining promotion steps, and the exclusions - in one document.
- docs/imf_portwatch.md+140
10Verdict
Merge candidate
pending CI and deployment
The data contract is proven; the operational one is not
The provider boundary, identity rules, checkpoint semantics and persistence are exercised against real payloads and against a completed 211,032-record backfill with zero duplicate natural keys. What has not happened is an Airflow deployment: both DAGs are checked in paused and have never been seen by a scheduler, and no required check has reported on this head while the pull request is still a draft.
U1 · DAGs unseen in production
no check has reported on this head
pull request still a draft
Remaining promotion
- Land the required checks on this head.
- Take the pull request out of draft and merge.
- Deploy, then confirm both DAGs appear paused in Airflow.
- Trigger exactly one manual DAG-backed shadow and compare its statistics with the recorded baseline.
- Unpause
imf-portwatch-dailyonly. Leaveimf-portwatch-backfillpaused.