Migration — Staying Current

This content is for v0.3.0. Switch to the latest version for up-to-date documentation.

Paxman follows Semantic Versioning. The capability set, the contract surface, and the data tables grow across releases — your code should be ready to move forward without surprise.

In plain language: small releases add things; breaking releases can change things. This page tells you which is which and what to do when you upgrade.

Versioning

Version bump	Meaning for your code	Example
PATCH (`0.0.X`)	Contract compatibility preserved. Docs and internal fixes; authority data corrections may change recognition, status, or `canonicalized_value` when specs evolve — re-run golden samples even on PATCH.	`0.1.0` → `0.1.1` may update CLDR/IDNA tables
MINOR (`0.X.0`)	Contract compatibility preserved (existing contracts still validate). Data-driven results may change when authority tables grow — pin `paxman` version and use `contract.year` (filters `publication_year <= year`) where point-in-time reproducibility matters, store `version_stamp`, and re-run golden samples.	`0.1.x` → `0.2.0` adds capabilities and data
MAJOR (`X.0.0`)	Breaking contract or flag semantics. Read the release notes — names, defaults, or canonical forms may change.	`0.x` → `1.0.0`

Contract compatibility (which contracts are accepted) is stable across PATCH and MINOR; result stability (which status/canonicalized_value you get) depends on data and is not promised when spec tables change. year filters rules by publication_year <= year; only pinned_rules and excluded_rules identify rules. Provenance and spec-version changes alone do not imply a MAJOR bump.

Determinism is per-installed-build: the version_stamp.paxman_version on every ExecutionResult records exactly which build produced the answer, so you can audit what changed across an upgrade. Since 0.2.0, version_stamp.recognition_revision (hash of the compiled matcher set + snapshot SHAs, per ADR-0009 §13) is the same-snapshot diff signal — if recognition_revision changes, recognition behavior changed for at least one capability even when paxman_version is unchanged (e.g., a lexicon token table update).

0.2.0 — Recognition Kernel (breaking, scoped)

Scope: This is a pre-1.0 minor bump (0.1.0 → 0.2.0) with one intentional breaking change: the F1 correctness fix for whole-input vocabulary matching (ADR-0009). All other inputs are byte-identical under the parity gate.

What changed: Country name_recognition moved from whole-input lookup to an in-text word-anchored trie on the CountryNameFold view. Short-code grammars (alpha2) now honestly compete with the name grammar instead of silently winning when the name was invisible.

Input class	Before (0.1.x pipeline)	After (0.2.0 kernel)
Exact name, whole input (`"United States"`)	`SUCCESS "US"`	`SUCCESS "US"` — unchanged
Name embedded in prose (`"Ship to United States please"`)	`SUCCESS "TO"` — wrong (Tonga)	`MultipleMentionsError` under a `single_value` contract; both mentions via `paxman.scan()`
Short code as ordinary word (`"to"` in prose)	recognized and validated as alpha-2 — silent win	recognized; competes with the name mention — no silent win
All other inputs	—	byte-identical (parity gate)

Migration snippet:

# Exact value — unchanged:
import paxman
from paxman.capabilities.Country import Country

paxman.register_all_shipped()
contract = Country.create_contract()
paxman.canonicalize("United States", contract)  # SUCCESS "US"

# Prose with embedded values — the new honest paths:
from paxman.core.errors import MultipleMentionsError

try:
    result = paxman.canonicalize("Ship to United States please", contract)
except MultipleMentionsError:
    # scan() shares one ScanContext substrate across all contracts in the batch
    mentions = paxman.scan("Ship to United States please", [contract])
    # mentions.mentions["country"] == [
    #   Mention(span=(5, 7), grammar="alpha2_recognition", notation=...),
    #   Mention(span=(8, 21), grammar="name_recognition", notation=...),
    # ]
    # Segment first — docs/recipes/segmentation.md remains valid
    for m in mentions.mentions["country"]:
        print(m.span, m.grammar, m.notation)

# Or segment-first (recipe) — `paxman.scan` is preferred; no direct
# `normalize_name` import is needed.

How to detect the change: Compare result.version_stamp.recognition_revision across builds. The F1 migration changes the compiled matcher set, so recognition_revision changes even if you stay on the same paxman_version snapshot. Store both paxman_version and recognition_revision for audit trails.

Why this is correct: 0.1.x returned a confident, provenance-backed, wrong answer on ordinary prose ("Ship to United States please" → Tonga). 0.2.0 surfaces the competition honestly; scan() turns the caller-owned split-then-canonicalize loop into an API (see docs/recipes/segmentation.md and paxman scan --help).

Other 0.2.0 additions (non-breaking):

paxman.scan(text, contracts) batch API + Mention/ScanResult model + paxman scan CLI (one substrate pass, see docs/user/api-reference.md).
ScanContext lazy views, MatcherSpec/LexiconMatcher trie (SIUnit 2.4–6.5× win at 650/820 tokens), BoundarySpec presets, AnchorSet T0 prefilter.
Snapshot rails (paxman/shared_data/*_snapshot.json + tools/regenerate_* + CI drift gate) and derived recognition keys (BIC country codes, Language IANA subset) per ADR-0009 §14.

0.2.0 — Two-array offset maps (A4 Rev.4, breaking in spans only)

Pre-0.2.0 the recognition kernel used a single-array D3 invariant offsets[s] -> offsets[e] with a len(text) sentinel. When a normalizer dropped source characters (CountryNameFold strips punctuation, StripSeparators drops ()-., IDNAFold drops tabs) the translated end absorbed the dropped tail: "United States." recognized as (0, 14) with raw_text == "United States.". ADR-0009 Rev.4 amends D3 to two arrays:

View.original_span(s, e) -> (starts[s], ends[e-1]) when mapped, (s, e) when None; empty (0, 0). Each normalizer now returns (subject, starts, ends) with len(starts)==len(ends)==len(subject) and 0 <= starts[i] < ends[i] <= len(text).

Visible change (Option 1, word-boundary-aligned mentions):

Input	Before	After
`"United States."` name mention	`(0, 14)` `raw_text="United States."`	`(0, 13)` `raw_text="United States"`
`"United States of America,"`	`(0, 25)` includes trailing `,`	`(0, 24)` trimmed
`"+1 (555) 123-4567"` via `StripSeparators`	`ends[-1]==len(text)` sentinel	`ends` per-char `s+1`, `original_span(0,n)==(0,17)` exact
All length-preserving views (`CaseFold` etc.)	`None`	`None, None` — zero-cost unchanged

Whole-input canonical values are unchanged (rules normalize); only span/raw_text presentation shifts, and scan() mentions no longer carry trailing dropped punctuation. raw_text == text[start:end] is now an engine invariant enforced for every emitted match.

Migrate: if you stored span for later slicing, re-derive it from the new ExecutionResult.span/Mention.span; do not add +1 for dropped chars. Golden samples that asserted (0, 14) for "United States." should assert (0, 13).

0.2.0 — Common-word suppression for scan (B1, ADR-0009 §16)

ADR-0009 §16 was deferred as non-binding; it now ships off by default as the suppression table for usable scan() on prose (R7: ~80% of prose scan() hits were short-code noise like to→Tonga). The change is additive and off-by-default — byte-identical for every existing caller.

Contract: CapabilityContract.suppress_common_words: bool = False (after extra_grammars; frozen no-slots). Every capability’s create_contract(..., suppress_common_words: bool = False) forwards it. Default False preserves existing canonicalize() / scan() results; True removes word-bounded short-code recognitions whose lowercased span is in the curated table — never canonicalizing, only suppressing recognition (provenance-neutral by construction).

Table: paxman/core/grammar/data/common_words.py:COMMON_WORDS frozenset[str] curated via Google 1000 (https://github.com/first20hours/google-10000-english, google-10000-english.txt first 1000 lines) ∩ (ISO 3166 α2/α3 + ISO 4217 + ISO 639-1/2/3) lowercased, reviewable, frozen with assert len(COMMON_WORDS)==67 and assert "USD" not in COMMON_WORDS. USD is deliberately not suppressed (not in Google 1000); currency scan() keeps USD while to/in etc. are removed for country/currency/language code shapes.

Matchers: short-code matchers marked suppressible=True (declaration, not per-grammar code): Country alpha2_recognition / alpha3_recognition / numeric_recognition, Currency code_recognition, Language language_code_recognition. Boundary is already word-bounded (BoundarySpec.WORD / WORD_SIGN — required), so suppression only fires on word-bounded hits.

Engine: paxman/core/grammar/engine_loop.py insertion between view.original_span and emit: when contract.suppress_common_words and matcher.suppressible and text[o_s:o_e].lower() in COMMON_WORDS → continue (skip emit).

Scan vs canonicalize:

scan("Ship to the United States of America, total 45.50 USD, weight 3.5 kg", [Country.create_contract(suppress_common_words=True)]) keeps only the name mention United States of America for the short-code shapes (plus numeric/already-word-bounded non-common-word hits); with the flag off the full current snapshot is preserved. Currency scan keeps USD in both modes.
canonicalize("to", Country.create_contract(suppress_common_words=False)) → SUCCESS "TO" (Tonga correct for bare code); with the flag on → MISSING (suppressed recognition, never validated).

CLI: paxman scan keeps default contracts (flag off) but now accepts --suppress-common-words — thin create_contract(suppress_common_words=True) construction only (see paxman scan --help). API remains the seam for canonicalize().

ADR: Rev.4 §16 “deferred, non-binding” is superseded — the table ships in 0.2.0 off by default. recognition_revision bumps for the new matcher marker (one-time, pre-release).

Migrate: no change required. Opt in only for scan() on prose where Tonga noise matters; keep bare-code canonicalize() contracts flag-off.

What can appear in a minor release

You do not need to change code for these — they are additive and backward compatible:

New capabilities (the set of importable names under paxman.capabilities grows — never treat a current count as final).
New contract flags of the form include_*, allow_*, or default_* (always optional, defaults preserve shipped behavior).
New offered output_format alternatives (the default rendering stays the same; pin output_format if you rely on a specific rendering).
Expanded authority tables (e.g. new CLDR entries, additional URL IDNA mappings) where the spec itself grew.

Keep your registration future-proof by preferring the explicit form when you care about the surface:

# Future-proof — only the capabilities you name
paxman.register_capability(Email())
paxman.register_capability(Date())

# Convenience — everything shipped in this build
paxman.register_all_shipped()  # convenient, but the set it registers grows over time

Either approach is supported; pick explicit when you want the upgrade to be a conscious decision, and bootstrap when you want the new capabilities automatically.

What signals a careful upgrade

These are major-bump signals — check the release notes and review the checklist below:

A DEFAULT_OUTPUT_FORMAT or OFFERED_OUTPUT_FORMATS change — the string behind canonicalized_value for the same input may differ even though status stays SUCCESS.
A rule’s provenance year or spec version changes — contract.year boundaries move and include_historical vs active coverage may shift.
A capability renamed, merged, or removed.

Upgrade checklist (copy-paste)

When moving to a new MINOR, glance; when moving to a new MAJOR, work through it:

Read the release notes — skim new capabilities, new contract flags, and any new output_format values. Decide whether a new capability belongs in your registration.
Review contracts — new optional flags default to shipped-preserving values. Verify that rule names in pinned_rules and excluded_rules still exist (a stale name raises ContractError), and separately confirm that year (filters publication_year <= year) still expresses the temporal window you intend — year is not a rule name.
Pin output if it matters — if your downstream expects a specific rendering (e.g. Phone rfc3966), construct the contract with output_format="rfc3966" rather than relying on the current default.
Re-run your golden samples — keep a small file of (text, contract) → canonicalized_value samples for the capabilities you use, assert them in CI, and compare after the upgrade. Determinism means a change is intentional, not noise.
Log or store version_stamp — for audit trails, persist result.version_stamp.paxman_version alongside canonicalized_value so you can explain which build produced which answer.
Segmentation review — if you added a new capability or flag, confirm the caller-owned split-then-canonicalize loop (see Segmentation) still routes each piece to the right capability/contract.

Minimal golden-sample harness:

import paxman
from paxman.capabilities import Email, Country
from paxman.core.domain import Resolution

paxman.register_all_shipped()

checks = [
    ("user@Example.COM", Email.create_contract(), "user@example.com"),
    ("United States", Country.create_contract(), "US"),
]

for text, contract, expected in checks:
    r = paxman.canonicalize(text, contract)
    assert r.status == Resolution.SUCCESS and r.canonicalized_value == expected, (
        text,
        r,
    )

Temporal filtering and data drift

Spec tables (CLDR, ISBN Range Message, URL IDNA) are regenerated from snapshots and live inside the library. When a spec evolves (new country names, new currency symbols, new IDNA mappings), the release notes will note it. Use contract.year to pin to specs published up to a given year when reproducibility against a point-in-time authority matters; combine it with a pinned paxman version in your environment for full reproducibility.

See Contracts for year filtering, Provenance for publication_year on each citation, and Execution Result for version_stamp.