Verification

This page describes how results of lunartrace.slope_quality/v1 are checked. It states what each check does and what it does not establish. Verification here supports numerical reproduction. It does not support scientific admission.

Standing

The Site01 result carries the standing NUMERIC_REPRODUCED. The receipt’s execution status is METHOD_EXECUTED. Neither is a scientific admission. See the Glossary.

Two implementations

The method has two implementations that share the specification but no code.

	Primary	Independent oracle
Identifier	IMPLEMENTATION_ID = "primary-gdal"	implementation: "oracle-numpy"
Source	src/lunartrace/science/slope_quality.py	research/oracles/slope_quality_oracle.py
Raster I/O	rasterio (GDAL)	tifffile, with GeoKeys read by geokeys and read_raster
Shapefile I/O	pyogrio (OGR), parsed by shapely	pyshp
ROI membership	make_valid(linework), buffer 0.5 m, GDAL centre-inside rasterisation	exact rational scanline crossings on the integer lattice, with every edge lattice point added explicitly
Dependencies	numpy, rasterio, pyogrio, shapely, affine	numpy, tifffile, pyshp and the standard library

The oracle’s own docstring states that it uses “exact rational scanline intersections on the integer cell-centre lattice” with “all lattice points on every edge” added explicitly (research/oracles/slope_quality_oracle.py). It is written against the specification, not the primary code. Its tests run under task oracle (Taskfile.yml).

The primary module states that the independent oracle “uses none of” GDAL, pyogrio or shapely (slope_quality.py). The two paths therefore differ in both geometry and I/O.

Agreement criteria

tools/compare_slope_quality.py compares two canonical result files. Its rules are:

• Exact. Integers, histogram bins, quantiles, nulls, strings and the roi, tau and method fields must be exactly equal, with matching Python types.
• Tolerant. slope_mean and included_fraction must agree to a relative tolerance of $10^{-12}$ (REL_TOL). The absolute tolerance is zero.
• Ignored. The implementation field may differ and is skipped (IGNORED).
• Structure. Keys present in only one file, and lists of different lengths, are reported as differences.

The comparison exits 0 and prints AGREE only when no difference is found. Otherwise it exits 1 and lists every difference.

Agreement on the Site01 witness

The receipt docs/r2/R2-SCIENTIFIC-KERNEL-NATIVE-RECEIPT.md records the agreement for the frozen Site01 witness. It reports a primary result with 24 rows (ROIs 0–6 and the union, at three thresholds). It also records the comparison command and its output, AGREE: 24 rows, with all 16 non-null means bit-identical (maximum relative difference 0.0). The method guide repeats the claim, “all counts, histogram bins, and quantiles match exactly” (docs/guide/method.md).

These statements are claims recorded in the cited documents. This documentation set does not re-run them, and the comparison output was not regenerated while writing this page.

Hash-gated inputs

Inputs are accepted only when their bytes match a pinned SHA-256 digest from the frozen manifest. The check happens at three points.

1. Primary loaders. load_grid and load_rois hash the file first and raise MethodInputError on a mismatch, before the raster or shapefile is opened (load_grid and load_rois in slope_quality.py). run_paths forwards the pins from its frozen argument.
2. STUDY plane. resolve_inputs in src/lunartrace/science/study.py resolves each role to a blob at cas/sha256/<2>/<64> in the content-addressed store. It re-hashes the blob and compares the digest with the manifest entry, and it requires the roles slope, slope_rms and the four roi_* sidecars. _stage_shapefile copies the sidecars to roi.shp, roi.shx, roi.dbf and roi.prj in a temporary directory, so that OGR finds them by suffix.
3. Oracle. verify_inputs in the oracle checks the digests of the two rasters and the four shapefile members against the manifest entries before any computation. It refuses a manifest whose status is not SOURCE_FROZEN and a duplicate role.

What the pins cover

The digest of the ROI shapefile covers the .shp file. The primary loader checks the sibling .prj for the required CRS. The .shx, .dbf and .prj sidecars are pinned in the STUDY and oracle paths, but load_rois only pins .shp. The source manifest records this scope too (load_rois notes).

Byte-identical replay

Two kinds of output are distinguished.

• result.json is deterministic. run_study writes result.canonical_bytes(), which is canonical JSON with sorted keys, compact separators, allow_nan=False and a trailing LF (study.py; slope_quality.py). The same inputs and thresholds give the same bytes. build_projection checks that stored result bytes are already canonical before wrapping them (study.py).
• receipt.json is not deterministic. It records timestamps, git state and environment versions (study.py). Its docstring says it “is not deterministic”. It is an execution record and is not compared byte for byte.

The relevant tests are test_canonical_bytes_are_deterministic in tests/science/test_slope_quality.py and test_synthetic_union_deduplicates_and_cli_replays_byte_identically in research/oracles/test_slope_quality_oracle.py. The method guide records five cold result runs on one Windows host with byte-identical results. That record does not establish replay on other platforms.

Property-based tests and boundary failures

The Hypothesis suites in tests/property/ generate many synthetic inputs and check stated invariants, including awkward boundaries that are easy to miss in a small hand-written fixture set. They are tests of sampled generated cases, not exhaustive verification or a proof of the method.

test_slope_quality_properties.py generates small float grids and masks with values on and around validity edges: 0° and 90°, the adjacent representable values, signed zero, NaN and infinities, zero and subnormal RMS, and τ boundaries. It checks row-count inequalities, histogram totals, null-versus-zero semantics, monotonic inclusion as τ increases, sorted observed-sample quantiles, consistency between raster inclusion codes and row counts, and lossless float32 projection encoding (or refusal when narrowing changes a value, including the sign bit of zero). Its lattice-ring generator builds closed walks from unit horizontal, vertical, diagonal, and zero-length steps, then closes the path; this produces self-touching walks, spurs, backtracks and collinear runs. Properties check closed-rule boundary coverage, bounding-box confinement, invariance to ring start/orientation, and inclusion of the even-odd interior. A separate affine-grid generator varies exact and decimal pixel sizes and origins to compare the primary’s lattice-vertex decision with the independent oracle’s cell-centre test. It caught a regression where the primary multiplied by a rounded inverse pixel size; the fix uses division to match the oracle’s coordinate calculation.

test_workbench_properties.py uses an in-memory synthetic site and drives the actual ASGI sidecar boundary with generated rings, numbers, headers, JSON bodies and query strings. It checks that accepted evaluations have one row per τ, finite JSON, stable ring-order behavior, bounded deterministic samples, and no silently overwritten inclusion columns; cell picks are idempotent when repeated at the returned cell centre. Adversarial examples exercise specific former failure mechanisms:

• A Unicode superscript digit such as ² reports str.isdigit() as true but cannot be parsed by int(). The Content-Length guard now requires ASCII decimal digits, so this malformed header gets a deliberate error response instead of an unhandled exception.
• Deeply nested JSON can make json.loads raise RecursionError; parsing converts that to a deliberate invalid-input response rather than a server error.
• Converting an integer such as 10**400 to float raises OverflowError; numeric parsing rejects it as out of range rather than leaking an unhandled exception.
• JavaScript JSON serialization turns -0 into 0. The ring-identity property checks that a served ring and its browser-echoed form have the same digest after zero normalization.
• Different τ values can share a %g label (for example 2.0 and 2.0000001), which would collide in keyed sample columns or cell-inclusion maps. The input is refused rather than silently overwriting one value; repeated τ values are also refused.
• /figure map requests must have a finite, non-negative τ. Generated invalid values such as NaN, infinity and negative values are rejected with HTTP 422, not rendered as an empty-looking map.

The workbench boundary properties require deliberate HTTP outcomes (among 200, 403, 404, 413, 422 and 503) for generated requests, not that every request succeeds. The related historical change is commit 755434a (test(property): Hypothesis property suites; fix failure mechanisms they found), which records these fixes and tests. That commit message is historical evidence of what was changed; it is not by itself evidence that the current checkout passes. Current property-test source describes the present assertions, while current verification still requires running the relevant suite. Neither the commit nor passing generated examples proves correctness for all inputs.

Test suites

Command (Taskfile.yml)	What runs	Reference
uv run --frozen pytest -q	Primary tests: synthetic closed-rule, self-touching, lattice fail-closed, NaN and range, quantile, histogram, null and input-failure cases, plus real Site01 counts	tests/science/test_slope_quality.py, which includes test_real_site01_counts_match_pick_and_union_is_disjoint_sum
task oracle	Oracle tests: synthetic inputs, union de-duplication, CLI replay and fail-closed input changes	research/oracles, run with python -I -m pytest research/oracles -q
python -I tools/compare_slope_quality.py primary.json oracle.json	Agreement check between one primary and one oracle result	tools/compare_slope_quality.py
task ci	Aggregates the above with lint, typecheck and secrets scans	Taskfile.yml (ci task)

Tests that use synthetic inputs check the rules on data invented for the test. Tests that use the real Site01 data run only when the frozen local inputs are present.

What each check establishes, and what it does not

Check	Establishes	Does not establish
Primary unit tests	The implementation follows the rules on the cases it covers	That the rules are scientifically correct for the source data
Oracle tests	The second implementation reproduces the same rules on its own cases	That the two implementations agree on untested geometry
Agreement comparison	Exact counts and histograms, and means and fractions within $10^{-12}$, for the compared result files	Agreement for other witnesses, other thresholds, or other platforms
Hash gating	The exact bytes of the inputs match the frozen digests	That the upstream product is current, or that it is the correct product
Canonical replay	Repeated runs on one host produce identical result bytes	Replay on other hosts or library versions
Receipt	Which inputs and library versions produced a result, and that the method was executed	A scientific standing, or that the receipt is authentic (it is unsigned)

What this does not establish

• Not scientific admission. Agreement between two implementations is numerical reproduction. The Site01 standing is NUMERIC_REPRODUCED, which is not admission.
• Not correctness of the rules. Both implementations encode the same specification. Their agreement does not show that the specification is the right question.
• Not coverage. One witness, one site and one set of thresholds were compared.
• Not source rights. The hash pins say what bytes were used. They do not verify redistribution rights, which are recorded as UNVERIFIED; the bytes are not redistributed.
• Not backup or custody. The local store is single-copy and is outside Git.
• Not an external audit. No third-party review of the code or the comparison is recorded in the documents read for this page.

Related pages

• Method specification, the rules that both implementations follow.
• Source data, the frozen witness and its digests.
• Limitations and FAQ.