Skip to content

test(7.15): reconcile the branch with what rc29 actually pins, plus the chain_id regression - #212

Merged
BitHighlander merged 8 commits into
reconcile/upstream-syncfrom
release/7.15-pyk-reconcile
Aug 15, 2026
Merged

test(7.15): reconcile the branch with what rc29 actually pins, plus the chain_id regression#212
BitHighlander merged 8 commits into
reconcile/upstream-syncfrom
release/7.15-pyk-reconcile

Conversation

@BitHighlander

@BitHighlander BitHighlander commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Brings reconcile/upstream-sync up to the python-keepkey commit firmware
rc29 is actually built against, and adds the one regression test that was
missing for the 7.15 EIP-1559 fix.

Why this exists

SOP is that before a signed release the firmware pins the canonical branch of
the single open PR into master
— not master, and not a fork branch. Auditing
rc29 against that standard:

submodule rc29 pin on its canonical branch?
deps/device-protocol cf308fd yes — upstream/up/release-protocol
deps/python-keepkey 417a613 no

git rev-list --left-right --count reconcile/upstream-sync...417a613 returned
0 5: this branch had zero commits the pin lacked, and the pin carried
five this branch had never seen. Lineage was fine — the canonical branch is
fully contained — but rc29 sat five commits past what was under review, and
those five were only reachable from a fork branch
(BitHighlander:fix/catalog-pin-kdf-rename).

The five, now included here:

b44f1b3 test(reset): display_random is accepted and ignored
74768b0 report: catalog the 7.15 seed-generation evidence and state the report's scope
5761392 report: say which Zcash shielded tests never touch a device
3bbf996 test(zcash): sign a shielded transaction on an actual device
417a613 fix(report): K8 named a storage test that no longer exists

They touch only scripts/generate-test-report.py,
tests/test_msg_resetdevice.py, and tests/test_msg_zcash_sign_pczt_device.py
no keepkeylib/. That distinction matters: firmware CMake executes
deps/python-keepkey/keepkeylib/eth/*.py at build time, so python-keepkey can
change the firmware binary. These do not. rc29's binary is unaffected.

What was affected is the evidence trail: three of the five change the zoo report
generator, including one whose entire point is "say which Zcash shielded tests
never touch a device."
The binary is clean; the report describing it was
produced by tooling outside the canonical PR.

New in this PR

test_ethereum_eip_1559_multibyte_chain_id — firmware ed6db167 ("EIP-1559
chainId hashing wrong for multi-byte chain IDs") shipped with no test at a chain
id that reproduces it. Every EIP-1559 case in that file uses chain_id 1 or 3,
both single-byte, so the bug had no coverage in the file that tests the feature.

A golden r/s would need a device run, so it is a differential instead: sign
one identical transaction under two chain ids the buggy firmware cannot
distinguish.

8453 = 0x2105   low byte 0x05, two-byte value
4357 = 0x1105   low byte 0x05, two-byte value

Same low byte and same RLP length header, so the broken code hashes a
byte-identical pre-image for both; signing is deterministic (RFC 6979), so it
returns the same signature twice and the assertion fails. Correct firmware
hashes 0x21 0x05 vs 0x11 0x05. No golden value and no new dependencies —
the repo has ecdsa but no keccak, so recovering the signer address was not
available.

Comparing against chain_id=5 would not work: the RLP length was always
derived from the full value, so the buggy pre-image for 8453 is malformed rather
than equal to a well-formed single-byte encoding. The twin has to match on both
low byte and byte-width.

Version-gated to 7.15.0, so it SKIPs rather than fails on firmware predating
the fix. Worth confirming in CI that it ran rather than skipped — a green
tick alone does not distinguish them.

Reviewer note

Vault cannot reach the EIP-1559 path on affected chains at all: plain sends are
unconditionally legacy (txbuilder/evm.ts:205) and swaps force legacy for
chainId >= 256 (eip1559Ok = chainId < 256). The vault REST API does pass
caller-supplied maxFeePerGas straight through, and the browser extension
builds type-2 with no chainId guard — which is why the failures only ever
appeared there.

Merges as a fast-forward (0 8 against this base).

Review round 1 — all four technical findings fixed (2cf5edc)

  1. Version gate removed real coverage. Correct, and my regression. The gate skipped the whole of test_reset_device_pin/test_failed_pin on the 7.10 emulator — PIN entry, EntropyRequest/Ack, mnemonic derivation — to silence one assertion. Now branches on version: pre-7.15 acks the Internal Entropy ButtonRequest that legitimately still exists, both paths converge on PinMatrixRequest, everything downstream keeps running everywhere.
  2. Z17 contradicted Z22. Correct. Scoped the claim to that module and pointed at Z22.
  3. Skips reported as executed. Correct and the most consequential. ran counted skips; this very run was 613 collected / 252 skipped, a 41% overstatement. Now tracks skipped separately, says collected, and states inline that a skip is not evidence a feature works.
  4. OLED capture before settle. Correct. Settle moved inside the wrapper before read_layout() — reading after would be worse, since the original callback presses the button and advances the screen. Made unconditional because these are structural assertions, not screenshot evidence.
  5. Stale 0 6. Updated above rather than squashed — the review history is worth keeping on a reconcile PR whose whole purpose is that these commits were never reviewed.

Also worth flagging from this round: test_ethereum_eip_1559_multibyte_chain_id SKIPPED in CI (emulator predates 7.15.0) while test_ethereum_eip_1559 passed beside it. It is inert until the CI emulator moves up — which is exactly the condition finding 3 now makes visible in the report instead of hiding.

Firmware no longer renders the Internal Entropy screen -- internal entropy
is seed pre-image material, and a host that supplies ext_entropy and reads
that screen once can compute SHA256(shown || ext) and derive the seed.

test_reset_device_pin and test_failed_pin asserted the ButtonRequest for
that screen, so they failed against the new firmware. Rather than dropping
display_random from the request, they keep sending it =True and now assert
the NEXT message is PinMatrixRequest -- which is a direct test of the
compatibility claim: the field stays decodable on the wire and changes
nothing.

Verified 6/6 against an emulator built from the paired firmware branch.
…t's scope

The PDF is the artifact a release review actually reads, and it was quietly
claiming more than it knew. Two defects, one visible consequence.

parse_junit only emitted a 'mod::meth' key when the JUnit classname contained
a dotted test_msg_*/test_sign_*/test_verify_* module. Native gtest suites carry
a bare classname ("Dice", "Storage"), so they produced no such key, and _lookup
has no bare-method fallback by design. CI merged the firmware-unit XMLs into the
report input and every one of the 432 native tests was then structurally
impossible to reference from SECTIONS. Bare classnames are now keyed as
'Suite::Test'.

The header reported "N/N PASSED" against the catalog with nothing saying the
catalog is a subset. A 7.15 RC audit grepped this PDF for feature keywords,
found no hits for dice and PIN KDF, and reported both as having zero coverage.
Both had in fact run green in the same CI run: test_reset_device_dice passed,
and so did all five Dice unit tests and the PIN KDF rewrap tests. The header now
states catalogued-vs-executed and says outright that absence here is not
evidence of absence.

New section K catalogues what that audit went looking for: the dice flow
end-to-end (digest equals SHA256 of exactly the injected rolls, then the
mnemonic is derived from post-mix internal entropy, which is what proves the
rolls reached the seed), the aborted-reset EntropyAck disarm regression, the
five Dice known-answer and independence vectors, and the v16->v19 PIN KDF
rewrap plus storage migration.

Verified against the 7.15.0 RC artifacts from run 31284108490: dice went from 0
to 13 occurrences in the rendered PDF, section K renders 11/11 passed, and
poisoning Dice::MixDependsOnRolls in the merged JUnit turns the header red and
fails --validate-junit, so the entries are wired to real results.

Needs the companion firmware change: the CI trigger validated against the
Python JUnit alone, where every native entry resolves to "missing".
Chasing a rendering defect on the per-output shielded confirm turned up
something worse than a missing screenshot: ZcashSignPCZT is never sent to a
device anywhere in this suite. Every test in test_msg_zcash_sign_pczt drives
a ScriptedTransport with canned responses -- they are offline contract tests
that prove the client builds and orders its messages correctly, and prove
nothing whatsoever about firmware behaviour.

The device-driven Zcash tests cover transparent signing, display-address,
FVK derivation and the seed fingerprint. None of them signs a shielded
output. So the on-device shielded path -- including the confirm screen that
fsm_msg_zcash.h designates as the verification gate for Orchard output
values, since total_amount is "a summary prompt" taken from the host -- has
no automated coverage at all.

The catalog gave no hint of this. The section Z entries read exactly like
the device tests around them, and that is how a screen nobody has ever
rendered sat behind seven green checks.

Say it in the entry instead. No screenshot hint: requesting frames from a
test that cannot reach a device would produce silently zero of them, which
is the same class of empty-but-green evidence this whole pass exists to
remove.

Verified: --screenshot-filter does not select it, and the report still
renders 325 tests and passes --validate-junit against the RC artifacts.
ZcashSignPCZT had never been sent to a device by anything in this suite.
Every test in test_msg_zcash_sign_pczt drives a ScriptedTransport with canned
responses; the device-driven Zcash tests cover transparent signing,
display-address, FVK derivation and the seed fingerprint. So the on-device
shielded path had no automated coverage at all, and seven green checks in
section Z read exactly like device coverage while proving only that the
client serialises its messages in the right order.

That is how a confirm screen which cannot physically fit its amount line
shipped unnoticed. The per-output shielded confirm is the verification gate
for Orchard output values -- total_amount on the summary prompt is taken
straight from the host message -- and a unified address is 106 characters,
three full body rows, against a three-row body. The amount never rendered.

The fixtures are the firmware's own known-answer vectors from
unittests/firmware/zcash.cpp, so the device's cmx recomputation accepts them
without needing a Pallas implementation in Python. The same note under both
pools commits to a different value, which is what makes the pool tests
possible at all.

Four tests:
  - the output review is two screens, and they render differently and
    non-blank (read_layout returns a framebuffer, not text, so the assertions
    are structural rather than OCR)
  - a one-bit change to the recipient breaks the commitment
  - the Orchard commitment is refused when Ironwood is declared
  - the Ironwood commitment for that same note is accepted

Two gates the offline fixtures do not satisfy had to be met for real
firmware: the header digest is recomputed and compared, and for a
shielded-only transaction the verified fee reduces to orchard_value_balance
and must equal the declared fee. Both are computed here rather than canned.

VERIFIED AS A REGRESSION TEST, not just written: run against the shipped
7.15.0 RC emulator (docker image from run 31284108490, the 27970b0c6 build)
it fails with "expected 2 ConfirmOutput screens, got 1", while the
commitment-binding and pool-selection tests pass. It reproduces the defect on
the firmware that has it.

Catalogued as Z22-Z25 with screenshot hints, so the shielded confirm screens
finally appear in the report -- the RC run captured 1037 OLED frames and not
one came from a shielded flow.
The firmware test was renamed PinKdfV16RewrapsToV19AfterCorrectPin ->
PinKdfRewrapsToActiveVersionAfterCorrectPin when the STORAGE_PIN_KDF_V19 gate
went to 0, because it is no longer v19-specific: it now asserts BOTH sides of
the gate, which is what makes it meaningful in the shipping build where v19 is
off. Catalog validation failed against the old name.

Adds K8b for PinUnlocksAfterRebootUnderV17, the end-to-end create/set-PIN/
serialize/reload/unlock/decrypt regression. It belongs in a curated catalog
precisely because every other storage test stays in RAM, and the wallet lockout
it guards lived on the serialize/reboot boundary.
Firmware fix ed6db167 shipped without a test at a chain id that reproduces it.
Every EIP-1559 case in this file uses chain_id 1 or 3 — both single-byte — so
the bug had zero coverage in the file that tests the feature.

hash_rlp_field((uint8_t*)&chain_id, 1) fed only the least-significant byte into
keccak on little-endian ARM. Base (8453 = 0x2105) hashed 0x05 and the signature
recovered to an unrelated address. RLP length was correct; the legacy EIP-155
path was correct; only the EIP-1559 hash was wrong.

A golden r/s needs a device run, so this is a differential: sign one identical
tx under 8453 (0x2105) and 4357 (0x1105). Same low byte AND same RLP length
header, so broken firmware hashes an identical pre-image and — signing being
deterministic — returns the same signature twice, failing the assertion. Fixed
firmware hashes 0x21 0x05 vs 0x11 0x05. No golden value, no new deps (the repo
has ecdsa but no keccak, so recovering the signer address was not available).

Version-gated to 7.15.0 so it SKIPs rather than fails on older firmware.
CI red on #212: test_reset_device_pin and test_failed_pin both failed with
'AssertionError: code: ButtonRequest_ResetDevice'.

Not a bug in b44f1b3 — that commit is correct. Firmware 7.15.0 removed the
Internal Entropy screen (fw 320f0eb5, 'auditable entropy source, on-device
dice, and no entropy display'), because internal entropy is seed pre-image
material: a host that sets display_random and reads the screen once can compute
SHA256(shown || ext) and derive the seed. Confirmed by inspection — reset.c on
the rc29 line contains no display_random references at all, while the pinned
older submodule still has 'if (display_random) {'.

So the assertion 'next message is PinMatrixRequest' is right for 7.15.0+ and
wrong for anything earlier, where that ButtonRequest legitimately still exists.
The tests carried NO version gate, so on CI's older emulator they failed
instead of skipping.

This is the same root cause as test_ethereum_eip_1559_multibyte_chain_id
skipping in the same run: CI's emulator predates 7.15.0. That test was gated
and skipped cleanly; these two were not and went red. Per the house rule that
feature tests must version-gate to SKIP, gate both at 7.15.0.

Worth noting the failure was only visible because these commits finally entered
a PR — they were reachable solely from a fork branch that rc29 pinned directly,
so the breakage rode along unreviewed.
[1] Version gate removed real reset coverage. My requires_firmware("7.15.0")
skipped the WHOLE of test_reset_device_pin / test_failed_pin on the 7.10
emulator — PIN entry, EntropyRequest/Ack, and mnemonic derivation checked
against a locally computed expected value — to silence one assertion. Branch on
the version instead: pre-7.15 acks the Internal Entropy ButtonRequest that
legitimately still exists there, then both paths converge on
PinMatrixRequest. Everything downstream keeps running on every firmware.

[2] Z17 claimed 'ZcashSignPCZT is not sent to a device anywhere in this suite
... no automated coverage at all' while 3bbf996 in the same branch added
exactly that device test. Scoped the claim to THIS module and pointed at Z22.
Notable because the comment above the scope block explains that saying so is
'the difference between evidence and a misleading completeness claim'.

[3] The report said CI 'executed' JUNIT_CENSUS['ran'], which increments for
every collected testcase including skips. The run behind this PR was 613
collected / 252 skipped — the report would have overstated execution by 41%,
and a version-gated test that SKIPs on an old emulator is not evidence the
feature works. Track skipped separately and say 'collected', with the skip
count and why skips happen stated inline.

[4] _capture_button_screens read the framebuffer BEFORE delegating to the
original callback, which is where the render-settle delay lives — so it could
capture a partially drawn or previous screen. Reading after would be worse
(the original presses the button and advances). Settle inside the wrapper
before reading. Unconditional, unlike client.callback_ButtonRequest's
SCREENSHOT-only sleep, because these are structural assertions rather than
screenshot evidence and need a settled layout on every run.

[5] handled in the PR description.
@BitHighlander
BitHighlander merged commit 3e68c86 into reconcile/upstream-sync Aug 15, 2026
3 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant