Skip to content

Auto-wrap popups and composite their video as an overlay - #36

Merged
mmkal merged 13 commits into
mainfrom
popup-overlay
Aug 14, 2026
Merged

Auto-wrap popups and composite their video as an overlay#36
mmkal merged 13 commits into
mainfrom
popup-overlay

Conversation

@mmkal

@mmkal mmkal commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Replaces #33 (stack tooling wouldn't let its base move off the now-closed #32). Complete popup support, from the approved design in tasks/complete/2026-08-13-popup-overlay-video.md — includes what was #32 (popup wrapping + per-instance videoMode artifact namespacing) plus:

  1. Auto-wrap: pages wrapped with addPlugins automatically wrap popups they open (recursively) — spinner-waiting, error reporting etc. apply to the popup with zero test wiring. Plugins control their popup behavior via a forPopup(ctx) hook; opt out wholesale with popups: false. Wrapping an already-wrapped page now throws.
  2. Overlay video: instead of a separate -2 video, the popup renders as an overlay in the main page's video — it slides up from the bottom over the dimmed page, popup clicks and typed fill reveals are annotated inside it (focus ring included), one cursor glides between page and popup, and it slides away on close. Implemented as a two-pass render: a composite pass over the raw screencasts, then the existing annotation machinery unchanged. Popup facts land in video-mode.json under children (schema v2).
// test code stays completely ordinary:
const popupPromise = page.waitForEvent("popup");
await page.getByRole("button", { name: "Sign in" }).click();
const popup = await popupPromise; // already wrapped — no addPlugins call
await popup.getByLabel("Username").fill("mmkal");
await popup.getByRole("button", { name: "Sign in" }).click();

Demo

One composed video from spec/popup-overlay-demo.spec.ts — the sign-in popup slides up over the dimmed dashboard, the cursor types the username and password into their focused fields, clicks Sign in, the popup slides away, and the dashboard shows the signed-in state:

popup-signin-overlay-demo.mp4

Deferred (noted in the task file): popup dialog annotations; child pans degrade to plain box highlights inside the overlay.

Full suite: 142 passed. The popups: false path keeps the #32 standalone-video behavior, covered by the migrated specs.

🤖 Generated with Claude Code

Session: b7f6f792-6606-44be-9ec3-207eb762c4b6


Note

High Risk
Large changes to plugin lifecycle and the ffmpeg video pipeline (composite overlays, projected highlights, fill reveals in popups); regressions could affect all video-mode renders, not only popup flows.

Overview
Popups get middleware by default. addPlugins now listens for "popup" and recursively wraps child windows with the same plugin list (or what each plugin’s new forPopup(ctx) hook returns). Opt out with popups: false for manual wrapping; calling addPlugins on an already-wrapped page throws. Parent dispose finalizes popup children first.

videoMode treats popups as one composed film. Auto-wrapped popups are recorded on the parent clock into video-mode.json children (schema v2). Post-render runs a composite pass (dimmed parent, scaled popup screencast, slide/fade) then the existing annotation pass on that composite—pointer/fill highlights inside the overlay, unified cursor, parent actions shifted past exit fades. Manual popups: false + a second videoMode() still yields separate -2 artifacts.

README Popups section, shared auth demo routes, and specs cover auto-wrap, overlay pixels, and standalone-video escape hatch. Popup alert/confirm/prompt in video mode remain unannotated.

Reviewed by Cursor Bugbot for commit e6a7948. Bugbot is set up for automated code reviews on this repo. Configure here.

mmkal and others added 13 commits August 13, 2026 12:34
…act gap

Investigation findings: middleware plugins already support popups via a
second addPlugins call (per-page dispatch with fall-through). videoMode
needs work: a reused instance wipes the main timeline on beforeTest, and
a fresh instance per popup collides on fixed artifact filenames in
testInfo.outputDir. Spec for the intended behavior comes next.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two tests against an auth-popup demo app. The first passes today:
wrapping the popup with a second addPlugins call runs its actions
through its own plugins, with timelines isolated in-memory. The second
is an intended-behavior spec that fails today: after both pages
finalize, each videoMode instance should own its artifacts, but both
resolve the same fixed filenames in testInfo.outputDir so the last
finalize clobbers the first.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Each videoMode() registered within a test now gets its own artifact
namespace: the first instance keeps the legacy unsuffixed filenames,
later ones get -2, -3, ... on metadata, raw/rendered video, player HTML,
highlight/pan/fill images, dialog/final frames, .ass files, and
attachment names. This lets an auth popup carry its own fresh videoMode
instance without clobbering the main page's artifacts - Playwright
screencasts each page separately, so separate videos per page is the
natural model.

Reusing one active instance on a second page now throws a clear error
instead of silently wiping the first page's timeline in beforeTest.

Specs: popup.spec.ts (artifact isolation now green, plus the reuse
guard), popup-video.spec.ts (video: 'on' end to end - popup gets its own
raw/rendered webm), shared demo app in auth-demo-app.ts. README gains a
Popups section. Task moved to complete.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Used by the gitignored popup-demo spec that records the PR demo videos;
the popup specs assert the same roles/text either way.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Design settled interactively (plannotator grill): parent-owned composition,
default-on auto-wrap with a forPopup plugin hook, unified piece timeline
with source tags, 90%-fit dimmed overlay, double-wrap error instead of an
escape hatch. Full decision log in the task file.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wrapped pages listen for the popup event and wrap the popup
automatically (recursively for nested popups). Plugins may declare
forPopup(ctx) to produce the popup's plugin - videoMode returns null for
now (its parent-bound child recorder is phase 2) - and plugins without
the hook are re-registered as-is, which is safe for the stateless ones.
Opt out with popups: false; children dispose before the parent
finalizes.

addPlugins on an already-wrapped page now throws instead of silently
replacing plugin state mid-flight - this also guards accidental double
wrapping of main pages. Manual popup wrapping (the #32 pattern) becomes
the popups: false path; specs migrated accordingly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
videoMode.forPopup now returns a parent-bound child recorder: popup
actions record highlights on the parent clock (so child->parent time
mapping is trivial), waits merge into the parent's unified dead air, and
at finalize the popup recorder settles, closes the popup, calibrates
recordingEndedAt, and copies the popup's raw screencast to
video-raw-popup-N.webm. Facts land in a new children array in
video-mode.json (schemaVersion 2). Grandchild popups recurse into the
same flat children list. Rendering is untouched - the composite overlay
is phase 3.

The action middleware is extracted into videoModeActionMiddleware,
shared between an instance (recording onto its own state) and its child
recorders (recording onto a child state that shares the parent's clock
and dead-air spans).

Popup dialogs are not annotated yet - noted in the task file.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two-pass render when a test recorded popups. Pass A composites each
popup screencast onto the parent's raw footage: dimmed backdrop (40%
black), scaled to fit 90% of the frame (never upscaled), centered,
alpha-faded in/out over 200ms, windowed to the popup's open/close span,
stacked newest-on-top. The composite shares the parent raw timeline
exactly, so pass B - the existing piece/hold/dead-air/cursor machinery -
runs on it unchanged; holds freeze the composite, so it never matters
which source triggered them.

Popup highlights project into composite coordinates and join the render
plan as plain box/pointer highlights (child-frame pixel treatments -
pans, fill reveals, screenshot stills - drop away). Child raw time maps
to the parent timeline via the settled-recorder calibration when the
popup was closed by the recorder, or the screencast start approximation
when the popup closed itself; the auth demo popup now self-closes after
Approve like a real OAuth popup.

Frame-sampling spec verifies the rendered output: frames with a dimmed
page background and the popup's bright card centered above it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…spec

Three compositing fixes found by eyeballing the rendered demo:

- Static pages emit sparse screencast frames, and overlay emits output
  only at primary-input frame times. Resample BOTH chains to a
  continuous fps: without it the child frame that passed fade mid-ramp
  ghosts at partial alpha for the whole window, and the popup window can
  contain zero composite frames entirely (showing post-close footage).
- Run the exit fade AFTER close, using the screencast's padded final
  frame: a self-closing popup otherwise puts its own Approve click - and
  the click hold's freeze frame - inside the fade by construction.
- An instant click's source slice is a few ms wide, often between frame
  ticks, so the hold trim came up empty and bled the next piece's
  footage. Anchor the slice back from close and widen it to two frames.

spec/popup-overlay-demo.spec.ts replaces the gitignored demo spec: full
watchable treatment (pointer, captions, overlay), light assertions on
the child span and rendered output. Overlay frame assertions calibrated
against measured downscale blends.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The popup overlay now slides up from the bottom edge on enter (300ms,
quadratic ease-out) and slides back down after close, alongside the
alpha fade. Piece planning gains keepSpans: the overlapping-hold skip
used to jump source footage straight across the popup's enter/exit
animations, hard-cutting them out of the output; skips that would cross
a popup transition are cancelled. Parent highlights that start inside a
popup's exit window shift past it, so their holds can't freeze a
mid-fade ghost (the flash after the popup disappeared).

Fill reveals now work inside overlays: the typed-text reveal renders
over a frozen composite frame (popup risen, field empty, backdrop
intact) with the child screenshot's content rect scaled and positioned
through the overlay transform, synced to cursor arrival like main-page
fills. highlightCursorPoint projects fillReveal.initialRect through the
overlay transform too - the I-beam used to land offset from the field.

Child raw footage anchoring: a self-closing popup's screencast t=0 is
its first captured frame, which lags the popup event by the initial
paint, so footage played early (fields filled before the reveal). The
raw video's padded end is the better anchor: closedAt + 1s minimum
final-frame padding - duration, floored at openedAt.

Demo: the popup is now a realistic sign-in form (username/password +
Sign in) on indigo, over a teal app page so the dimmed backdrop reads;
MIDDLEWRIGHT_DEBUG_PIECES=1 dumps the render piece plan.

Includes previously staged demo/styling work reviewed via the local
video (not the GitHub PR).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The reveal base predates the fill, so the field typed with no focus
ring, and the ring then flashed in from live footage after the hold.
At reveal start the field's ring region from the post-fill screenshot
overlays the base, its text immediately covered by the pre-fill
screenshot's empty content box, and the reveal bands type over that -
ring appears when the cursor lands, letters arrive inside it, and the
post-piece footage continues the ring seamlessly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Password inputs render one bullet per character, and the reveal only
ever shows the screenshot's dots - so there was no reason to fall back
to an instant fill. Measure bullet glyphs instead of the value's
graphemes for the reveal stops.

The reveal base now rewinds a single frame (a longer rewind could cross
the previous fill's completion and wipe its value from the frozen
frame), and the pre-fill empty content box covers the field from t=0 so
anchor imprecision can't leak early-typed text.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@pkg-pr-new

pkg-pr-new Bot commented Aug 14, 2026

Copy link
Copy Markdown

Open in StackBlitz

pnpm add https://pkg.pr.new/middlewright@36

commit: e6a7948

@mmkal
mmkal marked this pull request as ready for review August 14, 2026 13:17
@mmkal
mmkal merged commit d706188 into main Aug 14, 2026
4 of 5 checks passed

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit e6a7948. Configure here.

Comment thread src/plugin-system.ts
@mmkal

mmkal commented Aug 14, 2026

Copy link
Copy Markdown
Contributor Author

🤖 Correction for the record: the child-dispose fix discussed in the review thread here landed on the branch a few minutes after the squash-merge snapshot, so it missed main — re-landed as #37.

mmkal added a commit that referenced this pull request Aug 14, 2026
Follow-up to #36 (`popup-overlay`) — its popup auto-wrap exposed the
explicit-timeout bug fixed here.

### 1. Explicit timeouts are the author’s budget — stop overriding them
with the 1ms fast-fail

`iterate/iterate`’s preview e2e went fully red on the first run with
#33’s build: every mobile spec’s OAuth popup action failed with
`TimeoutError: locator.click: Timeout 1ms exceeded`. Those specs pass `{
timeout: 15_000 }` on popup actions precisely because the auth pages
render no spinner-visible loading UI — and popups used to be raw pages
where that timeout was honored. Once popups auto-wrapped,
spinner-waiter’s no-spinner fast-fail replaced the author’s 15s budget
with 1ms:

```ts
await popup.getByRole("button", { name: "Allow access" }).click({ timeout: 15_000 });
// → TimeoutError: locator.click: Timeout 1ms exceeded.
```

Now an explicitly passed `timeout` passes straight through — the
per-action equivalent of `settings.run({ disabled: true })`.

### 2. Leave disappearance waits to Playwright

`waitFor({ state: "detached" | "hidden" })` drives toward the target
leaving the page. Spinner-waiter’s appear-oriented model does not apply,
so these waits now pass through unchanged: satisfied waits resolve
normally and failures use Playwright’s configured action timeout rather
than the 1ms fast-fail.

### Risk map

- Explicit-timeout actions skip all spinner-waiter behavior. That is
deliberate: the author owns the supplied budget.
- Disappearance waits use vanilla Playwright behavior rather than
spinner detection.
- Review order: `src/plugins/spinner-waiter.ts`, then the three
regression specs in `spec/spinner-waiter.spec.ts`.

Spinner-waiter suite green locally (13 passed).

### Video


https://github.com/user-attachments/assets/24ccb234-7927-4cc8-a2d7-3f9e51debfe9

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Session id: `7b51d95e-5871-4a8f-b4c9-996a84e1beea`

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> Middleware pass-through changes when explicit timeouts are set skip
all spinner extension for those actions; disappearance waits bypass
fast-fail—behavioral change in a widely used plugin path but aligned
with author intent and downstream iterate usage.
> 
> **Overview**
> **Spinner-waiter** no longer overrides actions when the test author
sets an explicit `{ timeout }` or uses `waitFor({ state: "hidden" |
"detached" })`.
> 
> For **explicit timeouts**, middleware now passes straight to
Playwright instead of applying the no-spinner **1ms fast-fail**. That
restores per-action budgets (e.g. OAuth popup clicks with `{ timeout:
15_000 }`) that broke once popup auto-wrap routed those actions through
spinner-waiter.
> 
> For **disappearance waits**, the appear-oriented spinner logic is
skipped so those calls get normal Playwright timeouts—satisfied waits
resolve, failures use the configured action timeout rather than 1ms.
> 
> `spec/spinner-waiter.spec.ts` adds three tests covering honored
explicit timeout, exceeded explicit timeout messaging, and disappearance
wait pass-through.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
be26f5a. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant