From da0b1e405dae453c6e4065133d8c8e67aaf4acf8 Mon Sep 17 00:00:00 2001 From: Misha Kaletsky <15040698+mmkal@users.noreply.github.com> Date: Thu, 13 Aug 2026 12:34:45 +0100 Subject: [PATCH 01/13] Add popup-plugins task: wrap popups with addPlugins, video-mode artifact gap Investigation findings: middleware plugins already support popups via a second addPlugins call (per-page dispatch with fall-through). videoMode needs work: a reused instance wipes the main timeline on beforeTest, and a fresh instance per popup collides on fixed artifact filenames in testInfo.outputDir. Spec for the intended behavior comes next. Co-Authored-By: Claude Fable 5 --- tasks/popup-plugins.md | 43 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100644 tasks/popup-plugins.md diff --git a/tasks/popup-plugins.md b/tasks/popup-plugins.md new file mode 100644 index 0000000..6c0a83e --- /dev/null +++ b/tasks/popup-plugins.md @@ -0,0 +1,43 @@ +--- +status: in-progress +size: medium +branch: popup-plugins +--- + +# Popup support (auth popout windows) + +**Status summary**: investigation done. Wrapping a popup with `addPlugins` already works for middleware plugins — no code change needed there. Video mode has a real gap: two `videoMode()` instances in one test clobber each other's artifacts. Next step is a spec capturing the intended behavior; the fix (artifact namespacing) comes after. + +The pattern this should support: + +```ts +const popupPromise = page.waitForEvent("popup"); +await page.getByRole("button", { name: "Sign in" }).click(); +await using popup = await addPlugins({ + page: await popupPromise, + testInfo, + plugins: [spinnerWaiter(), videoMode()], // fresh instances for the popup +}); +``` + +## Findings: do we need a code change? + +- **Middleware plugins (spinnerWaiter, hydrationWaiter, uiErrorReporter, screenshot): no.** The `Locator.prototype` patch is global, but dispatch is per-page via `getPluginState(this.page())` (`src/plugin-system.ts`). An unwrapped popup falls through to original Playwright behavior; a second `addPlugins` call on the popup gives it its own plugin state. This is the designed seam. +- **videoMode: yes, sort of.** Two hazards, one per way you might wire it: + 1. **Reusing the same `videoMode()` instance** on page + popup is broken: the plugin holds one per-test `state` closure, and its `beforeTest` handler resets it — wrapping the popup mid-test wipes the main page's highlights/captions recorded so far. + 2. **A fresh `videoMode()` instance per popup** is the right model (Playwright screencasts each page separately; the instance's timebase is its creation time ≈ popup screencast start). But artifact filenames are fixed per `testInfo.outputDir` (`video-mode.json`, `video-raw.webm`, `video-mode-highlight-N.png`, …), so the two instances clobber each other: whichever page disposes last overwrites, and afterwards `metadata()`/`outputPaths()` on the popup instance read the *main page's* artifacts. +- A single combined video (main page + popup interleaved) would require compositing two screencasts and is out of scope. Separate videos per page is the natural model. + +## Checklist + +- [ ] spec: popup wrapped with `addPlugins` runs actions through its own plugins, timelines isolated in-memory during the test (should pass today) +- [ ] spec: after the test, each `videoMode` instance still owns its artifacts (intended-behavior spec, expected to fail today on the artifact collision) +- [ ] decide artifact namespacing for multiple `videoMode` instances per test (auto-index like `video-mode-2.json`, vs explicit `videoMode({ name: "popup" })`) +- [ ] implement the namespacing (metadata, raw/rendered video, highlight/pan images, player HTML, attachment names) +- [ ] guard against registering an already-active `videoMode` instance on a second page (clear error pointing at fresh-instance-per-page) +- [ ] ffmpeg-level spec with `video: "on"`: popup gets its own raw/rendered webm alongside the main page's +- [ ] README section on popups (fresh plugin instances per popup, `await using` so the popup finalizes) + +## Implementation log + +- 2026-08-13: investigated plugin-system + video-mode internals. Plan: `spec/popup.spec.ts` with an auth-popup demo app (`app.middlewright.test` opens `auth.middlewright.test`, Approve posts a message back to the opener). Assumption: intended behavior is *separate* artifacts per instance, not a composited single video. From db3f414bd9001999d9e799fc43f6da96968ca9ae Mon Sep 17 00:00:00 2001 From: Misha Kaletsky <15040698+mmkal@users.noreply.github.com> Date: Thu, 13 Aug 2026 12:35:50 +0100 Subject: [PATCH 02/13] Spec popup pages wrapped with addPlugins Two tests against an auth-popup demo app. The first passes today: wrapping the popup with a second addPlugins call runs its actions through its own plugins, with timelines isolated in-memory. The second is an intended-behavior spec that fails today: after both pages finalize, each videoMode instance should own its artifacts, but both resolve the same fixed filenames in testInfo.outputDir so the last finalize clobbers the first. Co-Authored-By: Claude Fable 5 --- spec/popup.spec.ts | 115 +++++++++++++++++++++++++++++++++++++++++ tasks/popup-plugins.md | 8 +-- 2 files changed, 119 insertions(+), 4 deletions(-) create mode 100644 spec/popup.spec.ts diff --git a/spec/popup.spec.ts b/spec/popup.spec.ts new file mode 100644 index 0000000..0fbb253 --- /dev/null +++ b/spec/popup.spec.ts @@ -0,0 +1,115 @@ +import { test, expect } from "@playwright/test"; +import type { BrowserContext } from "@playwright/test"; +import { addPlugins, videoMode } from "../src/index.ts"; + +test("a popup wrapped with addPlugins runs actions through its own plugins", async ({ + page: basePage, + context, +}, testInfo) => { + await routeAuthDemoApp(context); + const video = videoMode({ finalHold: 0, highlight: { mode: "outline", duration: 300 }, trimStart: "never" }); + await using page = await addPlugins({ page: basePage, testInfo, plugins: [video] }); + await page.goto("https://app.middlewright.test/"); + + const popupPromise = basePage.waitForEvent("popup"); + await page.getByRole("button", { name: "Sign in" }).click(); + const popupVideo = videoMode({ finalHold: 0, highlight: { mode: "outline", duration: 300 }, trimStart: "never" }); + await using popup = await addPlugins({ + page: await popupPromise, + testInfo, + plugins: [popupVideo], + }); + + await popup.getByRole("button", { name: "Approve" }).click(); + await page.getByText("Signed in as mmkal").waitFor(); + + // The popup's click went through the popup's own video-mode middleware... + await expect(popupVideo.metadata()).resolves.toMatchObject({ + highlights: [{ method: "click" }], + }); + // ...and the main page's timeline has only the main page's actions. + await expect(video.metadata()).resolves.toMatchObject({ + highlights: [{ method: "click" }, { method: "waitFor" }], + }); +}); + +test("each videoMode instance still owns its artifacts after the test", async ({ + page: basePage, + context, +}, testInfo) => { + await routeAuthDemoApp(context); + const video = videoMode({ finalHold: 0, highlight: { mode: "outline", duration: 300 }, trimStart: "never" }); + let popupVideo!: ReturnType; + { + await using page = await addPlugins({ page: basePage, testInfo, plugins: [video] }); + await page.goto("https://app.middlewright.test/"); + + const popupPromise = basePage.waitForEvent("popup"); + await page.getByRole("button", { name: "Sign in" }).click(); + popupVideo = videoMode({ finalHold: 0, highlight: { mode: "outline", duration: 300 }, trimStart: "never" }); + await using popup = await addPlugins({ + page: await popupPromise, + testInfo, + plugins: [popupVideo], + }); + + await popup.getByRole("button", { name: "Approve" }).click(); + await page.getByText("Signed in as mmkal").waitFor(); + } + + // Both pages have finalized. Each instance must write its own artifacts and + // read back its own timeline — not whichever page finalized last. + expect(popupVideo.outputPaths().metadata).not.toBe(video.outputPaths().metadata); + await expect(video.metadata()).resolves.toMatchObject({ + highlights: [{ method: "click" }, { method: "waitFor" }], + }); + await expect(popupVideo.metadata()).resolves.toMatchObject({ + highlights: [{ method: "click" }], + }); +}); + +/** + * app.middlewright.test shows a Sign in button that opens an auth popup on + * auth.middlewright.test; approving there posts a message back to the opener, + * which then shows who signed in. Routed on the context so the popup page is + * covered too. + */ +const routeAuthDemoApp = async (context: BrowserContext) => { + await context.route("https://app.middlewright.test/**", async (route) => { + await route.fulfill({ + body: ` +
+ + + +
+ `, + contentType: "text/html", + }); + }); + await context.route("https://auth.middlewright.test/**", async (route) => { + await route.fulfill({ + body: ` +
+

Authorize middlewright?

+ + +
+ `, + contentType: "text/html", + }); + }); +}; diff --git a/tasks/popup-plugins.md b/tasks/popup-plugins.md index 6c0a83e..81fdceb 100644 --- a/tasks/popup-plugins.md +++ b/tasks/popup-plugins.md @@ -6,7 +6,7 @@ branch: popup-plugins # Popup support (auth popout windows) -**Status summary**: investigation done. Wrapping a popup with `addPlugins` already works for middleware plugins — no code change needed there. Video mode has a real gap: two `videoMode()` instances in one test clobber each other's artifacts. Next step is a spec capturing the intended behavior; the fix (artifact namespacing) comes after. +**Status summary**: investigation done, spec written. Wrapping a popup with `addPlugins` already works for middleware plugins — no code change needed there, and a passing spec proves it. Video mode has a real gap: two `videoMode()` instances in one test clobber each other's artifacts; a failing intended-behavior spec captures it. The fix (artifact namespacing) is not implemented yet. The pattern this should support: @@ -30,8 +30,8 @@ await using popup = await addPlugins({ ## Checklist -- [ ] spec: popup wrapped with `addPlugins` runs actions through its own plugins, timelines isolated in-memory during the test (should pass today) -- [ ] spec: after the test, each `videoMode` instance still owns its artifacts (intended-behavior spec, expected to fail today on the artifact collision) +- [x] spec: popup wrapped with `addPlugins` runs actions through its own plugins, timelines isolated in-memory during the test _(`spec/popup.spec.ts`, passes — no code change needed for this half)_ +- [x] spec: after the test, each `videoMode` instance still owns its artifacts _(`spec/popup.spec.ts`, intended-behavior spec — fails today: both instances resolve `video-mode.json` in the same outputDir, and the two report attachments even share one content hash)_ - [ ] decide artifact namespacing for multiple `videoMode` instances per test (auto-index like `video-mode-2.json`, vs explicit `videoMode({ name: "popup" })`) - [ ] implement the namespacing (metadata, raw/rendered video, highlight/pan images, player HTML, attachment names) - [ ] guard against registering an already-active `videoMode` instance on a second page (clear error pointing at fresh-instance-per-page) @@ -40,4 +40,4 @@ await using popup = await addPlugins({ ## Implementation log -- 2026-08-13: investigated plugin-system + video-mode internals. Plan: `spec/popup.spec.ts` with an auth-popup demo app (`app.middlewright.test` opens `auth.middlewright.test`, Approve posts a message back to the opener). Assumption: intended behavior is *separate* artifacts per instance, not a composited single video. +- 2026-08-13: investigated plugin-system + video-mode internals. Wrote `spec/popup.spec.ts` with an auth-popup demo app (`app.middlewright.test` opens `auth.middlewright.test`, Approve posts a message back to the opener). Assumption: intended behavior is *separate* artifacts per instance, not a composited single video. Test run confirms: middleware test green, artifact test red at `outputPaths()` equality. From 94ef67c3ed22535ae6042dba9773164af7cc5b10 Mon Sep 17 00:00:00 2001 From: Misha Kaletsky <15040698+mmkal@users.noreply.github.com> Date: Thu, 13 Aug 2026 12:47:14 +0100 Subject: [PATCH 03/13] Namespace videoMode artifacts per instance to support popups Each videoMode() registered within a test now gets its own artifact namespace: the first instance keeps the legacy unsuffixed filenames, later ones get -2, -3, ... on metadata, raw/rendered video, player HTML, highlight/pan/fill images, dialog/final frames, .ass files, and attachment names. This lets an auth popup carry its own fresh videoMode instance without clobbering the main page's artifacts - Playwright screencasts each page separately, so separate videos per page is the natural model. Reusing one active instance on a second page now throws a clear error instead of silently wiping the first page's timeline in beforeTest. Specs: popup.spec.ts (artifact isolation now green, plus the reuse guard), popup-video.spec.ts (video: 'on' end to end - popup gets its own raw/rendered webm), shared demo app in auth-demo-app.ts. README gains a Popups section. Task moved to complete. Co-Authored-By: Claude Fable 5 --- README.md | 17 +++ spec/auth-demo-app.ts | 47 +++++++++ spec/popup-video.spec.ts | 48 +++++++++ spec/popup.spec.ts | 65 ++++-------- src/plugins/video-mode.ts | 117 ++++++++++++++++----- tasks/complete/2026-08-13-popup-plugins.md | 44 ++++++++ tasks/popup-plugins.md | 43 -------- 7 files changed, 264 insertions(+), 117 deletions(-) create mode 100644 spec/auth-demo-app.ts create mode 100644 spec/popup-video.spec.ts create mode 100644 tasks/complete/2026-08-13-popup-plugins.md delete mode 100644 tasks/popup-plugins.md diff --git a/README.md b/README.md index 909f4f2..fb18a14 100644 --- a/README.md +++ b/README.md @@ -351,6 +351,23 @@ export default defineConfig({ }); ``` +### Popups + +Pages you never wrap (popups, `context.newPage()`) fall through to plain Playwright. To get plugin behavior in a popup — an OAuth window, say — wrap it with a second `addPlugins` call, using **fresh plugin instances**: + +```ts +const popupPromise = page.waitForEvent("popup"); +await page.getByRole("button", { name: "Sign in" }).click(); +await using popup = await addPlugins({ + page: await popupPromise, + testInfo, + plugins: [spinnerWaiter(), videoMode()], +}); +await popup.getByRole("button", { name: "Approve" }).click(); +``` + +Playwright screencasts each page separately, so the popup's `videoMode` produces its own video; its artifacts get a `-2` suffix (`video-rendered-2.webm`, `video-mode-2.json`, …) so they sit next to the main page's in the same output dir. Reusing the main page's `videoMode` instance on the popup would wipe the main timeline, so it throws instead — one instance per page. See [spec/popup.spec.ts](spec/popup.spec.ts). + ## Writing your own plugin **Writing your own plugins is the intended way to use this package.** The bundled five exist because they were useful for one particular app; your app has its own loading conventions, error surfaces, and flake patterns. Each bundled plugin is one small self-contained file — use them as inspiration: [spinner-waiter](./src/plugins/spinner-waiter.ts) (conditional waiting + error enrichment + runtime settings via `AsyncLocalStorage`), [hydration-waiter](./src/plugins/hydration-waiter.ts) (the simplest one — start here), [ui-error-reporter](./src/plugins/ui-error-reporter.ts) (catch/enrich/rethrow), [video-mode](./src/plugins/video-mode.ts) (video annotations/artifacts + lifecycle hooks), [llm-recover](./src/plugins/llm-recover.ts) (recovery loops, artifacts, soft assertions). The source also ships inside the npm package, so it's right there in `node_modules/middlewright/src`. diff --git a/spec/auth-demo-app.ts b/spec/auth-demo-app.ts new file mode 100644 index 0000000..29d58f1 --- /dev/null +++ b/spec/auth-demo-app.ts @@ -0,0 +1,47 @@ +import type { BrowserContext } from "@playwright/test"; + +/** + * app.middlewright.test shows a Sign in button that opens an auth popup on + * auth.middlewright.test; approving there posts a message back to the opener, + * which then shows who signed in. Routed on the context so the popup page is + * covered too. + */ +export const routeAuthDemoApp = async (context: BrowserContext) => { + await context.route("https://app.middlewright.test/**", async (route) => { + await route.fulfill({ + body: ` +
+ + + +
+ `, + contentType: "text/html", + }); + }); + await context.route("https://auth.middlewright.test/**", async (route) => { + await route.fulfill({ + body: ` +
+

Authorize middlewright?

+ + +
+ `, + contentType: "text/html", + }); + }); +}; diff --git a/spec/popup-video.spec.ts b/spec/popup-video.spec.ts new file mode 100644 index 0000000..6f81c06 --- /dev/null +++ b/spec/popup-video.spec.ts @@ -0,0 +1,48 @@ +import { stat } from "node:fs/promises"; +import { test, expect } from "@playwright/test"; +import { addPlugins, videoMode } from "../src/index.ts"; +import { routeAuthDemoApp } from "./auth-demo-app.ts"; + +test.use({ video: "on" }); + +test("records separate videos for the main page and an auth popup", async ({ + page: basePage, + context, +}, testInfo) => { + await routeAuthDemoApp(context); + const video = videoMode({ finalHold: 0, highlight: { mode: "outline", duration: 300 }, trimStart: "never" }); + let popupVideo!: ReturnType; + { + await using page = await addPlugins({ page: basePage, testInfo, plugins: [video] }); + await page.goto("https://app.middlewright.test/"); + + const popupPromise = basePage.waitForEvent("popup"); + await page.getByRole("button", { name: "Sign in" }).click(); + popupVideo = videoMode({ finalHold: 0, highlight: { mode: "outline", duration: 300 }, trimStart: "never" }); + await using popup = await addPlugins({ + page: await popupPromise, + testInfo, + plugins: [popupVideo], + }); + + await popup.getByRole("button", { name: "Approve" }).click(); + await page.getByText("Signed in as mmkal").waitFor(); + } + + // Playwright screencasts each page separately, so each instance ends the + // test with its own raw recording and its own annotated render. + await expect(video.metadata()).resolves.toMatchObject({ + outputs: { raw: "video-raw.webm", rendered: "video-rendered.webm" }, + }); + await expect(popupVideo.metadata()).resolves.toMatchObject({ + outputs: { raw: "video-raw-2.webm", rendered: "video-rendered-2.webm" }, + }); + for (const path of [ + video.outputPaths().raw, + video.outputPaths().rendered, + popupVideo.outputPaths().raw, + popupVideo.outputPaths().rendered, + ]) { + expect((await stat(path)).size).toBeGreaterThan(0); + } +}); diff --git a/spec/popup.spec.ts b/spec/popup.spec.ts index 0fbb253..8f2fae3 100644 --- a/spec/popup.spec.ts +++ b/spec/popup.spec.ts @@ -1,6 +1,6 @@ import { test, expect } from "@playwright/test"; -import type { BrowserContext } from "@playwright/test"; import { addPlugins, videoMode } from "../src/index.ts"; +import { routeAuthDemoApp } from "./auth-demo-app.ts"; test("a popup wrapped with addPlugins runs actions through its own plugins", async ({ page: basePage, @@ -68,48 +68,21 @@ test("each videoMode instance still owns its artifacts after the test", async ({ }); }); -/** - * app.middlewright.test shows a Sign in button that opens an auth popup on - * auth.middlewright.test; approving there posts a message back to the opener, - * which then shows who signed in. Routed on the context so the popup page is - * covered too. - */ -const routeAuthDemoApp = async (context: BrowserContext) => { - await context.route("https://app.middlewright.test/**", async (route) => { - await route.fulfill({ - body: ` -
- - - -
- `, - contentType: "text/html", - }); - }); - await context.route("https://auth.middlewright.test/**", async (route) => { - await route.fulfill({ - body: ` -
-

Authorize middlewright?

- - -
- `, - contentType: "text/html", - }); - }); -}; +test("reusing one videoMode instance on a popup fails with a clear error", async ({ + page: basePage, + context, +}, testInfo) => { + await routeAuthDemoApp(context); + const video = videoMode({ finalHold: 0, highlight: { mode: "outline", duration: 300 }, trimStart: "never" }); + await using page = await addPlugins({ page: basePage, testInfo, plugins: [video] }); + await page.goto("https://app.middlewright.test/"); + + const popupPromise = basePage.waitForEvent("popup"); + await page.getByRole("button", { name: "Sign in" }).click(); + + // Wiring the same instance to a second page would wipe the main page's + // timeline, so it must fail loudly instead. + await expect( + addPlugins({ page: await popupPromise, testInfo, plugins: [video] }), + ).rejects.toThrow("create a fresh videoMode() instance for each page"); +}); diff --git a/src/plugins/video-mode.ts b/src/plugins/video-mode.ts index bc0e566..3f44357 100644 --- a/src/plugins/video-mode.ts +++ b/src/plugins/video-mode.ts @@ -265,6 +265,8 @@ export type VideoModeTrimStart = "auto" | "detect-blank" | "never" | ["selector" type VideoModeState = { addressBars: VideoModeAddressBar[]; + /** Distinguishes artifact filenames when a test has several instances. */ + artifactSuffix: string; captions: VideoModeCaption[]; deadAirDepth: number; deadAirSpans: VideoModeSpan[]; @@ -862,13 +864,35 @@ const recordCaption = async ( } }; -const videoModeOutputPaths = (testInfo: TestInfo): VideoModeOutputPaths => { +/** + * Insert a per-instance suffix before the extension: `video-mode.json` → + * `video-mode-2.json`. The first instance in a test keeps the unsuffixed + * names, so single-page tests are unaffected. + */ +const suffixArtifactFileName = (fileName: string, artifactSuffix: string) => { + if (!artifactSuffix) return fileName; + const extension = extname(fileName); + return `${fileName.slice(0, fileName.length - extension.length)}${artifactSuffix}${extension}`; +}; + +/** + * Registrations per testInfo.outputDir. Each videoMode instance added within + * one test (e.g. a fresh instance for a popup) gets its own artifact + * namespace so it can't clobber the main page's files. + */ +const videoModeRegistrationCounts = new Map(); + +const videoModeOutputPaths = ( + testInfo: TestInfo, + artifactSuffix: string, +): VideoModeOutputPaths => { + const name = (fileName: string) => suffixArtifactFileName(fileName, artifactSuffix); return { - metadata: join(testInfo.outputDir, VIDEO_MODE_METADATA_FILE), - player: join(testInfo.outputDir, VIDEO_MODE_PLAYER_FILE), - raw: join(testInfo.outputDir, VIDEO_MODE_RAW_FILE), - rendered: join(testInfo.outputDir, VIDEO_MODE_RENDERED_FILE), - reportPlayer: join(testInfo.outputDir, VIDEO_MODE_REPORT_PLAYER_FILE), + metadata: join(testInfo.outputDir, name(VIDEO_MODE_METADATA_FILE)), + player: join(testInfo.outputDir, name(VIDEO_MODE_PLAYER_FILE)), + raw: join(testInfo.outputDir, name(VIDEO_MODE_RAW_FILE)), + rendered: join(testInfo.outputDir, name(VIDEO_MODE_RENDERED_FILE)), + reportPlayer: join(testInfo.outputDir, name(VIDEO_MODE_REPORT_PLAYER_FILE)), }; }; @@ -1258,8 +1282,8 @@ const recordHighlight = async (options: { : undefined; const image = pan - ? `video-mode-pan-${options.state.highlightImageIndex}.png` - : `video-mode-highlight-${options.state.highlightImageIndex}.png`; + ? `video-mode-pan${options.state.artifactSuffix}-${options.state.highlightImageIndex}.png` + : `video-mode-highlight${options.state.artifactSuffix}-${options.state.highlightImageIndex}.png`; options.state.highlightImageIndex += 1; const imagePath = join(options.testInfo.outputDir, image); await mkdir(options.testInfo.outputDir, { recursive: true }); @@ -1546,7 +1570,7 @@ const recordFillReveal = async (options: { return; } - const image = `video-mode-fill-${options.state.highlightImageIndex}.png`; + const image = `video-mode-fill${options.state.artifactSuffix}-${options.state.highlightImageIndex}.png`; options.state.highlightImageIndex += 1; await mkdir(options.testInfo.outputDir, { recursive: true }); await options.locator.page().screenshot({ @@ -3580,7 +3604,7 @@ const playwrightReportAttachmentName = async (path: string) => { return `${createHash("sha1").update(data).digest("hex")}${extname(path)}`; }; -const videoModePlayerHtml = (options: { raw: string; rendered?: string }) => { +const videoModePlayerHtml = (options: { metadata: string; raw: string; rendered?: string }) => { const primary = options.rendered || options.raw; const primaryLabel = options.rendered ? "Rendered video" : "Raw video"; const primaryActiveKey = options.rendered ? "rendered" : "raw"; @@ -3724,7 +3748,7 @@ const videoModePlayerHtml = (options: { raw: string; rendered?: string }) => {
frame: 0
duration: ?s
Left/right steps one frame. Shift+left/right steps ten. Space toggles play.
-
${VIDEO_MODE_METADATA_FILE}
+
${options.metadata}
diff --git a/spec/popup-video.spec.ts b/spec/popup-video.spec.ts index 7865daf..6def818 100644 --- a/spec/popup-video.spec.ts +++ b/spec/popup-video.spec.ts @@ -1,5 +1,7 @@ +import { execFile as execFileCallback } from "node:child_process"; import { stat } from "node:fs/promises"; import { join } from "node:path"; +import { promisify } from "node:util"; import { test, expect } from "@playwright/test"; import { addPlugins, videoMode } from "../src/index.ts"; import { routeAuthDemoApp } from "./auth-demo-app.ts"; @@ -25,11 +27,13 @@ test("captures an auto-wrapped popup's raw screencast for the composite", async const metadata = await video.metadata(); expect(metadata.children).toMatchObject([ { + // The demo popup closes itself after Approve, like a real OAuth popup — + // closedAt comes from the close event, and there is no settled + // recordingEndedAt (the screencast start approximates the timeline). closedAt: expect.any(Number), highlights: [{ method: "click" }], openedAt: expect.any(Number), raw: "video-raw-popup-1.webm", - recordingEndedAt: expect.any(Number), viewport: { height: expect.any(Number), width: expect.any(Number) }, }, ]); @@ -38,6 +42,49 @@ test("captures an auto-wrapped popup's raw screencast for the composite", async expect((await stat(join(testInfo.outputDir, child.raw!))).size).toBeGreaterThan(0); }); +test("renders the popup as a dimmed overlay in one composed video", async ({ + page: basePage, + context, +}, testInfo) => { + await routeAuthDemoApp(context); + const video = videoMode({ + addressBar: false, + finalHold: 0, + highlight: { mode: "outline", duration: 500 }, + trimStart: "never", + }); + { + await using page = await addPlugins({ page: basePage, testInfo, plugins: [video] }); + await page.goto("https://app.middlewright.test/"); + // Let the screencast capture real frames on each side of the popup span — + // an instant flow lands entirely before the recorder's first frame. + await page.waitForTimeout(500); + + const popupPromise = basePage.waitForEvent("popup"); + await page.getByRole("button", { name: "Sign in" }).click(); + const popup = await popupPromise; + await popup.waitForTimeout(500); + await popup.getByRole("button", { name: "Approve" }).click(); + await page.getByText("Signed in as mmkal").waitFor(); + await page.waitForTimeout(500); + } + + await expect(video.metadata()).resolves.toMatchObject({ + outputs: { rendered: "video-rendered.webm" }, + }); + const frames = await videoFrameSamples(video.outputPaths().rendered); + // The demo app's background is a light gray throughout, so a darkened + // corner marks a frame where the popup backdrop dim is active (~40% black + // over ~244 gray lands around 146). + const dimmedFrames = frames.filter((frame) => frame.corner < 200); + const litFrames = frames.filter((frame) => frame.corner >= 200); + expect(dimmedFrames.length).toBeGreaterThan(0); + expect(litFrames.length).toBeGreaterThan(0); + // While dimmed, the popup's white card sits centered above the backdrop. + const overlayFrames = dimmedFrames.filter((frame) => frame.centerPeak > 220); + expect(overlayFrames.length).toBeGreaterThan(0); +}); + test("records separate videos for the main page and an auth popup", async ({ page: basePage, context, @@ -79,3 +126,49 @@ test("records separate videos for the main page and an auth popup", async ({ expect((await stat(path)).size).toBeGreaterThan(0); } }); + +const execFile = promisify(execFileCallback); + +/** + * Decode the video to small grayscale frames and sample each one: a pixel + * near the bottom-left corner (page background), and the brightest pixel of + * the central quarter (the popup card when the overlay is up). 0-255. + */ +const videoFrameSamples = async (path: string) => { + const size = 64; + const { stdout } = await execFile( + "ffmpeg", + [ + "-hide_banner", + "-loglevel", + "error", + "-i", + path, + "-vf", + `fps=10,scale=${size}:${size},format=gray`, + "-f", + "rawvideo", + "-pix_fmt", + "gray", + "pipe:1", + ], + { encoding: "buffer", maxBuffer: 64 * 1024 * 1024 }, + ); + const frameSize = size * size; + const frames: { centerPeak: number; corner: number }[] = []; + + for (let offset = 0; offset + frameSize <= stdout.length; offset += frameSize) { + let centerPeak = 0; + for (let y = Math.floor(size * 0.375); y < Math.floor(size * 0.625); y += 1) { + for (let x = Math.floor(size * 0.375); x < Math.floor(size * 0.625); x += 1) { + centerPeak = Math.max(centerPeak, stdout[offset + y * size + x]); + } + } + frames.push({ + centerPeak, + corner: stdout[offset + (size - 4) * size + 3], + }); + } + + return frames; +}; diff --git a/src/plugins/video-mode.ts b/src/plugins/video-mode.ts index 354f11d..a05ab3f 100644 --- a/src/plugins/video-mode.ts +++ b/src/plugins/video-mode.ts @@ -713,7 +713,7 @@ const translateVideoTimeline = (options: { text: caption.text, })), deadAir: options.deadAir.map((span) => translateVideoSpan(span, options.offsetMs)), - highlights: options.highlights.map((highlight) => { + highlights: options.highlights.map((highlight): VideoModeHighlight => { const start = Math.max(0, Math.round(highlight.start + options.offsetMs)); return { ...highlight, @@ -4105,6 +4105,183 @@ const renderVideo = async (options: { return true; }; +const VIDEO_MODE_COMPOSITE_FILE = "video-composite.webm"; +const CHILD_OVERLAY_MAX_FRACTION = 0.9; +const CHILD_OVERLAY_BACKDROP_OPACITY = 0.4; +const CHILD_OVERLAY_FADE_MS = 200; + +/** A popup screencast placed on the parent timeline as a scaled overlay. */ +type VideoModeChildLayer = { + child: VideoModeChild; + /** Shift applied to child-raw frames to land them in composite time (ms). */ + delayMs: number; + /** Overlay visibility window in composite time (ms). */ + enableFromMs: number; + enableToMs: number; + /** Scaled placement in composite pixels, centered and even-sized. */ + height: number; + width: number; + x: number; + y: number; + path: string; + rawInfo: VideoInfo; +}; + +const childCompositeLayers = async (options: { + children: VideoModeChild[]; + outputDir: string; + timelineOffsetMs: number; + video: VideoInfo; +}): Promise => { + const layers: VideoModeChildLayer[] = []; + + for (const child of options.children) { + if (!child.raw) continue; + const path = join(options.outputDir, child.raw); + const rawInfo = await videoInfo(path); + // A settled recorder maps the raw end to a known parent time. A popup + // that closed itself has no settled endpoint; its screencast started at + // page creation, which openedAt approximates. + const childOffsetMs = + child.recordingEndedAt === undefined + ? child.openedAt + : child.recordingEndedAt - rawInfo.durationMs; + const scale = Math.min( + 1, + (CHILD_OVERLAY_MAX_FRACTION * options.video.width) / rawInfo.width, + (CHILD_OVERLAY_MAX_FRACTION * options.video.height) / rawInfo.height, + ); + const width = Math.max(2, 2 * Math.round((rawInfo.width * scale) / 2)); + const height = Math.max(2, 2 * Math.round((rawInfo.height * scale) / 2)); + const enableFromMs = Math.max(0, child.openedAt + options.timelineOffsetMs); + const enableToMs = Math.min( + options.video.durationMs, + (child.closedAt === undefined ? child.openedAt + rawInfo.durationMs : child.closedAt) + + options.timelineOffsetMs, + ); + + if (enableToMs <= enableFromMs) continue; + + layers.push({ + child, + delayMs: childOffsetMs + options.timelineOffsetMs, + enableFromMs, + enableToMs, + height, + path, + rawInfo, + width, + x: Math.round((options.video.width - width) / 2), + y: Math.round((options.video.height - height) / 2), + }); + } + + return layers; +}; + +/** + * Pass A of the popup composite: overlay each popup screencast onto the + * parent's raw footage — dimmed backdrop, scaled to fit, alpha-faded in and + * out, windowed to the popup's open/close span, newest stacked on top. The + * output shares the parent raw timeline exactly, so the annotation render + * (pass B) runs on it unchanged; holds there freeze the composite, so it + * never matters which source triggered them. + */ +const compositeChildOverlays = async (options: { + inputPath: string; + layers: VideoModeChildLayer[]; + outputPath: string; +}) => { + const filters: string[] = []; + let currentLabel = "0:v"; + + options.layers.forEach((layer, index) => { + const from = formatSeconds(layer.enableFromMs); + const to = formatSeconds(layer.enableToMs); + const enable = `enable='between(t\\,${from}\\,${to})'`; + const dimLabel = `dim${index}`; + const childLabel = `popup${index}`; + const outLabel = `composite${index}`; + const fadeOutStartMs = Math.max(layer.enableFromMs, layer.enableToMs - CHILD_OVERLAY_FADE_MS); + + filters.push( + `[${currentLabel}]drawbox=x=0:y=0:w=iw:h=ih:color=black@${CHILD_OVERLAY_BACKDROP_OPACITY}:t=fill:${enable}[${dimLabel}]`, + ); + filters.push( + [ + `[${index + 1}:v]setpts=PTS+${formatSeconds(layer.delayMs)}/TB`, + `scale=w=${layer.width}:h=${layer.height}`, + "format=yuva420p", + `fade=t=in:st=${from}:d=${formatSeconds(CHILD_OVERLAY_FADE_MS)}:alpha=1`, + `fade=t=out:st=${formatSeconds(fadeOutStartMs)}:d=${formatSeconds(CHILD_OVERLAY_FADE_MS)}:alpha=1[${childLabel}]`, + ].join(","), + ); + filters.push( + `[${dimLabel}][${childLabel}]overlay=x=${layer.x}:y=${layer.y}:eof_action=pass:${enable}[${outLabel}]`, + ); + currentLabel = outLabel; + }); + + await execFile( + "ffmpeg", + [ + "-hide_banner", + "-loglevel", + "error", + "-y", + "-i", + options.inputPath, + ...options.layers.flatMap((layer) => ["-i", layer.path]), + "-filter_complex", + filters.join(";"), + "-map", + `[${currentLabel}]`, + "-an", + options.outputPath, + ], + { maxBuffer: 10 * 1024 * 1024 }, + ); +}; + +/** + * Project a popup highlight into composite coordinates. The child frame is + * scaled into an overlay box on the parent frame, so rects become + * parent-frame pixels; child-frame pixel treatments (pans, fill reveals, + * screenshot stills) drop away, leaving the plain box/pointer/freeze path. + */ +const projectChildHighlight = (options: { + highlight: VideoModeHighlight; + layer: VideoModeChildLayer; + video: VideoInfo; +}): VideoModeHighlight => { + const { highlight, layer } = options; + const viewport = options.layer.child.viewport || { + height: layer.rawInfo.height, + width: layer.rawInfo.width, + }; + const viewportToChildPixels = Math.min( + layer.rawInfo.width / viewport.width, + layer.rawInfo.height / viewport.height, + ); + const scale = viewportToChildPixels * (layer.width / layer.rawInfo.width); + + return { + ...highlight, + dialog: undefined, + fillReveal: undefined, + image: undefined, + pan: undefined, + rect: { + height: highlight.rect.height * scale, + width: highlight.rect.width * scale, + x: layer.x + highlight.rect.x * scale, + y: layer.y + highlight.rect.y * scale, + }, + sourceFrameAt: undefined, + viewport: { height: options.video.height, width: options.video.width }, + }; +}; + /** * The action-recording middleware, shared between a videoMode instance (which * records onto its own state) and its popup child recorders (which record onto @@ -4757,6 +4934,44 @@ export const videoMode = (options: VideoModeOptions = {}): VideoModePlugin => { highlights, offsetMs: timelineOffset, }); + + // Popup composite (pass A): overlay each popup's screencast onto the + // raw footage, then annotate that composite instead of the raw. The + // composite shares the raw timeline, so nothing downstream changes. + let renderInputPath = paths.raw; + const childLayers = await childCompositeLayers({ + children: metadataBeforeVideo.children, + outputDir: testInfo.outputDir, + timelineOffsetMs: timelineOffset, + video: rawVideoInfo, + }); + if (childLayers.length > 0) { + const compositePath = join( + testInfo.outputDir, + suffixArtifactFileName(VIDEO_MODE_COMPOSITE_FILE, state.artifactSuffix), + ); + await compositeChildOverlays({ + inputPath: paths.raw, + layers: childLayers, + outputPath: compositePath, + }); + renderInputPath = compositePath; + const projectedChildHighlights = childLayers.flatMap((layer) => + translateVideoTimeline({ + addressBars: [], + captions: [], + deadAir: [], + highlights: layer.child.highlights, + offsetMs: timelineOffset, + }).highlights.map((highlight) => + projectChildHighlight({ highlight, layer, video: rawVideoInfo }), + ), + ); + renderTimeline.highlights.push(...projectedChildHighlights); + renderTimeline.highlights.sort( + (left, right) => left.start - right.start || left.end - right.end, + ); + } // A selector-driven trim start resolves over the protocol and can // land a few milliseconds after a highlight recorded at effectively // the same moment. The race must not drop that highlight, so a start @@ -4838,7 +5053,7 @@ export const videoMode = (options: VideoModeOptions = {}): VideoModePlugin => { finalHoldMs: finalHold, highlightMode: highlight.mode === "pointer" ? "pointer" : "outline", highlights: renderTimeline.highlights, - inputPath: paths.raw, + inputPath: renderInputPath, outputDir: testInfo.outputDir, outputPath: paths.rendered, sourceRange, From 380ede960684e44be99c4f7f760a9a4b914ea3b6 Mon Sep 17 00:00:00 2001 From: Misha Kaletsky <15040698+mmkal@users.noreply.github.com> Date: Thu, 13 Aug 2026 15:09:12 +0100 Subject: [PATCH 09/13] Fix popup composite frame delivery and holds; permanent overlay demo spec Three compositing fixes found by eyeballing the rendered demo: - Static pages emit sparse screencast frames, and overlay emits output only at primary-input frame times. Resample BOTH chains to a continuous fps: without it the child frame that passed fade mid-ramp ghosts at partial alpha for the whole window, and the popup window can contain zero composite frames entirely (showing post-close footage). - Run the exit fade AFTER close, using the screencast's padded final frame: a self-closing popup otherwise puts its own Approve click - and the click hold's freeze frame - inside the fade by construction. - An instant click's source slice is a few ms wide, often between frame ticks, so the hold trim came up empty and bled the next piece's footage. Anchor the slice back from close and widen it to two frames. spec/popup-overlay-demo.spec.ts replaces the gitignored demo spec: full watchable treatment (pointer, captions, overlay), light assertions on the child span and rendered output. Overlay frame assertions calibrated against measured downscale blends. Co-Authored-By: Claude Fable 5 --- spec/popup-overlay-demo.spec.ts | 46 +++++++++++++++++++++++++++++++ spec/popup-video.spec.ts | 12 ++++---- src/plugins/video-mode.ts | 49 +++++++++++++++++++++++++++++---- 3 files changed, 97 insertions(+), 10 deletions(-) create mode 100644 spec/popup-overlay-demo.spec.ts diff --git a/spec/popup-overlay-demo.spec.ts b/spec/popup-overlay-demo.spec.ts new file mode 100644 index 0000000..7e55d7a --- /dev/null +++ b/spec/popup-overlay-demo.spec.ts @@ -0,0 +1,46 @@ +// Demo-grade popup flow with the full watchable treatment — pointer +// highlights, step captions, address bar, popup overlay composite. The +// rendered output doubles as the PR/README demo video. +import { stat } from "node:fs/promises"; +import { test, expect } from "@playwright/test"; +import { addPlugins, videoMode } from "../src/index.ts"; +import { routeAuthDemoApp } from "./auth-demo-app.ts"; + +test.use({ video: "on", viewport: { width: 960, height: 540 } }); + +test("auth popup demo", async ({ page: basePage, context }, testInfo) => { + await routeAuthDemoApp(context); + const video = videoMode(); + { + await using page = await addPlugins({ page: basePage, testInfo, plugins: [video] }); + + const popupPromise = basePage.waitForEvent("popup"); + await test.step("Open the sign-in popup", async () => { + await page.goto("https://app.middlewright.test/"); + // Real frames on each side of the popup span keep the composite honest + // (and the demo watchable) — an instant flow would land before the + // screencast's first frame. + await page.waitForTimeout(500); + await page.getByRole("button", { name: "Sign in" }).click(); + }); + + const popup = await popupPromise; + await test.step("Approve access in the popup", async () => { + await popup.waitForTimeout(500); + await popup.getByRole("button", { name: "Approve" }).click(); + }); + + await test.step("Back on the app, signed in", async () => { + await page.getByText("Signed in as mmkal").waitFor(); + await page.waitForTimeout(500); + }); + } + + const metadata = await video.metadata(); + expect(metadata).toMatchObject({ + children: [{ closedAt: expect.any(Number), openedAt: expect.any(Number) }], + outputs: { raw: "video-raw.webm", rendered: "video-rendered.webm" }, + }); + expect(metadata.children[0].closedAt!).toBeGreaterThan(metadata.children[0].openedAt); + expect((await stat(video.outputPaths().rendered)).size).toBeGreaterThan(0); +}); diff --git a/spec/popup-video.spec.ts b/spec/popup-video.spec.ts index 6def818..c40fbc6 100644 --- a/spec/popup-video.spec.ts +++ b/spec/popup-video.spec.ts @@ -73,11 +73,13 @@ test("renders the popup as a dimmed overlay in one composed video", async ({ outputs: { rendered: "video-rendered.webm" }, }); const frames = await videoFrameSamples(video.outputPaths().rendered); - // The demo app's background is a light gray throughout, so a darkened - // corner marks a frame where the popup backdrop dim is active (~40% black - // over ~244 gray lands around 146). - const dimmedFrames = frames.filter((frame) => frame.corner < 200); - const litFrames = frames.filter((frame) => frame.corner >= 200); + // The demo app's background is a light gray (~245) throughout, so a + // darkened corner marks a frame where the popup backdrop dim is active. + // The downscale blends the thin dim border with its bright neighbors, so + // dimmed corners read ~211 (overlay up) down to ~147 (exit fade), against + // ~245 when lit. + const dimmedFrames = frames.filter((frame) => frame.corner < 235); + const litFrames = frames.filter((frame) => frame.corner >= 235); expect(dimmedFrames.length).toBeGreaterThan(0); expect(litFrames.length).toBeGreaterThan(0); // While dimmed, the popup's white card sits centered above the backdrop. diff --git a/src/plugins/video-mode.ts b/src/plugins/video-mode.ts index a05ab3f..0219dbc 100644 --- a/src/plugins/video-mode.ts +++ b/src/plugins/video-mode.ts @@ -4113,9 +4113,11 @@ const CHILD_OVERLAY_FADE_MS = 200; /** A popup screencast placed on the parent timeline as a scaled overlay. */ type VideoModeChildLayer = { child: VideoModeChild; + /** Composite-time close (ms) — where the exit fade starts. */ + closeMs: number; /** Shift applied to child-raw frames to land them in composite time (ms). */ delayMs: number; - /** Overlay visibility window in composite time (ms). */ + /** Overlay visibility window in composite time (ms), including exit fade. */ enableFromMs: number; enableToMs: number; /** Scaled placement in composite pixels, centered and even-sized. */ @@ -4154,16 +4156,21 @@ const childCompositeLayers = async (options: { const width = Math.max(2, 2 * Math.round((rawInfo.width * scale) / 2)); const height = Math.max(2, 2 * Math.round((rawInfo.height * scale) / 2)); const enableFromMs = Math.max(0, child.openedAt + options.timelineOffsetMs); - const enableToMs = Math.min( + const closeMs = Math.min( options.video.durationMs, (child.closedAt === undefined ? child.openedAt + rawInfo.durationMs : child.closedAt) + options.timelineOffsetMs, ); + // The exit fade runs AFTER close (the screencast's padded final frame + // supplies footage): a popup that closes itself right after a click would + // otherwise put that click — and its hold's freeze frame — mid-fade. + const enableToMs = Math.min(options.video.durationMs, closeMs + CHILD_OVERLAY_FADE_MS); - if (enableToMs <= enableFromMs) continue; + if (closeMs <= enableFromMs) continue; layers.push({ child, + closeMs, delayMs: childOffsetMs + options.timelineOffsetMs, enableFromMs, enableToMs, @@ -4188,12 +4195,19 @@ const childCompositeLayers = async (options: { * never matters which source triggered them. */ const compositeChildOverlays = async (options: { + /** Continuous frame rate for the overlay chains (see fps note below). */ + fps: number; inputPath: string; layers: VideoModeChildLayer[]; outputPath: string; }) => { const filters: string[] = []; - let currentLabel = "0:v"; + // The parent screencast is as sparse as the child ones (a static page emits + // no frames), and overlay only emits output at primary-input frame times — + // without resampling, the whole popup window can contain zero composite + // frames. A continuous base gives every enable window frames to land on. + filters.push(`[0:v]fps=${formatFilterNumber(options.fps)}[base]`); + let currentLabel = "base"; options.layers.forEach((layer, index) => { const from = formatSeconds(layer.enableFromMs); @@ -4202,7 +4216,7 @@ const compositeChildOverlays = async (options: { const dimLabel = `dim${index}`; const childLabel = `popup${index}`; const outLabel = `composite${index}`; - const fadeOutStartMs = Math.max(layer.enableFromMs, layer.enableToMs - CHILD_OVERLAY_FADE_MS); + const fadeOutStartMs = layer.closeMs; filters.push( `[${currentLabel}]drawbox=x=0:y=0:w=iw:h=ih:color=black@${CHILD_OVERLAY_BACKDROP_OPACITY}:t=fill:${enable}[${dimLabel}]`, @@ -4210,6 +4224,11 @@ const compositeChildOverlays = async (options: { filters.push( [ `[${index + 1}:v]setpts=PTS+${formatSeconds(layer.delayMs)}/TB`, + // A mostly-static popup screencast has sparse frames; without + // resampling, the frame that happens to pass `fade` mid-ramp keeps + // its partial alpha while framesync repeats it for the whole window, + // ghosting the overlay. Continuous frames give fade real timestamps. + `fps=${formatFilterNumber(options.fps)}`, `scale=w=${layer.width}:h=${layer.height}`, "format=yuva420p", `fade=t=in:st=${from}:d=${formatSeconds(CHILD_OVERLAY_FADE_MS)}:alpha=1`, @@ -4264,9 +4283,28 @@ const projectChildHighlight = (options: { layer.rawInfo.height / viewport.height, ); const scale = viewportToChildPixels * (layer.width / layer.rawInfo.width); + // Holds clone the composite frame at actionEnd. An action that closes the + // popup (Approve on an OAuth popup) puts that frame past close — move the + // highlight's source slice back into stable popup-visible footage, and + // widen it to a couple of frames so the trim can't come up empty (an + // instant click's slice is a few ms — often between frame ticks). + const holdSafeEndMs = Math.floor(layer.closeMs - 2 * options.video.frameDurationMs); + let { actionEnd, end, start } = highlight; + if (actionEnd !== undefined && actionEnd > holdSafeEndMs) { + const holdMs = end - start; + const sliceMs = Math.max(2 * options.video.frameDurationMs, actionEnd - start); + const earliestStartMs = layer.enableFromMs + CHILD_OVERLAY_FADE_MS; + actionEnd = holdSafeEndMs; + start = Math.max(earliestStartMs, actionEnd - sliceMs); + actionEnd = Math.max(actionEnd, start + 1); + end = start + holdMs; + } return { ...highlight, + actionEnd, + end, + start, dialog: undefined, fillReveal: undefined, image: undefined, @@ -4951,6 +4989,7 @@ export const videoMode = (options: VideoModeOptions = {}): VideoModePlugin => { suffixArtifactFileName(VIDEO_MODE_COMPOSITE_FILE, state.artifactSuffix), ); await compositeChildOverlays({ + fps: 1000 / rawVideoInfo.frameDurationMs, inputPath: paths.raw, layers: childLayers, outputPath: compositePath, From a05ce83fbfb146403518d1abcec08bcc6a0e4230 Mon Sep 17 00:00:00 2001 From: Misha Kaletsky <15040698+mmkal@users.noreply.github.com> Date: Thu, 13 Aug 2026 15:09:50 +0100 Subject: [PATCH 10/13] Phase 4: popups README section, task completion Co-Authored-By: Claude Fable 5 --- README.md | 19 +++++++++++------- .../2026-08-13-popup-overlay-video.md} | 20 +++++++++++++------ 2 files changed, 26 insertions(+), 13 deletions(-) rename tasks/{popup-overlay-video.md => complete/2026-08-13-popup-overlay-video.md} (73%) diff --git a/README.md b/README.md index fb18a14..d1a42f4 100644 --- a/README.md +++ b/README.md @@ -353,20 +353,25 @@ export default defineConfig({ ### Popups -Pages you never wrap (popups, `context.newPage()`) fall through to plain Playwright. To get plugin behavior in a popup — an OAuth window, say — wrap it with a second `addPlugins` call, using **fresh plugin instances**: +Popups are wrapped automatically. When a wrapped page opens one — an OAuth window, say — the popup gets the same plugin treatment, no wiring: ```ts const popupPromise = page.waitForEvent("popup"); await page.getByRole("button", { name: "Sign in" }).click(); -await using popup = await addPlugins({ - page: await popupPromise, - testInfo, - plugins: [spinnerWaiter(), videoMode()], -}); +const popup = await popupPromise; // already wrapped await popup.getByRole("button", { name: "Approve" }).click(); ``` -Playwright screencasts each page separately, so the popup's `videoMode` produces its own video; its artifacts get a `-2` suffix (`video-rendered-2.webm`, `video-mode-2.json`, …) so they sit next to the main page's in the same output dir. Reusing the main page's `videoMode` instance on the popup would wipe the main timeline, so it throws instead — one instance per page. See [spec/popup.spec.ts](spec/popup.spec.ts). +In video mode, the popup renders as an overlay **in the main page's video**: scaled to fit 90% of the frame over the dimmed page, faded in and out on open/close, with popup clicks pointer-annotated inside the overlay. One composed video per test, popups included. The popup's facts land in `video-mode.json` under `children`. + +Details and escape hatches: + +- Plugins can control what a popup gets via the `forPopup(ctx)` hook — return a plugin for the popup, or `null` to skip. Hookless plugins are re-registered as-is (fine for stateless ones). +- `addPlugins({ ..., popups: false })` turns auto-wrap off. You can then wrap the popup manually with **fresh plugin instances** — a fresh `videoMode()` gives the popup its own standalone video, with `-2`-suffixed artifacts (`video-rendered-2.webm`, `video-mode-2.json`, …). +- Wrapping an already-wrapped page throws, as does reusing an active `videoMode` instance on a second page — one instance per page. +- Popup dialogs (`alert`/`confirm`/`prompt` opened by the popup) aren't annotated in video mode yet. + +See [spec/popup.spec.ts](spec/popup.spec.ts) and [spec/popup-overlay-demo.spec.ts](spec/popup-overlay-demo.spec.ts). ## Writing your own plugin diff --git a/tasks/popup-overlay-video.md b/tasks/complete/2026-08-13-popup-overlay-video.md similarity index 73% rename from tasks/popup-overlay-video.md rename to tasks/complete/2026-08-13-popup-overlay-video.md index a9807a0..e759ec4 100644 --- a/tasks/popup-overlay-video.md +++ b/tasks/complete/2026-08-13-popup-overlay-video.md @@ -1,20 +1,28 @@ # Popup overlay videos + auto-wrapped popup plugins --- -status: in-progress +status: done size: large branch: popup-overlay base: popup-plugins (PR #32) +pr: https://github.com/iterate/middlewright/pull/33 --- -**Status summary**: design settled via grill session (decisions below). Implementation in 4 phases; starting phase 1. +**Status summary**: done. Popups auto-wrap by default (plugin-system `forPopup` hook, `popups: false` opt-out, double-wrap error). Video mode records popups as child timelines and renders them as a dimmed 90%-fit overlay in ONE composed video via a two-pass render (composite pass, then the untouched annotation pass). Permanent demo spec + README updated. Deferred: popup dialog annotations; unified-piece holds inside the overlay use plain freezes (child pans/fill-reveals degrade to box highlights). ## Checklist -- [ ] Phase 1: plugin-system guard + auto-wrap (`popups: false` opt-out, `forPopup` hook, double-wrap error, spec migration) -- [ ] Phase 2: videoMode child recorder — facts + metadata `children` schema -- [ ] Phase 3: render integration — composited timeline, overlay transform, cursor projection -- [ ] Phase 4: docs + permanent demo spec + refreshed PR video +- [x] Phase 1: plugin-system guard + auto-wrap _(`popups: false`, `forPopup`, double-wrap error; specs migrated)_ +- [x] Phase 2: videoMode child recorder — facts + metadata `children` schema _(v2; parent-clock child highlights, raw screencast copy, close calibration)_ +- [x] Phase 3: render integration _(two-pass: composite with dim+fade+fps-resample, then existing piece machinery; projected child highlights; hold slices anchored clear of close)_ +- [x] Phase 4: docs + permanent demo spec (`spec/popup-overlay-demo.spec.ts`) + refreshed PR video + +## Implementation notes (what differed from the plan) + +- Decision 6's "unified piece timeline" landed as a **two-pass render**: pass A composites popup screencasts onto the parent raw footage (same timeline), pass B is the existing single-source piece/hold/cursor machinery over the composite. Holds freeze the composite as decided; child highlights project into composite coordinates as plain box/pointer highlights (child pans/fill-reveals/stills degrade — upgrade path stays open). +- Both ffmpeg chains need `fps` resampling: static pages emit sparse screencast frames, which ghosted fades and left the popup window without composite frames. +- Exit fade runs AFTER close (screencast's padded final frame supplies footage) — a self-closing popup otherwise puts its own click mid-fade. +- Deferred: popup dialog annotations, scale-zoom enter animation (alpha fade only). ## Context From 6556d5602ec85b182c2e4c41b93bd482cb205705 Mon Sep 17 00:00:00 2001 From: Misha Kaletsky <15040698+mmkal@users.noreply.github.com> Date: Fri, 14 Aug 2026 11:36:27 +0100 Subject: [PATCH 11/13] Animate popup overlays and sync fill reveals inside them The popup overlay now slides up from the bottom edge on enter (300ms, quadratic ease-out) and slides back down after close, alongside the alpha fade. Piece planning gains keepSpans: the overlapping-hold skip used to jump source footage straight across the popup's enter/exit animations, hard-cutting them out of the output; skips that would cross a popup transition are cancelled. Parent highlights that start inside a popup's exit window shift past it, so their holds can't freeze a mid-fade ghost (the flash after the popup disappeared). Fill reveals now work inside overlays: the typed-text reveal renders over a frozen composite frame (popup risen, field empty, backdrop intact) with the child screenshot's content rect scaled and positioned through the overlay transform, synced to cursor arrival like main-page fills. highlightCursorPoint projects fillReveal.initialRect through the overlay transform too - the I-beam used to land offset from the field. Child raw footage anchoring: a self-closing popup's screencast t=0 is its first captured frame, which lags the popup event by the initial paint, so footage played early (fields filled before the reveal). The raw video's padded end is the better anchor: closedAt + 1s minimum final-frame padding - duration, floored at openedAt. Demo: the popup is now a realistic sign-in form (username/password + Sign in) on indigo, over a teal app page so the dimmed backdrop reads; MIDDLEWRIGHT_DEBUG_PIECES=1 dumps the render piece plan. Includes previously staged demo/styling work reviewed via the local video (not the GitHub PR). Co-Authored-By: Claude Fable 5 --- spec/auth-demo-app.ts | 69 ++++++++ spec/popup-overlay-demo.spec.ts | 10 +- src/plugins/video-mode.ts | 303 +++++++++++++++++++++++++++++--- 3 files changed, 351 insertions(+), 31 deletions(-) diff --git a/spec/auth-demo-app.ts b/spec/auth-demo-app.ts index 8df3621..596fc57 100644 --- a/spec/auth-demo-app.ts +++ b/spec/auth-demo-app.ts @@ -16,6 +16,75 @@ const demoStyle = ` `; +/** + * Demo-video variant of the auth flow: same app page, but the popup is a + * realistic sign-in form (inert username/password fields, a Sign in button + * that notifies the opener and closes) on a visibly different background so + * the popout reads clearly in the rendered overlay. + */ +export const routeSignInDemoApp = async (context: BrowserContext) => { + await routeAuthDemoApp(context); + // The app page gets a colored background so the dimmed page under the + // popup overlay reads clearly in the rendered video. + await context.route("https://app.middlewright.test/**", async (route) => { + await route.fulfill({ + body: ` + ${demoStyle} + +
+

middlewright dashboard

+ + + +
+ `, + contentType: "text/html", + }); + }); + await context.route("https://auth.middlewright.test/**", async (route) => { + await route.fulfill({ + body: ` + ${demoStyle} + +
+

Sign in to middlewright

+ + + + + + +
+ `, + contentType: "text/html", + }); + }); +}; + export const routeAuthDemoApp = async (context: BrowserContext) => { await context.route("https://app.middlewright.test/**", async (route) => { await route.fulfill({ diff --git a/spec/popup-overlay-demo.spec.ts b/spec/popup-overlay-demo.spec.ts index 7e55d7a..c386301 100644 --- a/spec/popup-overlay-demo.spec.ts +++ b/spec/popup-overlay-demo.spec.ts @@ -4,12 +4,12 @@ import { stat } from "node:fs/promises"; import { test, expect } from "@playwright/test"; import { addPlugins, videoMode } from "../src/index.ts"; -import { routeAuthDemoApp } from "./auth-demo-app.ts"; +import { routeSignInDemoApp } from "./auth-demo-app.ts"; test.use({ video: "on", viewport: { width: 960, height: 540 } }); test("auth popup demo", async ({ page: basePage, context }, testInfo) => { - await routeAuthDemoApp(context); + await routeSignInDemoApp(context); const video = videoMode(); { await using page = await addPlugins({ page: basePage, testInfo, plugins: [video] }); @@ -25,9 +25,11 @@ test("auth popup demo", async ({ page: basePage, context }, testInfo) => { }); const popup = await popupPromise; - await test.step("Approve access in the popup", async () => { + await test.step("Sign in as mmkal", async () => { await popup.waitForTimeout(500); - await popup.getByRole("button", { name: "Approve" }).click(); + await popup.getByLabel("Username").fill("mmkal"); + await popup.getByLabel("Password").fill("hunter2"); + await popup.getByRole("button", { name: "Sign in" }).click(); }); await test.step("Back on the app, signed in", async () => { diff --git a/src/plugins/video-mode.ts b/src/plugins/video-mode.ts index 0219dbc..04fb781 100644 --- a/src/plugins/video-mode.ts +++ b/src/plugins/video-mode.ts @@ -142,6 +142,17 @@ export type VideoModeHighlight = VideoModeSpan & { fillReveal?: VideoModeFillReveal; image?: string; method?: OverrideableMethod; + /** + * Set at render time on projected popup highlights: maps child-frame + * coordinates (which `fillReveal` and its screenshots use) into the + * composite frame — scale plus the overlay's top-left corner. + */ + overlayTransform?: { + scale: number; + viewport: VideoModeViewport; + x: number; + y: number; + }; pan?: VideoModePan; rect: VideoModeRect; sourceFrameAt?: number; @@ -2315,6 +2326,12 @@ const videoPieces = (options: { addressBars: VideoModeAddressBar[]; frameDurationMs: number; highlights: VideoModeHighlight[]; + /** + * Source spans that must reach the output (popup enter/exit animations). + * Overlapping-hold skips normally jump the footage between two highlights; + * a skip is cancelled when it would leap across one of these. + */ + keepSpans: VideoModeSpan[]; preActionStabilizationMs: number; segments: RenderVideoSegment[]; }): VideoPiece[] => { @@ -2362,8 +2379,10 @@ const videoPieces = (options: { const nextHighlight = highlights[highlightIndex + 1]; if (highlight.start > cursor) { - const postAction = previousHighlight?.fillReveal ? previousHighlight : undefined; - const preAction = highlight.fillReveal ? highlight : undefined; + const stabilizable = (candidate: VideoModeHighlight | undefined) => + candidate?.fillReveal && !candidate.overlayTransform ? candidate : undefined; + const postAction = stabilizable(previousHighlight); + const preAction = stabilizable(highlight); // `trim` chooses whole source frames. Round down so the boundary frame // belongs to the stabilized piece instead of leaking from the raw gap. const preActionStart = preAction @@ -2417,7 +2436,13 @@ const videoPieces = (options: { let nextCursor = actionEnd; if (nextHighlight && highlight.end > nextHighlight.start) { - nextCursor = Math.max(nextCursor, nextHighlight.start); + const skipTo = Math.max(nextCursor, nextHighlight.start); + const skipCrossesKeptSpan = options.keepSpans.some( + (span) => span.start < skipTo && span.end > nextCursor, + ); + if (!skipCrossesKeptSpan) { + nextCursor = skipTo; + } } cursor = Math.min(segment.end, nextCursor); @@ -2427,7 +2452,10 @@ const videoPieces = (options: { if (segment.end > cursor) { pieces.push({ end: segment.end, - postAction: previousHighlight?.fillReveal ? previousHighlight : undefined, + postAction: + previousHighlight?.fillReveal && !previousHighlight.overlayTransform + ? previousHighlight + : undefined, speed: segment.speed, start: cursor, }); @@ -2656,8 +2684,20 @@ const highlightCursorPoint = ( highlight: VideoModeHighlight, video: { width: number; height: number }, ) => { - const rect = highlight.fillReveal - ? scaleVideoModeRect(highlight.fillReveal.initialRect, highlight.viewport, video) + // fillReveal rects live in child-frame coordinates on projected popup + // highlights — map them through the overlay transform first. + const transform = highlight.overlayTransform; + const projectedInitialRect = + highlight.fillReveal && transform + ? { + height: highlight.fillReveal.initialRect.height * transform.scale, + width: highlight.fillReveal.initialRect.width * transform.scale, + x: transform.x + highlight.fillReveal.initialRect.x * transform.scale, + y: transform.y + highlight.fillReveal.initialRect.y * transform.scale, + } + : highlight.fillReveal?.initialRect; + const rect = projectedInitialRect + ? scaleVideoModeRect(projectedInitialRect, highlight.viewport, video) : scaleHighlight(highlight, video); return { @@ -2959,6 +2999,7 @@ const renderedVideoFilter = (options: { highlightMode: "outline" | "pointer"; highlightInputs: HighlightInput[]; highlights: VideoModeHighlight[]; + keepSpans: VideoModeSpan[]; preActionStabilizationMs: number; segments: RenderVideoSegment[]; textPointerInput?: PointerInput; @@ -2971,10 +3012,23 @@ const renderedVideoFilter = (options: { addressBars: options.addressBars, frameDurationMs: options.video.frameDurationMs, highlights: options.highlights, + keepSpans: options.keepSpans, preActionStabilizationMs: options.preActionStabilizationMs, segments: options.segments, }); const renderedPieces = renderedVideoPieces(pieces); + if (process.env.MIDDLEWRIGHT_DEBUG_PIECES) { + for (const piece of renderedPieces) { + console.log( + `piece src[${piece.start}-${piece.end}] out[${Math.round(piece.outputStart)}-${Math.round(piece.outputEnd)}] speed=${piece.speed}` + + (piece.highlight ? ` highlight=${piece.highlight.method} hstart=${piece.highlight.start}` : "") + + (piece.addressBar ? " addressBar" : "") + + (piece.highlight?.overlayTransform ? " overlay" : "") + + (piece.highlight?.fillReveal ? " fillReveal" : "") + + (piece.highlight?.image ? ` image=${piece.highlight.image}` : ""), + ); + } + } const targets = cursorTargets({ highlights: options.highlights, pieces: renderedPieces, @@ -3126,6 +3180,126 @@ const renderedVideoFilter = (options: { continue; } + // Fill reveal inside a popup overlay: the base is a frozen composite + // frame from just before the fill (popup risen, field empty, dim and + // parent intact), and the typed reveal is the child screenshot's content + // rect scaled and positioned through the overlay transform. + if (piece.highlight && fillReveal && postFillInput && piece.highlight.overlayTransform) { + const transform = piece.highlight.overlayTransform; + const projectLength = (value: number) => Math.round(value * transform.scale); + const scaledImage = { + height: Math.max(2, projectLength(transform.viewport.height)), + width: Math.max(2, projectLength(transform.viewport.width)), + }; + const contentLocal = { + height: Math.max(1, Math.min(scaledImage.height, projectLength(fillReveal.contentRect.height))), + width: Math.max(1, Math.min(scaledImage.width, projectLength(fillReveal.contentRect.width))), + x: Math.max(0, projectLength(fillReveal.contentRect.x)), + y: Math.max(0, projectLength(fillReveal.contentRect.y)), + }; + const contentAbsolute = { + x: Math.round(transform.x + fillReveal.contentRect.x * transform.scale), + y: Math.round(transform.y + fillReveal.contentRect.y * transform.scale), + }; + const duration = renderedPieceDuration(piece); + const durationSeconds = formatSeconds(duration); + const revealStops = fillReveal.revealStops + .map((stop) => Math.max(1, Math.min(contentLocal.width, projectLength(stop)))) + .filter((stop, stopIndex, stops) => stopIndex === 0 || stop !== stops[stopIndex - 1]); + const revealSteps = fillReveal.revealBands.flatMap((band) => { + const y = Math.max(0, Math.min(contentLocal.height - 1, projectLength(band.y))); + const height = Math.max(1, Math.min(contentLocal.height - y, projectLength(band.height))); + return revealStops.map((width) => ({ height, width, y })); + }); + const target = plan.targets.find( + (candidate) => candidate.highlight === piece.highlight, + ); + const revealEnd = + options.highlightMode === "pointer" + ? Math.max(0, duration - TEXT_CURSOR_POINTER_TAIL_MS) + : duration; + const pointerArrival = target + ? Math.max(0, target.arriveAt - renderedPiece.outputStart) + : 0; + const availableAfterArrival = Math.max(0, revealEnd - pointerArrival); + const preRevealHold = Math.min( + TEXT_CURSOR_HOLD_IDEAL_MS, + availableAfterArrival / 2, + ); + const revealStart = Math.max( + 0, + Math.min(revealEnd, pointerArrival + preRevealHold), + ); + const baseLabel = `fillbase${index}`; + // A few frames of margin: the child anchor is approximate, and the + // safe failure mode is showing slightly earlier (still-empty) footage. + const freezeStart = Math.max(0, piece.start - 3 * options.video.frameDurationMs); + filters.push( + [ + `[0:v]trim=start=${formatSeconds(freezeStart)}:end=${formatSeconds( + freezeStart + options.video.frameDurationMs, + )}`, + "setpts=PTS-STARTPTS", + `tpad=stop_mode=clone:stop_duration=${formatSeconds( + Math.max(0, duration - options.video.frameDurationMs), + )}`, + `trim=start=0:end=${durationSeconds}`, + `setpts=PTS-STARTPTS[${baseLabel}]`, + ].join(","), + ); + + if (revealSteps.length === 0) { + filters.push(`[${baseLabel}]null[${label}]`); + continue; + } + + const splitLabels = revealSteps.map((_, stepIndex) => `fillpost${index}x${stepIndex}`); + filters.push( + [ + `[${postFillInput.inputIndex}:v]scale=w=${scaledImage.width}:h=${scaledImage.height}`, + `crop=w=${contentLocal.width}:h=${contentLocal.height}:x=${contentLocal.x}:y=${contentLocal.y}`, + `trim=start=0:end=${durationSeconds}`, + "setpts=PTS-STARTPTS", + `split=${revealSteps.length}${splitLabels.map((splitLabel) => `[${splitLabel}]`).join("")}`, + ].join(","), + ); + + let composedLabel = baseLabel; + for (let stepIndex = 0; stepIndex < revealSteps.length; stepIndex += 1) { + const step = revealSteps[stepIndex]; + const cropLabel = `fillcrop${index}x${stepIndex}`; + const nextLabel = `fillcomposed${index}x${stepIndex}`; + const showAt = + revealStart + + ((revealEnd - revealStart) * (stepIndex + 1)) / + (revealSteps.length + 1); + filters.push( + `${[ + `[${splitLabels[stepIndex]}]crop=w=${step.width}`, + `h=${step.height}`, + "x=0", + `y=${step.y}`, + ].join(":")}[${cropLabel}]`, + ); + filters.push( + [ + `[${composedLabel}][${cropLabel}]overlay=x=${contentAbsolute.x}`, + `y=${contentAbsolute.y + step.y}`, + `enable='gte(t\\,${formatSeconds(showAt)})'`, + `shortest=1[${nextLabel}]`, + ].join(":"), + ); + composedLabel = nextLabel; + } + + filters.push( + options.highlightMode === "outline" + ? `[${composedLabel}]${drawboxFilter(piece.highlight, options.video)}[${label}]` + : `[${composedLabel}]null[${label}]`, + ); + continue; + } + if (piece.highlight && fillReveal && preFillInput && postFillInput) { const scaledViewport = scaledViewportSize(piece.highlight.viewport, options.video); const contentRect = scaleVideoModeRect( @@ -3902,6 +4076,8 @@ const renderVideo = async (options: { highlightMode: "outline" | "pointer"; highlights: VideoModeHighlight[]; inputPath: string; + /** Source spans that must not be skipped (popup enter/exit animations). */ + keepSpans: VideoModeSpan[]; outputDir: string; outputPath: string; deadAir: VideoModeSpan[]; @@ -4010,6 +4186,7 @@ const renderVideo = async (options: { addressBars: options.addressBars, frameDurationMs: info.frameDurationMs, highlights: options.highlights, + keepSpans: options.keepSpans, preActionStabilizationMs, segments, }), @@ -4063,6 +4240,7 @@ const renderVideo = async (options: { highlightMode: options.highlightMode, highlightInputs, highlights: options.highlights, + keepSpans: options.keepSpans, preActionStabilizationMs, segments, textPointerInput, @@ -4106,9 +4284,12 @@ const renderVideo = async (options: { }; const VIDEO_MODE_COMPOSITE_FILE = "video-composite.webm"; +// Playwright extends a closed page's final screencast frame by the time since +// that frame arrived, with this minimum (see VIDEO_MODE_RECORDER_SETTLE_MS). +const RECORDER_FINAL_FRAME_MIN_PADDING_MS = 1000; const CHILD_OVERLAY_MAX_FRACTION = 0.9; const CHILD_OVERLAY_BACKDROP_OPACITY = 0.4; -const CHILD_OVERLAY_FADE_MS = 200; +const CHILD_OVERLAY_FADE_MS = 300; /** A popup screencast placed on the parent timeline as a scaled overlay. */ type VideoModeChildLayer = { @@ -4142,11 +4323,18 @@ const childCompositeLayers = async (options: { const path = join(options.outputDir, child.raw); const rawInfo = await videoInfo(path); // A settled recorder maps the raw end to a known parent time. A popup - // that closed itself has no settled endpoint; its screencast started at - // page creation, which openedAt approximates. + // that closed itself has no settled endpoint — and its screencast t=0 is + // the first *captured* frame, which lags the popup event by the initial + // paint. Playwright pads the final frame by >=1s on close, so the first + // frame lands near closedAt + padding - duration; openedAt is the floor. const childOffsetMs = child.recordingEndedAt === undefined - ? child.openedAt + ? Math.max( + child.openedAt, + (child.closedAt === undefined ? child.openedAt : child.closedAt) + + RECORDER_FINAL_FRAME_MIN_PADDING_MS - + rawInfo.durationMs, + ) : child.recordingEndedAt - rawInfo.durationMs; const scale = Math.min( 1, @@ -4235,8 +4423,22 @@ const compositeChildOverlays = async (options: { `fade=t=out:st=${formatSeconds(fadeOutStartMs)}:d=${formatSeconds(CHILD_OVERLAY_FADE_MS)}:alpha=1[${childLabel}]`, ].join(","), ); + // Slide up from the bottom edge on enter (cubic ease-out), slide back + // down after close (cubic ease-in), resting at the centered position + // between. Times are seconds; commas escaped for the filter graph. + const fadeSeconds = formatSeconds(CHILD_OVERLAY_FADE_MS); + const offscreenY = "H"; + const enterProgress = `min(max((t-${from})/${fadeSeconds}\\,0)\\,1)`; + const exitProgress = `min(max((t-${formatSeconds(layer.closeMs)})/${fadeSeconds}\\,0)\\,1)`; + const slideY = [ + `if(lt(t\\,${formatSeconds(layer.enableFromMs + CHILD_OVERLAY_FADE_MS)})`, + `\\,${layer.y}+(${offscreenY}-${layer.y})*pow(1-${enterProgress}\\,2)`, + `\\,if(gt(t\\,${formatSeconds(layer.closeMs)})`, + `\\,${layer.y}+(${offscreenY}-${layer.y})*pow(${exitProgress}\\,2)`, + `\\,${layer.y}))`, + ].join(""); filters.push( - `[${dimLabel}][${childLabel}]overlay=x=${layer.x}:y=${layer.y}:eof_action=pass:${enable}[${outLabel}]`, + `[${dimLabel}][${childLabel}]overlay=x=${layer.x}:y='${slideY}':eval=frame:eof_action=pass:${enable}[${outLabel}]`, ); currentLabel = outLabel; }); @@ -4283,21 +4485,25 @@ const projectChildHighlight = (options: { layer.rawInfo.height / viewport.height, ); const scale = viewportToChildPixels * (layer.width / layer.rawInfo.width); - // Holds clone the composite frame at actionEnd. An action that closes the - // popup (Approve on an OAuth popup) puts that frame past close — move the - // highlight's source slice back into stable popup-visible footage, and - // widen it to a couple of frames so the trim can't come up empty (an - // instant click's slice is a few ms — often between frame ticks). - const holdSafeEndMs = Math.floor(layer.closeMs - 2 * options.video.frameDurationMs); + // Holds clone the composite frame at actionEnd. The overlay stays fully + // visible until close (the exit animation runs after), so the freeze only + // needs two small guards: never sample past close, and keep the source + // slice wider than a frame tick so the trim can't come up empty (an + // instant click's slice is a few ms — often between ticks). Widening may + // nudge the highlight a few ms earlier; order-preserving in practice. + const closeFloorMs = Math.floor(layer.closeMs); + const minSliceMs = options.video.frameDurationMs + 10; let { actionEnd, end, start } = highlight; - if (actionEnd !== undefined && actionEnd > holdSafeEndMs) { - const holdMs = end - start; - const sliceMs = Math.max(2 * options.video.frameDurationMs, actionEnd - start); - const earliestStartMs = layer.enableFromMs + CHILD_OVERLAY_FADE_MS; - actionEnd = holdSafeEndMs; - start = Math.max(earliestStartMs, actionEnd - sliceMs); - actionEnd = Math.max(actionEnd, start + 1); - end = start + holdMs; + if (actionEnd !== undefined) { + actionEnd = Math.min(actionEnd, closeFloorMs); + if (actionEnd - start < minSliceMs) { + actionEnd = Math.min(closeFloorMs, start + minSliceMs); + } + if (actionEnd - start < minSliceMs) { + const shift = minSliceMs - (actionEnd - start); + start -= shift; + end -= shift; + } } return { @@ -4306,8 +4512,18 @@ const projectChildHighlight = (options: { end, start, dialog: undefined, - fillReveal: undefined, - image: undefined, + // Fill reveals keep their child-frame geometry and screenshots; the + // render projects them through overlayTransform. Screenshot stills + // without a reveal (e.g. password fallbacks) would render full-frame, so + // they drop away and the hold freezes the composite instead. + fillReveal: highlight.fillReveal, + image: highlight.fillReveal ? highlight.image : undefined, + overlayTransform: { + scale, + viewport, + x: layer.x, + y: layer.y, + }, pan: undefined, rect: { height: highlight.rect.height * scale, @@ -4977,6 +5193,9 @@ export const videoMode = (options: VideoModeOptions = {}): VideoModePlugin => { // raw footage, then annotate that composite instead of the raw. The // composite shares the raw timeline, so nothing downstream changes. let renderInputPath = paths.raw; + // Popup enter/exit animations must reach the output even when a + // hold's overlap-skip would jump across them. + const renderKeepSpans: VideoModeSpan[] = []; const childLayers = await childCompositeLayers({ children: metadataBeforeVideo.children, outputDir: testInfo.outputDir, @@ -4995,6 +5214,35 @@ export const videoMode = (options: VideoModeOptions = {}): VideoModePlugin => { outputPath: compositePath, }); renderInputPath = compositePath; + for (const layer of childLayers) { + renderKeepSpans.push( + { end: layer.enableFromMs + CHILD_OVERLAY_FADE_MS, start: layer.enableFromMs }, + { end: layer.enableToMs, start: layer.closeMs }, + ); + } + // A parent action right after a popup closes (waiting for the + // signed-in state, say) would hold a freeze frame from inside the + // overlay's exit animation — a ghost popup flashing back after it + // disappeared. Shift such highlights past the fade window. + for (const parentHighlight of renderTimeline.highlights) { + for (const layer of childLayers) { + const fadeEndMs = layer.enableToMs + rawVideoInfo.frameDurationMs; + if ( + parentHighlight.start >= layer.closeMs - rawVideoInfo.frameDurationMs && + parentHighlight.start < fadeEndMs + ) { + const shift = fadeEndMs - parentHighlight.start; + parentHighlight.start += shift; + parentHighlight.end += shift; + if (parentHighlight.actionEnd !== undefined) { + parentHighlight.actionEnd += shift; + } + if (parentHighlight.sourceFrameAt !== undefined) { + parentHighlight.sourceFrameAt += shift; + } + } + } + } const projectedChildHighlights = childLayers.flatMap((layer) => translateVideoTimeline({ addressBars: [], @@ -5093,6 +5341,7 @@ export const videoMode = (options: VideoModeOptions = {}): VideoModePlugin => { highlightMode: highlight.mode === "pointer" ? "pointer" : "outline", highlights: renderTimeline.highlights, inputPath: renderInputPath, + keepSpans: renderKeepSpans, outputDir: testInfo.outputDir, outputPath: paths.rendered, sourceRange, From eb34bdce42cfd84fa85001cb9d46f3ca5832c279 Mon Sep 17 00:00:00 2001 From: Misha Kaletsky <15040698+mmkal@users.noreply.github.com> Date: Fri, 14 Aug 2026 12:23:13 +0100 Subject: [PATCH 12/13] Show the focus ring while overlay fill reveals type The reveal base predates the fill, so the field typed with no focus ring, and the ring then flashed in from live footage after the hold. At reveal start the field's ring region from the post-fill screenshot overlays the base, its text immediately covered by the pre-fill screenshot's empty content box, and the reveal bands type over that - ring appears when the cursor lands, letters arrive inside it, and the post-piece footage continues the ring seamlessly. Co-Authored-By: Claude Fable 5 --- src/plugins/video-mode.ts | 61 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 61 insertions(+) diff --git a/src/plugins/video-mode.ts b/src/plugins/video-mode.ts index 04fb781..56b8957 100644 --- a/src/plugins/video-mode.ts +++ b/src/plugins/video-mode.ts @@ -3230,6 +3230,29 @@ const renderedVideoFilter = (options: { 0, Math.min(revealEnd, pointerArrival + preRevealHold), ); + // The base predates the fill, so it shows the field unfocused. The + // post-fill screenshot has the focus ring: overlay the field's ring + // region from it at reveal start, immediately cover its text with the + // pre-fill screenshot's empty content box, and let the reveal bands + // type over that — the ring appears when the cursor lands and the + // letters arrive inside it, continuous with the live footage after. + const ringPaddingPx = 4; + const ringSource = { + height: fillReveal.initialRect.height + 2 * ringPaddingPx, + width: fillReveal.initialRect.width + 2 * ringPaddingPx, + x: fillReveal.initialRect.x - ringPaddingPx, + y: fillReveal.initialRect.y - ringPaddingPx, + }; + const ringLocal = { + height: Math.max(1, Math.min(scaledImage.height, projectLength(ringSource.height))), + width: Math.max(1, Math.min(scaledImage.width, projectLength(ringSource.width))), + x: Math.max(0, projectLength(ringSource.x)), + y: Math.max(0, projectLength(ringSource.y)), + }; + const ringAbsolute = { + x: Math.round(transform.x + ringSource.x * transform.scale), + y: Math.round(transform.y + ringSource.y * transform.scale), + }; const baseLabel = `fillbase${index}`; // A few frames of margin: the child anchor is approximate, and the // safe failure mode is showing slightly earlier (still-empty) footage. @@ -3265,6 +3288,44 @@ const renderedVideoFilter = (options: { ); let composedLabel = baseLabel; + if (preFillInput) { + const ringLabel = `fillring${index}`; + const emptyLabel = `fillempty${index}`; + const revealStartEnable = `enable='gte(t\\,${formatSeconds(revealStart)})'`; + filters.push( + [ + `[${postFillInput.inputIndex}:v]scale=w=${scaledImage.width}:h=${scaledImage.height}`, + `crop=w=${ringLocal.width}:h=${ringLocal.height}:x=${ringLocal.x}:y=${ringLocal.y}`, + `trim=start=0:end=${durationSeconds}`, + `setpts=PTS-STARTPTS[${ringLabel}]`, + ].join(","), + ); + filters.push( + [ + `[${preFillInput.inputIndex}:v]scale=w=${scaledImage.width}:h=${scaledImage.height}`, + `crop=w=${contentLocal.width}:h=${contentLocal.height}:x=${contentLocal.x}:y=${contentLocal.y}`, + `trim=start=0:end=${durationSeconds}`, + `setpts=PTS-STARTPTS[${emptyLabel}]`, + ].join(","), + ); + filters.push( + [ + `[${composedLabel}][${ringLabel}]overlay=x=${ringAbsolute.x}`, + `y=${ringAbsolute.y}`, + revealStartEnable, + `shortest=1[fillringcomposed${index}]`, + ].join(":"), + ); + filters.push( + [ + `[fillringcomposed${index}][${emptyLabel}]overlay=x=${contentAbsolute.x}`, + `y=${contentAbsolute.y}`, + revealStartEnable, + `shortest=1[fillemptycomposed${index}]`, + ].join(":"), + ); + composedLabel = `fillemptycomposed${index}`; + } for (let stepIndex = 0; stepIndex < revealSteps.length; stepIndex += 1) { const step = revealSteps[stepIndex]; const cropLabel = `fillcrop${index}x${stepIndex}`; From e6a79483435370b04f23155f9616ffcfb00ede34 Mon Sep 17 00:00:00 2001 From: Misha Kaletsky <15040698+mmkal@users.noreply.github.com> Date: Fri, 14 Aug 2026 12:34:13 +0100 Subject: [PATCH 13/13] Animate password fill reveals Password inputs render one bullet per character, and the reveal only ever shows the screenshot's dots - so there was no reason to fall back to an instant fill. Measure bullet glyphs instead of the value's graphemes for the reveal stops. The reveal base now rewinds a single frame (a longer rewind could cross the previous fill's completion and wipe its value from the frozen frame), and the pre-fill empty content box covers the field from t=0 so anchor imprecision can't leak early-typed text. Co-Authored-By: Claude Fable 5 --- src/plugins/video-mode.ts | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/src/plugins/video-mode.ts b/src/plugins/video-mode.ts index 56b8957..cee6302 100644 --- a/src/plugins/video-mode.ts +++ b/src/plugins/video-mode.ts @@ -1466,8 +1466,7 @@ const recordFillReveal = async (options: { value.length === 0 || value.length > captureOptions.maxCharacters || style.direction === "rtl" || - !["left", "start"].includes(style.textAlign) || - (element instanceof HTMLInputElement && element.type === "password") + !["left", "start"].includes(style.textAlign) ) { return { ...geometry, kind: "fallback" as const }; } @@ -1574,10 +1573,16 @@ const recordFillReveal = async (options: { return { ...geometry, kind: "fallback" as const }; } context.font = style.font; - const graphemes = Array.from( - new Intl.Segmenter(undefined, { granularity: "grapheme" }).segment(value), - ({ segment }) => segment, - ); + // A password input renders one bullet per character, so measure those + // glyphs — the reveal only ever shows the screenshot's dots, never the + // value. + const masked = element instanceof HTMLInputElement && element.type === "password"; + const graphemes = masked + ? Array.from(value, () => "•") + : Array.from( + new Intl.Segmenter(undefined, { granularity: "grapheme" }).segment(value), + ({ segment }) => segment, + ); const letterSpacing = pixels(style.letterSpacing); const textIndent = pixels(style.textIndent); const revealStops = graphemes.map((_, index) => { @@ -3254,9 +3259,11 @@ const renderedVideoFilter = (options: { y: Math.round(transform.y + ringSource.y * transform.scale), }; const baseLabel = `fillbase${index}`; - // A few frames of margin: the child anchor is approximate, and the - // safe failure mode is showing slightly earlier (still-empty) footage. - const freezeStart = Math.max(0, piece.start - 3 * options.video.frameDurationMs); + // One frame back only: rewinding further can cross the previous fill's + // completion (wiping its value from the frozen base). The content box + // is covered with the pre-fill empty state from t=0 below, so anchor + // imprecision inside this field can't leak early-typed text either. + const freezeStart = Math.max(0, piece.start - options.video.frameDurationMs); filters.push( [ `[0:v]trim=start=${formatSeconds(freezeStart)}:end=${formatSeconds( @@ -3320,7 +3327,6 @@ const renderedVideoFilter = (options: { [ `[fillringcomposed${index}][${emptyLabel}]overlay=x=${contentAbsolute.x}`, `y=${contentAbsolute.y}`, - revealStartEnable, `shortest=1[fillemptycomposed${index}]`, ].join(":"), );