E2E: automatic teardown polish

[phyllis-sy-wu]

WHY are these changes introduced?

Improves reliability and speed of per-test teardown.

WHAT is this pull request doing?

Refactors teardown so phases have consistent structure, fixes/optimizes browser verification, and upgrades CLI helpers.

Now a complete store + app teardown looks like this:

Phase	Steps	Code/Logic	Retry
1	1. Uninstall app @admin/…/settings/apps	uninstallAppFromStore: click ⋯ → Uninstall → Confirm	Up to 3x, outer in teardown Phase 1
1	2. Confirm uninstall (reload admin/…/settings/apps)	uninstallAppFromStore: reload /apps, check app span gone → success	If fails, retry ↑↑↑
2	3. Delete store (gate: step 2 passed) @admin/…/settings/plan/cancel	Gate verifies isStoreAppsEmpty → safeToDelete = true → deleteStore	Up to 3x, outer in teardown Phase 2
2	4. Confirm store delete (reload admin/…/settings/plan/cancel)	Inside new deleteStore: reload /cancel, check if it gets redirect to admin/…/access_account → success	If fails, retry ↑↑↑
3	5. Delete app (gate: step 2 passed) @dashboard/../app/…/settings	Gate verifies uninstalled = true → proceed to delete app	Up to 3x, outer in teardown Phase 3
3	6. Confirm app delete (reload dashboard/../app/…/settings)	deleteAppFromDevDashboard: reload /settings, check if 404 → success	If fails, retry ↑↑↑

---

1. Faster app deletion verification (setup/app.ts)

Before: waitForURL(url !== urlBefore) burned up to 60s for the page to navigate away (usually takes 10-20 sec).

delete-app-automatic-redirect.mov

After: reloads {appUrl}/settings — 404 = deleted, other = fail/retry. Completes in seconds.

delete-app-force-refresh.mov

Also: early 404 short-circuit on initial load, scrollIntoViewIfNeeded for below-fold button, STILL_HAS_INSTALLS fail-fast (throw instead of spinning on disabled Delete button), optional DELETE confirmation input.

---

2. Faster store deletion verification (setup/store.ts)

Before: navigated to /settings/plan, clicked shadow DOM button, verified via waitForURL(/access_account/) — burned 60s on silent failure.
After: navigates to /settings/plan/cancel (auto-opens modal, no shadow DOM click needed), verifies via networkidle + reload → access_account redirect check.

Modal scoped by title (.Polaris-Modal-Dialog__Modal:has-text("Review before deleting store")) to avoid matching stale modals.

---

3. Consistent uninstall verification (setup/store.ts)

Before: checked stale DOM first, only reloaded as fallback.
After: always reloads → dismisses Dev Console → checks. Matches cleanup-stores pattern.

---

4. Teardown orchestrator cleanup (setup/teardown.ts)

Each helper now owns its own verification; teardown becomes the orchestrator:

• Phase 2 (delete store) skipped if Phase 1 failed
• Phase 3 (delete app) skipped if Phase 1 failed — catches STILL_HAS_INSTALLS and fails fast (cleanup-apps reaps orphans)
• Pre-flight gate: verifies isStoreAppsEmpty before attempting store delete

---

5. CLI helper upgrades (setup/app.ts)

• configLink rewritten — from exec with --client-id to interactive PTY spawn that answers org/app prompts (needed for tests that create apps from scratch)
• createApp auto-defaults --flavor javascript for reactRouter/remix (prevents PTY hang)
• versionsList accepts optional config (needed for multi-config tests)
• Consistent arg ordering — --path always last across all helpers

---

6. Minor fixes

• E2E_SKIP_CLEANUP → E2E_SKIP_TEARDOWN across 7 test specs (matches what it actually controls)
• refreshIfPageError: Internal Server Error → 500: Internal Server Error (tighter match)

---

How to test your changes?

Run an E2E test and confirm teardown completes cleanly:

E2E_HEADED=1 DEBUG=1 pnpm --filter e2e exec playwright test app-dev-server

Expand for complete log

cli % E2E_HEADED=1 DEBUG=1 pnpm --filter e2e exec playwright test app-dev-server
[e2e][auth] global setup starting

To run this command, log in to Shopify.
User verification code: QGWN-RFHK
👉 Open this link to start the auth process: https://accounts.shopify.com/activate-with-code?device_code%5Buser_code%5D=QGWN-RFHK
✔ Logged in.
✔ Current account: genghis-khan-identity-1-2025-03-31@shopify.com.
[e2e][auth] browser sessions established for admin + dev dashboard
[e2e][auth] global setup done, config at /Users/psyw/src/github.com/Shopify/cli/.e2e-tmp/global-auth/XDG_CONFIG_HOME

Running 1 test using 1 worker

     1 …ev starts, shows ready message, and quits with q

[e2e][w0] ----- TEST: dev starts, shows ready message, and quits with q ----- 
[e2e][auth] copying session from global setup

[e2e][w0] ----- Setup: store e2e-w0-1776933243084 ----- 
[e2e][w0][browser] store creating
[e2e][w0][browser] store form not loaded (attempt 1/3), url=https://admin.shopify.com/store-create/organization/161686155
[e2e][w0][browser] store plan=SHOPIFY_PLUS_APP_DEVELOPMENT
[e2e][w0][browser] store created e2e-w0-1776933243084.myshopify.com
[e2e][w0][cli] exec: node /Users/psyw/src/github.com/Shopify/cli/packages/create-app/bin/run.js
[e2e][w0][cli] app init --template none --name E2E-dev-1776933268386 --package-manager pnpm --local --organization-id 161686155 --path /Users/psyw/src/github.com/Shopify/cli/.e2e-tmp/e2e-trWnwm/app-5C8wvK
╭─ info ───────────────────────────────────────────────────────────────────────╮
│                                                                              │
│  Initializing project with `pnpm`                                            │
│  Use the `--package-manager` flag to select a different package manager.     │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯


╭─ success ────────────────────────────────────────────────────────────────────╮
│                                                                              │
│  e2-e-dev-1776933268386 is ready for you to build!                           │
│                                                                              │
│  Next steps                                                                  │
│    • Run `cd e2-e-dev-1776933268386`                                         │
│    • For extensions, run `shopify app generate extension`                    │
│    • To see your app, run `shopify app dev`                                  │
│                                                                              │
│  Reference                                                                   │
│    • Shopify docs [1]                                                        │
│    • Shopify Dev MCP, [2] connect your AI assistant to development           │
│      resources                                                               │
│    • For an overview of commands, run `shopify app --help`                   │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
[1] https://shopify.dev
[2] https://shopify.dev/docs/apps/build/devmcp

[e2e][w0][cli] spawn: node /Users/psyw/src/github.com/Shopify/cli/packages/cli/bin/run.js
[e2e][w0][cli] app dev --path /Users/psyw/src/github.com/Shopify/cli/.e2e-tmp/e2e-trWnwm/app-5C8wvK/e2-e-dev-1776933268386
╭─ info ───────────────────────────────────────────────────────────────────────╮
│                                                                              │
│  Using shopify.app.toml for default values:                                  │
│                                                                              │
│    • Org:             core-build-develop-admin-web-e2e                       │
│    • App:             E2E-dev-1776933268386                                  │
│    • Dev store:       e2e-w0-1776933243084.myshopify.com                     │
│    • Update URLs:     Yes                                                    │
│                                                                              │
│   You can pass `--reset` to your command to reset your app configuration.    │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯

04:34:49 │               app-preview │ Preparing dev preview on e2e-w0-1776933243084.myshopify.com
04:34:49 │                  graphiql │ GraphiQL server started on port 3457
04:34:49 │                     proxy │ Proxy server started on port 54238
04:34:51 │               app-preview │ ✅ Ready, watching for changes in your app 
04:34:51 │                app_access │ │ App has been installed
04:34:51 │                  app_home │ └ Using URL: https://shopify.dev/apps/default-app-home

╭─ info ───────────────────────────────────────────────────────────────────────╮
│                                                                              │
│  A preview of your development changes is still available on                 │
│  e2e-w0-1776933243084.myshopify.com.                                         │
│                                                                              │
│  Run `shopify app dev clean` to restore the latest released version of your  │
│   app.                                                                       │
│                                                                              │
│  Learn more about dev previews [1]                                           │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
[1] https://shopify.dev/beta/developer-dashboard/shopify-app-dev


[e2e][w0] ----- Teardown: store e2e-w0-1776933243084.myshopify.com ----- 
[e2e][w0][browser] uninstalling app from store
[e2e][w0][browser] app uninstalled
[e2e][w0][browser] deleting store
[e2e][w0][browser] store deleted

[e2e][w0] ----- Teardown: app E2E-dev-1776933268386 ----- 
[e2e][w0][browser] deleting app
[e2e][w0][browser] app found, deleting
[e2e][w0][browser] app deleted
  ✓  1 …ts, shows ready message, and quits with q (2.8m)

  1 passed (3.5m)

Checklist

• I've considered possible cross-platform impacts (Mac, Linux, Windows)
• I've considered possible documentation changes
• I've considered analytics changes to measure impact
• The change is user-facing, so I've added a changelog entry with pnpm changeset add

[phyllis-sy-wu]

• E2E - QA: init, deploy, versions list, config link, 2nd deploy #7367
• E2E: utility - cleanup all #7357
• E2E: utility - cleanup apps #7356
• E2E: utility - cleanup stores #7355
• E2E: automatic teardown polish #7358 👈 (View in Graphite)
• E2E: improve CLI log format #7364
• E2E: remove dead code #7360
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[byrichardpowell]

Nice.

Approving so you don't need a second review, but please check the comments and address anything that is important.

[Copilot]

Pull request overview

Improves E2E test reliability and speed by polishing per-test teardown, hardening browser automation checks, and upgrading CLI PTY helpers used in E2E flows.

Changes:

• Refactors teardown orchestration into gated phases with consistent retry structure and faster delete verification.
• Adds browser automation utilities (isVisibleWithin, main-frame status tracking) and updates store/app delete/uninstall flows to use them.
• Enhances CLI PTY waitForOutput to support cancellation and updates tests to use E2E_SKIP_TEARDOWN.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
packages/e2e/tests/toml-config.spec.ts	Rename skip env var to E2E_SKIP_TEARDOWN and update cleanup comment.
packages/e2e/tests/toml-config-invalid.spec.ts	Rename skip env var to E2E_SKIP_TEARDOWN and update cleanup comment.
packages/e2e/tests/multi-config-dev.spec.ts	Rename skip env var to E2E_SKIP_TEARDOWN and update cleanup comment.
packages/e2e/tests/dev-hot-reload.spec.ts	Rename skip env var to E2E_SKIP_TEARDOWN and update cleanup comment.
packages/e2e/tests/app-scaffold.spec.ts	Rename skip env var to E2E_SKIP_TEARDOWN and update cleanup comment.
packages/e2e/tests/app-dev-server.spec.ts	Rename skip env var to E2E_SKIP_TEARDOWN and update cleanup comment.
packages/e2e/tests/app-deploy.spec.ts	Rename skip env var to E2E_SKIP_TEARDOWN and update cleanup comment.
packages/e2e/setup/teardown.ts	Reworks teardown phase gating/retries and adds fail-fast handling for “still has installs”.
packages/e2e/setup/store.ts	Refines store create/uninstall/delete flows; uses isVisibleWithin and new delete verification.
packages/e2e/setup/global-auth.ts	Uses isVisibleWithin for account-picker handling during global auth.
packages/e2e/setup/cli.ts	Extends PTY waitForOutput API to support abort/cancellation and safer waiter iteration.
packages/e2e/setup/browser.ts	Adds main-frame HTTP status tracking, isVisibleWithin, and status-based error refresh.
packages/e2e/setup/app.ts	Updates CLI helper arg ordering, adds interactive configLink, and speeds app deletion verification via 404 status.
packages/e2e/helpers/browser-login.ts	Uses isVisibleWithin for login flow branching and submit confirmation detection.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

waitForOutput registers an AbortSignal abort listener but never removes it when the waiter resolves/rejects normally. If a long-lived signal is passed and never aborted, this retains closures (timer/output buffer refs) unnecessarily. Consider storing the abort handler function and calling signal.removeEventListener('abort', handler) inside both resolve and reject paths (and after timeout) to ensure cleanup even when the signal is never aborted.

[Copilot]

Typo in comment: "app is not on listed on the page" → "app is not listed on the page".

Suggested change

	// Success → app is not on listed on the page.
	// Success → app is not listed on the page.

[binks-code-reviewer]

Do not stop teardown when uninstall verification flakes

teardownAll now returns before Phase 3 when Phase 1 does not verify uninstall, but uninstallAppFromStore can return false because of transient admin UI failures rather than because the app is definitely still installed. One concrete path is when the actions menu opens but the Uninstall item never becomes visible in packages/e2e/setup/store.ts:161-163. That kind of eventual-consistency/UI flake is exactly what the rest of teardown retries around, so treating it as terminal causes store-backed specs to skip app deletion and leak dev-dashboard apps into the shared org. The fallback mentioned in the log message also appears to be unavailable, because there is no test:e2e-cleanup-apps script in the repository.

_{React with 👍/👎 — all feedback helps improve the agent.}