E2E: utility - cleanup stores

[phyllis-sy-wu]

WHY are these changes introduced?

E2E tests create dev stores that can accumulate when tests fail mid-run, CI times out, or teardown fails. This script automates bulk-clean for leftover stores.

WHAT is this pull request doing?

cleanup-stores.ts

Standalone cleanup script that finds leftover E2E dev stores, uninstalls their apps, and deletes them via browser automation.

pnpm --filter e2e exec tsx scripts/cleanup-stores.ts              # Full: uninstall apps + delete stores
pnpm --filter e2e exec tsx scripts/cleanup-stores.ts --list        # List stores with app counts
pnpm --filter e2e exec tsx scripts/cleanup-stores.ts --delete      # Delete only stores with 0 apps installed
pnpm --filter e2e exec tsx scripts/cleanup-stores.ts --headed      # Show browser window
pnpm --filter e2e exec tsx scripts/cleanup-stores.ts --pattern X   # Match stores containing "X" (default: "e2e-w")

Logic

Discovery phase:

1. Log in via completeLogin helper
2. Navigate to dev.shopify.com/dashboard/{orgId}/stores
3. Recover from transient 500/502 via refreshIfPageError
4. Wait for #stores-tbody tr to render, then scroll-to-bottom in a loop until row count stabilizes (lazy-loaded table)
5. Extract store FQDNs via regex on full page HTML — catches slugs in hrefs/attributes, not just visible text
6. Filter by name pattern (default: e2e-w), deduplicate via Set

--list mode (per store):

1. Navigate to admin.shopify.com/store/{slug}/settings/apps
2. Wait for page readiness via Promise.race — either empty state or first app menu button
3. Count menu buttons across all pages (paginate via button#nextURL)
4. Print store name + app count

Default mode (per store):

1. Navigate to admin.shopify.com/store/{slug}/settings/apps
2. Dismiss Dev Console if visible
3. Wait for page readiness via Promise.race — either empty state or first app menu button
4. Check for "Add apps to your store" empty state — only trust positive confirmation as proof of zero apps
5. If apps present: uninstall all apps (see uninstall logic below)
6. After uninstall: verify empty state is now visible — if not, skip store deletion and log warning
7. If safe to delete: call deleteStore() (from setup/store.ts) with up to 3 retries

--delete mode: same as default but skips stores that have apps installed (step 4 → skip if not empty).

Uninstall logic (uninstallAllAppsFromStore):

1. Check isStoreAppsEmpty — if true, done (primary termination)
2. Find button[aria-label="More actions"] at position consecutiveSkips
3. Extract app name from div[role="listitem"] → a span
4. Click menu button → click "Uninstall" → click confirm
5. If "Uninstall" option not in menu: press Escape, increment consecutiveSkips, try next button
6. If confirm never appears: increment consecutiveSkips (prevents infinite loop)
7. Reload page after each uninstall to refresh the list
8. When no more menu buttons visible: check for button#nextURL pagination → continue on next page

Features:

• Empty state safety: never deletes a store unless "Add apps to your store" is positively confirmed
• Page readiness wait via Promise.race before checking empty state (avoids false negatives from slow renders)
• Scroll-based store discovery handles lazy-loaded tables (stabilizes after 3 idle rounds)
• Per-store timing in output
• Exports cleanupStores() for use from other scripts

How is this different from per-test teardown?

• Per-test teardown (setup/teardown.ts) — knows the specific app name and store FQDN, uses direct URLs, no discovery. Runs automatically in test finally blocks.
• cleanup-stores.ts (bulk, manual) — discovers all matching stores via scroll-based lazy loading. Safety net for orphaned stores from failed/interrupted test runs.

How to test your changes?

1. Create leftover stores by skipping cleanup:

   E2E_SKIP_TEARDOWN=1 DEBUG=1 pnpm --filter e2e exec playwright test app

2. List them:

   pnpm --filter e2e exec tsx scripts/cleanup-stores.ts --list

3. Clean up:

   pnpm --filter e2e exec tsx scripts/cleanup-stores.ts --headed

Example: pnpm --filter e2e exec tsx scripts/cleanup-stores.ts --headed

cleanup-stores.mov

Expand for complete log

cli % pnpm --filter e2e exec tsx scripts/cleanup-stores.ts --headed

[cleanup-stores] Mode:    Uninstall apps + Delete stores
[cleanup-stores] Org:     161686155
[cleanup-stores] Pattern: "e2e-w"

[cleanup-stores] Logging in...
[cleanup-stores] Logged in successfully.
[cleanup-stores] Navigating to stores page...
[cleanup-stores]   ...loaded 20 stores
[cleanup-stores]   ...loaded 30 stores
[cleanup-stores]   ...loaded 40 stores
[cleanup-stores]   ...loaded 43 stores
[cleanup-stores] Found 41 store(s) matching pattern "e2e-w"

[cleanup-stores] [1/41] e2e-w8-1776939053026
  No apps installed (empty state confirmed)
  Deleting store...
  Deleted
  (29.2s)

[cleanup-stores] [2/41] e2e-w5-1776939053547
  No apps installed (empty state confirmed)
  Deleting store...
  Deleted
  (30.5s)

...

Post-release steps

Checklist

• I've considered possible cross-platform impacts (Mac, Linux, Windows)
• I've considered possible documentation changes
• I've considered analytics changes to measure impact
• The change is user-facing — I've identified the correct bump type (patch for bug fixes · minor for new features · major for breaking changes) and added a changeset with pnpm changeset add

[phyllis-sy-wu]

• E2E - QA: init, deploy, versions list, config link, 2nd deploy #7367
• E2E: utility - cleanup all #7357
• E2E: utility - cleanup apps #7356
• E2E: utility - cleanup stores #7355 👈 (View in Graphite)
• E2E: automatic teardown polish #7358
• E2E: improve CLI log format #7364
• E2E: remove dead code #7360
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[Copilot]

Pull request overview

Adds a standalone E2E utility script to bulk-discover and clean up leftover dev stores created by failing/aborted E2E runs, using Playwright browser automation against the Dev Dashboard + Store Admin.

Changes:

• Introduces packages/e2e/scripts/cleanup-stores.ts with --list, --delete, and default “full” cleanup modes.
• Implements store discovery via Dev Dashboard pagination and regex extraction of store FQDNs from page HTML.
• Automates app uninstall (with pagination) and store deletion flows with retries and safety checks (empty-state confirmation).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

isDirectRun compares process.argv[1] to fileURLToPath(import.meta.url) using strict string equality. When running via npx tsx packages/e2e/scripts/cleanup-stores.ts, process.argv[1] is typically a relative path, so this check can be false and main() won’t run at all. Consider normalizing both sides (e.g., path.resolve(process.argv[1])) or comparing URLs via pathToFileURL(path.resolve(process.argv[1])).href === import.meta.url.

[Copilot]

This script defines a local deleteStore() helper that largely overlaps with the existing deleteStore browser helper in packages/e2e/setup/store.ts. Having two different implementations with the same name risks divergence and makes future maintenance/debugging harder. Consider reusing the shared helper (and extending it if needed) or renaming this one to avoid confusion (e.g., deleteStoreViaPlanPage).

[binks-code-reviewer]

locator.isVisible({timeout}) is treated as a wait, so slow UI is misclassified as absent

The script uses locator.isVisible({timeout: ...}) as though it polls until visibility, but modern Playwright does not wait there. This repository already added isVisibleWithin() for that exact reason.

That pattern appears in multiple places around account selection, pagination, app menus, uninstall options, and confirmation buttons. Because these checks return the current state immediately, slightly slow admin pages can be treated as if the controls do not exist.

Reachable consequences include missing the account picker before store discovery, stopping pagination early in countInstalledApps(), and skipping app uninstall actions in uninstallAllAppsFromStore(). The impact is incomplete or inaccurate cleanup rather than a hard crash, which still makes --list unreliable and can require repeated cleanup runs.

_{React with 👍/👎 — all feedback helps improve the agent.}

[binks-code-reviewer]

Swallowing both branches of the apps-page race can falsely treat an unloaded page as empty

Both "wait for the apps page to settle" blocks wrap each branch of a Promise.race() in .catch(() => {}), so the code continues even when neither the empty state nor the first app row ever becomes visible.

After that, the script immediately calls isStoreAppsEmpty(). In packages/e2e/setup/store.ts, that helper falls back to treating the absence of More actions buttons as proof the store has no apps. On a page that never finished loading, redirected to login, or is showing an error/loading shell, that fallback becomes a false positive.

As a result, --list can report 0 apps for a store whose apps page never loaded, --delete can treat a non-empty store as safe to delete, and full cleanup can skip uninstall work entirely because the page was mistaken for an empty state. Shopify may still block deletion later, but this breaks the script's safety assumption that only a positively confirmed empty state proves zero apps.

_{React with 👍/👎 — all feedback helps improve the agent.}