E2E: utility - cleanup apps

[phyllis-sy-wu]

WHY are these changes introduced?

E2E tests create apps that can accumulate when tests fail mid-run, CI times out, or teardown fails. This script automates bulk-clean for leftover apps.

WHAT is this pull request doing?

cleanup-apps.ts

Standalone cleanup script that finds leftover E2E test apps on the Dev Dashboard, uninstalls them from all stores, and deletes them.

pnpm --filter e2e exec tsx scripts/cleanup-apps.ts              # Full cleanup: uninstall + delete
pnpm --filter e2e exec tsx scripts/cleanup-apps.ts --list        # List matching apps with install counts
pnpm --filter e2e exec tsx scripts/cleanup-apps.ts --uninstall   # Uninstall from all stores only (no delete)
pnpm --filter e2e exec tsx scripts/cleanup-apps.ts --delete      # Delete only apps with 0 installs
pnpm --filter e2e exec tsx scripts/cleanup-apps.ts --headed      # Show browser window
pnpm --filter e2e exec tsx scripts/cleanup-apps.ts --pattern X   # Match apps containing "X" (default: "E2E-")

Logic

The per-app mechanics delegate to the shared setup/ building blocks (uninstallAppFromStore, deleteAppFromDevDashboard, refreshIfPageError) — same primitives used by per-test teardown. The script adds bulk discovery, pagination, and per-app retry on top.

Discovery phase:

1. Log in via completeLogin helper
2. Navigate to Dev Dashboard, then loop refreshIfPageError up to 3× as extra resilience on 500/502 (hard-fails after 3 attempts)
3. Find app cards via a[href*=\"/apps/\"] selectors
4. Extract app name from card text (split on install count pattern), install count via regex, and URL from href
5. Filter by name pattern (default: E2E-)
6. Paginate via a[href*=\"next_cursor\"]; refreshIfPageError runs at the top of every iteration so error-page returns throw instead of silently yielding 0 apps

Uninstall (per app):

1. Navigate to {appUrl}/installs
2. Collect store slugs via FQDN regex on full page HTML (page.content())
3. If no FQDNs found on this page: fall back to table row text (store name = slug)
4. Paginate via button#nextURL and repeat 2–3 on each page
5. For each store slug: call uninstallAppFromStore(page, slug, appName) — the shared setup helper navigates to the store's /settings/apps, clicks the ⋯ menu → Uninstall → confirm, then reloads and verifies the app is gone. Returns true if gone (or already absent), false if still listed.
6. If any store returns false or throws: mark allUninstalled = false and log the store slug + error
7. Final verification: navigate back to {appUrl}/installs, scan every row across all pages — any non-empty row → return false
8. Return allUninstalled

Delete (per app):

1. Call deleteAppFromDevDashboard(page, appUrl) — the shared setup helper navigates to {appUrl}/settings, clicks Delete app (scroll + reload-once fallback for the button's propagation lag), types "DELETE" if the confirm input is present, clicks confirm, then reloads and returns true on 404 (deleted) or false otherwise. Throws STILL_HAS_INSTALLS if the Delete button stays disabled after reload (fail-fast signal — retries won't help).
2. Treat false as "deletion could not be verified" and retry via the outer loop.

Per-app retry wrapper:

1. Each app gets up to 3 attempts
2. On failure: log the error ((N/3) failed: ...), wait, re-navigate to dashboard, retry the full uninstall + delete flow
3. STILL_HAS_INSTALLS short-circuits the retry loop → record as skipped
4. Per-app elapsed time printed ((Xs)); summary printed at end: X succeeded, Y skipped, Z failed (Xs total)

--list mode: runs discovery only, prints app names and install counts.

--uninstall mode: runs uninstall only, skips apps with 0 installs.

--delete mode: runs delete only, skips apps with installs > 0.

Features:

• Reuses setup/ building blocks — single source of truth for per-store uninstall and per-app delete semantics (shared with per-test teardown)
• Dashboard error handling via shared refreshIfPageError helper — hard-fails after 3 consecutive 500/502 responses
• Store slug extraction via full HTML regex with per-page table text fallback
• Paginated final installs verification across all pages
• Error page differentiation: 404 = already deleted (success), 500/502 = server error (retry), disabled Delete button = STILL_HAS_INSTALLS (fail-fast skip)
• Exports cleanupAllApps() for use as a Playwright globalTeardown or from other scripts

How is this different from per-test teardown?

• Per-test teardown (setup/teardown.ts) — knows the specific app name and store FQDN, uses direct URLs, no discovery. Runs automatically in test `finally` blocks.
• `cleanup-apps.ts` (bulk, manual) — discovers all matching apps via dashboard pagination, discovers stores via installs page. Safety net for orphaned apps from failed/interrupted test runs.
• Both share the same underlying primitives in `setup/` (`uninstallAppFromStore`, `deleteAppFromDevDashboard`, `refreshIfPageError`), so browser automation semantics stay consistent.

How to test your changes?

1. Create leftover apps by skipping cleanup:

   E2E_SKIP_TEARDOWN=1 DEBUG=1 pnpm --filter e2e exec playwright test app

2. List them:

   pnpm --filter e2e exec tsx scripts/cleanup-apps.ts --list

3. Clean up:

   pnpm --filter e2e exec tsx scripts/cleanup-apps.ts --headed

Example

pnpm --filter e2e exec tsx scripts/cleanup-apps.ts --headed

cleanup-apps.mov

Expand for complete log

cli % pnpm --filter e2e exec tsx scripts/cleanup-apps.ts --headed

[cleanup-apps] Mode:    Uninstall + Delete
[cleanup-apps] Org:     161686155
[cleanup-apps] Pattern: "E2E-"

[cleanup-apps] Logging in...
[cleanup-apps] Logged in successfully.
[cleanup-apps] Navigating to dashboard...
[cleanup-apps] Dashboard loaded.
[cleanup-apps] Finding matching apps...
[cleanup-apps]   ...loaded 20 apps
[cleanup-apps]   ...loaded 40 apps
[cleanup-apps]   ...loaded 60 apps
[cleanup-apps]   ...loaded 73 apps
[cleanup-apps] Found 41 app(s) matching pattern "E2E-"

  1. E2E-dev-basic-1776959380611 (1 install)
  2. E2E-dev-basic-1776957041976 (0 installs)
  3. E2E-dev-basic-1776956913189 (0 installs)
  4. E2E-hot-create-1776956797812 (1 install)
  5. E2E-hot-reload-1776956796321 (1 install)
  6. E2E-dev-1776956791662 (1 install)
  7. E2E-multi-cfg-1776956790856 (1 install)
  8. E2E-dev-basic-1776955811113 (0 installs)
  9. E2E-hot-create-1776947144539 (1 install)
  10. E2E-dev-1776947140553 (1 install)
  11. E2E-hot-delete-1776947140856 (1 install)
  12. E2E-multi-cfg-1776947136959 (1 install)
  13. E2E-hot-reload-1776947140231 (1 install)
  14. E2E-multi-cfg-1776947138343 (0 installs)
  15. E2E-dev-1776947134834 (0 installs)
  16. E2E-hot-create-1776947136636 (0 installs)
  17. E2E-toml-dev-1776947144639 (1 install)
  18. E2E-multi-cfg-1776946913357 (0 installs)
  19. E2E-hot-create-1776943244407 (1 install)
  20. E2E-hot-reload-1776943242934 (1 install)
  21. E2E-multi-cfg-1776943243151 (0 installs)
  22. E2E-dev-1776942633118 (0 installs)
  23. E2E-multi-cfg-1776942630633 (0 installs)
  24. E2E-hot-create-1776942600305 (0 installs)
  25. E2E-hot-reload-1776942598644 (0 installs)
  26. E2E-dev-1776942598380 (1 install)
  27. E2E-hot-reload-1776942554353 (0 installs)
  28. E2E-dev-1776942552276 (1 install)
  29. E2E-hot-delete-1776942549780 (1 install)
  30. E2E-toml-deploy-1776942540447 (0 installs)
  31. E2E-hot-create-1776942343056 (0 installs)
  32. E2E-hot-reload-1776942340258 (1 install)
  33. E2E-multi-cfg-1776942333644 (1 install)
  34. E2E-scaffold-1776942269970 (0 installs)
  35. E2E-hot-reload-1776942269254 (0 installs)
  36. E2E-hot-delete-1776942264353 (1 install)
  37. E2E-dev-1776942284449 (0 installs)
  38. E2E-deploy-1776942269971 (0 installs)
  39. E2E-toml-dev-1776942268415 (0 installs)
  40. E2E-toml-deploy-1776942247534 (0 installs)
  41. E2E-ext-only-1776942269968 (0 installs)

[cleanup-apps] [1/41] E2E-dev-basic-1776959380611
  Uninstalling...
  Uninstalled
  Deleting...
  Deleted
  (15.0s)

[cleanup-apps] [2/41] E2E-dev-basic-1776957041976
  Not installed
  Deleting...
  Deleted
  (11.4s)

[cleanup-apps] [3/41] E2E-dev-basic-1776956913189
  Not installed
  Deleting...
  Deleted
  (11.3s)

...

Post-release steps

Checklist

• I've considered possible cross-platform impacts (Mac, Linux, Windows)
• I've considered possible documentation changes
• I've considered analytics changes to measure impact
• The change is user-facing — I've identified the correct bump type (`patch` for bug fixes · `minor` for new features · `major` for breaking changes) and added a changeset with `pnpm changeset add`

[phyllis-sy-wu]

• E2E - QA: init, deploy, versions list, config link, 2nd deploy #7367
• E2E: utility - cleanup all #7357
• E2E: utility - cleanup apps #7356 👈 (View in Graphite)
• E2E: utility - cleanup stores #7355
• E2E: automatic teardown polish #7358
• E2E: improve CLI log format #7364
• E2E: remove dead code #7360
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[binks-code-reviewer]

Pagination uses locator.isVisible({timeout}), which does not wait and can skip later pages

findAppsOnDashboard() assumes nextLink.isVisible({ timeout: ... }) waits for the pagination control to appear, but this codebase already documents that modern Playwright does not wait in locator.isVisible(). The script only does goto(..., { waitUntil: 'domcontentloaded' }) plus a fixed sleep before checking pagination, so if the dashboard renders the next link slightly later, discovery will stop on the current page and miss later apps. The same pattern is also used for button#nextURL in the installs-page flow, so later installs pages can be skipped too. This is consistent with existing project guidance: setup/browser.ts explicitly says to use isVisibleWithin(), and setup/app.ts already uses that helper for the same dashboard pagination case. The result is silent partial cleanup: operators can run the tool and never scan all apps or installs, leaving orphaned E2E apps in the dev org.

_{React with 👍/👎 — all feedback helps improve the agent.}