fix(test): Stream simulator test execution progress#389
Merged
cameroncooke merged 6 commits intomainfrom May 2, 2026
Merged
Conversation
Emit a test-running stage when two-phase simulator execution moves from build-for-testing to test-without-building so the CLI does not leave stale build status visible while UI tests are running. Keep static discovery observational, parse destination-suffixed XCTest result lines, and handle multiline Swift Testing arguments so discovery and final counts stay project-agnostic across Calculator and Weather. Fixes #384 Co-Authored-By: OpenAI Codex <noreply@openai.com>
commit: |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed:
parseXcodebuildSwiftTestingLineis now unreachable dead code- Removed the unreachable parseXcodebuildSwiftTestingLine function and its call site since parseTestCaseLine now matches all lines it would have matched.
Preview (3b14b40265)
diff --git a/CHANGELOG.md b/CHANGELOG.md
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,9 +4,12 @@
### Fixed
+- Fixed simulator test JSONL accuracy by keeping preflight discovery observational, preserving only user-supplied test selectors, discovering multiline parameterized Swift Testing tests, and parsing destination-suffixed xcodebuild test result lines.
- Removed stale physical-device log session status and shutdown cleanup for deprecated standalone device log capture, and corrected the device build-and-run tool description.
+- Fixed mixed Swift Testing and XCTest summaries so simulator test text output no longer overcounts parameterized Swift Testing results or issue lines.
- Fixed CLI test summaries showing false-positive compiler errors from xcodebuild NSError dump lines, and added compiler-error snapshot coverage for simulator, device, and macOS build-style flows ([#383](https://github.com/getsentry/XcodeBuildMCP/issues/383)).
- Fixed simulator OSLog helper cleanup so server and daemon startup reconcile same-workspace orphaned log streams without stopping helpers owned by live sessions in other workspaces ([#382](https://github.com/getsentry/XcodeBuildMCP/issues/382)).
+- Fixed Weather example test discovery and made CLI test progress visible while tests are running instead of leaving the last build phase displayed.
## [2.5.0-beta.1]
diff --git a/example_projects/Weather/Weather.xcodeproj/xcshareddata/xcschemes/Weather.xcscheme b/example_projects/Weather/Weather.xcodeproj/xcshareddata/xcschemes/Weather.xcscheme
new file mode 100644
--- /dev/null
+++ b/example_projects/Weather/Weather.xcodeproj/xcshareddata/xcschemes/Weather.xcscheme
@@ -1,0 +1,101 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<Scheme
+ LastUpgradeVersion = "1640"
+ version = "1.7">
+ <BuildAction
+ parallelizeBuildables = "YES"
+ buildImplicitDependencies = "YES"
+ buildArchitectures = "Automatic">
+ <BuildActionEntries>
+ <BuildActionEntry
+ buildForTesting = "YES"
+ buildForRunning = "YES"
+ buildForProfiling = "YES"
+ buildForArchiving = "YES"
+ buildForAnalyzing = "YES">
+ <BuildableReference
+ BuildableIdentifier = "primary"
+ BlueprintIdentifier = "8B9291402FA3FCC300B2E371"
+ BuildableName = "Weather.app"
+ BlueprintName = "Weather"
+ ReferencedContainer = "container:Weather.xcodeproj">
+ </BuildableReference>
+ </BuildActionEntry>
+ </BuildActionEntries>
+ </BuildAction>
+ <TestAction
+ buildConfiguration = "Debug"
+ selectedDebuggerIdentifier = "Xcode.DebuggerFoundation.Debugger.LLDB"
+ selectedLauncherIdentifier = "Xcode.DebuggerFoundation.Launcher.LLDB"
+ shouldUseLaunchSchemeArgsEnv = "YES">
+ <Testables>
+ <TestableReference
+ skipped = "NO"
+ parallelizable = "YES">
+ <BuildableReference
+ BuildableIdentifier = "primary"
+ BlueprintIdentifier = "8B92914D2FA3FCC400B2E371"
+ BuildableName = "WeatherTests.xctest"
+ BlueprintName = "WeatherTests"
+ ReferencedContainer = "container:Weather.xcodeproj">
+ </BuildableReference>
+ </TestableReference>
+ <TestableReference
+ skipped = "NO"
+ parallelizable = "YES">
+ <BuildableReference
+ BuildableIdentifier = "primary"
+ BlueprintIdentifier = "8B9291572FA3FCC400B2E371"
+ BuildableName = "WeatherUITests.xctest"
+ BlueprintName = "WeatherUITests"
+ ReferencedContainer = "container:Weather.xcodeproj">
+ </BuildableReference>
+ </TestableReference>
+ </Testables>
+ </TestAction>
+ <LaunchAction
+ buildConfiguration = "Debug"
+ selectedDebuggerIdentifier = "Xcode.DebuggerFoundation.Debugger.LLDB"
+ selectedLauncherIdentifier = "Xcode.DebuggerFoundation.Launcher.LLDB"
+ launchStyle = "0"
+ useCustomWorkingDirectory = "NO"
+ ignoresPersistentStateOnLaunch = "NO"
+ debugDocumentVersioning = "YES"
+ debugServiceExtension = "internal"
+ allowLocationSimulation = "YES">
+ <BuildableProductRunnable
+ runnableDebuggingMode = "0">
+ <BuildableReference
+ BuildableIdentifier = "primary"
+ BlueprintIdentifier = "8B9291402FA3FCC300B2E371"
+ BuildableName = "Weather.app"
+ BlueprintName = "Weather"
+ ReferencedContainer = "container:Weather.xcodeproj">
+ </BuildableReference>
+ </BuildableProductRunnable>
+ </LaunchAction>
+ <ProfileAction
+ buildConfiguration = "Release"
+ shouldUseLaunchSchemeArgsEnv = "YES"
+ savedToolIdentifier = ""
+ useCustomWorkingDirectory = "NO"
+ debugDocumentVersioning = "YES">
+ <BuildableProductRunnable
+ runnableDebuggingMode = "0">
+ <BuildableReference
+ BuildableIdentifier = "primary"
+ BlueprintIdentifier = "8B9291402FA3FCC300B2E371"
+ BuildableName = "Weather.app"
+ BlueprintName = "Weather"
+ ReferencedContainer = "container:Weather.xcodeproj">
+ </BuildableReference>
+ </BuildableProductRunnable>
+ </ProfileAction>
+ <AnalyzeAction
+ buildConfiguration = "Debug">
+ </AnalyzeAction>
+ <ArchiveAction
+ buildConfiguration = "Release"
+ revealArchiveInOrganizer = "YES">
+ </ArchiveAction>
+</Scheme>
diff --git a/src/utils/__tests__/simulator-test-execution.test.ts b/src/utils/__tests__/simulator-test-execution.test.ts
new file mode 100644
--- /dev/null
+++ b/src/utils/__tests__/simulator-test-execution.test.ts
@@ -1,0 +1,94 @@
+import { describe, expect, it } from 'vitest';
+import { createSimulatorTwoPhaseExecutionPlan } from '../simulator-test-execution.ts';
+import type { TestPreflightResult } from '../test-preflight.ts';
+
+function createPreflight(): TestPreflightResult {
+ return {
+ scheme: 'CalculatorApp',
+ configuration: 'Debug',
+ projectPath: '/tmp/CalculatorApp.xcodeproj',
+ destinationName: 'iPhone 17 Pro',
+ selectors: { onlyTesting: [], skipTesting: [] },
+ warnings: [],
+ completeness: 'complete',
+ totalTests: 2,
+ targets: [
+ {
+ name: 'CalculatorAppTests',
+ warnings: [],
+ files: [
+ {
+ path: '/tmp/CalculatorAppTests.swift',
+ tests: [
+ {
+ framework: 'xctest',
+ targetName: 'CalculatorAppTests',
+ typeName: 'CalculatorAppTests',
+ methodName: 'testAddition',
+ displayName: 'CalculatorAppTests/CalculatorAppTests/testAddition',
+ line: 10,
+ parameterized: false,
+ },
+ {
+ framework: 'swift-testing',
+ targetName: 'CalculatorAppTests',
+ typeName: 'ExpressionSuite',
+ methodName: 'evaluatesExpression',
+ displayName: 'CalculatorAppTests/ExpressionSuite/evaluatesExpression',
+ line: 20,
+ parameterized: true,
+ },
+ ],
+ },
+ ],
+ },
+ ],
+ };
+}
+
+describe('createSimulatorTwoPhaseExecutionPlan', () => {
+ it('keeps preflight discovery observational instead of synthesizing only-testing selectors', () => {
+ const plan = createSimulatorTwoPhaseExecutionPlan({
+ extraArgs: ['-parallel-testing-enabled', 'YES'],
+ preflight: createPreflight(),
+ resultBundlePath: '/tmp/Calculator.xcresult',
+ });
+
+ expect(plan.buildArgs).toEqual(['-parallel-testing-enabled', 'YES']);
+ expect(plan.testArgs).toEqual([
+ '-parallel-testing-enabled',
+ 'YES',
+ '-resultBundlePath',
+ '/tmp/Calculator.xcresult',
+ ]);
+ expect(plan.usesExactSelectors).toBe(false);
+ });
+
+ it('preserves user-supplied selector arguments in both simulator test phases', () => {
+ const plan = createSimulatorTwoPhaseExecutionPlan({
+ extraArgs: [
+ '-only-testing:CalculatorAppTests/CalculatorAppTests/testAddition',
+ '-skip-testing',
+ 'CalculatorAppTests/ExpressionSuite/evaluatesExpression',
+ ],
+ preflight: createPreflight(),
+ });
+
+ expect(plan.buildArgs).toEqual([
+ '-only-testing:CalculatorAppTests/CalculatorAppTests/testAddition',
+ '-skip-testing',
+ 'CalculatorAppTests/ExpressionSuite/evaluatesExpression',
+ ]);
+ expect(plan.testArgs).toEqual(plan.buildArgs);
+ expect(plan.usesExactSelectors).toBe(true);
+ });
+
+ it('keeps resultBundlePath out of build-for-testing args and includes it for test-without-building', () => {
+ const plan = createSimulatorTwoPhaseExecutionPlan({
+ extraArgs: ['-resultBundlePath', '/tmp/UserProvided.xcresult'],
+ });
+
+ expect(plan.buildArgs).toEqual([]);
+ expect(plan.testArgs).toEqual(['-resultBundlePath', '/tmp/UserProvided.xcresult']);
+ });
+});
diff --git a/src/utils/__tests__/swift-test-discovery.test.ts b/src/utils/__tests__/swift-test-discovery.test.ts
new file mode 100644
--- /dev/null
+++ b/src/utils/__tests__/swift-test-discovery.test.ts
@@ -1,0 +1,54 @@
+import { describe, expect, it } from 'vitest';
+import { createMockFileSystemExecutor } from '../../test-utils/mock-executors.ts';
+import { discoverSwiftTestsInFiles } from '../swift-test-discovery.ts';
+
+describe('discoverSwiftTestsInFiles', () => {
+ it('discovers Swift Testing functions with multiline parameterized Test attributes', async () => {
+ const filePath = '/tmp/CalculatorServiceTests.swift';
+ const fileSystemExecutor = createMockFileSystemExecutor({
+ readFile: async () => `
+import Testing
+
+struct CalculatorServiceTests {
+ @Test(
+ "evaluates decimal operations",
+ arguments: [
+ ("1 + 1", "2"),
+ ("4 / 2", "2"),
+ ]
+ )
+ func evaluatesDecimalOperations(expression: String, expected: String) async throws {}
+
+ @Test(arguments: ["+", "-", "×"])
+ func evaluatesOperators(symbol: String) async throws {}
+}
+`,
+ });
+
+ const files = await discoverSwiftTestsInFiles(
+ 'CalculatorAppFeatureTests',
+ [filePath],
+ fileSystemExecutor,
+ );
+
+ expect(files).toHaveLength(1);
+ expect(files[0].tests).toMatchObject([
+ {
+ framework: 'swift-testing',
+ targetName: 'CalculatorAppFeatureTests',
+ typeName: 'CalculatorServiceTests',
+ methodName: 'evaluatesDecimalOperations',
+ displayName: 'CalculatorAppFeatureTests/CalculatorServiceTests/evaluatesDecimalOperations',
+ parameterized: true,
+ },
+ {
+ framework: 'swift-testing',
+ targetName: 'CalculatorAppFeatureTests',
+ typeName: 'CalculatorServiceTests',
+ methodName: 'evaluatesOperators',
+ displayName: 'CalculatorAppFeatureTests/CalculatorServiceTests/evaluatesOperators',
+ parameterized: true,
+ },
+ ]);
+ });
+});
diff --git a/src/utils/__tests__/swift-testing-line-parsers.test.ts b/src/utils/__tests__/swift-testing-line-parsers.test.ts
--- a/src/utils/__tests__/swift-testing-line-parsers.test.ts
+++ b/src/utils/__tests__/swift-testing-line-parsers.test.ts
@@ -4,7 +4,6 @@
parseSwiftTestingIssueLine,
parseSwiftTestingRunSummary,
parseSwiftTestingContinuationLine,
- parseXcodebuildSwiftTestingLine,
} from '../swift-testing-line-parsers.ts';
describe('Swift Testing line parsers', () => {
@@ -259,42 +258,4 @@
expect(parseSwiftTestingContinuationLine('regular line')).toBeNull();
});
});
-
- describe('parseXcodebuildSwiftTestingLine', () => {
- it('should parse a passed test case', () => {
- const result = parseXcodebuildSwiftTestingLine(
- "Test case 'MCPTestTests/appNameIsCorrect()' passed on 'My Mac - MCPTest (78757)' (0.000 seconds)",
- );
- expect(result).toEqual({
- status: 'passed',
- rawName: 'MCPTestTests/appNameIsCorrect()',
- suiteName: 'MCPTestTests',
- testName: 'appNameIsCorrect()',
- durationText: '0.000s',
- });
- });
-
- it('should parse a failed test case', () => {
- const result = parseXcodebuildSwiftTestingLine(
- "Test case 'MCPTestTests/deliberateFailure()' failed on 'My Mac - MCPTest (78757)' (0.000 seconds)",
- );
- expect(result).toEqual({
- status: 'failed',
- rawName: 'MCPTestTests/deliberateFailure()',
- suiteName: 'MCPTestTests',
- testName: 'deliberateFailure()',
- durationText: '0.000s',
- });
- });
-
- it('should return null for XCTest format lines', () => {
- expect(
- parseXcodebuildSwiftTestingLine("Test Case '-[Suite test]' passed (0.001 seconds)."),
- ).toBeNull();
- });
-
- it('should return null for non-matching lines', () => {
- expect(parseXcodebuildSwiftTestingLine('random text')).toBeNull();
- });
- });
});
diff --git a/src/utils/__tests__/test-common.test.ts b/src/utils/__tests__/test-common.test.ts
--- a/src/utils/__tests__/test-common.test.ts
+++ b/src/utils/__tests__/test-common.test.ts
@@ -1,6 +1,59 @@
import { afterEach, describe, expect, it, vi } from 'vitest';
-import { resolveTestProgressEnabled } from '../test-common.ts';
+import type { ChildProcess } from 'node:child_process';
+import { createTestExecutor, resolveTestProgressEnabled } from '../test-common.ts';
+import type { CommandExecutor, CommandResponse } from '../command.ts';
+import { DefaultStreamingExecutionContext } from '../execution/index.ts';
+import type { AnyFragment } from '../../types/domain-fragments.ts';
+import type { TestPreflightResult } from '../test-preflight.ts';
+import { XcodePlatform } from '../xcode.ts';
+function createSuccessfulCommandResponse(): CommandResponse {
+ return {
+ success: true,
+ output: '',
+ process: { pid: 12345 } as ChildProcess,
+ exitCode: 0,
+ };
+}
+
+function createPreflight(): TestPreflightResult {
+ return {
+ scheme: 'Weather',
+ configuration: 'Debug',
+ projectPath: 'Weather.xcodeproj',
+ destinationName: 'iPhone 17 Pro',
+ selectors: {
+ onlyTesting: [],
+ skipTesting: [],
+ },
+ targets: [
+ {
+ name: 'WeatherTests',
+ files: [
+ {
+ path: 'WeatherTests/WeatherTests.swift',
+ tests: [
+ {
+ framework: 'swift-testing',
+ targetName: 'WeatherTests',
+ typeName: 'WeatherTests',
+ methodName: 'emptySearchReturnsNoResults',
+ displayName: 'WeatherTests/WeatherTests/emptySearchReturnsNoResults',
+ line: 12,
+ parameterized: false,
+ },
+ ],
+ },
+ ],
+ warnings: [],
+ },
+ ],
+ warnings: [],
+ totalTests: 1,
+ completeness: 'complete',
+ };
+}
+
describe('resolveTestProgressEnabled', () => {
const originalRuntime = process.env.XCODEBUILDMCP_RUNTIME;
@@ -39,3 +92,60 @@
expect(resolveTestProgressEnabled(false)).toBe(false);
});
});
+
+describe('createTestExecutor', () => {
+ it('emits RUN_TESTS before test-without-building starts in two-phase simulator execution', async () => {
+ const emitted: AnyFragment[] = [];
+ const actions: string[] = [];
+ const executor: CommandExecutor = async (command, _logPrefix, _useShell, opts) => {
+ const action = command.at(-1);
+ if (action) {
+ actions.push(action);
+ }
+
+ if (action === 'build-for-testing') {
+ opts?.onStdout?.('Ld /tmp/Weather.build/Weather normal arm64\n');
+ }
+
+ return createSuccessfulCommandResponse();
+ };
+
+ const executeTest = createTestExecutor(executor, {
+ preflight: createPreflight(),
+ toolName: 'test_sim',
+ target: 'simulator',
+ request: {
+ scheme: 'Weather',
+ projectPath: 'Weather.xcodeproj',
+ configuration: 'Debug',
+ platform: XcodePlatform.iOSSimulator,
+ },
+ });
+
+ await executeTest(
+ {
+ projectPath: 'Weather.xcodeproj',
+ scheme: 'Weather',
+ configuration: 'Debug',
+ simulatorId: 'A2C64636-37E9-4B68-B872-E7F0A82A5670',
+ platform: XcodePlatform.iOSSimulator,
+ },
+ new DefaultStreamingExecutionContext({
+ onFragment: (fragment) => emitted.push(fragment),
+ }),
+ );
+
+ expect(actions).toEqual(['build-for-testing', 'test-without-building']);
+
+ const stageEvents = emitted.filter((event) => event.fragment === 'build-stage');
+ expect(stageEvents.map((event) => event.stage)).toEqual(['LINKING', 'RUN_TESTS']);
+
+ const runTestsIndex = emitted.findIndex(
+ (event) => event.fragment === 'build-stage' && event.stage === 'RUN_TESTS',
+ );
+ const finalSummaryIndex = emitted.findIndex((event) => event.fragment === 'build-summary');
+
+ expect(runTestsIndex).toBeGreaterThan(-1);
+ expect(finalSummaryIndex).toBeGreaterThan(runTestsIndex);
+ });
+});
diff --git a/src/utils/__tests__/xcodebuild-event-parser.test.ts b/src/utils/__tests__/xcodebuild-event-parser.test.ts
--- a/src/utils/__tests__/xcodebuild-event-parser.test.ts
+++ b/src/utils/__tests__/xcodebuild-event-parser.test.ts
@@ -1,5 +1,6 @@
import { describe, expect, it } from 'vitest';
import { createXcodebuildEventParser } from '../xcodebuild-event-parser.ts';
+import { createXcodebuildRunState } from '../xcodebuild-run-state.ts';
import type { DomainFragment } from '../../types/domain-fragments.ts';
function collectEvents(
@@ -24,6 +25,43 @@
return events;
}
+function collectRunStateEvents(
+ lines: { source: 'stdout' | 'stderr'; text: string }[],
+): DomainFragment[] {
+ const events: DomainFragment[] = [];
+ const runState = createXcodebuildRunState({
+ operation: 'TEST',
+ onEvent: (event) => events.push(event),
+ });
+ const parser = createXcodebuildEventParser({
+ operation: 'TEST',
+ onEvent: (event) => {
+ switch (event.fragment) {
+ case 'build-stage':
+ case 'compiler-diagnostic':
+ case 'test-discovery':
+ case 'test-failure':
+ case 'test-progress':
+ case 'test-case-result':
+ runState.push(event);
+ break;
+ }
+ },
+ });
+
+ for (const { source, text } of lines) {
+ if (source === 'stdout') {
+ parser.onStdout(text);
+ } else {
+ parser.onStderr(text);
+ }
+ }
+
+ parser.flush();
+ runState.finalize(false);
+ return events;
+}
+
describe('xcodebuild-event-parser', () => {
it('emits status events for package resolution', () => {
const events = collectEvents('TEST', [{ source: 'stdout', text: 'Resolve Package Graph\n' }]);
@@ -75,6 +113,31 @@
});
});
+ it('emits running-tests status for test suite and case start lines', () => {
+ const events = collectEvents('TEST', [
+ {
+ source: 'stdout',
+ text: "Test suite 'WeatherUITests' started on 'Clone 1 of iPhone 17 Pro - WeatherUITests-Runner (12147)'\n",
+ },
+ {
+ source: 'stdout',
+ text: "Test case 'WeatherTests/emptySearchReturnsNoResults()' started on 'Clone 2 of iPhone 17 Pro - Weather (12472)'\n",
+ },
+ {
+ source: 'stdout',
+ text: '◇ Test "Calculator initializes with correct default values" started.\n',
+ },
+ ]);
+
+ const stages = events.filter((event) => event.fragment === 'build-stage');
+ expect(stages).toHaveLength(3);
+ expect(stages).toEqual(
+ expect.arrayContaining([
+ expect.objectContaining({ fragment: 'build-stage', stage: 'RUN_TESTS' }),
+ ]),
+ );
+ });
+
it('emits test-progress events with cumulative counts', () => {
const events = collectEvents('TEST', [
{ source: 'stdout', text: "Test Case '-[Suite testA]' passed (0.001 seconds)\n" },
@@ -115,6 +178,50 @@
});
});
+ it('parses modern xcodebuild Test Case lines with destinations', () => {
+ const events = collectEvents('TEST', [
+ {
+ source: 'stdout',
+ text: "Test Case '-[WeatherTests testForecast]' passed on 'iPhone 16 Pro' (0.010 seconds)\n",
+ },
+ {
+ source: 'stdout',
+ text: "Test case 'WeatherUITests/testSearch()' failed on 'Clone 1 of iPhone 16 Pro' (0.250 seconds)\n",
+ },
+ {
+ source: 'stdout',
+ text: "Test case 'WeatherUITests/testOfflineMode()' skipped on 'Clone 2 of iPhone 16 Pro' (0.001 seconds)\n",
+ },
+ ]);
+
+ const cases = events.filter((e) => e.fragment === 'test-case-result');
+ expect(cases).toHaveLength(3);
+ expect(cases[0]).toMatchObject({
+ suite: 'WeatherTests',
+ test: 'testForecast',
+ status: 'passed',
+ durationMs: 10,
+ });
+ expect(cases[1]).toMatchObject({
+ suite: 'WeatherUITests',
+ test: 'testSearch()',
+ status: 'failed',
+ durationMs: 250,
+ });
+ expect(cases[2]).toMatchObject({
+ suite: 'WeatherUITests',
+ test: 'testOfflineMode()',
+ status: 'skipped',
+ durationMs: 1,
+ });
+
+ const progressEvents = events.filter((e) => e.fragment === 'test-progress');
+ expect(progressEvents).toHaveLength(3);
+ expect(progressEvents[0]).toMatchObject({ completed: 1, failed: 0, skipped: 0 });
+ expect(progressEvents[1]).toMatchObject({ completed: 2, failed: 1, skipped: 0 });
+ expect(progressEvents[2]).toMatchObject({ completed: 3, failed: 1, skipped: 1 });
+ });
+
it('emits test-case-result events for Swift Testing passed/failed lines', () => {
const events = collectEvents('TEST', [
{ source: 'stdout', text: '✔ Test "passingTest()" passed after 0.005 seconds.\n' },
@@ -398,6 +505,101 @@
});
});
+ it('uses Swift Testing and XCTest summaries once for mixed Calculator test output', () => {
+ const xctestPassedLines = Array.from({ length: 21 }, (_, index) => ({
+ source: 'stdout' as const,
+ text: `Test Case '-[CalculatorAppTests.CalculatorAppTests testPassing${index + 1}]' passed (0.001 seconds).\n`,
+ }));
+ const events = collectRunStateEvents([
+ {
+ source: 'stdout',
+ text: '✔ Test "Adding single digit numbers" passed after 0.016 seconds.\n',
+ },
+ {
+ source: 'stdout',
+ text: '\u200B✔ Test "Adding decimal numbers" passed after 0.012 seconds.\n',
+ },
+ {
+ source: 'stdout',
+ text: '✔ Test "Addition operation" with 4 test cases passed after 0.005 seconds.\n',
+ },
+ {
+ source: 'stdout',
+ text: '✘ Test "This test should fail to verify error reporting" recorded an issue at CalculatorServiceTests.swift:37:9: Expectation failed: (calculator.display → "0") == "999"\n',
+ },
+ {
+ source: 'stdout',
+ text: '✘ Test "This test should fail to verify error reporting" failed after 0.029 seconds with 1 issue.\n',
+ },
+ {
+ source: 'stdout',
+ text: '✘ Test run with 34 tests in 9 suites failed after 0.047 seconds with 1 issue.\n',
+ },
+ ...xctestPassedLines,
+ {
+ source: 'stderr',
+ text: '/Volumes/Developer/XcodeBuildMCP/example_projects/iOS_Calculator/CalculatorAppTests/CalculatorAppTests.swift:52: error: -[CalculatorAppTests.CalculatorAppTests testCalculatorServiceFailure] : XCTAssertEqual failed: ("0") is not equal to ("999") - This test should fail - display should be 0, not 999\n',
+ },
+ {
+ source: 'stdout',
+ text: "Test Case '-[CalculatorAppTests.CalculatorAppTests testCalculatorServiceFailure]' failed (0.004 seconds).\n",
+ },
+ {
+ source: 'stderr',
+ text: '/Volumes/Developer/XcodeBuildMCP/example_projects/iOS_Calculator/CalculatorAppTests/CalculatorAppTests.swift:286: error: -[CalculatorAppTests.IntentionalFailureTests test] : XCTAssertTrue failed - This test should fail to verify error reporting\n',
+ },
+ {
+ source: 'stdout',
+ text: "Test Case '-[CalculatorAppTests.IntentionalFailureTests test]' failed (0.003 seconds).\n",
+ },
+ {
+ source: 'stdout',
+ text: '\t Executed 23 tests, with 2 failures (0 unexpected) in 0.654 (0.665) seconds\n',
+ },
+ ]);
+
+ const summary = events.filter((event) => event.fragment === 'build-summary').at(-1);
+ expect(summary).toMatchObject({
+ fragment: 'build-summary',
+ operation: 'TEST',
+ totalTests: 57,
+ passedTests: 54,
+ failedTests: 3,
+ skippedTests: 0,
+ });
+ });
+
+ it('reconciles separate Swift Testing run summaries independently', () => {
+ const events = collectRunStateEvents([
+ {
+ source: 'stdout',
+ text: '✔ Test "First target test" passed after 0.001 seconds.\n',
+ },
+ {
+ source: 'stdout',
+ text: '✔ Test run with 1 test in 1 suite passed after 0.001 seconds.\n',
+ },
+ {
+ source: 'stdout',
+ text: '✔ Test "Second target parameterized test" with 4 test cases passed after 0.002 seconds.\n',
+ },
+ {
+ source: 'stdout',
+ text: '✔ Test run with 1 test in 1 suite passed after 0.002 seconds.\n',
+ },
+ ]);
+
+ const summary = events.filter((event) => event.fragment === 'build-summary').at(-1);
+ expect(summary).toMatchObject({
+ fragment: 'build-summary',
+ operation: 'TEST',
+ totalTests: 2,
+ passedTests: 2,
+ failedTests: 0,
+ skippedTests: 0,
+ });
+ });
+
it('processes full test lifecycle', () => {
const events = collectEvents('TEST', [
{ source: 'stdout', text: 'Resolve Package Graph\n' },
diff --git a/src/utils/renderers/__tests__/cli-text-renderer.test.ts b/src/utils/renderers/__tests__/cli-text-renderer.test.ts
--- a/src/utils/renderers/__tests__/cli-text-renderer.test.ts
+++ b/src/utils/renderers/__tests__/cli-text-renderer.test.ts
@@ -145,6 +145,66 @@
expect(output).toContain('\u{2705} Resolving app path\n');
});
+ it('replaces interactive build-stage updates with test progress updates', () => {
+ const renderer = createCliTextRenderer({ interactive: true });
+
+ renderer.onFragment({
+ kind: 'test-result',
+ fragment: 'build-stage',
+ operation: 'TEST',
+ stage: 'LINKING',
+ message: 'Linking',
+ });
+
+ renderer.onFragment({
+ kind: 'test-result',
+ fragment: 'test-progress',
+ operation: 'TEST',
+ completed: 4,
+ failed: 0,
+ skipped: 0,
+ });
+
+ expect(reporter.update).toHaveBeenCalledWith('Linking...');
+ expect(reporter.update).toHaveBeenCalledWith(
+ 'Running tests (4 completed, 0 failures, 0 skipped)',
+ );
+ });
+
+ it('renders non-interactive test progress durably and deduplicates repeated counts', () => {
+ const output = renderCliTextTranscript({
+ items: [
+ {
+ kind: 'test-result',
+ fragment: 'test-progress',
+ operation: 'TEST',
+ completed: 1,
+ failed: 0,
+ skipped: 0,
+ },
+ {
+ kind: 'test-result',
+ fragment: 'test-progress',
+ operation: 'TEST',
+ completed: 1,
+ failed: 0,
+ skipped: 0,
+ },
+ {
+ kind: 'test-result',
+ fragment: 'test-progress',
+ operation: 'TEST',
+ completed: 2,
+ failed: 0,
+ skipped: 0,
+ },
+ ],
+ });
+
+ expect(output.match(/Running tests \(1 completed, 0 failures, 0 skipped\)/g)).toHaveLength(1);
+ expect(output).toContain('Running tests (2 completed, 0 failures, 0 skipped)');
+ });
+
it('renders grouped sad-path diagnostics before the failed summary', () => {
const stdoutWrite = vi.spyOn(process.stdout, 'write').mockImplementation(() => true);
const renderer = createCliTextRenderer({ interactive: false });
diff --git a/src/utils/renderers/cli-text-renderer.ts b/src/utils/renderers/cli-text-renderer.ts
--- a/src/utils/renderers/cli-text-renderer.ts
+++ b/src/utils/renderers/cli-text-renderer.ts
@@ -39,6 +39,7 @@
formatNextStepsEvent,
formatTestCaseResults,
formatTestDiscoveryEvent,
+ formatTestProgressEvent,
} from './event-formatting.ts';
import {
createXcodebuildEventParser,
@@ -112,6 +113,7 @@
let nextSteps: readonly NextStep[] = [];
let nextStepsRuntime: 'cli' | 'daemon' | 'mcp' | undefined;
let sawProgressNextSteps = false;
+ let lastRenderedTestProgressKey: string | null = null;
function writeDurable(text: string): void {
sink.clearTransient();
@@ -261,10 +263,16 @@
}
case 'test-progress': {
+ const renderedProgress = formatTestProgressEvent(item);
... diff truncated: showing 800 of 1129 linesYou can send follow-ups to the cloud agent here.
The updated parseTestCaseLine regex now matches both 'Test Case' and 'Test case' with an optional 'on' clause, making it strictly more permissive than parseXcodebuildSwiftTestingLine. Since parseTestCaseLine is checked first in processLine, parseXcodebuildSwiftTestingLine became unreachable dead code.
![sentry-warden[bot]](https://avatars.githubusercontent.com/in/2755838?s=60&v=4){kind=small}
Keep Swift Testing summary reconciliation from decrementing progress or re-counting xcodebuild-formatted per-test lines when summaries arrive later. Count summary-only failures as additional observed failures. Remove the now-unreachable xcodebuild Swift Testing parser path because the generic test-case parser handles destination-suffixed xcodebuild lines. Refs #384 Co-Authored-By: OpenAI Codex <noreply@openai.com>
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Unused
sourceparam causes XCTest/Swift Testing miscount- Gated the Swift Testing counter increments with a source check so XCTest results no longer inflate the count subtracted from Swift Testing summaries.
Preview (24a295a637)
diff --git a/CHANGELOG.md b/CHANGELOG.md
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,9 +4,12 @@
### Fixed
+- Fixed simulator test JSONL accuracy by keeping preflight discovery observational, preserving only user-supplied test selectors, discovering multiline parameterized Swift Testing tests, and parsing destination-suffixed xcodebuild test result lines.
- Removed stale physical-device log session status and shutdown cleanup for deprecated standalone device log capture, and corrected the device build-and-run tool description.
+- Fixed mixed Swift Testing and XCTest summaries so simulator test text output no longer overcounts parameterized Swift Testing results or issue lines.
- Fixed CLI test summaries showing false-positive compiler errors from xcodebuild NSError dump lines, and added compiler-error snapshot coverage for simulator, device, and macOS build-style flows ([#383](https://github.com/getsentry/XcodeBuildMCP/issues/383)).
- Fixed simulator OSLog helper cleanup so server and daemon startup reconcile same-workspace orphaned log streams without stopping helpers owned by live sessions in other workspaces ([#382](https://github.com/getsentry/XcodeBuildMCP/issues/382)).
+- Fixed Weather example test discovery and made CLI test progress visible while tests are running instead of leaving the last build phase displayed.
## [2.5.0-beta.1]
diff --git a/example_projects/Weather/Weather.xcodeproj/xcshareddata/xcschemes/Weather.xcscheme b/example_projects/Weather/Weather.xcodeproj/xcshareddata/xcschemes/Weather.xcscheme
new file mode 100644
--- /dev/null
+++ b/example_projects/Weather/Weather.xcodeproj/xcshareddata/xcschemes/Weather.xcscheme
@@ -1,0 +1,101 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<Scheme
+ LastUpgradeVersion = "1640"
+ version = "1.7">
+ <BuildAction
+ parallelizeBuildables = "YES"
+ buildImplicitDependencies = "YES"
+ buildArchitectures = "Automatic">
+ <BuildActionEntries>
+ <BuildActionEntry
+ buildForTesting = "YES"
+ buildForRunning = "YES"
+ buildForProfiling = "YES"
+ buildForArchiving = "YES"
+ buildForAnalyzing = "YES">
+ <BuildableReference
+ BuildableIdentifier = "primary"
+ BlueprintIdentifier = "8B9291402FA3FCC300B2E371"
+ BuildableName = "Weather.app"
+ BlueprintName = "Weather"
+ ReferencedContainer = "container:Weather.xcodeproj">
+ </BuildableReference>
+ </BuildActionEntry>
+ </BuildActionEntries>
+ </BuildAction>
+ <TestAction
+ buildConfiguration = "Debug"
+ selectedDebuggerIdentifier = "Xcode.DebuggerFoundation.Debugger.LLDB"
+ selectedLauncherIdentifier = "Xcode.DebuggerFoundation.Launcher.LLDB"
+ shouldUseLaunchSchemeArgsEnv = "YES">
+ <Testables>
+ <TestableReference
+ skipped = "NO"
+ parallelizable = "YES">
+ <BuildableReference
+ BuildableIdentifier = "primary"
+ BlueprintIdentifier = "8B92914D2FA3FCC400B2E371"
+ BuildableName = "WeatherTests.xctest"
+ BlueprintName = "WeatherTests"
+ ReferencedContainer = "container:Weather.xcodeproj">
+ </BuildableReference>
+ </TestableReference>
+ <TestableReference
+ skipped = "NO"
+ parallelizable = "YES">
+ <BuildableReference
+ BuildableIdentifier = "primary"
+ BlueprintIdentifier = "8B9291572FA3FCC400B2E371"
+ BuildableName = "WeatherUITests.xctest"
+ BlueprintName = "WeatherUITests"
+ ReferencedContainer = "container:Weather.xcodeproj">
+ </BuildableReference>
+ </TestableReference>
+ </Testables>
+ </TestAction>
+ <LaunchAction
+ buildConfiguration = "Debug"
+ selectedDebuggerIdentifier = "Xcode.DebuggerFoundation.Debugger.LLDB"
+ selectedLauncherIdentifier = "Xcode.DebuggerFoundation.Launcher.LLDB"
+ launchStyle = "0"
+ useCustomWorkingDirectory = "NO"
+ ignoresPersistentStateOnLaunch = "NO"
+ debugDocumentVersioning = "YES"
+ debugServiceExtension = "internal"
+ allowLocationSimulation = "YES">
+ <BuildableProductRunnable
+ runnableDebuggingMode = "0">
+ <BuildableReference
+ BuildableIdentifier = "primary"
+ BlueprintIdentifier = "8B9291402FA3FCC300B2E371"
+ BuildableName = "Weather.app"
+ BlueprintName = "Weather"
+ ReferencedContainer = "container:Weather.xcodeproj">
+ </BuildableReference>
+ </BuildableProductRunnable>
+ </LaunchAction>
+ <ProfileAction
+ buildConfiguration = "Release"
+ shouldUseLaunchSchemeArgsEnv = "YES"
+ savedToolIdentifier = ""
+ useCustomWorkingDirectory = "NO"
+ debugDocumentVersioning = "YES">
+ <BuildableProductRunnable
+ runnableDebuggingMode = "0">
+ <BuildableReference
+ BuildableIdentifier = "primary"
+ BlueprintIdentifier = "8B9291402FA3FCC300B2E371"
+ BuildableName = "Weather.app"
+ BlueprintName = "Weather"
+ ReferencedContainer = "container:Weather.xcodeproj">
+ </BuildableReference>
+ </BuildableProductRunnable>
+ </ProfileAction>
+ <AnalyzeAction
+ buildConfiguration = "Debug">
+ </AnalyzeAction>
+ <ArchiveAction
+ buildConfiguration = "Release"
+ revealArchiveInOrganizer = "YES">
+ </ArchiveAction>
+</Scheme>
diff --git a/src/utils/__tests__/simulator-test-execution.test.ts b/src/utils/__tests__/simulator-test-execution.test.ts
new file mode 100644
--- /dev/null
+++ b/src/utils/__tests__/simulator-test-execution.test.ts
@@ -1,0 +1,94 @@
+import { describe, expect, it } from 'vitest';
+import { createSimulatorTwoPhaseExecutionPlan } from '../simulator-test-execution.ts';
+import type { TestPreflightResult } from '../test-preflight.ts';
+
+function createPreflight(): TestPreflightResult {
+ return {
+ scheme: 'CalculatorApp',
+ configuration: 'Debug',
+ projectPath: '/tmp/CalculatorApp.xcodeproj',
+ destinationName: 'iPhone 17 Pro',
+ selectors: { onlyTesting: [], skipTesting: [] },
+ warnings: [],
+ completeness: 'complete',
+ totalTests: 2,
+ targets: [
+ {
+ name: 'CalculatorAppTests',
+ warnings: [],
+ files: [
+ {
+ path: '/tmp/CalculatorAppTests.swift',
+ tests: [
+ {
+ framework: 'xctest',
+ targetName: 'CalculatorAppTests',
+ typeName: 'CalculatorAppTests',
+ methodName: 'testAddition',
+ displayName: 'CalculatorAppTests/CalculatorAppTests/testAddition',
+ line: 10,
+ parameterized: false,
+ },
+ {
+ framework: 'swift-testing',
+ targetName: 'CalculatorAppTests',
+ typeName: 'ExpressionSuite',
+ methodName: 'evaluatesExpression',
+ displayName: 'CalculatorAppTests/ExpressionSuite/evaluatesExpression',
+ line: 20,
+ parameterized: true,
+ },
+ ],
+ },
+ ],
+ },
+ ],
+ };
+}
+
+describe('createSimulatorTwoPhaseExecutionPlan', () => {
+ it('keeps preflight discovery observational instead of synthesizing only-testing selectors', () => {
+ const plan = createSimulatorTwoPhaseExecutionPlan({
+ extraArgs: ['-parallel-testing-enabled', 'YES'],
+ preflight: createPreflight(),
+ resultBundlePath: '/tmp/Calculator.xcresult',
+ });
+
+ expect(plan.buildArgs).toEqual(['-parallel-testing-enabled', 'YES']);
+ expect(plan.testArgs).toEqual([
+ '-parallel-testing-enabled',
+ 'YES',
+ '-resultBundlePath',
+ '/tmp/Calculator.xcresult',
+ ]);
+ expect(plan.usesExactSelectors).toBe(false);
+ });
+
+ it('preserves user-supplied selector arguments in both simulator test phases', () => {
+ const plan = createSimulatorTwoPhaseExecutionPlan({
+ extraArgs: [
+ '-only-testing:CalculatorAppTests/CalculatorAppTests/testAddition',
+ '-skip-testing',
+ 'CalculatorAppTests/ExpressionSuite/evaluatesExpression',
+ ],
+ preflight: createPreflight(),
+ });
+
+ expect(plan.buildArgs).toEqual([
+ '-only-testing:CalculatorAppTests/CalculatorAppTests/testAddition',
+ '-skip-testing',
+ 'CalculatorAppTests/ExpressionSuite/evaluatesExpression',
+ ]);
+ expect(plan.testArgs).toEqual(plan.buildArgs);
+ expect(plan.usesExactSelectors).toBe(true);
+ });
+
+ it('keeps resultBundlePath out of build-for-testing args and includes it for test-without-building', () => {
+ const plan = createSimulatorTwoPhaseExecutionPlan({
+ extraArgs: ['-resultBundlePath', '/tmp/UserProvided.xcresult'],
+ });
+
+ expect(plan.buildArgs).toEqual([]);
+ expect(plan.testArgs).toEqual(['-resultBundlePath', '/tmp/UserProvided.xcresult']);
+ });
+});
diff --git a/src/utils/__tests__/swift-test-discovery.test.ts b/src/utils/__tests__/swift-test-discovery.test.ts
new file mode 100644
--- /dev/null
+++ b/src/utils/__tests__/swift-test-discovery.test.ts
@@ -1,0 +1,54 @@
+import { describe, expect, it } from 'vitest';
+import { createMockFileSystemExecutor } from '../../test-utils/mock-executors.ts';
+import { discoverSwiftTestsInFiles } from '../swift-test-discovery.ts';
+
+describe('discoverSwiftTestsInFiles', () => {
+ it('discovers Swift Testing functions with multiline parameterized Test attributes', async () => {
+ const filePath = '/tmp/CalculatorServiceTests.swift';
+ const fileSystemExecutor = createMockFileSystemExecutor({
+ readFile: async () => `
+import Testing
+
+struct CalculatorServiceTests {
+ @Test(
+ "evaluates decimal operations",
+ arguments: [
+ ("1 + 1", "2"),
+ ("4 / 2", "2"),
+ ]
+ )
+ func evaluatesDecimalOperations(expression: String, expected: String) async throws {}
+
+ @Test(arguments: ["+", "-", "×"])
+ func evaluatesOperators(symbol: String) async throws {}
+}
+`,
+ });
+
+ const files = await discoverSwiftTestsInFiles(
+ 'CalculatorAppFeatureTests',
+ [filePath],
+ fileSystemExecutor,
+ );
+
+ expect(files).toHaveLength(1);
+ expect(files[0].tests).toMatchObject([
+ {
+ framework: 'swift-testing',
+ targetName: 'CalculatorAppFeatureTests',
+ typeName: 'CalculatorServiceTests',
+ methodName: 'evaluatesDecimalOperations',
+ displayName: 'CalculatorAppFeatureTests/CalculatorServiceTests/evaluatesDecimalOperations',
+ parameterized: true,
+ },
+ {
+ framework: 'swift-testing',
+ targetName: 'CalculatorAppFeatureTests',
+ typeName: 'CalculatorServiceTests',
+ methodName: 'evaluatesOperators',
+ displayName: 'CalculatorAppFeatureTests/CalculatorServiceTests/evaluatesOperators',
+ parameterized: true,
+ },
+ ]);
+ });
+});
diff --git a/src/utils/__tests__/swift-testing-line-parsers.test.ts b/src/utils/__tests__/swift-testing-line-parsers.test.ts
--- a/src/utils/__tests__/swift-testing-line-parsers.test.ts
+++ b/src/utils/__tests__/swift-testing-line-parsers.test.ts
@@ -4,7 +4,6 @@
parseSwiftTestingIssueLine,
parseSwiftTestingRunSummary,
parseSwiftTestingContinuationLine,
- parseXcodebuildSwiftTestingLine,
} from '../swift-testing-line-parsers.ts';
describe('Swift Testing line parsers', () => {
@@ -259,42 +258,4 @@
expect(parseSwiftTestingContinuationLine('regular line')).toBeNull();
});
});
-
- describe('parseXcodebuildSwiftTestingLine', () => {
- it('should parse a passed test case', () => {
- const result = parseXcodebuildSwiftTestingLine(
- "Test case 'MCPTestTests/appNameIsCorrect()' passed on 'My Mac - MCPTest (78757)' (0.000 seconds)",
- );
- expect(result).toEqual({
- status: 'passed',
- rawName: 'MCPTestTests/appNameIsCorrect()',
- suiteName: 'MCPTestTests',
- testName: 'appNameIsCorrect()',
- durationText: '0.000s',
- });
- });
-
- it('should parse a failed test case', () => {
- const result = parseXcodebuildSwiftTestingLine(
- "Test case 'MCPTestTests/deliberateFailure()' failed on 'My Mac - MCPTest (78757)' (0.000 seconds)",
- );
- expect(result).toEqual({
- status: 'failed',
- rawName: 'MCPTestTests/deliberateFailure()',
- suiteName: 'MCPTestTests',
- testName: 'deliberateFailure()',
- durationText: '0.000s',
- });
- });
-
- it('should return null for XCTest format lines', () => {
- expect(
- parseXcodebuildSwiftTestingLine("Test Case '-[Suite test]' passed (0.001 seconds)."),
- ).toBeNull();
- });
-
- it('should return null for non-matching lines', () => {
- expect(parseXcodebuildSwiftTestingLine('random text')).toBeNull();
- });
- });
});
diff --git a/src/utils/__tests__/test-common.test.ts b/src/utils/__tests__/test-common.test.ts
--- a/src/utils/__tests__/test-common.test.ts
+++ b/src/utils/__tests__/test-common.test.ts
@@ -1,6 +1,59 @@
import { afterEach, describe, expect, it, vi } from 'vitest';
-import { resolveTestProgressEnabled } from '../test-common.ts';
+import type { ChildProcess } from 'node:child_process';
+import { createTestExecutor, resolveTestProgressEnabled } from '../test-common.ts';
+import type { CommandExecutor, CommandResponse } from '../command.ts';
+import { DefaultStreamingExecutionContext } from '../execution/index.ts';
+import type { AnyFragment } from '../../types/domain-fragments.ts';
+import type { TestPreflightResult } from '../test-preflight.ts';
+import { XcodePlatform } from '../xcode.ts';
+function createSuccessfulCommandResponse(): CommandResponse {
+ return {
+ success: true,
+ output: '',
+ process: { pid: 12345 } as ChildProcess,
+ exitCode: 0,
+ };
+}
+
+function createPreflight(): TestPreflightResult {
+ return {
+ scheme: 'Weather',
+ configuration: 'Debug',
+ projectPath: 'Weather.xcodeproj',
+ destinationName: 'iPhone 17 Pro',
+ selectors: {
+ onlyTesting: [],
+ skipTesting: [],
+ },
+ targets: [
+ {
+ name: 'WeatherTests',
+ files: [
+ {
+ path: 'WeatherTests/WeatherTests.swift',
+ tests: [
+ {
+ framework: 'swift-testing',
+ targetName: 'WeatherTests',
+ typeName: 'WeatherTests',
+ methodName: 'emptySearchReturnsNoResults',
+ displayName: 'WeatherTests/WeatherTests/emptySearchReturnsNoResults',
+ line: 12,
+ parameterized: false,
+ },
+ ],
+ },
+ ],
+ warnings: [],
+ },
+ ],
+ warnings: [],
+ totalTests: 1,
+ completeness: 'complete',
+ };
+}
+
describe('resolveTestProgressEnabled', () => {
const originalRuntime = process.env.XCODEBUILDMCP_RUNTIME;
@@ -39,3 +92,60 @@
expect(resolveTestProgressEnabled(false)).toBe(false);
});
});
+
+describe('createTestExecutor', () => {
+ it('emits RUN_TESTS before test-without-building starts in two-phase simulator execution', async () => {
+ const emitted: AnyFragment[] = [];
+ const actions: string[] = [];
+ const executor: CommandExecutor = async (command, _logPrefix, _useShell, opts) => {
+ const action = command.at(-1);
+ if (action) {
+ actions.push(action);
+ }
+
+ if (action === 'build-for-testing') {
+ opts?.onStdout?.('Ld /tmp/Weather.build/Weather normal arm64\n');
+ }
+
+ return createSuccessfulCommandResponse();
+ };
+
+ const executeTest = createTestExecutor(executor, {
+ preflight: createPreflight(),
+ toolName: 'test_sim',
+ target: 'simulator',
+ request: {
+ scheme: 'Weather',
+ projectPath: 'Weather.xcodeproj',
+ configuration: 'Debug',
+ platform: XcodePlatform.iOSSimulator,
+ },
+ });
+
+ await executeTest(
+ {
+ projectPath: 'Weather.xcodeproj',
+ scheme: 'Weather',
+ configuration: 'Debug',
+ simulatorId: 'A2C64636-37E9-4B68-B872-E7F0A82A5670',
+ platform: XcodePlatform.iOSSimulator,
+ },
+ new DefaultStreamingExecutionContext({
+ onFragment: (fragment) => emitted.push(fragment),
+ }),
+ );
+
+ expect(actions).toEqual(['build-for-testing', 'test-without-building']);
+
+ const stageEvents = emitted.filter((event) => event.fragment === 'build-stage');
+ expect(stageEvents.map((event) => event.stage)).toEqual(['LINKING', 'RUN_TESTS']);
+
+ const runTestsIndex = emitted.findIndex(
+ (event) => event.fragment === 'build-stage' && event.stage === 'RUN_TESTS',
+ );
+ const finalSummaryIndex = emitted.findIndex((event) => event.fragment === 'build-summary');
+
+ expect(runTestsIndex).toBeGreaterThan(-1);
+ expect(finalSummaryIndex).toBeGreaterThan(runTestsIndex);
+ });
+});
diff --git a/src/utils/__tests__/xcodebuild-event-parser.test.ts b/src/utils/__tests__/xcodebuild-event-parser.test.ts
--- a/src/utils/__tests__/xcodebuild-event-parser.test.ts
+++ b/src/utils/__tests__/xcodebuild-event-parser.test.ts
@@ -1,5 +1,6 @@
import { describe, expect, it } from 'vitest';
import { createXcodebuildEventParser } from '../xcodebuild-event-parser.ts';
+import { createXcodebuildRunState } from '../xcodebuild-run-state.ts';
import type { DomainFragment } from '../../types/domain-fragments.ts';
function collectEvents(
@@ -24,6 +25,43 @@
return events;
}
+function collectRunStateEvents(
+ lines: { source: 'stdout' | 'stderr'; text: string }[],
+): DomainFragment[] {
+ const events: DomainFragment[] = [];
+ const runState = createXcodebuildRunState({
+ operation: 'TEST',
+ onEvent: (event) => events.push(event),
+ });
+ const parser = createXcodebuildEventParser({
+ operation: 'TEST',
+ onEvent: (event) => {
+ switch (event.fragment) {
+ case 'build-stage':
+ case 'compiler-diagnostic':
+ case 'test-discovery':
+ case 'test-failure':
+ case 'test-progress':
+ case 'test-case-result':
+ runState.push(event);
+ break;
+ }
+ },
+ });
+
+ for (const { source, text } of lines) {
+ if (source === 'stdout') {
+ parser.onStdout(text);
+ } else {
+ parser.onStderr(text);
+ }
+ }
+
+ parser.flush();
+ runState.finalize(false);
+ return events;
+}
+
describe('xcodebuild-event-parser', () => {
it('emits status events for package resolution', () => {
const events = collectEvents('TEST', [{ source: 'stdout', text: 'Resolve Package Graph\n' }]);
@@ -75,6 +113,31 @@
});
});
+ it('emits running-tests status for test suite and case start lines', () => {
+ const events = collectEvents('TEST', [
+ {
+ source: 'stdout',
+ text: "Test suite 'WeatherUITests' started on 'Clone 1 of iPhone 17 Pro - WeatherUITests-Runner (12147)'\n",
+ },
+ {
+ source: 'stdout',
+ text: "Test case 'WeatherTests/emptySearchReturnsNoResults()' started on 'Clone 2 of iPhone 17 Pro - Weather (12472)'\n",
+ },
+ {
+ source: 'stdout',
+ text: '◇ Test "Calculator initializes with correct default values" started.\n',
+ },
+ ]);
+
+ const stages = events.filter((event) => event.fragment === 'build-stage');
+ expect(stages).toHaveLength(3);
+ expect(stages).toEqual(
+ expect.arrayContaining([
+ expect.objectContaining({ fragment: 'build-stage', stage: 'RUN_TESTS' }),
+ ]),
+ );
+ });
+
it('emits test-progress events with cumulative counts', () => {
const events = collectEvents('TEST', [
{ source: 'stdout', text: "Test Case '-[Suite testA]' passed (0.001 seconds)\n" },
@@ -115,6 +178,50 @@
});
});
+ it('parses modern xcodebuild Test Case lines with destinations', () => {
+ const events = collectEvents('TEST', [
+ {
+ source: 'stdout',
+ text: "Test Case '-[WeatherTests testForecast]' passed on 'iPhone 16 Pro' (0.010 seconds)\n",
+ },
+ {
+ source: 'stdout',
+ text: "Test case 'WeatherUITests/testSearch()' failed on 'Clone 1 of iPhone 16 Pro' (0.250 seconds)\n",
+ },
+ {
+ source: 'stdout',
+ text: "Test case 'WeatherUITests/testOfflineMode()' skipped on 'Clone 2 of iPhone 16 Pro' (0.001 seconds)\n",
+ },
+ ]);
+
+ const cases = events.filter((e) => e.fragment === 'test-case-result');
+ expect(cases).toHaveLength(3);
+ expect(cases[0]).toMatchObject({
+ suite: 'WeatherTests',
+ test: 'testForecast',
+ status: 'passed',
+ durationMs: 10,
+ });
+ expect(cases[1]).toMatchObject({
+ suite: 'WeatherUITests',
+ test: 'testSearch()',
+ status: 'failed',
+ durationMs: 250,
+ });
+ expect(cases[2]).toMatchObject({
+ suite: 'WeatherUITests',
+ test: 'testOfflineMode()',
+ status: 'skipped',
+ durationMs: 1,
+ });
+
+ const progressEvents = events.filter((e) => e.fragment === 'test-progress');
+ expect(progressEvents).toHaveLength(3);
+ expect(progressEvents[0]).toMatchObject({ completed: 1, failed: 0, skipped: 0 });
+ expect(progressEvents[1]).toMatchObject({ completed: 2, failed: 1, skipped: 0 });
+ expect(progressEvents[2]).toMatchObject({ completed: 3, failed: 1, skipped: 1 });
+ });
+
it('emits test-case-result events for Swift Testing passed/failed lines', () => {
const events = collectEvents('TEST', [
{ source: 'stdout', text: '✔ Test "passingTest()" passed after 0.005 seconds.\n' },
@@ -398,6 +505,163 @@
});
});
+ it('uses Swift Testing and XCTest summaries once for mixed Calculator test output', () => {
+ const xctestPassedLines = Array.from({ length: 21 }, (_, index) => ({
+ source: 'stdout' as const,
+ text: `Test Case '-[CalculatorAppTests.CalculatorAppTests testPassing${index + 1}]' passed (0.001 seconds).\n`,
+ }));
+ const events = collectRunStateEvents([
+ {
+ source: 'stdout',
+ text: '✔ Test "Adding single digit numbers" passed after 0.016 seconds.\n',
+ },
+ {
+ source: 'stdout',
+ text: '\u200B✔ Test "Adding decimal numbers" passed after 0.012 seconds.\n',
+ },
+ {
+ source: 'stdout',
+ text: '✔ Test "Addition operation" with 4 test cases passed after 0.005 seconds.\n',
+ },
+ {
+ source: 'stdout',
+ text: '✘ Test "This test should fail to verify error reporting" recorded an issue at CalculatorServiceTests.swift:37:9: Expectation failed: (calculator.display → "0") == "999"\n',
+ },
+ {
+ source: 'stdout',
+ text: '✘ Test "This test should fail to verify error reporting" failed after 0.029 seconds with 1 issue.\n',
+ },
+ {
+ source: 'stdout',
+ text: '✘ Test run with 34 tests in 9 suites failed after 0.047 seconds with 1 issue.\n',
+ },
+ ...xctestPassedLines,
+ {
+ source: 'stderr',
+ text: '/Volumes/Developer/XcodeBuildMCP/example_projects/iOS_Calculator/CalculatorAppTests/CalculatorAppTests.swift:52: error: -[CalculatorAppTests.CalculatorAppTests testCalculatorServiceFailure] : XCTAssertEqual failed: ("0") is not equal to ("999") - This test should fail - display should be 0, not 999\n',
+ },
+ {
+ source: 'stdout',
+ text: "Test Case '-[CalculatorAppTests.CalculatorAppTests testCalculatorServiceFailure]' failed (0.004 seconds).\n",
+ },
+ {
+ source: 'stderr',
+ text: '/Volumes/Developer/XcodeBuildMCP/example_projects/iOS_Calculator/CalculatorAppTests/CalculatorAppTests.swift:286: error: -[CalculatorAppTests.IntentionalFailureTests test] : XCTAssertTrue failed - This test should fail to verify error reporting\n',
+ },
+ {
+ source: 'stdout',
+ text: "Test Case '-[CalculatorAppTests.IntentionalFailureTests test]' failed (0.003 seconds).\n",
+ },
+ {
+ source: 'stdout',
+ text: '\t Executed 23 tests, with 2 failures (0 unexpected) in 0.654 (0.665) seconds\n',
+ },
+ ]);
+
+ const summary = events.filter((event) => event.fragment === 'build-summary').at(-1);
+ expect(summary).toMatchObject({
+ fragment: 'build-summary',
+ operation: 'TEST',
+ totalTests: 57,
+ passedTests: 54,
+ failedTests: 3,
+ skippedTests: 0,
+ });
+ });
+
+ it('reconciles separate Swift Testing run summaries independently', () => {
+ const events = collectRunStateEvents([
+ {
+ source: 'stdout',
+ text: '✔ Test "First target test" passed after 0.001 seconds.\n',
+ },
+ {
+ source: 'stdout',
+ text: '✔ Test run with 1 test in 1 suite passed after 0.001 seconds.\n',
+ },
+ {
+ source: 'stdout',
+ text: '✔ Test "Second target parameterized test" with 4 test cases passed after 0.002 seconds.\n',
+ },
+ {
+ source: 'stdout',
+ text: '✔ Test run with 1 test in 1 suite passed after 0.002 seconds.\n',
+ },
+ ]);
+
+ const summary = events.filter((event) => event.fragment === 'build-summary').at(-1);
+ expect(summary).toMatchObject({
+ fragment: 'build-summary',
+ operation: 'TEST',
+ totalTests: 5,
+ passedTests: 5,
+ failedTests: 0,
+ skippedTests: 0,
+ });
+ });
+
+ it('keeps Swift Testing summary progress monotonic when per-test lines exceed the summary', () => {
+ const events = collectEvents('TEST', [
+ {
+ source: 'stdout',
+ text: '✔ Test "First observed case" passed after 0.001 seconds.\n',
+ },
+ {
+ source: 'stdout',
+ text: '✔ Test "Second observed case" passed after 0.001 seconds.\n',
+ },
+ {
+ source: 'stdout',
+ text: '✔ Test run with 1 test in 1 suite passed after 0.001 seconds.\n',
+ },
+ ]);
+
+ const progress = events.filter((event) => event.fragment === 'test-progress');
+ expect(progress).toEqual([
+ expect.objectContaining({ completed: 1, failed: 0, skipped: 0 }),
+ expect.objectContaining({ completed: 2, failed: 0, skipped: 0 }),
+ expect.objectContaining({ completed: 2, failed: 0, skipped: 0 }),
+ ]);
+ });
+
+ it('does not double-count xcodebuild-formatted test lines when a Swift Testing summary follows', () => {
+ const events = collectEvents('TEST', [
+ {
+ source: 'stdout',
+ text: "Test case 'WeatherTests/emptySearchReturnsNoResults()' passed on 'Clone 1' (0.001 seconds)\n",
+ },
+ {
+ source: 'stdout',
+ text: '✔ Test run with 1 test in 1 suite passed after 0.001 seconds.\n',
+ },
+ ]);
+
+ const progress = events.filter((event) => event.fragment === 'test-progress');
+ expect(progress).toEqual([
+ expect.objectContaining({ completed: 1, failed: 0, skipped: 0 }),
+ expect.objectContaining({ completed: 1, failed: 0, skipped: 0 }),
+ ]);
+ });
+
+ it('counts additional failures reported only by a Swift Testing summary', () => {
+ const events = collectEvents('TEST', [
+ {
+ source: 'stdout',
+ text: '✘ Test "Individually reported failure" failed after 0.001 seconds with 1 issue.\n',
+ },
+ {
+ source: 'stdout',
+ text: '✘ Test run with 2 tests in 1 suite failed after 0.001 seconds with 2 issues.\n',
+ },
+ ]);
+
+ const progress = events.filter((event) => event.fragment === 'test-progress');
+ expect(progress).toEqual([
+ expect.objectContaining({ completed: 1, failed: 1, skipped: 0 }),
+ expect.objectContaining({ completed: 2, failed: 2, skipped: 0 }),
+ ]);
+ });
+
it('processes full test lifecycle', () => {
const events = collectEvents('TEST', [
{ source: 'stdout', text: 'Resolve Package Graph\n' },
diff --git a/src/utils/renderers/__tests__/cli-text-renderer.test.ts b/src/utils/renderers/__tests__/cli-text-renderer.test.ts
--- a/src/utils/renderers/__tests__/cli-text-renderer.test.ts
+++ b/src/utils/renderers/__tests__/cli-text-renderer.test.ts
@@ -145,6 +145,66 @@
expect(output).toContain('\u{2705} Resolving app path\n');
});
+ it('replaces interactive build-stage updates with test progress updates', () => {
+ const renderer = createCliTextRenderer({ interactive: true });
+
+ renderer.onFragment({
+ kind: 'test-result',
+ fragment: 'build-stage',
+ operation: 'TEST',
+ stage: 'LINKING',
+ message: 'Linking',
+ });
+
+ renderer.onFragment({
+ kind: 'test-result',
+ fragment: 'test-progress',
+ operation: 'TEST',
+ completed: 4,
+ failed: 0,
+ skipped: 0,
+ });
+
+ expect(reporter.update).toHaveBeenCalledWith('Linking...');
+ expect(reporter.update).toHaveBeenCalledWith(
+ 'Running tests (4 completed, 0 failures, 0 skipped)',
+ );
+ });
+
... diff truncated: showing 800 of 1192 linesYou can send follow-ups to the cloud agent here.
…ource parameter The recordTestCaseResult function now only increments testCasesCompletedSinceSwiftTestingSummary and testCasesFailedSinceSwiftTestingSummary when source is 'swift-testing', preventing XCTest results from being incorrectly subtracted from Swift Testing summary counts.
Swift Testing result lines can include parameterized case metadata such as "with 4 test cases", but Xcode's discovery and run summaries count that output as one reported test item. Count each result line once so CLI progress and final summaries stay aligned with discovered tests. Fixes GH-384 Co-Authored-By: OpenAI Codex <codex@openai.com>
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Resolved by another fix: Xcodebuild-format Swift Testing lines double-counted with summaries
- The bug was already fixed in the remote branch with a more sophisticated pattern check that also validates the Swift Testing name format.
You can send follow-ups to the cloud agent here.
Reviewed by Cursor Bugbot for commit 3038cac. Configure here.
Lowercase xcodebuild test case lines can represent Swift Testing results when they use slash-separated Swift test identifiers. Treat those lines as Swift Testing observations so a following Swift Testing run summary does not count the same tests again. Fixes GH-384 Co-Authored-By: OpenAI Codex <codex@openai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Make simulator test runs visibly enter the test-execution phase instead of leaving the last build stage on screen while
test-without-buildingis running.This fixes the Weather example case from #384, where parallel UI tests run for a long time but xcodebuild delays per-test result lines until near the end. The CLI now emits a
RUN_TESTSstage as soon as the two-phase simulator flow moves frombuild-for-testingtotest-without-building, then replaces that with parsed per-test progress when xcodebuild eventually emits test-case lines.The patch also keeps preflight discovery observational, improves Swift Testing/XCTest discovery and parsing coverage, and adds a shared Weather scheme so the example project can be used as a regression fixture for this path.
Validated with the unit suite, typecheck, lint, format check, build, and a Weather JSONL smoke run that now streams
LINKINGfollowed byRUN_TESTSbefore the delayed per-test progress events.Fixes #384