Skip to content

fix(core): preserve type through offloading + reject removed messageTypeField#429

Open
kibertoad wants to merge 3 commits intomainfrom
types/reject-removed-message-type-field
Open

fix(core): preserve type through offloading + reject removed messageTypeField#429
kibertoad wants to merge 3 commits intomainfrom
types/reject-removed-message-type-field

Conversation

@kibertoad
Copy link
Copy Markdown
Owner

@kibertoad kibertoad commented Apr 28, 2026

Summary

Two related fixes for the same regression class — silent drop of the message `type` field through payload offloading.

1. Type-level rejection of the removed messageTypeField option

messageTypeField was removed in core 25.x in favor of messageTypeResolver. Until now the option was simply absent from CommonQueueOptions, but TypeScript's excess-property checks are weakened across the long Omit/&/generic chains used by publisher options (e.g. SnsPublisherManager.newPublisherOptions), so callers who hadn't migrated kept silently passing a no-op option.

Fix: type `messageTypeField` as a removal-marker string-literal (`'REMOVED_USE_messageTypeResolver_INSTEAD'`). Passing it now fails compilation with a diagnostic that surfaces the migration hint at the call site.

2. Consistent type-field preservation through offloading

AbstractQueueService.offloadMessagePayloadIfNeeded preserved the type field only when messageTypeResolver was the messageTypePath flavor. With no resolver, or with literal / resolver modes, the field was silently stripped. This was inconsistent with:

  • the rest of the offloader, which copies messageIdField / messageTimestampField / messageDeduplicationIdField / messageDeduplicationOptionsField unconditionally based on their defaulted field names (`'id'`, `'timestamp'`, ...);
  • MessageSchemaContainer.resolveSchema, which falls back to a single default schema when no resolver is configured.

Fix: default the type-preservation path to the conventional top-level `type` when the resolver doesn't provide a body path. The literal/resolver modes still drive routing/dispatch as before; this only keeps the field present for downstream consumers (notably SNS subscriptions whose FilterPolicy filters on the body `type`).

Production-confirmed regression in file-storage-service: after the 24.x → 25.x bump, large `statistic.generated` messages were getting offloaded successfully and SNS-published successfully, but never reached SQS because the SQS subscription's `FilterPolicyScope: 'MessageBody'` filter on `type` no longer matched. See lokalise/file-storage-service#1027.

Test plan

  • `tsc --noEmit` clean in `packages/core` and `packages/sns` after the change.
  • New unit tests in `packages/core/test/queues/AbstractQueueService.offload.spec.ts` cover all four resolver modes (none / messageTypePath / literal / resolver), the nested path case, and the no-`type` source case. Verified that 3 of these fail without the offload fix and all pass with it.
  • New type-level tests in `packages/core/test/types/queueOptionsTypes.types.spec.ts` and `packages/sns/test/SnsPublisherManager.types.spec.ts` lock in compile-time rejection of the legacy option.
  • Full core test suite (`vitest run packages/core`): 168 passed.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Documentation

    • Deprecated the legacy messageTypeField option; migrate to messageTypeResolver (messageTypePath, literal, or resolver).
  • Bug Fixes

    • Offloaded message payloads now preserve the original message type when present, avoiding downstream filter mismatches.
  • Tests

    • Added compile-time type tests and regression tests to ensure the deprecated option errors and offloading preserves the type.

…time

`messageTypeField` was removed in core 25.x in favor of
`messageTypeResolver`. Until now the option was simply absent from
`CommonQueueOptions`, but TypeScript's excess-property checks are
weakened across the long `Omit`/`&`/generic chains used by publisher
options (`SnsPublisherManager.newPublisherOptions` etc.), so callers
who hadn't migrated were silently passing a no-op option.

The runtime impact is severe and silent: with payload offloading
enabled, the offload step in `AbstractQueueService` only preserves the
message `type` field on the offloaded body when `messageTypeResolver`
is configured. Callers still on `messageTypeField` end up publishing
SNS messages without `type` in the body, and any subscription with
`FilterPolicyScope: 'MessageBody'` filtering on `type` silently drops
the message before SQS delivery. Production-confirmed against
file-storage-service after its 24.x → 25.x bump.

Type the field as `?: never` with deprecation guidance so it now
fails compilation with a clear error pointing at the migration. Add
type-level guards in core and sns test suites that lock the contract
in place.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Apr 28, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: bd3d68f0-dd17-40d4-a5a6-eb5fe0809f35

📥 Commits

Reviewing files that changed from the base of the PR and between e8358a7 and df6dcf9.

📒 Files selected for processing (3)
  • packages/core/lib/queues/AbstractQueueService.ts
  • packages/core/lib/types/queueOptionsTypes.ts
  • packages/core/test/queues/AbstractQueueService.offload.spec.ts

📝 Walkthrough

Walkthrough

Adds a deprecated removal-marker messageTypeField to core types, enforces its absence via new type-level tests, updates AbstractQueueService offload logic to preserve top-level type into offloaded payloads, and adds runtime/type tests for the offload behavior and SNS publisher options.

Changes

Cohort / File(s) Summary
Core types
packages/core/lib/types/queueOptionsTypes.ts
Adds deprecated removal-marker messageTypeField?: '__REMOVED_USE_messageTypeResolver_INSTEAD__' to CommonQueueOptions to force compile-time errors for legacy usage.
Type-level tests (core + sns)
packages/core/test/types/queueOptionsTypes.types.spec.ts, packages/sns/test/SnsPublisherManager.types.spec.ts
New *.types.spec.ts files asserting messageTypeField triggers @ts-expect-error and validating allowed messageTypeResolver shapes and publisher option typings.
Offload logic
packages/core/lib/queues/AbstractQueueService.ts
Changes offloading to compute a typePath that falls back to top-level 'type' and copies that value into the offloaded payload when present, ensuring type is preserved unless absent.
Offload tests
packages/core/test/queues/AbstractQueueService.offload.spec.ts
New regression tests forcing payload offload and asserting the offloaded payload retains original type when present and does not fabricate it when absent.

Sequence Diagram(s)

sequenceDiagram
  participant Publisher as Publisher
  participant Queue as AbstractQueueService
  participant Store as PayloadStore
  participant SNS as SNS (topic)

  Publisher->>Queue: publish(message, options)
  Queue->>Queue: determine messageTypePath (resolver or fallback 'type')
  alt message exceeds threshold -> offload
    Queue->>Store: store(payload) -> returns payloadRef
    Store-->>Queue: payloadRef
    Queue->>SNS: send({ payloadRef, type: message[typePath]? })
  else small message -> inline
    Queue->>SNS: send(message)
  end
  SNS-->>Publisher: ack
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

  • Improved message logging #385: Modifies AbstractQueueService.offloadMessagePayloadIfNeeded similarly to preserve message metadata/type during offload.
  • Handle edge cases #374: Related removal of legacy messageTypeField and migration toward messageTypeResolver handling and offload behavior.
  • Add flexible type resolution #373: Continues migration from messageTypeField to messageTypeResolver, overlapping type and offload adjustments.

Suggested labels

minor

Suggested reviewers

  • kjamrog
  • CarlosGamero

Poem

🐰 I found a field that hopped away,
I marked it gone with strings that say,
The type hops home when payload's stored,
No more surprises on the board,
Hooray — a tidy trail today! 🎉

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately describes the two primary changes: preserving type through offloading and rejecting the removed messageTypeField option at compile time.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch types/reject-removed-message-type-field

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@packages/core/lib/types/queueOptionsTypes.ts`:
- Around line 117-126: The field messageTypeField?: never does not fully prevent
callers from passing messageTypeField: undefined unless TypeScript's
exactOptionalPropertyTypes is enabled; either enable
"exactOptionalPropertyTypes": true in the packages/core/tsconfig.json to make
optional properties typed as exact and enforce the never ban, or replace the
declaration with a stricter pattern (e.g., a branded/deprecated type or separate
deprecated interface) so passing undefined is a compile-time error—update the
definition of messageTypeField in types/queueOptionsTypes.ts (and update any
related interfaces) to use the chosen strict approach.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 496c146c-d9b6-4c07-8a99-6577601ad056

📥 Commits

Reviewing files that changed from the base of the PR and between c78b37c and e8358a7.

📒 Files selected for processing (3)
  • packages/core/lib/types/queueOptionsTypes.ts
  • packages/core/test/types/queueOptionsTypes.types.spec.ts
  • packages/sns/test/SnsPublisherManager.types.spec.ts

Comment thread packages/core/lib/types/queueOptionsTypes.ts Outdated
@kibertoad kibertoad requested a review from CarlosGamero April 28, 2026 20:31
Igor Savin and others added 2 commits April 28, 2026 23:35
…ssage

Per PR review: `messageTypeField?: never` produces a generic "Type
'string' is not assignable to type 'never'" diagnostic. Switch to a
removal-marker string literal so the diagnostic surfaces the migration
hint right at the offending call site, e.g. "Type 'type' is not
assignable to type '__REMOVED_USE_messageTypeResolver_INSTEAD__ |
undefined'".

Note on the `messageTypeField: undefined` loophole: closing it would
require enabling `exactOptionalPropertyTypes` package-wide, which
produces ~13 unrelated type errors in core (HandlerContainer,
MessageSchemaContainer, AbstractQueueService, startupResourcePolling)
and is a separate refactor. Passing `undefined` is operationally
identical to omitting the field, so this guard catches the realistic
regression — `messageTypeField: '<name>'`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…mode

`offloadMessagePayloadIfNeeded` previously only re-attached the message
`type` field on the offloaded payload when `messageTypeResolver` was a
`messageTypePath` config. With no resolver, or with `literal` /
`resolver` modes, the field was silently stripped.

This was inconsistent with the rest of the offloader, which copies
identity fields (`messageIdField`, `messageTimestampField`,
`messageDeduplicationIdField`, `messageDeduplicationOptionsField`)
unconditionally based on their defaulted field names ('id',
'timestamp', ...), and with `MessageSchemaContainer.resolveSchema`,
which falls back to a single default schema when no resolver is
configured.

Symptoms: large messages get offloaded to S3 and the SNS publish
succeeds, but any subscription with `FilterPolicyScope: 'MessageBody'`
filtering on `type` silently drops them — the body the filter sees
no longer contains a `type` field. Production-confirmed against
file-storage-service after its 24.x → 25.x bump
(lokalise/file-storage-service#1027).

Default the type-preservation path to the conventional top-level `type`
when the resolver doesn't provide a body path. The literal/resolver
modes still drive routing/dispatch as before; this is purely about
keeping the field available for downstream filters.

Add direct unit tests for the offload behavior across all four resolver
configurations.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@kibertoad kibertoad changed the title types: reject removed messageTypeField at compile time fix(core): preserve type through offloading + reject removed messageTypeField Apr 28, 2026
Comment on lines +30 to +35
it('rejects the removed `messageTypeField` option', () => {
const _bad: NewPublisherOptions = {
// @ts-expect-error - `messageTypeField` was removed in core 25.x; use `messageTypeResolver` instead
messageTypeField: 'type',
messageIdField: 'id',
}
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we use the expect type of option from vitest?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants