seo(robots): disallow tag-filtered blog URLs

[emir-karabeg]

Summary

Adds Disallow: /blog*tag= to the generated robots.txt to prevent crawlers from indexing tag-filtered blog views (both /blog?tag=X and paginated /blog?page=N&tag=X). Per SEO team request to reduce duplicate content surfaced from tag filters.

Type of Change

• Other: SEO / robots.txt

Testing

Visit /robots.txt after deploy and confirm the new Disallow: /blog*tag= line is present.

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs	Skipped		Apr 21, 2026 10:08pm

[cursor]

PR Summary

Low Risk
Low risk SEO-only change that only adjusts robots.txt crawling rules; it could affect blog discoverability if the pattern is overly broad.

Overview
Updates the generated robots.txt to disallow crawling of tag-filtered blog URLs by adding '/blog*tag=' to the disallow list, preventing indexing of /blog?tag=... (including paginated variants).

^{Reviewed by Cursor Bugbot for commit eb3281b. Configure here.}

[greptile-apps]

Greptile Summary

Adds Disallow: /blog*tag= to the generated robots.txt via Next.js MetadataRoute.Robots, preventing crawlers from indexing tag-filtered blog pages (e.g. /blog?tag=X, /blog?page=N&tag=X). The wildcard pattern is a widely-supported Google/Bing extension and Next.js renders it verbatim in the output file.

Confidence Score: 5/5

Safe to merge — single-line addition to an SEO config file with no logic changes.

The change is minimal and correct: the wildcard pattern /blog*tag= is a widely-supported robots.txt extension that matches all tag-filtered blog URLs as intended. Next.js renders disallow strings verbatim so the * is preserved. No logic, auth, or data paths are affected.

No files require special attention.

Important Files Changed

Filename	Overview
apps/sim/app/robots.ts	Adds /blog*tag= to the disallow list to prevent crawlers from indexing tag-filtered blog URLs; single-line, low-risk SEO change.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Crawler requests /robots.txt] --> B[Next.js robots function]
    B --> C{URL pattern match}
    C -->|matches /blog*tag=| D[Disallowed - not crawled]
    C -->|matches /api/ or /workspace/ etc| D
    C -->|no disallow match| E[Allowed - crawled and indexed]
    D --> F[Crawler skips URL]
    E --> G[Crawler indexes page]

Loading

_{Reviews (1): Last reviewed commit: "seo(robots): disallow tag-filtered blog ..." | Re-trigger Greptile}