Adblock Compiler (currently in code freeze at v0.79.4 for e2e testing and refactoring)

Compiler-as-a-Service for adblock filter lists. Transform, optimize, and combine filter lists from multiple sources with real-time progress tracking.

🌐 Try the Admin Dashboard | 🔧 Compiler UI | 🚀 API Endpoint | 📚 Documentation | WIP

✨ Features

• 🎯 Multi-Source Compilation - Combine filter lists from URLs, files, or inline rules
• ⚡ Performance - Gzip compression (70-80% cache reduction), request deduplication, smart caching
• 🔄 Circuit Breaker - Automatic retry with exponential backoff for unreliable sources
• 📊 Visual Diff - See what changed between compilations
• 🎪 Batch Processing - Compile up to 10 lists in parallel
• 📡 Real-time Updates - Server-Sent Events (SSE) and WebSocket support
• 🔔 Async Notifications - Get notified when background jobs complete
• 🌐 Admin Dashboard - Monitor metrics, queue depth, and system health
• 📖 OpenAPI 3.0 Specification - Full API documentation with contract tests
• 🌍 Universal - Works in Deno, Node.js, Cloudflare Workers, browsers
• 🖥️ Angular 21 SPA - Production frontend with zoneless change detection, Material Design 3, and SSR
• 🎨 11 Transformations - Deduplicate, compress, validate, and more
• 📝 Structured Logging - Production-ready JSON logs for observability (CloudWatch, Datadog, Splunk)
• 🚨 Error Reporting - Centralized error tracking with Sentry and Cloudflare Analytics Engine
• ✅ Zod Validation - Runtime schema validation for all configurations and API inputs
• 🧩 AST Rule Parsing - Full abstract syntax tree parsing via @adguard/agtree
• ⚙️ Cloudflare Workflows - Durable execution for long-running and background compilations

📚 Documentation

Full documentation is available in mdBook format at this link and in the /docs directory.

🚀 Quick Start

Installation

Run directly without installation:

deno run --allow-read --allow-write --allow-net jsr:@jk-com/adblock-compiler -c config.json -o output.txt

Or install globally:

deno install --allow-read --allow-write --allow-net -n adblock-compiler jsr:@jk-com/adblock-compiler/cli

Or run with Docker:

docker compose up -d

Access the web UI at http://localhost:8787

📚 Quick Start Guide | 📚 Docker Deployment Guide

Basic Usage

Quick hosts conversion

adblock-compiler -i hosts.txt -i hosts2.txt -o output.txt

Build a configurable blocklist from multiple sources

adblock-compiler -c configuration.json -o output.txt

CLI Options

Usage: adblock-compiler [options]

General:
  -c, --config <file>          Path to the compiler configuration file
  -i, --input <source>         URL or file path to convert (repeatable)
  -t, --input-type <type>      Input format: hosts|adblock [default: hosts]
  -v, --verbose                Enable verbose logging
  -b, --benchmark              Show performance benchmark report
  -q, --use-queue              Use asynchronous queue-based compilation
      --priority <level>       Queue priority: standard|high [default: standard]
      --version                Show version number
  -h, --help                   Show help

Output:
  -o, --output <file>          Path to the output file [required unless --stdout]
      --stdout                 Write output to stdout instead of a file
      --append                 Append to output file instead of overwriting
      --format <format>        Output format
      --name <file>            Compare output against existing file and print diff
      --max-rules <n>          Truncate output to at most <n> rules

Transformations:
      --no-deduplicate         Skip the Deduplicate transformation
      --no-validate            Skip the Validate transformation
      --no-compress            Skip the Compress transformation
      --no-comments            Skip the RemoveComments transformation
      --invert-allow           Apply the InvertAllow transformation
      --remove-modifiers       Apply the RemoveModifiers transformation
      --allow-ip               Use ValidateAllowIp instead of Validate
      --convert-to-ascii       Apply the ConvertToAscii transformation
      --transformation <name>  Explicit transformation pipeline (repeatable,
                               overrides all other transformation flags)

Filtering:
      --exclude <pattern>      Exclude rules matching pattern (repeatable)
      --exclude-from <file>    Load exclusions from file (repeatable)
      --include <pattern>      Include only rules matching pattern (repeatable)
      --include-from <file>    Load inclusions from file (repeatable)

Networking:
      --timeout <ms>           HTTP request timeout in milliseconds
      --retries <n>            Number of HTTP retry attempts
      --user-agent <string>    Custom HTTP User-Agent header

Authentication (when using --use-queue with a remote API):
      --api-key <key>          API key (abc_...) for authenticated requests
      --bearer-token <token>   Clerk JWT Bearer token for authenticated requests
      --api-url <url>          Worker API base URL [default: http://localhost:8787]

Examples:
  adblock-compiler -c config.json -o output.txt
      compile a blocklist and write the output to output.txt

  adblock-compiler -i hosts.txt -i hosts2.txt --stdout
      compile from multiple inputs and stream to stdout

  adblock-compiler -c config.json -o output.txt --no-compress --allow-ip
      compile without compression, keeping IP-address rules

  adblock-compiler -i https://example.org/hosts.txt -o output.txt \
      --transformation RemoveComments --transformation Deduplicate
      compile with an explicit transformation pipeline

📚 See the CLI Reference for the full flag reference and advanced examples.

📖 Further Reading

Topic	Doc
CLI reference	docs/usage/CLI.md
Configuration reference	docs/usage/CONFIGURATION.md
Transformations reference	docs/usage/TRANSFORMATIONS.md
TypeScript API & Zod validation	docs/api/README.md
OpenAPI specification	docs/api/OPENAPI_TOOLING.md
Platform support & custom fetchers	docs/api/PLATFORM_SUPPORT.md
Extensibility	docs/development/EXTENSIBILITY.md
Structured logging & OpenTelemetry	docs/development/LOGGING.md
Error reporting	docs/development/ERROR_REPORTING.md
Docker deployment	docs/deployment/docker.md
Cloudflare Workers deployment	docs/deployment/cloudflare-pages.md
Angular frontend	frontend/README.md

⚙️ CI/CD Workflows

Workflow	Trigger	Purpose
ci.yml	Push/PR to main, workflow_dispatch	Lint, type-check, test, build, deploy to Cloudflare, publish to JSR. Includes ZTA security lint via the zta-checks composite action.
docker-publish.yml	Push/PR to main on Dockerfile/Dockerfile.container paths, v*.*.* tags, daily schedule	Build and push Docker images to GHCR and Docker Hub; build Cloudflare Containers image; cosign signing; daily security rebuilds.
release.yml	v* and compiler-v* tags, workflow_dispatch	Compile cross-platform binaries and create GitHub Releases. Docker images are handled by docker-publish.yml.
bench.yml	Push/PR to main on src/** changes, workflow_dispatch	Run deno bench and upload results as artifacts.
lighthouse.yml	After CI completes on main, workflow_dispatch	Lighthouse performance audit of the deployed frontend.
codeql.yml	Push/PR to main, weekly schedule	CodeQL static analysis security scan.
api-shield-scan.yml	Push/PR to main on API spec changes, workflow_dispatch	Upload OpenAPI schema to Cloudflare API Shield.
sentry-frontend.yml	Push to main on frontend/**, v* tags, workflow_dispatch	Upload Angular browser + SSR source maps to Sentry.
sentry-worker.yml	Push to main on worker/**, v* and compiler-v* tags, workflow_dispatch	Build worker bundle and upload source maps to Sentry.
version-bump.yml	Push to main and master, workflow_dispatch	Automatically bump deno.json version from conventional commit messages.
frontend-version-bump.yml	Push to main on frontend/**, workflow_dispatch	Sync frontend/package.json version with deno.json.
create-version-tag.yml	Version-bump PR merged, workflow_dispatch	Create a Git tag after a version bump.
gradual-deploy.yml	workflow_dispatch	Gradually roll out a Worker deployment via Cloudflare traffic management.
db-migrate.yml	Push/PR to main on migration paths, release published, workflow_dispatch	Run database migrations against the production Neon instance.
cloudflare-dep-update.yml	Daily schedule, workflow_dispatch	Auto-update Cloudflare dependency pins and open a PR.
mdbook.yml	Push to main on docs/**, workflow_dispatch	Build and deploy the mdBook documentation site.
neon-branch-create.yml	PR opened/synchronize on Prisma paths, workflow_dispatch	Create a Neon database branch for preview environments.
neon-branch-cleanup.yml	PR closed on Prisma paths, branch deleted, workflow_dispatch	Delete the Neon preview branch.
cleanup-branches.yml	PR closed, workflow_dispatch	Delete head branches after PR merge and clean stale branches.
lint-workflows.yml	Push/PR on .github/workflows/**, .github/actions/**	Lint workflow YAML files.
claude.yml	Issue/PR comments, issues, PR review events, push on .github/workflows/claude.yml	Claude AI code assistant.

Note: zta-lint.yml was removed — its ZTA security checks are fully consolidated into the zta-checks composite action run by ci.yml. The build-docker job was removed from release.yml — Docker image builds are owned by docker-publish.yml. This table summarizes the primary triggers; for the complete configuration check each workflow file's on: block.

🏗️ Tech Stack

Layer	Technology	Notes
Runtime	Deno 2.x, Cloudflare Workers, Typescript 6.0	Edge-first execution, Deno runtime, Typescript 6.x
Framework	Hono, tRPC	Lightweight web framework for Workers
Database	Neon PostgreSQL (primary), Cloudflare D1 (edge cache)	Neon via Hyperdrive connection pooling
ORM	Prisma 7.x	@prisma/adapter-pg for PostgreSQL, @prisma/adapter-d1 for D1
Authentication	Better Auth (primary)	Better Auth for new sessions; Clerk for legacy tokens during migration
Storage	Cloudflare R2, KV	R2 for compiled outputs, KV for caching & rate limits
Frontend	Angular 21, Vite, TailwindCSS	Zoneless change detection, Material Design 3, SSR on Workers
Validation	Zod	Runtime schema validation at all trust boundaries
API	OpenAPI 3.0, Postman	OpenAPI 3.0 specification
Routing	Hono Router	Hono router for Typescript
Testing	Deno 2.x, Vitest, Postman, Playwright	Deno test for backend, Vitest for frontend, Postman for API, Playwright for e2e
Security	Full ZTA, JWT	Zero-trust architecture at every layer/API call via JWT + Better Auth
Observability	Sentry, Prometheus, Cloudflare Observatory	Prometheus, Sentry, and Cloudflare Observatory log pushing
DevOps	Github, CodeQL, Lighthouse	DevOps hosted on Github with CodeQL and Lighthouse integration
Architecture	Edge First, Workers Modules, Cloud Native, ZTA, Microservices	Cloudlfare-native microservices hosted in Workers, ZTA at every boundary

🔧 Development

Prerequisites

• Deno 2.0 or later

Available tasks

Note: Always use deno task test instead of deno test directly. The tests require specific permissions (--allow-read, --allow-write, --allow-net, --allow-env) that are configured in the task.

# Run in development mode with watch
deno task dev

# Run the compiler
deno task compile

# Build standalone executable
deno task build

# Run tests (all tests co-located with source files in src/)
deno task test

# Run tests in watch mode
deno task test:watch

# Run tests with coverage
deno task test:coverage

# Run specific test file (with required permissions)
deno test --allow-read --allow-write --allow-net --allow-env src/cli/ArgumentParser.test.ts

# Lint code
deno task lint

# Format code
deno task fmt

# Check formatting
deno task fmt:check

# Type check
deno task check

# Cache dependencies
deno task cache

# Generate the TypeScript API reference (into book/api-reference/)
deno task docs:api

# Build the full mdBook site + API reference (requires mdBook CLI installed)
deno task docs:build

# Live-preview the mdBook (does not include API reference; requires mdBook CLI installed)
deno task docs:serve

Angular Frontend Development

The primary frontend is an Angular 21 SPA in frontend/. It uses:

• Angular 21 with zoneless change detection, signals, rxResource, linkedSignal
• Angular Material 3 for UI components and theming
• SSR via @angular/ssr on Cloudflare Workers
• Vitest for unit testing

# Development server (http://localhost:4200)
pnpm --filter adblock-frontend run start

# Production build
pnpm --filter adblock-frontend run build

# Run tests
pnpm --filter adblock-frontend run test

For full-stack local development:

deno task wrangler:dev        # Worker on :8787
pnpm --filter adblock-frontend run start  # Angular on :4200 → proxies /api to :8787

Pages:

• Dashboard — live metrics from /api/metrics and /api/health
• Compiler — filter list compilation with JSON and SSE streaming modes, drag-and-drop
• Performance — real-time compilation latency and throughput
• Validation — AGTree-powered filter rule validation
• API Docs — HTTP endpoint reference
• Admin — KV/R2/D1 storage management (requires admin key)

The public/ directory contains the original vanilla HTML frontend. It will be removed once the Angular migration is complete.

📄 SPA Benefits Analysis - Analysis of SPA benefits for this application

Project structure

mindmap
  root((Project structure))
    src["src/"]
      cli["cli/ — Command-line interface (with *.test.ts files)"]
      compiler["compiler/ — Core compilation logic (with *.test.ts files)"]
      configuration["configuration/ — Configuration validation (with *.test.ts files)"]
      downloader["downloader/ — Filter list downloading (with *.test.ts files)"]
      platform["platform/ — Platform abstraction layer (with *.test.ts files)"]
      transformations["transformations/ — Rule transformations (with *.test.ts files)"]
      types["types/ — TypeScript type definitions"]
      utils["utils/ — Utility functions (with *.test.ts files)"]
      indexTs["index.ts — Main library exports"]
      modTs["mod.ts — Deno module exports"]
    worker["worker/ — Cloudflare Worker implementation (production-ready)"]
      workerTs["worker.ts — Main worker with API endpoints"]
      htmlTs["html.ts — Fallback HTML templates"]
    frontend["frontend/ — Angular 21 SPA (replaces public/)"]
      app["src/app/ — Components, services, guards, interceptors"]
      indexHtml["src/index.html — Root HTML with Cloudflare analytics"]
      angularJson["angular.json — Build configuration (SSR + browser)"]
    public["public/ — Legacy static web UI (to be removed)"]
    examples["examples/"]
      cloudflareWorker["cloudflare-worker/ — Legacy deployment reference"]

Loading

Publishing to JSR

# Dry run to verify everything is correct
deno publish --dry-run

# Publish to JSR
deno publish

🤝 Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines.

Quick Start for Contributors

1. Fork and clone the repository
2. Make your changes following our commit message guidelines
3. Use conventional commits: feat:, fix:, docs:, etc.
4. Submit a pull request

Automatic Version Bumping: When your PR is merged, the version will be automatically bumped based on your commit messages. See docs/reference/AUTO_VERSION_BUMP.md for details.

📄 License

MIT - See LICENSE for details.