Skip to content

feat(tests): add pytest-codeblocks for documentation code snippet CI#198

Open
haoyu-haoyu wants to merge 4 commits intodotimplement:mainfrom
haoyu-haoyu:feat/doc-code-ci
Open

feat(tests): add pytest-codeblocks for documentation code snippet CI#198
haoyu-haoyu wants to merge 4 commits intodotimplement:mainfrom
haoyu-haoyu:feat/doc-code-ci

Conversation

@haoyu-haoyu
Copy link
Copy Markdown

Summary

Add automated testing infrastructure for documentation code examples using pytest-codeblocks, so code snippets stay in sync with the evolving codebase.

Changes

New dependency

  • pytest-codeblocks>=0.17.0,<0.18 added to dev dependencies

CI integration

  • New step in .github/workflows/ci.yml runs pytest --codeblocks on docs/quickstart.md, docs/cookbook/, and docs/tutorials/
  • Runs on Python 3.12 only (pytest-codeblocks not verified for 3.13)
  • Uses continue-on-error: true so snippets can be incrementally enabled

Skip markers

  • 56 Python code blocks across 11 doc files marked with <!--pytest.mark.skip--> (these require healthchain package + external services)
  • Files covered: quickstart, all 5 cookbooks, 6 tutorial pages

pytest configuration

  • Added [tool.pytest.ini_options] with codeblocks marker in pyproject.toml

Usage

# Test all doc snippets
uv run pytest --codeblocks docs/quickstart.md docs/cookbook/ docs/tutorials/

# Test a specific file
uv run pytest --codeblocks docs/quickstart.md

To make a snippet testable, remove its <!--pytest.mark.skip--> marker and ensure it runs standalone.

Design decisions

  • Skip-first approach: All existing snippets are skipped initially since they depend on the healthchain package, FHIR servers, spaCy models, etc. Snippets can be incrementally enabled as they are made self-contained.
  • Scoped to curated docs: Only quickstart, cookbook, and tutorials are tested — reference/API docs are excluded to avoid executing auto-generated content.
  • Single Python version: Runs on 3.12 only to avoid compatibility issues with the pytest-codeblocks package.

Closes #164

Add automated testing infrastructure for documentation code examples
using pytest-codeblocks:

- Add pytest-codeblocks>=0.17.0 to dev dependencies
- Add [tool.pytest.ini_options] with codeblocks marker
- Add doc snippet test step to CI workflow
- Mark 56 Python code blocks across 11 doc files with skip markers
  (these require healthchain package + external services to run)

The CI step runs with continue-on-error initially so snippets can be
incrementally enabled as they are made self-contained.

Usage:
  uv run pytest --codeblocks docs/   # Test all doc snippets
  uv run pytest --codeblocks docs/quickstart.md  # Test specific file

To make a snippet testable, remove its <!-- pytest-codeblocks:skip -->
marker and ensure it runs standalone (no external dependencies).

Closes dotimplement#164
- Fix skip marker format: replace legacy `pytest-codeblocks:skip`
  with current `pytest.mark.skip` syntax (56 markers updated)
- Restrict CI doc test to Python 3.12 only (pytest-codeblocks not
  verified for 3.13)
- Narrow CI scope to quickstart + cookbook + tutorials (avoids
  executing shell blocks in reference docs)
- Update pyproject.toml comment with correct skip syntax
- Add skip markers to 3 missed cookbook files (format_conversion,
  index, setup_fhir_sandboxes) — 10 Python blocks
- Add skip markers to all bash/sh/shell blocks in scoped docs
  to prevent pytest-codeblocks from executing install/setup scripts
Copy link
Copy Markdown
Contributor

@adamkells adamkells left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @haoyu-haoyu great work on this! Apologies for the delay in reviewing. I left a couple of minor comments. I think it would be great to have at least one example unskipped so that we can verify that the changes are working correctly.

Comment thread pyproject.toml Outdated
# Snippets can be skipped with: <!--pytest.mark.skip-->
# Entire files can be skipped with: <!--pytest-codeblocks:skipfile-->
markers = [
"codeblocks: marks tests extracted from documentation code blocks",
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if this option does anything as we don't use a codeblocks marker in any of our existing pytests. Can you confirm what this addition does?

Comment thread docs/cookbook/ml_model_deployment.md Outdated

```bash
<!--pytest.mark.skip-->
```bash
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are a few instances like this where the indentation has been thrown off a little. Can you check these?

1. Unskip 3 runnable examples in docs/quickstart.md so CI actually
   verifies pytest-codeblocks integration:
     - Pipeline quick-inline (line ~61)
     - sandbox.list_available_datasets (line ~207)
     - fhir.create_condition helper (line ~264)
   All three are in-memory only (no network, no filesystem deps).

   Also fix a latent bug in the pipeline snippet: create_condition()
   requires subject=..., and problem_list is a getter that returns a
   fresh list per call — append() on it doesn't persist.  Changed to
   the setter: `doc.fhir.problem_list = [condition]`.

2. Remove dead [tool.pytest.ini_options] markers block from
   pyproject.toml.  pytest-codeblocks uses the --codeblocks CLI flag,
   not a pytest marker, and no tests reference @pytest.mark.codeblocks.

3. Fix markdown code-fence indent mismatches introduced by the earlier
   skip-marker insertion pass.  Walked every docs/**/*.md, matched
   each opener/closer pair, re-indented openers to match closer
   indent.  18 fixes across 7 files (ml_model_deployment,
   multi_ehr_aggregation, setup_fhir_sandboxes, and 4 tutorial
   files).  Restores correct rendering of snippets nested under
   admonitions / numbered lists.

4. Drop `continue-on-error: true` from the codeblocks CI step.
   Skipped snippets are already non-blocking by design, so the flag
   only masked real failures in the now-live examples.

Verified:
  uv run pytest --codeblocks docs/quickstart.md docs/cookbook/ \\
    docs/tutorials/                             → 3 passed, 107 skipped
  uv run pytest                                 → 746 passed, 13 skipped
  uv run ruff check .                           → clean
@haoyu-haoyu
Copy link
Copy Markdown
Author

Thanks @adamkells! Pushed fixes for all three in bf857b6:

1. At least one real unskipped example — unskipped 3 snippets in docs/quickstart.md:

  • Pipeline quick-inline (line ~61) — also fixed a latent doc bug: create_condition() requires subject=, and problem_list is a getter that returns a fresh list per call so .append() didn't persist. Switched to the setter: doc.fhir.problem_list = [condition] with a short comment explaining why.
  • sandbox.list_available_datasets() (line ~207)
  • fhir.create_condition helper (line ~264)

All three are in-memory only (no network, no filesystem deps), so CI won't get flaky.

2. Dead [tool.pytest.ini_options] block — good catch, removed. You were right: pytest-codeblocks uses the --codeblocks CLI flag, not a marker, and no tests used @pytest.mark.codeblocks. It was pure cargo-culting on my part.

3. Indentation — there were 18 mismatched fence pairs across 7 files (all from my earlier skip-marker insertion pass — I un-indented openers but left content/closers at the original indent, breaking any block nested under admonitions or lists). Wrote a pass over docs/**/*.md that matched each fence pair and re-indented the opener to match the closer. Verified 0 mismatches remain.

Bonus — also dropped continue-on-error: true from the codeblocks CI step. Skipped snippets are already non-blocking by design, so the flag only masked real failures in the now-live examples.

Verified locally:

uv run pytest --codeblocks docs/quickstart.md docs/cookbook/ docs/tutorials/
→ 3 passed, 107 skipped

uv run pytest  → 746 passed, 13 skipped
uv run ruff check .  → All checks passed

Happy to unskip more examples in follow-up PRs as we confirm they work end-to-end (the cookbook ones mostly need external services so they'll stay skipped for now).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add CI for documentation code snippets

2 participants