Skip to content

feat: add evaluations client passthrough and tests#430

Open
nborges-aws wants to merge 2 commits intomainfrom
feat/evaluationsClient
Open

feat: add evaluations client passthrough and tests#430
nborges-aws wants to merge 2 commits intomainfrom
feat/evaluationsClient

Conversation

@nborges-aws
Copy link
Copy Markdown
Contributor

Description of changes:

  • Add __getattr__ passthrough with accept_snake_case_kwargs to EvaluationClient
  • Control plane allowlisted methods (10):
    • Evaluator CRUD: create_evaluator, get_evaluator, list_evaluators, update_evaluator, delete_evaluator
    • Online evaluation config CRUD: create_online_evaluation_config, get_online_evaluation_config, list_online_evaluation_configs, update_online_evaluation_config, delete_online_evaluation_config
  • Data plane allowlisted methods (1):
    • evaluate
  • Add *_and_wait polling methods using shared wait_until/wait_until_deleted utilities:
    • create_evaluator_and_wait, update_evaluator_and_wait, delete_evaluator_and_wait
    • create_online_evaluation_config_and_wait, update_online_evaluation_config_and_wait, delete_online_evaluation_config_and_wait
  • All *_and_wait methods accept WaitConfig for configurable polling behavior and apply convert_kwargs for snake_case support
  • Changes are additive — existing run() method and its helpers are untouched

Test plan:

  • Unit tests (13 new):
    • Passthrough: CP forwarding, DP forwarding, snake_case conversion, AttributeError on unknown method, CP allowlist contents, DP allowlist contents
    • *_and_wait: create_evaluator_and_wait (success + failure), update_evaluator_and_wait, create_online_evaluation_config_and_wait, update_online_evaluation_config_and_wait, delete_evaluator_and_wait, delete_online_evaluation_config_and_wait
  • Integration tests (13 total):
    • Passthrough (5): list_evaluators, list_evaluators with snake_case, get_evaluator on built-in, list_online_evaluation_configs, non-allowlisted method raises
    • Evaluator CRUD (4): create_evaluator_and_wait → get_evaluator → update_evaluator_and_wait → delete_evaluator_and_wait
    • Online evaluation config CRUD (4): create_online_evaluation_config_and_wait → get_online_evaluation_config → update_online_evaluation_config_and_wait → delete_online_evaluation_config_and_wait

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@nborges-aws nborges-aws requested a review from a team April 23, 2026 18:44
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 23, 2026

✅ No Breaking Changes Detected

No public API breaking changes found in this PR.

QUERY_TIMEOUT_SECONDS = 60
POLL_INTERVAL_SECONDS = 2

_EVALUATOR_FAILED_STATUSES = {"CREATE_FAILED", "UPDATE_FAILED"}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we consolidate these two? Also, is there a DELETE_FAILED? I know there are for other primitives but not confident on evaluators.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep will consolidate. I checked the service models, evaluators has DELETING, but no DELETE_FAILED

evaluatorId=self.evaluator_ids[0],
description="updated by integ test",
)
assert updated["status"] == "ACTIVE"
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we verify the description actually updated?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep will add assertion

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants