Introduction
A rubric is a set of programmatic checks, like length limits, required terms, or schema validation, that an agent's output must satisfy. RubricLoop runs those checks in a loop: when an output fails, the failure details go back to the agent so it can self-correct before the result is used. This helps you achieve safety, control, and cost saving by catching bad outputs early and stopping runaway token spend.
Most verification relies on deterministic AI checks using exact code, schemas, and sandboxes so tests stay fast, repeatable, and cost-free without calling extra models. For scenarios where code alone cannot evaluate subtle criteria, or when an agent gets stuck in a loop, a hybrid approach pairs deterministic rules with a scoped LLM judge to provide targeted guidance while keeping hard rules as the final authority.
Quick start
Install the library, set your OpenAI API key, and pass your agent function to verify(). When an output fails a rule, RubricLoop sends the failure diagnostic back to the model so it can self-correct within your budget limits.
# Install RubricLoop and the OpenAI client
pip install rubricloop openai
# Set your API key
export OPENAI_API_KEY="sk-..."from openai import OpenAI
from rubricloop import AgentRequest, AgentResponse, Budget, rubric, verify
from rubricloop.checks import forbidden_terms, required_terms, word_count
client = OpenAI()
rules = rubric(
"support/reply",
word_count("reply.length", minimum=40, maximum=70),
required_terms("reply.timeline", ["timeline"]),
forbidden_terms("reply.promise", ["guaranteed refund"]),
)
def support_agent(request: AgentRequest) -> AgentResponse:
if request.feedback:
prompt = (
f"{request.prompt}\n\n"
"Revise the reply using these measured failures:\n"
f"{request.feedback}\n\n"
"Return 40 to 70 words total with no greeting, signature, or placeholders."
)
else:
prompt = (
f"{request.prompt}\n\n"
"For the first draft, use fewer than 25 words, include the exact phrase "
"'guaranteed refund', and do not use the word 'timeline'."
)
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{
"role": "system",
"content": (
"You write customer support replies. Follow correction diagnostics "
"from the latest user message. Return only the concise reply."
),
},
{"role": "user", "content": prompt},
],
temperature=0,
)
usage = response.usage
return AgentResponse(
response.choices[0].message.content or "",
prompt_tokens=usage.prompt_tokens if usage else 0,
completion_tokens=usage.completion_tokens if usage else 0,
)
run = verify(
support_agent,
(
"Write a helpful response to a customer whose order is delayed. Explain what "
"happens next without promising a refund."
),
rules,
budget=Budget(max_iterations=3, max_tokens=2_000),
)
print("Passed:", run.passed)
print("Iterations:", len(run.iterations))
print("Stop reason:", run.stop_reason)
print("Tokens:", run.total_tokens)
print("Final reply:\n", run.output)
if not run.passed:
raise SystemExit("The reply did not satisfy the rubric within the budget.")RubricLoop does not proxy your LLM calls or charge for tokens. Your agent function calls OpenAI directly using your own key. You can swap client.chat.completions.create with Anthropic, Ollama, or any custom model at any time.
How it works
Instead of static eval after the run is over, RubricLoop runs an in-flight verification loop. The agent drafts an answer, rules score it, and any failures become targeted feedback for the next turn.
Agent
Your generator function. It receives the prompt and previous feedback, then returns candidate output.
Rubric
A set of deterministic rules. Each rule returns a pass, an actionable failure, or an escalation.
Loop
Retries the agent until all checks pass, failures stop improving, or the token and iteration budget runs out.
Sandbox
An isolated execution environment (like SQLite) to test code or queries safely before touching production.
Write a rule
Rules are ordinary Python functions. They take the candidate output and any context you pass in, then return CheckResult.passed() or CheckResult.failed(). No custom DSL required.
from rubricloop import CheckResult, Rule, rubric
def order_total_matches(candidate, context):
order = context.data["order"]
expected = sum(item["price"] * item["quantity"] for item in order["items"])
if order["total"] != expected:
return CheckResult.failed(
f"total is {order['total']}, expected {expected}",
"recalculate total from the line items",
)
return CheckResult.passed("order total matches")
rules = rubric(
"commerce/order",
Rule("order.total", "order total", order_total_matches),
)Rules should verify logic using data already in memory or in a sandbox. Avoid calling external LLMs or third-party APIs inside a check so the retry loop stays fast and reliable.
Check SQL
When your agent generates database queries, run them against an in-memory SQLite sandbox first. The built-in sql_safe pack checks that the query is read-only, uses valid tables, and produces the expected columns.
from rubricloop import verify
from rubricloop.packs import sql_safe
from rubricloop.sandboxes import SQLiteSandbox
sandbox = SQLiteSandbox(
ddl="""
CREATE TABLE users (
id INTEGER PRIMARY KEY,
name TEXT NOT NULL,
country TEXT NOT NULL
);
""",
seed_sql="""
INSERT INTO users (name, country)
VALUES ('Ada', 'DE'), ('Grace', 'US');
""",
)
rules = sql_safe(
schema={"users": ["id", "name", "country"]},
reference_query="SELECT name FROM users WHERE country = 'DE'",
columns=["name"],
)
run = verify(
sql_agent,
"Return the names of users in Germany.",
rules,
sandbox=sandbox,
)SELECT *Policy gates
Some operations shouldn't be blindly retried by an LLM. For sensitive actions like refunds, payments, or data deletions, run a deterministic gate check before hitting your production API.
from datetime import datetime, timezone
from rubricloop.packs import RefundOrder, RefundPolicy, check_refund
policy = RefundPolicy(
refund_window_days=30,
max_auto_approve=50,
deny_above=500,
allowed_reasons=("damaged", "wrong_item", "not_received"),
non_refundable_skus=("FINAL-1",),
)
result = check_refund(
{
"action": "issue_refund",
"order_id": "ORD-100",
"amount": 80,
"reason": "damaged",
},
policy=policy,
order=RefundOrder(
id="ORD-100",
sku="STANDARD-1",
paid_amount=120,
refunded_amount=20,
purchased_at=datetime(2026, 9, 13, tzinfo=timezone.utc),
),
)
# Returns "approval" to keep the original amount and route to a human
print(result.decision)All rules passed. Safe to trigger the action automatically.
Request is valid, but exceeds auto-approval limits. Route to a human.
Violates hard policy constraints. Reject immediately.
If a check flags a request for human approval, route the original request to a person. Don't feed it back to the model to lower the amount or manipulate inputs to force a pass.
Read results
Calling verify() returns a RunResult with the final output, overall pass status, and residuals for any checks that didn't resolve.
run.passedTrue when every scored rule passed.run.outputThe final output string or payload.run.stop_reasonWhy the loop stopped: passed, max_iterations, token_budget, or cycle_rollback.run.residualsList of rules that still failed on the final output.run.iterationsFull history of prompts, outputs, and failures per turn.run.to_dict()Serialized dictionary for logging and traces.Stream events
Pass an on_event callback to receive real-time progress updates during execution. Useful for printing live terminal feedback or streaming status to a frontend.
def on_event(event):
if event["type"] == "iteration":
print(f"Iteration {event['index']}: passed={event['passed']} failures={event['failures']}")
run = verify(
agent,
prompt,
rules,
on_event=on_event,
)Production guards
Protect your systems against runaway loops, unexpected token costs, and hallucinated actions before deploying to users.
Always set max_iterations and max_tokens in Budget. If the agent runs out of turns, route to a human instead of guessing.
Populate sandboxes with synthetic seed data or read-only test fixtures. Never pass live database credentials into a sandbox.
Pin the exact rubric release or package digest in your code so behavior remains deterministic between deployments.
Rules are best for clear, objective criteria. When an agent faces subjective judgment calls, fail open to a human operator.
Package and share
To reuse rubrics across multiple repositories or share them with the open-source community, bundle them into signed .rlpack files with built-in tests and checksums.
# Validate bundle structure and checksums
rubricloop validate dist/policy-checks.rlpack
# Run test suites bundled in the package
rubricloop test dist/policy-checks.rlpack
# Push to the registry with a version tag
rubricloop --server https://api.rubricloop.dev push dist/policy-checks.rlpack --tag latest
# Request automated vetting for community release
rubricloop --server https://api.rubricloop.dev vet acme/policy-checks@latest
# Publish the approved digest to the community catalog
rubricloop --server https://api.rubricloop.dev publish acme/policy-checks@sha256:<bundle-digest> --scope public --reason "Ready for community use"rubricloop validate checks bundle integrity. rubricloop test executes all test suites defined in the pack before distribution.Hosted API
Run verifications over HTTP from any language or runtime, such as Node.js, Go, LangChain graphs, or n8n workflows. Send your output payload and context, and receive an instant decision.
curl -X POST https://api.rubricloop.dev/v1/verify \
-H "Content-Type: application/json" \
-H "X-API-Key: $RUBRICLOOP_API_KEY" \
-d '{
"pack_id": "support/refund-policy-v1",
"output": {
"action": "issue_refund",
"order_id": "ORD-100",
"amount": 40,
"reason": "damaged"
},
"context": {
"policy": {
"refund_window_days": 30,
"max_auto_approve": 50,
"deny_above": 500,
"allowed_reasons": ["damaged", "wrong_item"]
},
"order": {
"id": "ORD-100",
"sku": "STANDARD-1",
"paid_amount": 120,
"refunded_amount": 20,
"purchased_at": "2026-09-13T09:00:00Z"
}
}
}'POST /v1/verify lets you evaluate agents built in any framework without embedding the Python runtime.
Registry & CLI
The rubricloop CLI connects your local environment to the package registry. When you're ready to push packs or manage repositories, log in with an API key from your dashboard.
# Install the CLI (included with the Python package)
pip install rubricloop
# Authenticate with your API key
rubricloop --server https://api.rubricloop.dev login
# Scaffold a new rubric package
rubricloop init policy-checks
# Create its registry repository
rubricloop --server https://api.rubricloop.dev create \
acme/policy-checks \
--visibility privaterubricloop login stores tokens in your operating system's secure credential store. In CI/CD pipelines, provide RUBRICLOOP_API_KEY as an environment variable.
registry:pullDownload packages you have access to.registry:pushCreate repositories, push releases, and manage tags.vetting:runSubmit package releases for automated security checks.verify:runExecute verifications via the hosted cloud engine.Public vs private
Control visibility on a per-repository basis. Keep proprietary internal rubrics private to your team, or publish community packs for the open-source ecosystem.
Accessible only to authenticated members of your organization and invited collaborators.
Discoverable and pullable by anyone in the community once vetted and approved.
# Pull the latest release by tag
rubricloop --server https://api.rubricloop.dev pull acme/policy-checks@latest
# Pull an exact, immutable release by content digest
rubricloop --server https://api.rubricloop.dev pull acme/policy-checks@sha256:<bundle-digest>
# Run a vetted package directly on the hosted engine
rubricloop --server https://api.rubricloop.dev run acme/policy-checks@sha256:<bundle-digest> --input request.jsonTags like latest can be updated over time. For production stability, pull by sha256 digest or commit rubricloop.lock to ensure every build uses the exact same package.
Team workspace
Collaborate with your team, manage scoped API keys, and review audit logs across all agent verification runs.
Assign owner, admin, or member roles to control who can publish packages, generate keys, or change visibility.
Generate minimal-privilege keys for CI runners, developers, and production environments. Keys can be revoked at any time.
Monitor pass rates, retry counts, and latency across all active rubrics with live performance telemetry.
Prompt and output payloads are never stored on RubricLoop servers unless explicitly enabled by an admin for debugging.
Ready to share rubrics or use the hosted API?
Create a free account to get an API key, publish community packages, or collaborate with your team.