Docs
Rules
A rule is a business invariant that must hold after every step, such as: total refunds never exceed the amount paid. Bob proposes rules, you approve them, and the engine searches only for the ones you approved.
How a rule gets approved
- The Lawmaker proposes rules, each with evidence from your code: the place that suggests the rule should hold.
- The engine checks every proposed rule. Its check must parse with the safe evaluator, use only the state and fields the model defines, and hold on a fresh app. A rule that fails is rejected with the engine's reason and goes no further.
- The Rule Critic judges what the remaining rules mean: a real invariant, too weak, or a duplicate. It can approve, reject or revise a rule. A revised rule is checked by the engine again, and the Critic can never rescue a rule the engine rejected.
- You approve all of the accepted rules, none of them, or the ones you pick. Until then they are saved with the status proposed.
- The rules you approved are saved in rook/rook.yaml with the status approved. The search checks these and no others.
Bob's part ends at proposing and reviewing. Whether a rule is broken is decided later by the engine alone, by running your app.
Rules the app already breaks
A state rule that is false on a fresh app is rejected: it is a wrong rule. A response rule is different, because it is judged on a single request. If the app breaks it on the very first request, Roook cannot tell a one-step bug from a wrong rule, so it keeps the rule and flags it already_broken.
The flag is shown with the rule and its reason. A person may approve a flagged rule. Auto mode never does.
Auto mode
With rook run --auto, Roook answers its own questions inside limits that are enforced in code:
- It approves only rules the Rule Critic approved or revised, and never one flagged
already_broken. - It applies a fix only for a diagnosis the Diagnosis Reviewer approved.
- The fix still lands on its own rook/fix-* branch, never on your default branch.
- It never invents a setup value: those come from
rook run --setup NAMEor from you.
rook run --ci without --auto approves the same rules, but answers no to fixing and shipping: it reports and changes no code.
An example rook/rook.yaml
This is tests/fixtures/minishop/rook.yaml from the Roook repo, unchanged: the hand-written model of minishop, Roook's own small test shop, which Roook's engine tests run against. It has four approved rules, one for each bug planted in that shop.
# Rook model for the minishop fixture (format: docs/02_ARCHITECTURE.md section 6).
version: 1
app:
base_url: "{{sandbox.base_url}}"
health: {method: GET, path: /health, expect_status: 200}
isolation: fresh_entities
actors:
- name: customer
setup:
- {method: POST, path: /auth/signup, json: {email: "{{fresh.email}}", password: "pw-{{fresh.id}}"}}
- {method: POST, path: /auth/login, json: {email: "{{fresh.email}}", password: "pw-{{fresh.id}}"},
capture: {token: "$.token"}}
auth: {header: Authorization, value: "Bearer {{actor.token}}"}
- name: admin
setup:
- {method: POST, path: /auth/login, json: {email: admin@demo.local, password: "{{env.MINISHOP_ADMIN_PASSWORD}}"},
capture: {token: "$.token"}}
auth: {header: Authorization, value: "Bearer {{actor.token}}"}
actions:
- name: create_product
actor: admin
request: {method: POST, path: /products, json: {name: "item-{{fresh.id}}", price: "{{p.price}}", stock: "{{p.stock}}"}}
params:
price: {int: [1, 500], edges: [1, 100, 500]}
stock: {int: [0, 5], edges: [0, 1]}
capture: {product_id: "$.id"}
weight: 0.5
- name: buy
actor: customer
request: {method: POST, path: /orders, json: {product_id: "{{ref.product_id}}", quantity: "{{p.quantity}}"}}
params: {quantity: {int: [1, 3], edges: [1]}}
requires: [product_id]
capture: {order_id: "$.id"}
weight: 1.0
- name: refund
actor: customer
request: {method: POST, path: "/orders/{{ref.order_id}}/refunds", json: {amount: "{{p.amount}}"}}
params: {amount: {int: [1, 500], edges: [1, 50, 60, 100]}}
requires: [order_id]
weight: 1.0
- name: cancel
actor: customer
request: {method: POST, path: "/orders/{{ref.order_id}}/cancel"}
requires: [order_id]
weight: 0.5
- name: ship
actor: customer
request: {method: POST, path: "/orders/{{ref.order_id}}/ship"}
requires: [order_id]
weight: 0.5
- name: admin_export
actor: customer
request: {method: GET, path: /admin/export}
weight: 0.3
state:
- name: order
each: order_id
request: {method: GET, path: "/orders/{{order_id}}", actor: admin}
fields: {paid: "$.paid", refunded: "$.refunded_total", status: "$.status", shipped: "$.shipped"}
- name: product
each: product_id
request: {method: GET, path: "/products/{{product_id}}", actor: admin}
fields: {stock: "$.stock"}
rules:
- id: refund_le_paid
text: "Total refunds never exceed the amount paid"
kind: state
scope: order
check: "order.refunded <= order.paid"
evidence: ["tests/fixtures/minishop/app.py: refund() compares each refund with paid on its own"]
status: approved
- id: stock_non_negative
text: "Product stock never goes below zero"
kind: state
scope: product
check: "product.stock >= 0"
evidence: ["tests/fixtures/minishop/app.py: buy() awaits between the stock check and the decrement"]
status: approved
- id: cancelled_never_ships
text: "A cancelled order is never shipped"
kind: state
scope: order
check: 'not (order.status == "cancelled" and order.shipped)'
evidence: ["tests/fixtures/minishop/app.py: ship() does not check for a cancelled order"]
status: approved
- id: admin_export_forbidden
text: "A normal user never opens the admin export"
kind: response
when: {action: admin_export, actor: customer}
check: "response.status in (401, 403)"
evidence: ["tests/fixtures/minishop/app.py: admin_export() only requires a login"]
status: approved
In a run on your repo, the Mapper and the Lawmaker write this file and you approve its rules, so the names are theirs. The recorded run on the Counterexamples page named the same refund rule refunded_total_le_paid.
The file, part by part
The file is validated strictly: an unknown field, a duplicate name or a reference to an actor, action or state reader that does not exist is an error. version is 1.
app
- base_url
- Always {{sandbox.base_url}}. Roook fills it in at run time with the sandbox's address; any other value is rejected.
- health
- The request Roook polls until the app is up: a method, a path and the expected status.
- isolation
- fresh_entities (the default): every sequence creates its own actors and entities, so nothing is reset between sequences. restart: the sandbox is restarted instead.
actors
Who sends the requests. Each actor has a name, optional setup requests that run at the start of a sequence to create a fresh actor (sign up, log in), a capture that reads values such as a token out of a response with a JSONPath, and the auth header its requests carry.
actions
- name, actor
- What the action is called, and which actor sends it.
- request
- A declarative HTTP template: method, path, and optionally json, query and headers. The path is relative to the sandbox. An absolute URL anywhere in a request is rejected.
- params
- The values the search generates.
int: [lo, hi]with optionaledges(values tried more often),choicewith a list of values, orstringwith apatternor a list ofvalues. - requires
- The captured variables that must exist before this action can run. refund requires an order_id, so it can only follow buy.
- capture
- Values read from the response and added to the sequence's variables, such as the id of the order just created.
- weight
- How often the search picks this action, relative to the others. The default is 1.0.
state
State readers are how Roook sees your app's state: plain HTTP reads. A reader has a name, each (the captured variable it is read for, such as every order_id the sequence created), the request, and fields picked out of the response with JSONPaths. Rules refer to these fields, as in order.refunded.
rules
- id, text
- A short identifier, and the rule in plain words.
- kind: state
- A check on state after every step.
scopenames the state reader it applies to, or global. - kind: response
- A check on the response to one action. It needs
when: the action, and optionally the actor who sends it. - check
- The expression that must be true, at most 500 characters.
- evidence
- Where in the code the rule comes from.
- status
- proposed, approved or rejected. Only approved rules are searched.
Templates
- {{p.name}}
- A generated param of this action.
- {{ref.name}}
- A value captured earlier in the sequence, picked from those available.
- {{fresh.name}}
- A value that is unique to the sequence, such as a new email address.
- {{env.NAME}}
- A setup value from the sandbox's environment, such as an admin password. It is never written into the file.
- {{actor.token}}
- A value the current actor captured in its setup.
Check expressions and the safe evaluator
A check is one expression over the values Roook read from your app. A state rule sees its state reader by name, as in order.refunded <= order.paid. A response rule sees response, as in response.status in (401, 403).
- Allowed: comparisons,
and,or,not,in, arithmetic, tuples and lists, indexing, reading a field with a dot, and generator expressions and list comprehensions. - Allowed calls, and no others: sum, len, min, max, abs, all, any, round.
- Refused: any other call, names starting with an underscore, lambdas, assignment, f-strings, conditional expressions, slices and the power operator.
Rules come from Bob, so Roook treats them as untrusted. A check is parsed into a syntax tree, every node is compared with an allowlist, and a small interpreter inside Roook walks the tree with limits on depth and work. Python's own eval and exec are never used, on checks or on anything else Bob writes.
What is not built yet
- In the web app, editing a single rule is not built yet: you approve or reject the rules as proposed.
- Roook does not yet read rules you wrote by hand before a run: the Lawmaker proposes them in each run, and you choose.