Skip to content

Docs

Rules

A rule is a business invariant that must hold after every step, such as: total refunds never exceed the amount paid. Bob proposes rules, you approve them, and the engine searches only for the ones you approved.

How a rule gets approved

  1. The Lawmaker proposes rules, each with evidence from your code: the place that suggests the rule should hold.
  2. The engine checks every proposed rule. Its check must parse with the safe evaluator, use only the state and fields the model defines, and hold on a fresh app. A rule that fails is rejected with the engine's reason and goes no further.
  3. The Rule Critic judges what the remaining rules mean: a real invariant, too weak, or a duplicate. It can approve, reject or revise a rule. A revised rule is checked by the engine again, and the Critic can never rescue a rule the engine rejected.
  4. You approve all of the accepted rules, none of them, or the ones you pick. Until then they are saved with the status proposed.
  5. The rules you approved are saved in rook/rook.yaml with the status approved. The search checks these and no others.

Bob's part ends at proposing and reviewing. Whether a rule is broken is decided later by the engine alone, by running your app.

Rules the app already breaks

A state rule that is false on a fresh app is rejected: it is a wrong rule. A response rule is different, because it is judged on a single request. If the app breaks it on the very first request, Roook cannot tell a one-step bug from a wrong rule, so it keeps the rule and flags it already_broken.

The flag is shown with the rule and its reason. A person may approve a flagged rule. Auto mode never does.

Auto mode

With rook run --auto, Roook answers its own questions inside limits that are enforced in code:

  • It approves only rules the Rule Critic approved or revised, and never one flagged already_broken.
  • It applies a fix only for a diagnosis the Diagnosis Reviewer approved.
  • The fix still lands on its own rook/fix-* branch, never on your default branch.
  • It never invents a setup value: those come from rook run --setup NAME or from you.

rook run --ci without --auto approves the same rules, but answers no to fixing and shipping: it reports and changes no code.

An example rook/rook.yaml

This is tests/fixtures/minishop/rook.yaml from the Roook repo, unchanged: the hand-written model of minishop, Roook's own small test shop, which Roook's engine tests run against. It has four approved rules, one for each bug planted in that shop.

tests/fixtures/minishop/rook.yaml · Unchanged from the Roook repo
# Rook model for the minishop fixture (format: docs/02_ARCHITECTURE.md section 6).
version: 1
app:
  base_url: "{{sandbox.base_url}}"
  health: {method: GET, path: /health, expect_status: 200}
  isolation: fresh_entities
actors:
  - name: customer
    setup:
      - {method: POST, path: /auth/signup, json: {email: "{{fresh.email}}", password: "pw-{{fresh.id}}"}}
      - {method: POST, path: /auth/login, json: {email: "{{fresh.email}}", password: "pw-{{fresh.id}}"},
         capture: {token: "$.token"}}
    auth: {header: Authorization, value: "Bearer {{actor.token}}"}
  - name: admin
    setup:
      - {method: POST, path: /auth/login, json: {email: admin@demo.local, password: "{{env.MINISHOP_ADMIN_PASSWORD}}"},
         capture: {token: "$.token"}}
    auth: {header: Authorization, value: "Bearer {{actor.token}}"}
actions:
  - name: create_product
    actor: admin
    request: {method: POST, path: /products, json: {name: "item-{{fresh.id}}", price: "{{p.price}}", stock: "{{p.stock}}"}}
    params:
      price: {int: [1, 500], edges: [1, 100, 500]}
      stock: {int: [0, 5], edges: [0, 1]}
    capture: {product_id: "$.id"}
    weight: 0.5
  - name: buy
    actor: customer
    request: {method: POST, path: /orders, json: {product_id: "{{ref.product_id}}", quantity: "{{p.quantity}}"}}
    params: {quantity: {int: [1, 3], edges: [1]}}
    requires: [product_id]
    capture: {order_id: "$.id"}
    weight: 1.0
  - name: refund
    actor: customer
    request: {method: POST, path: "/orders/{{ref.order_id}}/refunds", json: {amount: "{{p.amount}}"}}
    params: {amount: {int: [1, 500], edges: [1, 50, 60, 100]}}
    requires: [order_id]
    weight: 1.0
  - name: cancel
    actor: customer
    request: {method: POST, path: "/orders/{{ref.order_id}}/cancel"}
    requires: [order_id]
    weight: 0.5
  - name: ship
    actor: customer
    request: {method: POST, path: "/orders/{{ref.order_id}}/ship"}
    requires: [order_id]
    weight: 0.5
  - name: admin_export
    actor: customer
    request: {method: GET, path: /admin/export}
    weight: 0.3
state:
  - name: order
    each: order_id
    request: {method: GET, path: "/orders/{{order_id}}", actor: admin}
    fields: {paid: "$.paid", refunded: "$.refunded_total", status: "$.status", shipped: "$.shipped"}
  - name: product
    each: product_id
    request: {method: GET, path: "/products/{{product_id}}", actor: admin}
    fields: {stock: "$.stock"}
rules:
  - id: refund_le_paid
    text: "Total refunds never exceed the amount paid"
    kind: state
    scope: order
    check: "order.refunded <= order.paid"
    evidence: ["tests/fixtures/minishop/app.py: refund() compares each refund with paid on its own"]
    status: approved
  - id: stock_non_negative
    text: "Product stock never goes below zero"
    kind: state
    scope: product
    check: "product.stock >= 0"
    evidence: ["tests/fixtures/minishop/app.py: buy() awaits between the stock check and the decrement"]
    status: approved
  - id: cancelled_never_ships
    text: "A cancelled order is never shipped"
    kind: state
    scope: order
    check: 'not (order.status == "cancelled" and order.shipped)'
    evidence: ["tests/fixtures/minishop/app.py: ship() does not check for a cancelled order"]
    status: approved
  - id: admin_export_forbidden
    text: "A normal user never opens the admin export"
    kind: response
    when: {action: admin_export, actor: customer}
    check: "response.status in (401, 403)"
    evidence: ["tests/fixtures/minishop/app.py: admin_export() only requires a login"]
    status: approved

In a run on your repo, the Mapper and the Lawmaker write this file and you approve its rules, so the names are theirs. The recorded run on the Counterexamples page named the same refund rule refunded_total_le_paid.

The file, part by part

The file is validated strictly: an unknown field, a duplicate name or a reference to an actor, action or state reader that does not exist is an error. version is 1.

app

base_url
Always {{sandbox.base_url}}. Roook fills it in at run time with the sandbox's address; any other value is rejected.
health
The request Roook polls until the app is up: a method, a path and the expected status.
isolation
fresh_entities (the default): every sequence creates its own actors and entities, so nothing is reset between sequences. restart: the sandbox is restarted instead.

actors

Who sends the requests. Each actor has a name, optional setup requests that run at the start of a sequence to create a fresh actor (sign up, log in), a capture that reads values such as a token out of a response with a JSONPath, and the auth header its requests carry.

actions

name, actor
What the action is called, and which actor sends it.
request
A declarative HTTP template: method, path, and optionally json, query and headers. The path is relative to the sandbox. An absolute URL anywhere in a request is rejected.
params
The values the search generates. int: [lo, hi] with optional edges (values tried more often), choice with a list of values, or string with a pattern or a list of values.
requires
The captured variables that must exist before this action can run. refund requires an order_id, so it can only follow buy.
capture
Values read from the response and added to the sequence's variables, such as the id of the order just created.
weight
How often the search picks this action, relative to the others. The default is 1.0.

state

State readers are how Roook sees your app's state: plain HTTP reads. A reader has a name, each (the captured variable it is read for, such as every order_id the sequence created), the request, and fields picked out of the response with JSONPaths. Rules refer to these fields, as in order.refunded.

rules

id, text
A short identifier, and the rule in plain words.
kind: state
A check on state after every step. scope names the state reader it applies to, or global.
kind: response
A check on the response to one action. It needs when: the action, and optionally the actor who sends it.
check
The expression that must be true, at most 500 characters.
evidence
Where in the code the rule comes from.
status
proposed, approved or rejected. Only approved rules are searched.

Templates

{{p.name}}
A generated param of this action.
{{ref.name}}
A value captured earlier in the sequence, picked from those available.
{{fresh.name}}
A value that is unique to the sequence, such as a new email address.
{{env.NAME}}
A setup value from the sandbox's environment, such as an admin password. It is never written into the file.
{{actor.token}}
A value the current actor captured in its setup.

Check expressions and the safe evaluator

A check is one expression over the values Roook read from your app. A state rule sees its state reader by name, as in order.refunded <= order.paid. A response rule sees response, as in response.status in (401, 403).

  • Allowed: comparisons, and, or, not, in, arithmetic, tuples and lists, indexing, reading a field with a dot, and generator expressions and list comprehensions.
  • Allowed calls, and no others: sum, len, min, max, abs, all, any, round.
  • Refused: any other call, names starting with an underscore, lambdas, assignment, f-strings, conditional expressions, slices and the power operator.

Rules come from Bob, so Roook treats them as untrusted. A check is parsed into a syntax tree, every node is compared with an allowlist, and a small interpreter inside Roook walks the tree with limits on depth and work. Python's own eval and exec are never used, on checks or on anything else Bob writes.

What is not built yet

  • In the web app, editing a single rule is not built yet: you approve or reject the rules as proposed.
  • Roook does not yet read rules you wrote by hand before a run: the Lawmaker proposes them in each run, and you choose.
Rules · Roook