Skip to content
Hexel StudioDocs

Swarms

Run many agents on one goal under one budget, one concurrency limit and one failure rule, as a map, a lead with workers, a pipeline or a task board.

On this page

A swarm runs many agents on one goal. Its members are tasks of their own, started while the swarm has room, under the swarm's budget, concurrency limit and failure rule. Titan governs each member like any task, and can narrow every member's access.

PatternShape
mapOne member per item, then an optional reduce step over their outputs
lead_workersA lead agent that starts workers as it goes
pipelineStages that feed each other, item by item or all at once
boardMembers that claim items from a shared board, post results and add items
Bash
hexel harness swarm estimate -f swarm.yaml    # price it first
hexel harness swarm run -f swarm.yaml --follow
hexel harness swarm status swm_123
hexel harness swarm members swm_123
hexel harness swarm cancel swm_123

The file is the body of POST /harness/swarms, in YAML or JSON.

Common fields

FieldDescription
patternmap, lead_workers, pipeline or board.
goalEach member's goal (a lead's goal for lead_workers).
agentThe members' agent, as name@channel or name@v<n>.
budgetRequired. The whole swarm's budget, for example {"max_usd": 10}. It also takes max_estimate_usd (Budgets and estimates).
member_budgetOptional. Each member's budget; by default max_usd is an equal share per slot.
max_concurrency1 to 100 members working at once (default 4). A lead does not take a slot.
failure_ruleWhat happens when a member fails (below).
accessOptional. {"capabilities": ["harness.sandbox.read_file"]}: Titan narrows every member to these capabilities.
sandbox_templateOptional.

Map

summarise.yaml
pattern: map
goal: Summarise the file
agent: auditor@prod
items: [{path: a.md}, {path: b.md}]
reduce: {goal: Combine the summaries, agent: editor@prod}
budget: {max_usd: 10}
max_concurrency: 4
failure_rule: {on_failure: retry, max_attempts: 2, then: skip}

items is 1 to 1,000 objects, one member each. The swarm's outputs are its reduce step's outputs.

Lead and workers

plan.yaml
pattern: lead_workers
goal: Plan and audit the repository
agent: planner@prod
worker_agent: auditor@prod
inputs: {repo: acme/api}
budget: {max_usd: 20}

The lead runs with inputs and starts workers as it goes. A worker never gets access its lead does not have (403 access_widens). The swarm's outputs are the lead's.

Pipeline

brief.yaml
pattern: pipeline
goal: Brief on the documents
agent: auditor@prod
items: [{doc: a.md}, {doc: b.md}]
stages:
  - {name: extract, goal: Extract the facts}
  - {name: check, goal: Check each fact, agent: checker@prod}
  - {name: brief, goal: Write one brief, mode: all}
budget: {max_usd: 10}
failure_rule: {on_failure: skip}
  • 2 to 10 stages, in order. An each stage (the default) has one member per item; an all stage has one member over every output of the stage before. The first stage is each, and every stage after an all stage is all. At most 1,000 members in all.
  • A member of a later stage is blocked until the members feeding it are done; it starts with their outputs, read in your runtime.
  • An item whose member is skipped or fails is dropped from later stages (end_reason: upstream_skipped). An all member is never skipped: the swarm fails instead.
  • The pipeline succeeds once every member is final and a member of its last stage is done. progress.stages shows each stage.

Task board

fix-tests.yaml
pattern: board
goal: Fix every failing test
agent: fixer@prod
board:
  items: [{key: test_a, inputs: {test: tests/test_a.py}}]
  members: 4
  item_ttl_s: 600
  max_item_attempts: 2
  on_item_failure: fail
budget: {max_usd: 10}

Members claim items under leases, post results, add new items and share a working memory. Members start once every item is on the board, and each claims items until the board is empty.

FieldDescription
items0 to 1,000 items. key is 1 to 64 of A-Z a-z 0-9 . _ @ - (default: the item's index); put content in inputs, not in the key.
members1 to 100 (default max_concurrency).
item_ttl_sA claim's lease, 30 to 3,600 seconds, renewed while the item is worked on.
max_item_attemptsClaims an item gets, 1 to 10.
on_item_failurefail (an item that fails for good fails the swarm) or skip.

The swarm succeeds once every member and every item is final (with on_item_failure: fail, none failed). progress.items is {total, open, done, failed}.

The board is kept in Titan, as a coordination mission. Item keys and the board's working memory hold references (keys, digests and sizes), while the content stays in your runtime.

Failure rules

failure_rule is {on_failure, max_attempts, then}:

on_failureWhat happens to a failed member
retryIts task runs again.
reassignA new task takes its place.
skipIt is skipped.
escalateThe swarm waits for a person (waiting_reason: escalation).
failThe swarm fails.

After max_attempts (1 to 10), then (skip, escalate or fail) applies. Decide an escalated member with:

Bash
hexel harness swarm resolve swm_123 smb_456 --action retry   # or skip, fail

Progress and states

A swarm is running, waiting, succeeded, failed or cancelled. A member is queued, working, done, failed, skipped, escalated or cancelled (and blocked in a pipeline).

progress is {total, done, working, queued, failed, skipped, escalated, cancelled, cost_usd, budget_usd, summary}, where summary reads like "32 of 40 done · 3 working · $4.10". GET /harness/swarms/{id}/events streams a swarm.progress snapshot whenever it changes (and swarm.stage for a pipeline), then end.

Each member is a task with swarm_id set: read, stream, steer and approve it with the task routes. While its swarm is open, a member cannot be retried, resumed with a new budget or deleted (409 swarm_member). Cancelling the swarm cancels its working members, and queued ones never start.

An organization can have at most 100 open swarms (429 too_many_swarms).