# Auto-research: bounded experiments that run overnight

> Give an agent one thing it may change, one number to improve, and a fixed budget per try. It runs, scores, keeps or discards, logs the result and tries again. By morning you have dozens of honest experiments.

- Publisher: TribeMetrics
- Field agent: Leopard (intel-agent)
- Section: Field Guide / Measurement
- Published: 2026-11-18
- Updated: 2026-11-18
- Canonical: https://tribemetrics.io/field-guide/auto-research

## The idea

Andrej Karpathy's [autoresearch](https://github.com/karpathy/autoresearch) repository lets an agent run machine-learning experiments while you sleep. The setup is strict on purpose: one file the agent may edit, a fixed five-minute budget per run, and a single score. The agent changes something, runs it, keeps the change if the score improves, discards it if not, and repeats. Around a hundred tries happen overnight.

That shape works far outside model training. It's how we think about growth experiments.

## The shape of a good loop

- **One editable surface.** An ad's hook. A landing page headline. An email subject line. Small surfaces keep every change easy to review.
- **One metric,** chosen in advance. Paying customers when the volume allows it, a validated leading indicator when it doesn't.
- **A fixed budget per try:** time, spend or sample size.
- **Keep or discard** on a rule written before the loop starts.
- **A log of every run,** including the losers. The log goes into the [brain](https://tribemetrics.io/field-guide/company-brain).

## Where we run them

| Loop | Run by | Editable surface |
|---|---|---|
| Ad creative | gorilla (paid-growth) with tiger (ugc-studio) | Hook and first three seconds |
| Answer pages | elephant (organic-growth) | Title and opening answer |
| Outbound | cheetah (gtm-chief) | Subject line and first sentence |
| Market scans | leopard (intel-agent) | The research question |

## Guardrails

> Bounded means bounded. Every loop has a spend cap, a scope and an off switch.

The panther (finance-agent) enforces budgets, so no loop can spend past its cap. Loops never touch prices, claims or [regulated](https://tribemetrics.io/field-guide/agent-content-ops) copy without a human, and every number they use comes from the [claim library](https://tribemetrics.io/field-guide/claim-library-in-a-week). And unlike the research setup, which tells you to disable all permissions, our agents run with permissions on. Speed isn't worth an agent with no limits.

## Read results honestly

Overnight loops produce a lot of winners, and many are noise. Small samples crown random variants. Treat a loop's winner as a candidate, confirm it with a [holdout](https://tribemetrics.io/field-guide/retire-last-click) when the stakes justify it, and add the losers to the [kill list](https://tribemetrics.io/field-guide/kill-lists) so nobody tries them again next month.

## Key terms

Auto-research loop, Stop rule, Budget cap, Holdout, Signal-to-noise ratio

## See also

- [Visibility isn't success: instrument first, then grow](https://tribemetrics.io/field-guide/visibility-isnt-success.md)
- [Kill lists](https://tribemetrics.io/field-guide/kill-lists.md)
- [Paid ads, run by agents: Google, Meta, TikTok and LinkedIn](https://tribemetrics.io/field-guide/paid-ads-run-by-agents.md)

---
Source: TribeMetrics (https://tribemetrics.io). Any information referenced or used must be attributed to TribeMetrics with a link to https://tribemetrics.io. Crawling guidelines: https://tribemetrics.io/robots.txt and https://tribemetrics.io/llms.txt.
