10.9 Agentic Data Flywheel: Letting Agents Improve Themselves

🔄 "The best training data is not manually labeled in isolation; it is the trajectory left behind when Agents succeed, fail, recover, and complete tasks in real environments."

Why the Data Flywheel Matters

Agentic RL needs feedback from real tasks. But high-quality Agent data is expensive because it includes multi-step reasoning, tool calls, observations, failures, and final outcomes.

A data flywheel turns production usage into training improvement:

Deploy Agent
    ↓
Collect trajectories
    ↓
Evaluate outcomes
    ↓
Filter and label data
    ↓
Train / fine-tune / optimize
    ↓
Deploy improved Agent
    ↓
Collect better trajectories

The more the Agent is used, the more opportunities you have to improve it—if the data pipeline is designed correctly.

What Data Should Be Collected?

A useful Agent trajectory includes more than the final answer:

from dataclasses import dataclass, field


@dataclass
class AgentTrajectory:
    task_id: str
    user_query: str
    steps: list[dict] = field(default_factory=list)
    final_answer: str = ""
    success: bool | None = None
    user_feedback: int | None = None
    cost: float = 0.0
    latency_ms: float = 0.0
    metadata: dict = field(default_factory=dict)

Each step should record:

• model input and output;
• tool name and arguments;
• tool observation;
• errors and retries;
• intermediate decisions;
• timestamps and token usage.

Without trajectories, you cannot train better Agent behavior. You can only train better final text.

Feedback Sources

A mature system combines multiple signals instead of relying on a single score.

Data Filtering

Not every production trace should become training data. Filter aggressively:

def is_training_candidate(trace: AgentTrajectory) -> bool:
    if trace.success is False and trace.user_feedback is None:
        return False
    if trace.cost > 5.0:
        return False
    if trace.metadata.get("contains_sensitive_data"):
        return False
    if len(trace.steps) == 0:
        return False
    return True

Recommended filters:

• remove sensitive or private data;
• remove corrupted traces;
• deduplicate near-identical tasks;
• keep both successes and informative failures;
• balance categories and difficulty levels;
• separate training, validation, and regression sets.

Turning Trajectories into Training Data

Different training methods need different data transformations:

Example preference pair:

{
  "prompt": "Analyze this CSV and find anomalies.",
  "chosen": "Uses Python tool, validates columns, reports anomalies with evidence.",
  "rejected": "Guesses anomalies without reading the file.",
  "reason": "Chosen trajectory grounds the answer in tool output."
}

Reward Design

Agent rewards should combine outcome quality with process quality.

def compute_agent_reward(trace: AgentTrajectory) -> float:
    reward = 0.0
    if trace.success:
        reward += 1.0
    if trace.user_feedback is not None:
        reward += 0.2 * trace.user_feedback

    # Penalize inefficient or unstable behavior.
    reward -= 0.01 * len(trace.steps)
    reward -= 0.1 * count_tool_errors(trace)
    reward -= 0.001 * trace.cost

    return max(min(reward, 2.0), -1.0)

Reward design should be monitored carefully. A poorly designed reward can teach the Agent to optimize metrics while hurting real user value.

Human-in-the-Loop Data Improvement

The most valuable traces are often failure cases. A reviewer can convert them into training data by:

1. identifying the first wrong decision;
2. correcting the tool call or reasoning step;
3. adding the corrected trajectory to SFT data;
4. adding the original failure as a rejected preference example;
5. adding the task to the regression suite.

Failed trajectory
      ↓
Human correction
      ↓
SFT example + preference pair + regression test

This turns incidents into long-term capability improvements.

Flywheel Governance

A data flywheel can be dangerous without governance:

• it may store private user data;
• it may amplify biased or low-quality behavior;
• it may train on prompt-injection traces;
• it may overfit to frequent but low-value tasks;
• it may silently degrade rare but important workflows.

Governance checklist:

• user consent and data retention policy;
• PII detection and redaction;
• safety filtering;
• dataset versioning;
• human review for high-risk domains;
• separate evaluation sets that are never trained on.

Production Data Flywheel Architecture

Agent Runtime
   ↓ emits traces
Trace Store
   ↓
Privacy Filter / Redaction
   ↓
Outcome Evaluator
   ↓
Data Selection
   ├── SFT Dataset
   ├── Preference Dataset
   ├── Reward Dataset
   └── Regression Dataset
   ↓
Training / Fine-Tuning
   ↓
Offline Evaluation
   ↓
A/B Test
   ↓
Production Rollout

The key principle is separation: raw traces, filtered data, training sets, and evaluation sets should be versioned independently.

Best Practices

• Collect full trajectories, not only final answers.
• Treat production traces as raw material, not automatically trusted training data.
• Keep informative failures and human corrections.
• Use privacy filtering before any training pipeline.
• Build separate datasets for SFT, preferences, rewards, and regression.
• Version datasets, prompts, tools, models, and evaluation results together.
• Close the loop with A/B testing before production rollout.

Chapter Takeaways

• An Agentic data flywheel converts real Agent usage into better training data.
• The most valuable data includes trajectories, tool calls, observations, failures, and outcomes.
• Feedback can come from users, tools, humans, judges, and runtime metrics.
• Data filtering and privacy governance are mandatory.
• A mature flywheel turns failures into SFT examples, preference pairs, reward data, and regression tests.

Previous: 10.8 Agent-Specific Fine-Tuning