9.1 What Is Harness Engineering?

🏇 "Harness engineering is the idea that anytime you find an agent makes a mistake, you take the time to engineer a solution such that the agent will not make that mistake again in the future."
— Mitchell Hashimoto, Co-founder of HashiCorp, November 2025

Starting from a Real Frustration

Suppose you've already built an AI programming assistant Agent, based on GPT-5 or Claude 4, that performs flawlessly in demos. But once it's handed over to the team for real use, problems start piling up:

• The Agent fixed Bug A but quietly introduced Bug B, and didn't notice
• Faced with a refactoring task requiring changes to 10 files, the Agent did 5 and quietly "completed" it
• To make tests pass, the Agent directly deleted the test cases (rather than fixing the code)
• A long-running task got stuck in an infinite loop at the 80% mark

Your first instinct might be: switch to a better model?

But after switching to GPT-5, the problems didn't disappear — they just became more subtle.

This is because the root cause of these problems is not that the model isn't smart enough, but that the system lacks sufficient constraints and feedback mechanisms.

This is exactly what Harness Engineering is designed to solve.

Core Definition

The core philosophy of Harness Engineering can be summarized in one sentence:

Whenever you find that an Agent has made a certain type of mistake, take the time to engineer a solution at the system level so that the Agent will not make the same type of mistake in the future.

Note the key words in this definition:

1. "Whenever": this is a continuous iterative process, not a one-time design
2. "Engineer a solution at the system level": not changing the Prompt to tell the Agent to "remember not to do this," but using code, constraints, and validation mechanisms to enforce it
3. "Will not make the same type of mistake in the future": the goal is to systematically eliminate a category of errors, not to fix a specific case

This is consistent with the philosophy of "continuous improvement" in traditional software engineering, but the subject has expanded from code logic to AI Agent behavior.

Core Formula: Agent = Model + Harness

The simplest way to understand Harness Engineering is this equation:

Agent = Model + Harness

The diagram clearly shows the division of responsibilities:

Model (Intelligence Engine) provides three core capabilities:

• Language understanding: parsing semantics in natural language, code, and documents
• Reasoning and planning: analyzing problems, formulating strategies, making decisions
• Content generation: outputting code, text, structured data

Harness (Control System) provides six engineering guarantees:

• Context architecture: controls what information enters the model, prevents information overload
• Architectural constraints: uses code to enforce rules, doesn't rely on the model's "self-discipline"
• Self-verification loops: forces validation checks before the Agent claims completion
• Context isolation: prevents erroneous information from spreading across Agents in multi-Agent collaboration
• Entropy management: combats technical debt accumulation from AI's rapid code generation
• Detachability: gracefully removes constraints that are no longer needed as model capabilities improve

🏎️ Racing car analogy: the Model is the engine (provides power), the Harness is the steering wheel + brakes + dashboard (ensures safety). No matter how powerful the engine, without a control system, it cannot be driven safely.

The relationship between the two: there is bidirectional interaction between Model and Harness — the Model outputs execution results to the Harness, and the Harness feeds back constraint information and validation conclusions to the Model, guiding the next step. This closed-loop feedback mechanism is the core distinction between Harness Engineering and traditional Prompt Engineering.

Differences from Other Engineering Paradigms

Harness Engineering is often confused with two related concepts, so it's necessary to clearly distinguish them:

Difference from Prompt Engineering

# Prompt Engineering approach:
# Modify the prompt to make the model "know" to run tests
system_prompt = """
You are a programming assistant.
Important: After modifying code, please be sure to run tests to confirm correctness.
Please remember to run tests!!!
"""
# Problem: relies on the model's "self-discipline," easily forgotten in complex tasks

# Harness Engineering approach:
# Enforce validation at the code level
class PreCompletionChecklist:
    """Intercept completion signals, force execution of checklist"""
    
    def intercept_completion(self, agent_output):
        if agent_output.intent == "task_complete":
            checks = [
                self.verify_tests_run(),
                self.verify_no_test_deletion(),
                self.verify_linter_passed(),
            ]
            if not all(checks):
                return self.force_validation_step()
        return agent_output

Key difference: Prompt Engineering relies on "soft constraints" (linguistic persuasion), Harness Engineering uses "hard constraints" (code enforcement).

Relationship with Context Engineering

Context Engineering ⊂ Harness Engineering

Context Engineering (Chapter 8) is a subset of Harness Engineering: it focuses on input-level optimization, answering the question "what information to give the model."

Harness Engineering has a broader scope, also including:

In short: context engineering tells the Agent "what it should know," while Harness Engineering further ensures the Agent "acts correctly and validates results."

Relationship with LLMOps

Harness Engineering ⊂ LLMOps

LLMOps (LLM Operations) is a broader concept, covering the full lifecycle of model deployment, monitoring, A/B testing, version management, and more. Harness Engineering is the specific practice within LLMOps that focuses on Agent reliability engineering.

The containment relationship can be understood as:

LLMOps (full model lifecycle operations)
  └── Harness Engineering (Agent runtime reliability engineering)
        └── Context Engineering (model input information optimization)

As an Agent developer, you don't need to master the complete LLMOps system, but Harness Engineering is a core capability you must master.

Five Core Problems: What Does Harness Engineering Solve?

Harness Engineering emerged to solve five categories of systemic problems encountered when running Agents in production. These problems share a common characteristic: switching to a better model cannot solve them — because they are system design flaws, not model capability flaws.

Let's analyze each problem and its Harness solution in depth:

Problem 1: Context Anxiety

# Experimental data (2026 industry research)
context_performance = {
    "0-40% utilization":  "✅ Reasoning quality stable",
    "40-70% utilization": "⚠️ Slight quality degradation, omissions begin",
    "70-90% utilization": "⚠️ Noticeable quality degradation, Agent starts 'skipping steps'",
    "90-100% utilization": "❌ Severe quality degradation, Agent develops 'context anxiety'",
}

# Manifestations of context anxiety:
# - Silently skipping steps (without saying "I skipped this," just skipping)
# - Simplifying output ("The code here is similar to above, omitted")
# - Premature completion (claiming completion before the task is truly done)

Harness solution: actively monitor context utilization, trigger compression at 40% to prevent entering the danger zone.

Problem 2: Completion Bias

Research shows that LLMs have a systematic "optimism bias" — they tend to believe tasks are complete even when validation steps haven't passed.

# Typical manifestation of completion bias
agent_log = """
[Step 15] Modified the calculate_discount function in user_service.py
[Step 16] I believe the task is complete. The code should work correctly.
          (Note: Agent didn't run any tests!)
"""

# Harness solution: forced validation checkpoints
class ValidationGate:
    """Any 'task complete' signal must pass through the validation gate"""
    
    REQUIRED_CHECKS = [
        "unit_tests_passed",
        "integration_tests_passed", 
        "linter_clean",
        "no_test_modifications",  # prevent cheating by deleting tests
    ]
    
    def allow_completion(self, agent_state) -> bool:
        return all(
            agent_state.checks.get(check, False) 
            for check in self.REQUIRED_CHECKS
        )

Problem 3: Multi-Agent Contamination Effect

In multi-Agent systems, one Agent's errors can "contaminate" other Agents' decisions through message passing:

# Contamination effect example
class MultiAgentSystem:
    def route_task(self, task):
        # Agent A produces an incorrect architectural decision
        agent_a_output = self.agent_a.process(task)
        # agent_a_output contains erroneous information: "should use MongoDB"
        # (the actual project uses PostgreSQL)
        
        # Agent B receives this erroneous information and treats it as "fact"
        agent_b_output = self.agent_b.process(agent_a_output)
        # agent_b_output starts designing a MongoDB-based solution, error is spreading...
        
        # Agent C further amplifies this error...
        agent_c_output = self.agent_c.process(agent_b_output)

# Harness solution: context isolation
# Each Agent only receives "sanitized" information relevant to its task
# Agents communicate through structured interfaces, not raw conversation history

Problem 4: Doom Loop

# Typical form of doom loop
for attempt in range(infinity):
    result = agent.fix_bug(bug_description)
    if test_passes(result):
        break
    # Problem: Agent keeps trying with the same approach, will never succeed
    # Documented case: one production incident consumed 47 iterations, costing $23 in API fees

# Harness solution: loop detection + strategy switching
class LoopDetector:
    def __init__(self, max_same_attempts: int = 3):
        self.edit_history = defaultdict(int)
        self.max_same_attempts = max_same_attempts
    
    def track_edit(self, file_path: str, edit_type: str) -> bool:
        """Returns True if a doom loop is detected"""
        key = f"{file_path}:{edit_type}"
        self.edit_history[key] += 1
        
        if self.edit_history[key] > self.max_same_attempts:
            return True  # trigger strategy switch
        return False
    
    def suggest_alternative(self, stuck_context: str) -> str:
        """Inject new ideas into the Agent"""
        return f"""
You seem to be repeating the same approach. Please try a completely different approach:
- Current stuck problem: {stuck_context}
- Suggestion: consider analyzing the root cause from a different angle
- If still unable to resolve, please flag for human intervention
"""

Problem 5: System Entropy

# AI generates code far faster than humans can understand it
# Without control mechanisms, the codebase quickly becomes chaotic

# Typical entropy accumulation path:
# Week 1: AI-generated code has consistent style, meets standards
# Week 2: A few naming convention violations, but still acceptable
# Week 4: Various patterns start mixing, documentation starts falling behind
# Month 2: New AI starts learning from old "bad code,"
#           because it appears more frequently in the codebase...
# Month 3: Human engineers start unable to understand code they "wrote"

# Harness solution: periodic entropy management Agent
class EntropyGardener:
    """Runs periodically to maintain codebase health"""
    
    def weekly_scan(self):
        """Comprehensive health check run weekly"""
        issues = []
        issues += self.check_doc_sync()          # is documentation in sync with code
        issues += self.check_convention_drift()   # are there naming convention drifts
        issues += self.check_dead_code()          # is there accumulating dead code
        issues += self.check_dependency_health()  # do dependencies need updating
        
        for issue in issues:
            self.create_cleanup_pr(issue)  # automatically submit cleanup PRs
        
        return f"Found and fixed {len(issues)} entropy issues"

The Fundamental Shift in the Engineer's Role

Harness Engineering is not just a change in technical methodology — it marks a fundamental transformation in the software engineer's role.

Traditional Role              →         Harness Era Role
────────────────────────────────────────────────────────
"Code Craftsman"                          "Systems Architect"
· Personally write business logic  →     · Design Agent runtime environments
· Debug specific bugs              →     · Analyze error patterns, improve Harness
· Maintain code quality            →     · Build self-maintaining constraint systems
· Write feature code               →     · Write linter rules, validation logic

The shift in core value:

From "How fast and well can I write code"

To "How smart and robust a system can I design to reliably tame AI Agents"

This shift doesn't mean engineers are "unemployed" — it means a dramatic increase in engineer leverage — designing a good Harness system can enable AI Agents to reliably complete work that previously required an entire team.

Section Summary

💡 Key insight: the essence of Harness Engineering is to systematize and engineer "human lessons learned," making them persistently effective constraint mechanisms rather than "verbal reminders" that need to be repeated to the model every time it runs.

References

[1] HASHIMOTO M. Harness engineering[EB/OL]. X/Twitter, 2025-11.

[2] OPENAI ENGINEERING TEAM. Harness engineering: leveraging Codex in an engineering organization[EB/OL]. OpenAI Blog, 2026-02.

[3] LANGCHAIN TEAM. How to think about harness engineering[EB/OL]. LangChain Blog, 2026-01.

Next: 9.2 Six Engineering Pillars