{"harness": {"recipes": 3, "summary": "OpenHands/OpenHands is the Agent Canvas web frontend plus Node launchers (bin/agent-canvas.mjs, scripts/dev-*.mjs); it contains no task-running CLI. The runnable agent is the pinned `openhands-agent-server` (software-agent-sdk, version from config/defaults.json versions.agentServer = 1.49.6) that the launchers and docker/entrypoint.sh spawn and then drive over its REST API. The overlay installs that pinned server (sdk/tools/workspace pinned alike, plus the posthog constraint the launcher adds) into a uv venv under /opt/harness and copies the repo to /opt/harness/src so the agent-server imports the repo's tools/canvas_ui_tool.py at startup exactly like scripts/dev-safe.mjs and docker/entrypoint.sh do. The run command starts `agent-server --host 127.0.0.1 --port 18000 --import-modules canvas_ui_tool`, then POSTs one conversation (`agent_settings.llm` = model/api_key/base_url with api_mode=chat and native tool calling, LocalWorkspace = the task cwd, worktree=false, initial_message with run=true, NeverConfirm, autotitle off), polls execution_status until finished/error/stuck/idle-after-running, dumps all events to /out/trajectory.json plus conversation.json and agent-server.log, prints the final agent message, and exits 0 on finished/idle. The SDK talks to the model through litellm; with an `openai/<name>` model and base_url=$PROXY_URL/v1 it sends non-streaming OpenAI chat-completions with tool calls (terminal, file_editor, task_tracker, finish, think) \u2014 verified against a mock server in both a bookworm and a jammy-based image.", "recipe": "openhands-openhands@7dc6805406ea3c76cb4a3ce407c3c72d481b0ac6", "tasks_passed": 4, "harness": "openhands-openhands", "first_run": "2026-09-23T15:10:59", "domains": ["swe", "devops-sre"], "runs": 7, "base_image": "python:3.12-slim-bookworm", "finished": 7, "last_run": "2026-09-25T16:09:04", "repo": "https://github.com/OpenHands/OpenHands", "passes": 5, "last_run_id": "20260925T090253-openhands-op-python-12", "use_case": "Runs a web UI to manage and monitor AI coding agents that edit files, run terminal commands, and automate development workflows across local or cloud backends", "tasksets": ["aider_polyglot", "bird-bench", "humanevalfix", "swebench-verified"], "results": {"bird-bench/card_games__372": {"passes": 1, "last": "2026-09-24T23:09:17", "last_run": "20260924T230831-sw-openhands-card_games__372", "last_tests": null, "last_outcome": "pass", "last_reward": 1, "runs": 1}, "aider_polyglot/polyglot_python_bowling": {"passes": 1, "last": "2026-09-23T15:10:59", "last_run": "20260923T145420-openhands-op-polyglot_python_bowling", "last_tests": {"summary": "31 passed", "total": 31, "passed": 31, "failed": 0, "agent_written": 0, "failed_names": []}, "last_outcome": "pass", "last_reward": 1, "runs": 1}, "aider_polyglot/polyglot_go_palindrome-products": {"passes": 0, "last": "2026-09-24T22:55:36", "last_run": "20260924T225250-sw-openhands-polyglot_go_palindrome-products", "last_tests": null, "last_outcome": "fail", "last_reward": 0, "runs": 1}, "swebench-verified/psf__requests-5414": {"passes": 1, "last": "2026-09-24T23:24:16", "last_run": "20260924T232412-sw-openhands-psf__requests-5414", "last_tests": {"summary": "131 passed, 1 xfailed, 158 errors", "total": 0, "passed": 0, "failed": 0, "agent_written": 0, "failed_names": []}, "last_outcome": "pass", "last_reward": 1, "runs": 2}, "humanevalfix/python-12": {"passes": 2, "last": "2026-09-25T16:09:04", "last_run": "20260925T090253-openhands-op-python-12", "last_tests": {"summary": "1 passed", "total": 1, "passed": 1, "failed": 0, "agent_written": 0, "failed_names": []}, "last_outcome": "pass", "last_reward": 1, "runs": 2}}, "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 5, "api_style": "openai", "commit": "7dc6805406ea3c76cb4a3ce407c3c72d481b0ac6"}, "profile": {"evidence": "README.md lines 1-50 describe it as a \"self-hosted developer control center for coding agents and automations\" that runs OpenHands, Claude Code, Codex, or Gemini agents. src/api/agent-server-adapter.ts:145 shows DEFAULT_TOOL_NAMES = [\"terminal\", \"file_editor\", \"task_tracker\"] and BROWSER_TOOL_SET_NAME at line 146. scripts/dev-with-automation.mjs orchestrates the full stack including agent-server via uvx. docs/architecture.md lines 8-19 confirms it manages conversations, terminal, browser, files, and automation UI. AGENTS.md shows integration with GitHub, Slack, Linear for automations. bin/agent-canvas.mjs launches the complete stack.", "harness": "openhands-openhands", "domains": ["swe", "devops-sre"], "source": "claude -p", "cost_usd": 0.37737810000000005, "capabilities": ["edits-files", "runs-shell", "browses-web", "uses-git", "multi-agent", "long-horizon", "calls-apis"], "seconds": 100, "languages": [], "use_case": "Runs a web UI to manage and monitor AI coding agents that edit files, run terminal commands, and automate development workflows across local or cloud backends", "at": "2026-09-25T16:11:34", "not_for": ["This is a UI and orchestrator, not the agent itself - actual agent execution happens in the separate OpenHands Agent Server (software-agent-sdk)", "Does not directly call LLMs - delegates to agent backends", "Not for direct command-line agent interaction without the web UI"], "commit": "7dc6805406ea3c76cb4a3ce407c3c72d481b0ac6"}, "recommendations": {"model": "haiku", "at": "2026-09-25T16:12:24", "profile_source": "claude -p", "harness": "openhands-openhands", "recs": [{"why": "Tests whether Go's failure was language-specific; C++ is a compiled language like Go but with different idioms.", "taskset": "aider_polyglot", "language": "cpp", "task": "polyglot_cpp_allergies", "score": null, "domain": "swe"}, {"language": "javascript", "taskset": "aider_polyglot", "domain": "swe", "task": "polyglot_javascript_triangle", "score": null, "why": "Expands language coverage to JavaScript\u2014relevant for a web-UI orchestrator\u2014and continues polyglot validation."}, {"domain": "data-sql", "why": "Tests multi-step data transformation workflows beyond single-file code editing, exercising shell integration and long-horizon reasoning.", "language": null, "task": "divvy001", "score": null, "taskset": "spider2-dbt"}, {"taskset": "swebench-verified", "score": null, "why": "Real-world repository complexity with git workflows; shows whether the harness scales beyond toy problems to actual codebases.", "language": null, "domain": "swe", "task": "sphinx-doc__sphinx-8595"}, {"domain": "swe", "taskset": "quixbugs", "task": "quixbugs-python-sieve", "language": "python", "score": null, "why": "Establishes a baseline for pure bug-fixing (vs. feature implementation) in a tight scope, bridging easy and harder tasks."}], "source": "llm", "cost_usd": 0.03717925, "based_on_run": "20260925T090253-openhands-op-python-12"}, "runs": [{"run": "20260925T090253-openhands-op-python-12", "started": "2026-09-25T16:09:04", "finished": "2026-09-25T16:09:49", "status": "done", "kind": "harbor", "harness": "openhands-openhands", "task": {"taskset": "humanevalfix", "name": "python-12"}, "model": "bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0", "reward": 1, "verifier_rc": 0, "tests": {"summary": "1 passed", "total": 1, "passed": 1, "failed": 0, "agent_written": 0, "failed_names": []}, "calls": 9, "seconds": 40, "input_tokens": 81736, "output_tokens": 1786, "errors": 0, "last_action": "file_editor: view", "outcome": "scored", "verifier_says": "reward 1 \u00b7 1 passed"}, {"run": "20260924T232412-sw-openhands-psf__requests-5414", "started": "2026-09-24T23:24:16", "finished": "2026-09-24T23:30:34", "status": "done", "kind": "harbor", "harness": "openhands-openhands", "task": {"taskset": "swebench-verified", "name": "psf__requests-5414"}, "model": "bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0", "reward": 1, "verifier_rc": 0, "tests": {"summary": "131 passed, 1 xfailed, 158 errors", "total": 0, "passed": 0, "failed": 0, "agent_written": 0, "failed_names": []}, "calls": 94, "seconds": 364, "input_tokens": 3836096, "output_tokens": 24519, "errors": 0, "last_action": "finish: {\"message\": \"## Fix Complete: UnicodeError \\u2192 InvalidURL for Invalid Hostnames\\n\\n### Problem\\nWhen attempting to ge\u2026", "outcome": "scored", "verifier_says": "reward 1 \u00b7 131 passed, 1 xfailed, 158 errors"}, {"run": "20260924T230831-sw-openhands-card_games__372", "started": "2026-09-24T23:09:17", "finished": "2026-09-24T23:09:48", "status": "done", "kind": "harbor", "harness": "openhands-openhands", "task": {"taskset": "bird-bench", "name": "card_games__372"}, "model": "bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0", "reward": 1, "verifier_rc": 0, "tests": null, "calls": 5, "seconds": 24, "input_tokens": 46406, "output_tokens": 441, "errors": 0, "last_action": "terminal: cat /app/answer.sql", "outcome": "scored", "verifier_says": "reward 1"}, {"run": "20260924T230708-sw-openhands-python-12", "started": "2026-09-24T23:07:51", "finished": "2026-09-24T23:08:29", "status": "done", "kind": "harbor", "harness": "openhands-openhands", "task": {"taskset": "humanevalfix", "name": "python-12"}, "model": "bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0", "reward": 1, "verifier_rc": 0, "tests": {"summary": "1 passed", "total": 1, "passed": 1, "failed": 0, "agent_written": 0, "failed_names": []}, "calls": 9, "seconds": 34, "input_tokens": 81632, "output_tokens": 2320, "errors": 0, "last_action": "terminal: cd /workspace && python -c \" def longest(strings): if not strings: return None maxlen = max(len(x) for x in strings) for\u2026", "outcome": "scored", "verifier_says": "reward 1 \u00b7 1 passed"}, {"run": "20260924T225744-sw-openhands-psf__requests-5414", "started": "2026-09-24T22:58:44", "finished": "2026-09-24T23:07:05", "status": "done", "kind": "harbor", "harness": "openhands-openhands", "task": {"taskset": "swebench-verified", "name": "psf__requests-5414"}, "model": "bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0", "reward": 0, "verifier_rc": 1, "tests": {"summary": "7 failed, 124 passed, 1 xfailed, 158 errors", "total": 0, "passed": 0, "failed": 0, "agent_written": 0, "failed_names": []}, "calls": 83, "seconds": 336, "input_tokens": 2755758, "output_tokens": 20948, "errors": 0, "last_action": "finish: {\"message\": \"I've successfully fixed the issue where `requests.get(\\\"http://.example.com\\\")` was raising a `UnicodeError\u2026", "outcome": "scored", "verifier_says": "reward 0 \u00b7 7 failed, 124 passed, 1 xfailed, 158 errors \u00b7 verifier exited 1"}, {"run": "20260924T225250-sw-openhands-polyglot_go_palindrome-products", "started": "2026-09-24T22:55:36", "finished": "2026-09-24T22:57:43", "status": "done", "kind": "harbor", "harness": "openhands-openhands", "task": {"taskset": "aider_polyglot", "name": "polyglot_go_palindrome-products"}, "model": "bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0", "reward": 0, "verifier_rc": 0, "tests": null, "calls": 21, "seconds": 93, "input_tokens": 269552, "output_tokens": 5097, "errors": 0, "last_action": "file_editor: view", "outcome": "scored", "verifier_says": "reward 0"}, {"run": "20260923T145420-openhands-op-polyglot_python_bowling", "started": "2026-09-23T15:10:59", "finished": "2026-09-23T15:15:57", "status": "done", "kind": "harbor", "harness": "openhands-openhands", "task": {"taskset": "aider_polyglot", "name": "polyglot_python_bowling"}, "model": "bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0", "reward": 1, "verifier_rc": 0, "tests": {"summary": "31 passed", "total": 31, "passed": 31, "failed": 0, "agent_written": 0, "failed_names": []}, "calls": 49, "seconds": 293, "input_tokens": 2007965, "output_tokens": 35692, "errors": 0, "last_action": "terminal: cd /app && python3 -c \" from bowling import BowlingGame # Quick sanity checks game = BowlingGame() # Test error conditio\u2026", "outcome": "scored", "verifier_says": "reward 1 \u00b7 31 passed"}]}