{"harnesses": [{"recipes": 1, "summary": "Aider is a terminal AI pair-programming harness written in Python (package `aider-chat`, console entrypoint `aider = aider.main:main`). It edits files in the current working directory (using git for its repo map and change tracking) and makes every LLM call through litellm: `aider/llm.py` lazy-loads litellm and `Model.send_completion` in `aider/models.py` calls `litellm.completion` with the model name, streaming flag and optional tool spec. Using an `openai/<name>` model makes litellm's OpenAI chat-completions client honour `OPENAI_API_BASE`/`OPENAI_BASE_URL`, so all requests (main, weak and editor models) go to `$PROXY_URL/v1/chat/completions`. A single non-interactive task is run with `--message \"$TASK\"` plus `--yes-always`, which answers every confirmation, applies the edits and exits after one turn; chat and full LLM request/response history files are written under /out.", "tasks_passed": 3, "harness": "aider-ai-aider", "first_run": "2026-09-23T11:32:24", "domains": ["swe"], "runs": 19, "base_image": "python:3.12-slim-bookworm", "finished": 19, "last_run": "2026-09-25T16:09:45", "repo": "https://github.com/Aider-AI/aider", "passes": 8, "last_run_id": "20260925T090301-aider-ai-aid-psf__requests-5414", "use_case": "A terminal pair-programming assistant: you add files from your git repo to the chat, describe a change or bug fix, and the LLM edits the code with SEARCH/REPLACE diffs, auto-commits the change, and can re-run your linter/tests to fix failures.", "tasksets": ["aider_polyglot", "bird-bench", "humanevalfix", "swebench-verified"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 5, "api_style": "openai", "commit": "5dc9490bb35f9729ef2c95d00a19ccd30c26339c"}, {"recipes": 1, "summary": "mini-SWE-agent is a minimal Python SWE agent (package `mini-swe-agent`, src/minisweagent). Its CLI entrypoint is the `mini` console script (minisweagent.run.mini:app, typer), which builds a config from src/minisweagent/config/mini.yaml plus CLI flags, instantiates a LitellmModel (default model class), a LocalEnvironment (runs each bash action as a subprocess in the current working directory) and the InteractiveAgent, then loops: it calls litellm.completion() with a single `bash` function tool, executes the returned tool calls, feeds the JSON-rendered output back as tool messages, and stops when the model runs `echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT`. The model is chosen by MSWEA_MODEL_NAME (or -m); with an `openai/<name>` model name litellm speaks the OpenAI chat-completions API and honors OPENAI_BASE_URL/OPENAI_API_BASE and OPENAI_API_KEY (the config also passes model_kwargs.api_base explicitly). The trajectory (all messages, tool outputs, exit status) is written as JSON to the path given by -o.", "tasks_passed": 7, "harness": "swe-agent-mini-swe-agent", "first_run": "2026-09-23T10:31:10", "domains": ["swe"], "runs": 16, "base_image": "python:3.12-slim-bookworm", "finished": 16, "last_run": "2026-09-25T22:09:23", "repo": "https://github.com/SWE-agent/mini-swe-agent", "passes": 7, "last_run_id": "20260925T220613-swe-agent-mi-b7a243ece02c-polyglot_python_connect", "use_case": "solves GitHub issues and programming tasks in a codebase by executing bash commands in sandboxed environments", "tasksets": ["aider_polyglot", "algotune", "bird-bench", "humanevalfix", "quixbugs", "swebench-verified"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0", "bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0"], "tasks_tried": 9, "api_style": "openai", "commit": "04d809ceab9df28f9adaed044884180159172930"}, {"recipes": 3, "summary": "Kilo CLI (packages/opencode, a fork of OpenCode; core services in packages/core) is a Bun/TypeScript coding agent. Its entrypoint is packages/opencode/src/index.ts (yargs CLI, script name `kilo`); the non-interactive one-shot mode is `kilo run --format json --auto --model provider/model \"<prompt>\"`, which boots the in-process HTTP server, creates a session rooted at the cwd, streams one JSON event per line, auto-approves permission requests, and exits. Model calls go through the Vercel AI SDK; providers come from JSON config (KILO_CONFIG_CONTENT holds inline config), and a custom provider with npm \"@ai-sdk/openai-compatible\" plus options.baseURL sends OpenAI chat-completions requests (streaming + tool calls) to the proxy, exactly the way the repo's own CLI subprocess tests do (test/lib/test-provider.ts, test/lib/cli-process.ts). The overlay installs Bun 1.3.14 under /opt/harness/bun, copies the repo to /opt/harness/src, runs `bun install --frozen-lockfile`, and writes a /opt/harness/bin/kilo shim (symlinked into /usr/local/bin and added via profile.d so `bash -lc` keeps it on PATH) that launches src/index.ts with the same bun flags as the repo's dev/test tooling, with the @opentui/solid preload resolved to an absolute path at build time and offline-safe KILO_DISABLE_* defaults exported so the CLI never fetches models.dev, plugins, or updates.", "harness": "kilo-org-kilocode", "first_run": "2026-09-23T14:22:32", "domains": ["swe"], "runs": 8, "base_image": "debian:bookworm-slim", "finished": 8, "last_run": "2026-09-25T23:38:46", "repo": "https://github.com/kilo-org/kilocode", "last_run_id": "20260925T233225-kilo-org-kil-d996c3e6b2fa-polyglot_go_palindrome-products", "use_case": "Writes, edits, and debugs code across multiple files in any language using natural language instructions from VS Code, JetBrains, or the terminal.", "tasksets": ["aider_polyglot", "algotune", "bigcodebench_hard_complete", "bird-bench", "humanevalfix", "swebench-verified"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 7, "api_style": "openai", "commit": "24167bd703f56b8adc39cf94263cb85736963777", "scored": 8}, {"recipes": 3, "summary": "Codex CLI (openai/codex) is a Rust coding agent. Its CLI is the codex-rs workspace crate codex-cli (binary `codex`); the npm package in codex-cli/ only wraps prebuilt binaries. The non-interactive entrypoint is `codex exec [OPTIONS] -- \"<prompt>\"`, which runs one headless session. `--json` streams JSONL events, `-o FILE` writes the final message, `--dangerously-bypass-approvals-and-sandbox` turns off approvals and the sandbox, `--skip-git-repo-check` allows non-git dirs, and CODEX_HOME holds the config and the rollout trajectories. Compiling the workspace needs more than 8 GB of RAM, and the previous build was OOM-killed. The overlay therefore installs the official statically linked musl `codex` binary that upstream CI builds from this workspace (GitHub release rust-v0.156.1, x86_64 or aarch64 chosen by `uname -m`). The repo source is still COPYed to /opt/harness/src, and `--build-arg CODEX_INSTALL=source` compiles from source on a large builder instead. Codex calls models only through the OpenAI Responses API (`POST <base_url>/responses`, SSE); `wire_api=\"chat\"` is rejected. So the run command starts a stdlib-Python shim on 127.0.0.1:8790 using a uv-managed Python under /opt/harness. The shim translates Responses requests into streaming chat-completions calls to $PROXY_URL/v1/chat/completions and converts the results back. Function tools map 1:1, and freeform custom tools such as apply_patch become functions with a single `input` string. Codex reaches the shim through a custom provider in CODEX_HOME/config.toml, with model gpt-5.5.", "tasks_passed": 4, "harness": "openai-codex", "first_run": "2026-09-23T23:05:28", "domains": ["swe"], "runs": 7, "base_image": "python:3.12-slim-bookworm", "finished": 7, "last_run": "2026-09-25T16:10:25", "repo": "https://github.com/openai/codex", "passes": 4, "last_run_id": "20260925T090258-openai-codex-polyglot_go_palindrome-products", "use_case": "Helps developers write, debug, test, and refactor code in any programming language from the terminal", "tasksets": ["aider_polyglot", "bird-bench", "humanevalfix", "swebench-verified"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 5, "api_style": "openai", "commit": "55543d87724bb66bdd51bde65254feb9b4c9ed10"}, {"recipes": 3, "summary": "OpenHands/OpenHands is the Agent Canvas web frontend plus Node launchers (bin/agent-canvas.mjs, scripts/dev-*.mjs); it contains no task-running CLI. The runnable agent is the pinned `openhands-agent-server` (software-agent-sdk, version from config/defaults.json versions.agentServer = 1.49.6) that the launchers and docker/entrypoint.sh spawn and then drive over its REST API. The overlay installs that pinned server (sdk/tools/workspace pinned alike, plus the posthog constraint the launcher adds) into a uv venv under /opt/harness and copies the repo to /opt/harness/src so the agent-server imports the repo's tools/canvas_ui_tool.py at startup exactly like scripts/dev-safe.mjs and docker/entrypoint.sh do. The run command starts `agent-server --host 127.0.0.1 --port 18000 --import-modules canvas_ui_tool`, then POSTs one conversation (`agent_settings.llm` = model/api_key/base_url with api_mode=chat and native tool calling, LocalWorkspace = the task cwd, worktree=false, initial_message with run=true, NeverConfirm, autotitle off), polls execution_status until finished/error/stuck/idle-after-running, dumps all events to /out/trajectory.json plus conversation.json and agent-server.log, prints the final agent message, and exits 0 on finished/idle. The SDK talks to the model through litellm; with an `openai/<name>` model and base_url=$PROXY_URL/v1 it sends non-streaming OpenAI chat-completions with tool calls (terminal, file_editor, task_tracker, finish, think) \u2014 verified against a mock server in both a bookworm and a jammy-based image.", "tasks_passed": 4, "harness": "openhands-openhands", "first_run": "2026-09-23T15:10:59", "domains": ["swe", "devops-sre"], "runs": 7, "base_image": "python:3.12-slim-bookworm", "finished": 7, "last_run": "2026-09-25T16:09:04", "repo": "https://github.com/OpenHands/OpenHands", "passes": 5, "last_run_id": "20260925T090253-openhands-op-python-12", "use_case": "Runs a web UI to manage and monitor AI coding agents that edit files, run terminal commands, and automate development workflows across local or cloud backends", "tasksets": ["aider_polyglot", "bird-bench", "humanevalfix", "swebench-verified"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 5, "api_style": "openai", "commit": "7dc6805406ea3c76cb4a3ce407c3c72d481b0ac6"}, {"recipes": 0, "summary": null, "tasks_passed": 3, "harness": "google-gemini-gemini-cli", "first_run": "2026-09-23T15:38:03", "domains": null, "runs": 6, "base_image": null, "finished": 6, "last_run": "2026-09-24T23:23:40", "repo": "https://github.com/google-gemini/gemini-cli", "passes": 3, "last_run_id": "20260924T232100-sw-gemini-cli-card_games__372", "use_case": null, "tasksets": ["aider_polyglot", "bird-bench", "humanevalfix", "swebench-verified"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 5, "api_style": "openai", "commit": "bedef96ef42905bd84a86dbec021c706168e7e2f"}, {"recipes": 0, "summary": null, "tasks_passed": 0, "harness": "apurv101-website-builder", "first_run": "2026-09-24T16:37:18", "domains": null, "runs": 3, "base_image": null, "finished": 3, "last_run": "2026-09-24T17:00:12", "repo": "https://github.com/apurv101/website-builder", "passes": 0, "last_run_id": "20260924T165657-apurv101-web-polyglot_python_bowling", "use_case": null, "tasksets": ["aider_polyglot"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "openai", "commit": "c6c6cb0116c7deafc0c760984f9fceb64d62d81a"}, {"recipes": 0, "summary": null, "tasks_passed": 1, "harness": "vivekhaldar-seed", "first_run": "2026-09-23T15:33:40", "domains": null, "runs": 3, "base_image": null, "finished": 2, "last_run": "2026-09-24T16:22:45", "repo": "https://github.com/vivekhaldar/seed", "passes": 1, "last_run_id": "20260924T162242-vivekhaldar--polyglot_python_bowling", "use_case": null, "tasksets": ["aider_polyglot"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "openai", "commit": "879244dec4000c04f72afaea93584bee9488b114"}, {"recipes": 0, "summary": null, "tasks_passed": 1, "harness": "yc-software-qm", "first_run": "2026-09-23T15:22:48", "domains": null, "runs": 3, "base_image": null, "finished": 3, "last_run": "2026-09-24T18:08:22", "repo": "https://github.com/yc-software/qm", "passes": 2, "last_run_id": "20260924T180430-yc-software--polyglot_python_bowling", "use_case": null, "tasksets": ["aider_polyglot"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "anthropic", "commit": "a61db71b91158d7239047738ae0a07c8c5a829bc"}, {"recipes": 0, "summary": null, "tasks_passed": 1, "harness": "aaif-goose-goose", "first_run": "2026-09-23T11:55:29", "domains": null, "runs": 2, "base_image": null, "finished": 2, "last_run": "2026-09-24T15:40:36", "repo": "https://github.com/aaif-goose/goose", "passes": 1, "last_run_id": "20260924T154034-aaif-goose-g-polyglot_python_bowling", "use_case": null, "tasksets": ["aider_polyglot"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "anthropic", "commit": "ac6222029fe526a4ac15540cb69e2e49cf3a8878"}, {"recipes": 0, "summary": null, "tasks_passed": 1, "harness": "dlants-magenta.nvim", "first_run": "2026-09-23T12:10:37", "domains": null, "runs": 2, "base_image": null, "finished": 2, "last_run": "2026-09-24T18:58:49", "repo": "https://github.com/dlants/magenta.nvim", "passes": 1, "last_run_id": "20260924T185619-dlants-magen-polyglot_python_bowling", "use_case": null, "tasksets": ["aider_polyglot"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "anthropic", "commit": "5af1a2139a26a6a652bfa9a4b69d4b8f4c418e8f"}, {"recipes": 0, "summary": null, "tasks_passed": 0, "harness": "gadievron-raptor", "first_run": "2026-09-23T13:38:01", "domains": null, "runs": 2, "base_image": null, "finished": 2, "last_run": "2026-09-24T15:40:36", "repo": "https://github.com/gadievron/raptor", "passes": 0, "last_run_id": "20260924T154034-gadievron-ra-polyglot_python_bowling", "use_case": null, "tasksets": ["aider_polyglot"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "anthropic", "commit": "7c4551596c0a2c8f4f75b4e14f5ec339650f9bfe"}, {"recipes": 0, "summary": null, "tasks_passed": 0, "harness": "nanocoai-nanoclaw", "first_run": "2026-09-23T14:28:33", "domains": null, "runs": 2, "base_image": null, "finished": 2, "last_run": "2026-09-24T15:40:36", "repo": "https://github.com/nanocoai/nanoclaw", "passes": 0, "last_run_id": "20260924T154034-nanocoai-nan-polyglot_python_bowling", "use_case": null, "tasksets": ["aider_polyglot"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "anthropic", "commit": "143db6c907c652773a536c7c9e96269fdad0a4a4"}, {"recipes": 0, "summary": null, "tasks_passed": 1, "harness": "omnigent-ai-omnigent", "first_run": "2026-09-23T14:44:48", "domains": null, "runs": 2, "base_image": null, "finished": 2, "last_run": "2026-09-24T15:41:02", "repo": "https://github.com/omnigent-ai/omnigent", "passes": 1, "last_run_id": "20260924T154101-omnigent-ai--polyglot_python_bowling", "use_case": null, "tasksets": ["aider_polyglot"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "anthropic", "commit": "ae8cbf5b1c1c9842a497d15f18a6ea2bc70e7550"}, {"recipes": 0, "summary": null, "tasks_passed": 0, "harness": "onecli-onecli", "first_run": "2026-09-23T14:54:02", "domains": null, "runs": 2, "base_image": null, "finished": 2, "last_run": "2026-09-24T15:42:00", "repo": "https://github.com/onecli/onecli", "passes": 0, "last_run_id": "20260924T154158-onecli-onecl-polyglot_python_bowling", "use_case": null, "tasksets": ["aider_polyglot"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "anthropic", "commit": "8ea47cd88973742e482791600f71199834b54b22"}, {"recipes": 0, "summary": null, "tasks_passed": 1, "harness": "opencoven-coven", "first_run": "2026-09-23T14:54:21", "domains": null, "runs": 2, "base_image": null, "finished": 2, "last_run": "2026-09-24T15:43:08", "repo": "https://github.com/OpenCoven/coven", "passes": 1, "last_run_id": "20260924T154306-opencoven-co-polyglot_python_bowling", "use_case": null, "tasksets": ["aider_polyglot"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "anthropic", "commit": "5c614c6b79faaf32e5828058d0ce4272cd7fb60e"}, {"recipes": 0, "summary": null, "tasks_passed": 0, "harness": "swe-agent-swe-agent", "first_run": "2026-09-23T15:20:05", "domains": null, "runs": 2, "base_image": null, "finished": 2, "last_run": "2026-09-24T15:44:05", "repo": "https://github.com/SWE-agent/SWE-agent", "passes": 0, "last_run_id": "20260924T154403-swe-agent-sw-polyglot_python_bowling", "use_case": null, "tasksets": ["aider_polyglot"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "openai", "commit": "3ea751c087f32b16e039a2233dd6eefecef325d5"}, {"recipes": 0, "summary": null, "tasks_passed": 0, "harness": "agent-of-empires-agent-of-empires", "first_run": "2026-09-23T14:15:15", "domains": null, "runs": 1, "base_image": null, "finished": 1, "last_run": "2026-09-23T14:15:15", "repo": "https://github.com/agent-of-empires/agent-of-empires", "passes": 0, "last_run_id": "20260923T135733-agent-of-emp-polyglot_python_bowling", "use_case": null, "tasksets": ["aider_polyglot"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "anthropic", "commit": "70a474cb6a42bb12e90c959f49b8f9d0e203104a"}, {"recipes": 0, "summary": null, "tasks_passed": 0, "harness": "ale-aguirre-claude-job-hunter", "first_run": "2026-09-23T11:55:30", "domains": null, "runs": 1, "base_image": null, "finished": 1, "last_run": "2026-09-23T11:55:30", "repo": "https://github.com/ale-aguirre/claude-job-hunter", "passes": 0, "last_run_id": "20260923T115529-ale-aguirre--polyglot_python_bowling", "use_case": null, "tasksets": ["aider_polyglot"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "anthropic", "commit": "c085f0962f8ad208448d614477692e7aa6fc3e99"}, {"recipes": 0, "summary": null, "tasks_passed": 0, "harness": "anomalyco-opencode", "first_run": "2026-09-23T12:07:20", "domains": null, "runs": 1, "base_image": null, "finished": 1, "last_run": "2026-09-23T12:07:20", "repo": "https://github.com/anomalyco/opencode", "passes": 0, "last_run_id": "20260923T115542-anomalyco-op-polyglot_python_bowling", "use_case": null, "tasksets": ["aider_polyglot"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "anthropic", "commit": "1d6c3c0e29f6a6ceff5204fa40ea3fb54338f159"}, {"recipes": 1, "summary": "Crush (github.com/charmbracelet/crush) is a terminal-based AI coding assistant built in Go. Its CLI entrypoint is `crush run [prompt]` for non-interactive use, which sends the prompt to an LLM and streams the response while the agent uses tools (view, edit, bash, glob, grep, etc.) to complete coding tasks. The harness is built from source using `golang:latest` with `CGO_ENABLED=0 GOEXPERIMENT=greenteagc`. A crushrc config file at `$CRUSH_GLOBAL_CONFIG/crushrc` configures a custom OpenAI-compatible provider pointing to `$PROXY_URL/v1` with model `claude-3-5-sonnet-20241022`, disables the default provider catalog and auto-update to avoid network calls, and pre-approves all tool permissions so the agent can run without user interaction. The run command is `crush run --quiet \"$TASK\"` from the task working directory.", "compatibility": null, "harness": "charmbracelet-crush", "first_run": "2026-09-26T07:48:48", "domains": ["swe"], "runs": 1, "base_image": "buildpack-deps:jammy", "finished": 1, "last_run": "2026-09-26T07:48:48", "repo": "https://github.com/charmbracelet/crush", "last_run_id": "20260926T073808-charmbracele-308ef1539f9a-polyglot_cpp_allergies", "use_case": "Writes, edits, tests, and debugs code in a terminal by executing shell commands, reading and modifying files, and using LSP for code intelligence.", "tasksets": ["aider_polyglot"], "scored": 0, "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "openai", "commit": "c9b45348a4f1b7695e70830a4408aa4c31688851"}, {"recipes": 0, "summary": null, "tasks_passed": 0, "harness": "cline-cline", "first_run": "2026-09-23T13:22:01", "domains": null, "runs": 1, "base_image": null, "finished": 1, "last_run": "2026-09-23T13:22:01", "repo": "https://github.com/cline/cline", "passes": 0, "last_run_id": "20260923T125958-cline-cline-polyglot_python_bowling", "use_case": null, "tasksets": ["aider_polyglot"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "openai", "commit": "3c8fde80c349d712b689c0b1b21fc70882e5b817"}, {"recipes": 0, "summary": null, "tasks_passed": 0, "harness": "code-yeongyu-lazycodex", "first_run": "2026-09-23T13:45:18", "domains": null, "runs": 1, "base_image": null, "finished": 1, "last_run": "2026-09-23T13:45:18", "repo": "https://github.com/code-yeongyu/lazycodex", "passes": 0, "last_run_id": "20260923T130851-code-yeongyu-polyglot_python_bowling", "use_case": null, "tasksets": ["aider_polyglot"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "openai", "commit": "86eff5079f555f3a67513cd487ec1410b29cc8f6"}, {"recipes": 0, "summary": null, "tasks_passed": 0, "harness": "continuedev-continue", "first_run": "2026-09-23T13:20:38", "domains": null, "runs": 1, "base_image": null, "finished": 1, "last_run": "2026-09-23T13:20:38", "repo": "https://github.com/continuedev/continue", "passes": 0, "last_run_id": "20260923T130853-continuedev--polyglot_python_bowling", "use_case": null, "tasksets": ["aider_polyglot"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "anthropic", "commit": "5522c6f44ca0ac3528b37244818fbfa39b5af470"}, {"recipes": 0, "summary": null, "tasks_passed": 1, "harness": "earendil-works-pi", "first_run": "2026-09-23T13:20:57", "domains": null, "runs": 1, "base_image": null, "finished": 1, "last_run": "2026-09-23T13:20:57", "repo": "https://github.com/earendil-works/pi", "passes": 1, "last_run_id": "20260923T130856-earendil-wor-polyglot_python_bowling", "use_case": null, "tasksets": ["aider_polyglot"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "anthropic", "commit": "7fd564cbb78f35f3de14d5382fea692b87ec4026"}, {"recipes": 0, "summary": null, "tasks_passed": 0, "harness": "emanuelecz-docent", "first_run": "2026-09-23T13:32:22", "domains": null, "runs": 1, "base_image": null, "finished": 1, "last_run": "2026-09-23T13:32:22", "repo": "https://github.com/emanuelecz/Docent", "passes": 0, "last_run_id": "20260923T132057-emanuelecz-d-polyglot_python_bowling", "use_case": null, "tasksets": ["aider_polyglot"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "anthropic", "commit": "33561785cc1915dad25a1759149caf56dc815925"}, {"recipes": 0, "summary": null, "tasks_passed": 0, "harness": "exlee-rik", "first_run": "2026-09-23T11:51:02", "domains": null, "runs": 1, "base_image": null, "finished": 1, "last_run": "2026-09-23T11:51:02", "repo": "https://github.com/exlee/rik", "passes": 0, "last_run_id": "20260923T114213-exlee-rik-polyglot_python_bowling", "use_case": null, "tasksets": ["aider_polyglot"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "openai", "commit": "6a393cacd39df2ee6581b4fc1d915e06b7c7fde0"}, {"recipes": 0, "summary": null, "tasks_passed": 1, "harness": "exoharness-exo", "first_run": "2026-09-23T13:46:56", "domains": null, "runs": 1, "base_image": null, "finished": 1, "last_run": "2026-09-23T13:46:56", "repo": "https://github.com/exoharness/exo", "passes": 1, "last_run_id": "20260923T132508-exoharness-e-polyglot_python_bowling", "use_case": null, "tasksets": ["aider_polyglot"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "anthropic", "commit": "1110ec2f9b9767c0f072181473d2b680fcc41335"}, {"recipes": 0, "summary": null, "tasks_passed": 0, "harness": "getworkloop-gambit", "first_run": "2026-09-23T13:17:22", "domains": null, "runs": 1, "base_image": null, "finished": 1, "last_run": "2026-09-23T13:17:22", "repo": "https://github.com/getworkloop/gambit", "passes": 0, "last_run_id": "20260923T125758-getworkloop--polyglot_python_bowling", "use_case": null, "tasksets": ["aider_polyglot"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "openai", "commit": "d413df73f7262f7eb9dee56721249caeb8e759a2"}, {"recipes": 1, "summary": "gptme is a terminal-first AI agent CLI with tool use. The entrypoint is `gptme` (Python console script from `gptme.cli.main:main`). It is invoked with `gptme --non-interactive \"$TASK\"`, which starts a non-interactive session: no confirmation prompts, no tty required. The harness uses the Anthropic provider: `ANTHROPIC_API_KEY=proxy` provides the key, `LLM_PROXY_URL=$PROXY_URL` redirects all Anthropic SDK calls to the proxy (the Anthropic SDK appends `/v1/messages` to this base URL, matching the proxy's Anthropic endpoint), and `GPTME_MODEL=anthropic/claude-sonnet-4-6` selects the model. A summary/title model (claude-haiku-4-5) is also called automatically. The harness is installed via `uv` into `/opt/harness/venv` from the repo source.", "compatibility": null, "harness": "gptme-gptme", "first_run": "2026-09-26T07:31:42", "domains": ["swe", "devops-sre"], "runs": 1, "base_image": "buildpack-deps:jammy", "finished": 1, "last_run": "2026-09-26T07:31:42", "repo": "https://github.com/gptme/gptme", "last_run_id": "20260926T072636-gptme-gptme-c876fe156da2-polyglot_cpp_allergies", "use_case": "Runs a coding agent in any terminal to write code, fix tests, edit files, run shell commands, browse the web, and operate autonomously on a schedule", "tasksets": ["aider_polyglot"], "scored": 0, "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "anthropic", "commit": "9e3ab7cf5418ae623d4b01ad7710fc9cfcf43a9c"}, {"recipes": 0, "summary": null, "tasks_passed": 0, "harness": "huangruiteng-loopx", "first_run": "2026-09-23T13:49:43", "domains": null, "runs": 1, "base_image": null, "finished": 1, "last_run": "2026-09-23T13:49:43", "repo": "https://github.com/huangruiteng/loopx", "passes": 0, "last_run_id": "20260923T133805-huangruiteng-polyglot_python_bowling", "use_case": null, "tasksets": ["aider_polyglot"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "openai", "commit": "f3d1e8c4ff1958faf3484bf3dc651a040a41a741"}, {"recipes": 0, "summary": null, "tasks_passed": 0, "harness": "irthomasthomas-llm-consortium", "first_run": "2026-09-23T15:30:59", "domains": null, "runs": 1, "base_image": null, "finished": 1, "last_run": "2026-09-23T15:30:59", "repo": "https://github.com/irthomasthomas/llm-consortium", "passes": 0, "last_run_id": "20260923T152744-irthomasthom-polyglot_python_bowling", "use_case": null, "tasksets": ["aider_polyglot"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "openai", "commit": "11c7f93325ec9e8809c3976708527c8a4d58452d"}, {"recipes": 0, "summary": null, "tasks_passed": 0, "harness": "kstenerud-yoloai", "first_run": "2026-09-23T13:57:27", "domains": null, "runs": 1, "base_image": null, "finished": 1, "last_run": "2026-09-23T13:57:27", "repo": "https://github.com/kstenerud/yoloai", "passes": 0, "last_run_id": "20260923T134534-kstenerud-yo-polyglot_python_bowling", "use_case": null, "tasksets": ["aider_polyglot"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "anthropic", "commit": "8fa6150d21ff04f975eb9545b05f16ca60c02759"}, {"recipes": 0, "summary": null, "tasks_passed": 0, "harness": "letta-ai-letta-code", "first_run": "2026-09-23T14:03:17", "domains": null, "runs": 1, "base_image": null, "finished": 1, "last_run": "2026-09-23T14:03:17", "repo": "https://github.com/letta-ai/letta-code", "passes": 0, "last_run_id": "20260923T134857-letta-ai-let-polyglot_python_bowling", "use_case": null, "tasksets": ["aider_polyglot"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "anthropic", "commit": "02c5437c74c9875ba7bd783979dfef1f938454c6"}, {"recipes": 0, "summary": null, "tasks_passed": 0, "harness": "litellm-labs-lite-harness", "first_run": "2026-09-23T14:00:06", "domains": null, "runs": 1, "base_image": null, "finished": 1, "last_run": "2026-09-23T14:00:06", "repo": "https://github.com/LiteLLM-Labs/lite-harness", "passes": 0, "last_run_id": "20260923T135024-litellm-labs-polyglot_python_bowling", "use_case": null, "tasksets": ["aider_polyglot"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "anthropic", "commit": "dd99cfdfc68dbb6b3f7f986d54efd42572373a6c"}, {"recipes": 0, "summary": null, "tasks_passed": 1, "harness": "mrsirg97-rgb-rig", "first_run": "2026-09-23T14:08:45", "domains": null, "runs": 1, "base_image": null, "finished": 1, "last_run": "2026-09-23T14:08:45", "repo": "https://github.com/mrsirg97-rgb/rig", "passes": 1, "last_run_id": "20260923T140038-mrsirg97-rgb-polyglot_python_bowling", "use_case": null, "tasksets": ["aider_polyglot"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "openai", "commit": "8441fd8534edcca00e472da5c159131f6fe11dcd"}, {"recipes": 0, "summary": null, "tasks_passed": 0, "harness": "mudler-nib", "first_run": "2026-09-23T13:35:01", "domains": null, "runs": 1, "base_image": null, "finished": 1, "last_run": "2026-09-23T13:35:01", "repo": "https://github.com/mudler/nib", "passes": 0, "last_run_id": "20260923T132417-mudler-nib-polyglot_python_bowling", "use_case": null, "tasksets": ["aider_polyglot"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "openai", "commit": "75c77ce0d100d2f9c8d4ced31a506742cd877a41"}, {"recipes": 0, "summary": null, "tasks_passed": 0, "harness": "nex-crm-wuphf", "first_run": "2026-09-23T14:23:06", "domains": null, "runs": 1, "base_image": null, "finished": 1, "last_run": "2026-09-23T14:23:06", "repo": "https://github.com/nex-crm/wuphf", "passes": 0, "last_run_id": "20260923T141137-nex-crm-wuph-polyglot_python_bowling", "use_case": null, "tasksets": ["aider_polyglot"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "anthropic", "commit": "f396ad5ff7be9dc06583077c8c6ac2cbfe161bb7"}, {"recipes": 0, "summary": null, "tasks_passed": 0, "harness": "nousresearch-hermes-agent", "first_run": "2026-09-23T14:30:03", "domains": null, "runs": 1, "base_image": null, "finished": 1, "last_run": "2026-09-23T14:30:03", "repo": "https://github.com/NousResearch/hermes-agent", "passes": 0, "last_run_id": "20260923T141731-nousresearch-polyglot_python_bowling", "use_case": null, "tasksets": ["aider_polyglot"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "openai", "commit": "0087827510b6a184d512c90b08d4566fcf68b346"}, {"recipes": 0, "summary": null, "tasks_passed": 1, "harness": "roocodeinc-roo-code", "first_run": "2026-09-23T15:10:23", "domains": null, "runs": 1, "base_image": null, "finished": 1, "last_run": "2026-09-23T15:10:23", "repo": "https://github.com/RooCodeInc/Roo-Code", "passes": 1, "last_run_id": "20260923T150100-roocodeinc-r-polyglot_python_bowling", "use_case": null, "tasksets": ["aider_polyglot"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "anthropic", "commit": "b867ec9145750d0ae1ff7f02d35406e9bf2a0b16"}, {"recipes": 0, "summary": null, "tasks_passed": 0, "harness": "ruvnet-ruflo", "first_run": "2026-09-23T15:31:54", "domains": null, "runs": 1, "base_image": null, "finished": 1, "last_run": "2026-09-23T15:31:54", "repo": "https://github.com/ruvnet/ruflo", "passes": 0, "last_run_id": "20260923T151336-ruvnet-ruflo-polyglot_python_bowling", "use_case": null, "tasksets": ["aider_polyglot"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "anthropic", "commit": "0a96fb8857dabd343d71d76c3ca703100a2923bc"}, {"recipes": 1, "summary": "Company Brain is a Cloudflare Workers-based Slack bot that uses Durable Objects (CompanyBrainAgent) to maintain conversational state. It handles Slack events via HTTP webhooks and makes LLM calls through the Vercel AI SDK with support for Anthropic (@ai-sdk/anthropic), OpenAI (@ai-sdk/openai), Google (@ai-sdk/google), and xAI (@ai-sdk/xai) providers, routing through Cloudflare AI Gateway when configured. The application has no CLI entrypoint - src/worker.ts exports a Hono HTTP handler, not a task runner. Model selection happens in src/brain/turn/brain-model.ts via getBrainModel(), which resolves providers based on available API keys and routes through wrapBrainGateway().", "harness": "supermemoryai-company-brain", "first_run": "2026-09-26T04:19:17", "domains": ["other"], "runs": 1, "base_image": "buildpack-deps:jammy", "finished": 1, "last_run": "2026-09-26T04:19:17", "repo": "https://github.com/supermemoryai/company-brain", "last_run_id": "20260926T041003-supermemorya-6304726e1230-polyglot_cpp_allergies", "use_case": "Deploys a Slack bot that remembers company conversations, answers from memory, and acts in connected tools like GitHub and Linear", "tasksets": ["aider_polyglot"], "scored": 0, "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "anthropic", "commit": "0071d6164991ce5dccddbd645bcac631ee477572"}, {"recipes": 0, "summary": null, "tasks_passed": 0, "harness": "tarruda-neoagent", "first_run": "2026-09-23T15:26:01", "domains": null, "runs": 1, "base_image": null, "finished": 1, "last_run": "2026-09-23T15:26:01", "repo": "https://github.com/tarruda/neoagent", "passes": 0, "last_run_id": "20260923T152255-tarruda-neoa-polyglot_python_bowling", "use_case": null, "tasksets": ["aider_polyglot"], "models": ["bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"], "tasks_tried": 1, "api_style": "openai", "commit": "69dc9a11992f33a07d02243e03c9ad249bb8e9bb"}]}