{"taskset": {"n_runnable": 0, "name": "Finance Agent Benchmark", "url_data": "https://huggingface.co/datasets/vals-ai/finance_agent_benchmark", "catalog_id": "vals-finance-agent", "owner_type": "company", "cited_by": ["Gemini 3.8 Flash", "Kimi K3", "Nemotron 3 Ultra"], "taskset": "financeagent_terminal", "domain": "finance", "task_kind": "financial research on SEC filings with tools", "environment": "live-web", "grading": "llm-judge", "path": "datasets/financeagent_terminal", "n_single_container": 0, "n_tasks": 50, "domain_raw": "finance", "sample_instruction": "You are a financial agent. Today is April 07, 2025. You are given a question and you need to answer it using the tools provided.\nYou can only use the built-in and provided tools. The provided tools are:\n1. google_web_search\n\nSearch the web for information using Google Search. Results are limited to content before April 7, 2025.\n\n**Usage:**\n```bash\npython3 /app/tools/google_search.py --search-query \"your query here\"\n```\n\n**Required Arguments:**\n- `--search-query`: The query to search for\n\n**Note:** Returns up to 10 results by default. All results are from dates before April 7, 2025.\n\n---\n\n2. edgar_search\n\nSearch the EDGAR Database through the SEC API. You should provide a query, a list of form types, a list of CIKs, a start date, an end date, a page number, and a top N results. The results are returned as a list of dictionaries, each containing the metadata for a filing. It does not contain the full text of the filing.\n\n**Usage:**\n```bash\npython3 /app/tools/edgar_search.py   --query \"search keywords\"   --form-types '[\"10-K\", \"10-Q\"]'   --ciks '[\"0000320193\"]'   --start-date \"2024-01-01\"   --end-date \"2024-12-31\"   --page \"1\"   --top-n-results 10\n```\n\n**Required Arguments (ALL 7 are required):**\n- `--query`: The keyword or phrase to search, such as 'substantial doubt' OR 'material weakness'\n- `--form-types`: Limits search to specific SEC form types (e.g., ['8-K', '10-Q']) list of strings. Default is None (all form types). Use `'[]'` for all form types.\n- `--ciks`: Filters resul", "owner_org": "Vals AI", "url_repo": "https://github.com/vals-ai/finance-agent", "url_paper": "https://arxiv.org/abs/2508.00828", "license": "CC-BY-4.0", "languages": [], "difficulties": {"?": 50}, "categories": ["financeQA"]}, "tasks": [{"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-0", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-1", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-10", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-11", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-12", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-13", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-14", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-15", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-16", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-17", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-18", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-19", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-2", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-20", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-21", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-22", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-23", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-24", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-25", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-26", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-27", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-28", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-29", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-3", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-30", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-31", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-32", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-33", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-34", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-35", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-36", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-37", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-38", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-39", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-4", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-40", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-41", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-42", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-43", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-44", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-45", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-46", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-47", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-48", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-49", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-5", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-6", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-7", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-8", "taskset": "financeagent_terminal"}, {"agent_timeout": 3600, "category": "financeQA", "compose": true, "difficulty": "", "language": "", "oracle": null, "runnable": false, "tags": ["finance,version:terminal"], "task": "financeagent-9", "taskset": "financeagent_terminal"}], "next": null}