Benchmarks for agentic scientific research

OpenScience leads every public benchmark for agentic scientific research that we have run. Each benchmark runs in its native environment, with its own verifiers or judge, through Harbor on Modal, and we report each in its own metric. Every other entry shown is from the benchmark's public leaderboard or a published comparison on the same tasks and grader.²

Terminal-Bench Science

Terminal-Bench Science is 70 long-horizon research tasks, each with its own verifier and an 8-hour limit, in fields from cosmological inference and spatial transcriptomics to formal proof and hydrology. OpenScience solves 75.7%. The strongest public entry, Codex with the same lead model (GPT-6 Astra), solves 68.1%; the strongest Claude Code entry (Opus 5.5) solves 63.3%.

Figure 1.OpenScience and all 17 entries on the public Terminal-Bench Science leaderboard.

The margin is largest in the life sciences, 73.7% against 59.6% for Codex. Engineering, with nine tasks, is the one domain where Codex is ahead (70.4% against 55.6%).

Figure 2.Resolution rate by scientific domain for OpenScience and the strongest Codex and Claude Code entries. Numbers above the rows are OpenScience's.

OpenScience also solved 2 tasks, cmb-cross-inference and variable-star-vetting, that no leaderboard entry solved in any of its 51 attempts.

Figure 3.Every Terminal-Bench Science task, grouped by domain. Cell shade is the fraction of attempts that passed (OpenScience one, leaderboard entries three); open cells never passed. Triangles mark tasks that only OpenScience solved.

Figure 4.OpenScience's wall-clock time per task (spoke length, log scale; the dashed ring is the 8-hour limit), ordered by time. Filled dots passed. The two tasks marked * were regraded after the run and have no recorded time.

Terminal-Bench 4.0

On the 14 science tasks of Terminal-Bench 4.0 OpenScience solves 71.4%. We scored all 27 public entries on the same tasks from their published per-trial results: the best is Claude Code with Fable 5.1 (high) at 60%, and the best Codex entry reaches 45.7%.

Figure 5.OpenScience and every public Terminal-Bench 4.0 entry on the 14 science tasks. Reasoning effort in parentheses.

BiomniBench-DA

BiomniBench-DA asks for open-ended analyses of published biomedical datasets. The agent may not read the source paper, and a Gemini 3.1 Pro judge grades the full trace against a rubric written with the paper's authors. Over the 50 public tasks OpenScience averages 82.2, with 20 perfect scores. AIPOCH Open Science reports 81.04 under the same judge, and OmicOS 76.6.

Figure 6.OpenScience and the published BiomniBench-DA results on the same 50 tasks and judge. The row marked * is OpenScience run by the BiomniBench-AI4S authors with DeepSeek V4 Pro, the model every entry below AIPOCH uses.

Figure 7.OpenScience's score on each of the 50 BiomniBench-DA tasks.

OpenScience Bench

The public benchmarks each test part of the work. OpenScience Bench, our internal benchmark, measures end-to-end scientific research, and every entry is scored pass@3. OpenScience with GPT-6 Astra scores 47.3%, ahead of Claude Science with Fable 5.1 (40.6%), K-Dense (BYOK) with GPT-6 Astra (40.3%) and Codex with GPT-6 Astra (38.1%); Grok Build with Grok 4.7 scores 12.2%. Three of those entries run the same model, so their gap is the harness: 7.0 points over K-Dense and 9.2 over Codex.

Figure 8.OpenScience Bench, pass@3 for every entry. The model each harness ran is after the dot.

How OpenScience works

At the centre is a small, general agent loop. The science lives around it: in skills, in the agent definitions, and in a prompt header for each model family. A few optional harness units add checks at the end of a turn: one re-reads a written report against the original request in a fresh context, another enforces the budget. A unit stays only if it improves measured outcomes.

Because the loop is model-agnostic, the product is too. The science instructions are identical across model families, no built-in agent is tied to a model, and each lab can run the model it trusts.

Figure 9.How a request moves through OpenScience. The lead agent loads skills as a task needs them, sends briefs to workers and gets back reports with a verification, calls tools and compute, and returns Results with their provenance.

An agent built for scientific work

Scientific work follows procedures, and OpenScience carries 371 of them as skills in 18 categories: how a field runs a single-cell analysis, a docking screen, a DESeq2 contrast or a Lean proof. The agent always sees an index and loads a skill's full text when a task needs it. Much of the library comes from other groups: 180 skills from K-Dense, 79 from Orchestra Research, and more from Hugging Face, Anthropic and NVIDIA BioNeMo.

Figure 10.The skill library, one square per skill, grouped by category and coloured by field.

It reaches data through 42 connectors to scientific databases in 6 domains, and literature through a tool that searches OpenAlex and arXiv and then reads the full text of the paper. It runs code in long-lived Python and R kernels and a shell, and sends heavier jobs to an SSH host, a Slurm or PBS cluster, or Modal as durable jobs that outlive the conversation. Each remote dispatch needs your approval.

Figure 11.The 42 database connectors, grouped by domain.

A lab of specialists

The lead agent can hand bounded tasks to seven workers in parallel: explore, a scout for literature, data and code; five domain specialists (ml, biology, physics, chemistry, data); and general. Workers cannot start other workers or publish anything. A worker's report without a verification section counts as a claim, and when two workers compute the same quantity independently and disagree, the disagreement is reported as a finding.

Figure 12 shows one Terminal-Bench Science run, hysteretic-aquifer-control: calibrate a four-state aquifer model, identify miscalibrated devices, forecast all four states and design a pump-and-treat intervention. The lead brought in ml, data and physics workers. The run took 3.1 hours and passed the task's verifier.

Figure 12.Tool calls over time in one run, one row per session, labelled with the brief the lead wrote.

Logistic map x_n+1 = r x_n(1 − x_n), 2.85 ≤ r ≤ 4

Availability

OpenScience is released under Apache 2.0. It works with your own API keys (22 provider SDKs are bundled), with local models through Ollama or LM Studio, or with Ace, the pay-as-you-go service we run. Usage through your own keys or a local model is never billed through Ace.

Install the desktop app, or use the terminal:

$ npm install -g @synsci/openscience
$ openscience ~/research/my-project

# connect a model: your own key, or a local endpoint
$ openscience keys add
$ openscience local add

Then describe the task as you would to a colleague:

Inspect data/samples.csv for missing values and inconsistent labels. Keep the original data unchanged. Save a quality report and a plot in results/, with the code needed to reproduce them.

The source is on GitHub, the documentation at docs, and contributions are welcome.

Notes

Figures and counts are computed by the launch post's scripts from OpenScience (commit 132dfdf), benchmarks-openscience (commit 61328d9) and snapshots of the public leaderboards taken 2026-09-26. The build checks the recomputed scores against the benchmarks repository's README.
OpenScience runs: Terminal-Bench Science with a GPT-6 Astra lead, GPT-6 Sol workers and 8 hours per task; BiomniBench-DA with GPT-6 Sol and 1 hour per task; Terminal-Bench 4.0 with a GPT-6 Astra lead and GPT-5.6 Sol workers. OpenScience rows report one counted trial per task. Leaderboard entries average 3 trials per task on Terminal-Bench Science and 5 on Terminal-Bench 4.0. Other entries come from the Terminal-Bench Science and Terminal-Bench 4.0 leaderboards, the BiomniBench-AI4S comparison (Gemini 3.1 Pro judge column) and AIPOCH's published result.

Citation

@misc{gangwani2026openscience,
  title        = {OpenScience: The #1 Scientific Agent, Open Source},
  author       = {Gangwani, Ishaan and Bansal, Aayam},
  year         = {2026},
  month        = sep,
  organization = {Synthetic Sciences},
  howpublished = {\url{https://github.com/synthetic-sciences/OpenScience}}
}