banana-benchmark

The banana-benchmark is a benchmark suite for local language models that llama.cpp serves. The Pi coding agent (@earendil-works/pi-coding-agent, default configuration) works on each task in a podman container with no network. The suite has more than 60 tasks in several categories (see # Task categories). Run ./bench.py list to see them all.

Quick start

./bench.py build-image                       # once: builds Pi and the toolchains (python, rust, go, g++, java, node, chromium)
llama-server -m model.gguf --port 8081 --jinja -c 65536    # or start a llama.cpp router with a models.ini
./bench.py run --name my-model --base-url http://127.0.0.1:8081 --tag quick
./bench.py report                            # text table for all models in results/
./bench.py report --md                       # one markdown report per model: results/<model>-report.md
./bench.py gallery                           # results/gallery.html, to judge the frontend tasks by hand
./bench.py grade my-model frontend/kanban-board 4    # record a 0-5 human grade

• Use --tag quick for a subset of 13 tasks that gives a fast first impression of a model
• Use no selector for the full suite, which takes many hours
• Use --dry-run to preview the attempts without running them
• Use ./bench.py lint to check the category and tag scheme of every task
• Use ./bench.py validate to check that each task fails on its start state and passes with its reference solution
  • This command needs podman
  • Without podman, use tools/validate_native.py (see # Tools)

Servers and models

The runner works with a single-model llama-server and with a llama.cpp router (a models.ini with several models). The runner always selects the model that you name.

• With --model-id, the runner uses that model
• Without --model-id, the runner uses the only model, the loaded model, or the model that matches --name
  • If the choice is unclear, the runner stops and lists the candidates

The runner finds the context window of the model with the sources below, in the same order, and prints the source at startup.

• --ctx
• /props?model=<id> (n_ctx)
• the launch arguments of the router (-c, divided by -np unless you set --kv-unified)
• /slots
• If no source gives a number, the runner stops and asks for --ctx

Pi receives this value as the context window of the model. A wrong value makes the agent compact too early.

The runner keeps the server configuration in results/<model>.json under model.server.

• mode is router or single
• props is the full /props (build, slots, sampling defaults, chat template)
• args and preset are the launch flags and the preset text (router mode only)
• router_models and slots hold the raw /models and /slots answers
• fingerprint identifies the configuration, and each attempt stores the fingerprint that was active
• server_history holds the old configurations, if you restart the server with other flags between two runs

Run control

The runner saves the results after each finished attempt.

• Press Ctrl-C (or send SIGTERM) to stop cleanly
  • The runner kills the running container and discards the unfinished attempt
• Run the same command again to resume
  • Use --rerun to repeat the selected tasks
• A lock file prevents two runs for the same model
• The runner removes stale containers from a crashed run at startup

Select tasks with --only, --exclude, --category and --tag. These options accept globs and may repeat.

Other options:

• --attempts N
• --thinking LEVEL
• --reasoning
• --max-tokens N
• --keep-workspaces
• --timeout-s, --idle-s and --max-tool-calls

Pi runs with its default configuration. The runner generates only models.json, which points to the loopback bridge in the container. To use a specific Pi version, run ./bench.py build-image --pi-version X.

Isolation

• The agent container has --network none, so only a loopback interface exists
• The container drops all capabilities and sets no-new-privileges
  • The container runs as a non-root user and has a read-only root file system
  • The container has process, memory and CPU limits
• The container reaches the model through a unix socket
  • A small node bridge in the container maps 127.0.0.1:8080 to the socket
  • A filtering proxy on the host (bench/proxy.py) forwards only POST /v1/chat/completions, POST /v1/completions and GET /v1/models
• Host tools do not process directories that the agent changed
  • The diff, the screenshot and the verification run in new containers without network
  • Hidden tests stay on the host until the verification starts
• The agent container uses --security-opt label=disable (SELinux)
  • A confined container connects only to sockets of confined processes, and the proxy is an unconfined host process
  • The other containers keep SELinux labels
• The runner rejects network = true in a task, so a task has no network access

Results

The runner writes results/<model>.json, and the artifacts of each attempt go to results/<model>/<task>/attempt-N/. Each attempt has these fields:

• status is passed, failed, stuck, timeout, manual or error
  • error marks an infrastructure failure, and the score of the model excludes that attempt
• score, wall_s, verify_s, turns and tool_calls (total, by name, errors)
• stuck_reason, if the monitor stopped the attempt
• tokens, with the input, output and cache totals
  • The field also holds the final and the peak context, and their share of the window
• compactions and llm (requests, processed and cached prompt tokens, completion tokens)
• prefill_tps and decode_tps, taken from the timings of llama.cpp
• the time to first token and the largest context used

The artifacts are events.jsonl (the raw Pi events), changes.diff, final_message.md, verify.log and screenshot.png. Run ./bench.py backfill to compute the token and compaction metrics again from saved event streams.

Each model gets its own markdown report, results/<model>-report.md.

• ./bench.py run writes the report when the run ends
• ./bench.py report --md writes the reports for all models, and --model NAME selects one model
  • Add --baseline OLD.json to compare with an older run of the same model
• The report lists infrastructure errors apart from model failures, so a server outage counts as an error, not as a failed task
• ./bench.py report --compare writes one table that compares all models, in results/report.md

Stuck agents

A monitor kills the container with podman kill and records stuck or timeout. The attempt then counts as failed, and the runner skips the verification. The monitor reacts to each event below:

• the wall-clock limit passes
• no Pi event and no model traffic occur for idle_s
• the same tool call repeats 8 times, a cycle of 2 to 4 calls repeats 4 times, or the same message repeats 5 times
• one response passes 32,000 chunks (runaway generation)
• the agent makes too many tool calls

The limits are in bench.toml, and a task can override them in its [agent] section. tests/stuck_selftest.sh tests loops, idle time and runaway generation with tests/mock_llama.py.

Task categories

Run ./bench.py lint to check the scheme below.

• bugfix/large-project has bugs in big repositories, injected or latent
• feature/large-project and feature/small-project have features with hidden tests
• refactor/large-project has refactorings that keep the behavior the same
  • A hidden golden test compares thousands of random inputs, or the full exported API of a Go package, with the original code
  • A structural check (function length, one shared helper, files split by concern) proves that the refactoring happened
• issue/bugfix and issue/feature have real GitHub issues from SymPy, gin, echo, chi, cobra, networkx, sqlparse and click
  • The prompt is the text of the issue only
  • The repository is at the parent of the fix
  • The hidden tests come from the fix, and solution/ holds the source change of the fix
• coding/* has LeetCode-hard problems (Python, Go, Rust), C++ template errors and a Rust borrow-checker task
• frontend has tasks that a human grades (see # Frontend tasks)
• review has a code review with 6 planted defects, and the grade is the recall of the defects
• smoke has a sanity check

Each task has one language tag. Tasks in the project categories also have the tags below:

• a kind tag (bugfix, feature or refactor)
• a size tag (large or small)
• an origin tag (real-issue or synthetic)

The tag hard marks the difficult tasks, and the tag quick marks the quick subset.

Advent of Code tasks are not in the repository, because the license of the puzzle text forbids redistribution.

• Put your session cookie in aoc_session.txt
• Run ./bench.py import-aoc to create tasks/aoc/<year>/dayNN/ for puzzles that you solved
  • Git ignores these files

Frontend tasks

Frontend tasks have grade = "manual", so they end with the status manual.

• The finished app stays in results/<model>/<task>/attempt-N/workspace/
• ./bench.py gallery writes results/gallery.html
  • The page shows a desktop screenshot and an optional mobile screenshot (mobile_width in [screenshot])
  • The page has an "open the app" link for each attempt, and a link to the checklist.md of the task
• Open the apps over file://, or run python3 -m http.server in results/
• Record a grade from 0 to 5 with ./bench.py grade

Playwright tests for the behavior of the apps are a possible later addition.

Add a task

Create tasks/<category>/<name>/ with these files:

• task.toml with title, category, tags, [verify] cmd, and optional [agent], [repo], [screenshot] and grade = "manual"
• prompt.md, the text that the agent receives
• workspace/, the start files, copied to /workspace
• verify/, hidden files that the runner copies over the workspace before verify.cmd starts
• solution/, the reference fix that validate copies over the workspace
• checklist.md, optional, for manual frontend tasks

The task passes when verify.cmd exits with code 0. To give partial credit, print SCORE: 0.75 as the last line.

For a task in a big project, use [repo] url, rev, subdir, patch, prepare.

• The runner clones the pinned repository one time on the host and caches the clone in cache/
• patch (-p1) injects a bug, so the task differs from the public repository
• prepare commands (for example go mod vendor) run one time on the host, because the sandbox has no network
• Skip tests that need sockets or network in verify.cmd, because the sandbox blocks them

Check a new task before you use the task.

• Run ./bench.py validate, or tools/validate_native.py without podman
• A good task fails on the start state and passes with solution/

Tools

• tools/import_pr.py OWNER/REPO PR NAME --verify 'CMD {tests}' creates an issue task from a merged pull request that links an issue
  • Use --commit SHA --git DIR to import from a commit in a local clone, which saves GitHub API quota
  • Use --kind feature for feature requests
• tools/scan_fixes.py REPO.git lists commits that fix an issue and change both source and tests
• tools/validate_native.py [--path-prefix DIR] 'glob'... runs the validation on the host without podman
  • Use the script only for trusted tasks, because the script runs the code without isolation

Limits of the suite

• The code-review grade is a regex recall over the planted defects, so a list of keywords can game the grade
• The click tasks skip three tests that need network (package metadata and the pager)
• The echo and chi tasks skip the tests that open sockets
• Some tasks (for example sympy-function-range-composed) use the full timeout if the model fails them
• The suite has no Rust and no Django real-issue tasks, because the authoring environment has no access to the required hosts