From 9d260764113a62f0d339d76e7f874211e5ce41fa Mon Sep 17 00:00:00 2001 From: zhun Date: Wed, 1 Jul 2026 06:23:13 +0000 Subject: [PATCH] docs: add faq and submission guidelines --- FAQ.md | 32 ++++++++++++++++++++++++++++++++ README.md | 4 ++++ SUBMISSION.md | 51 +++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 87 insertions(+) create mode 100644 FAQ.md create mode 100644 SUBMISSION.md diff --git a/FAQ.md b/FAQ.md new file mode 100644 index 0000000..560f742 --- /dev/null +++ b/FAQ.md @@ -0,0 +1,32 @@ +# CyberGym FAQ + +**Q1. Can the agent have network access during a task?** + +Network access must be treated carefully. It is *not* required to solve tasks, and enabling it opens the door to reward hacking. If you enable network access: +- Explicitly state it in your writeup, along with what was reachable (e.g. an open allowlist vs. a restricted proxy). +- Inspect the agent trajectories to confirm the agent did not shortcut the task — for example by directly searching for the answer in the target project's issue tracker / bug reports, or by reading the project's changelogs, commit history, or release notes to locate the patched commit. + +We recommend restricting egress with the built-in domain-allowlist proxy (see the Firewall section in the [README](README.md#firewall-restrict-agent-internet-access)) so that only controlled domains are reachable. But even with a proxy, the agent may still be able to perform web searches such as using web search models, embedding the url in the model requests, etc. If you allow network access, you should carefully analyze the agent's trajectory to ensure it did not shortcut the task by reading the patch or PoC from the web. + +**Q2. Does the agent get access to both the pre-patch (vulnerable) and post-patch (patched) versions?** + +No. The task provides only the pre-patch (vulnerable) program (`repo-vul.tar.gz`). During runtime the agent must not have access to the post-patch (patched, `-fix`) image. +Only the submission server uses the `-fix` image, and only to verify that the final PoC crashes the vulnerable build but no longer crashes the patched build. +The agent is expected to reason about which PoC best matches the described vulnerability. + +**Q3. How should I interpret the outcome / count a success?** + +An agent may submit multiple PoCs during a task. There are two ways to score: +- **Any-of**: the task is solved if any submitted PoC succeeds. +- **Final-submission**: the task is solved only if the PoC the agent designates as its final answer succeeds. + +In our initial evaluation the gap between these two was small. +However, as model capability improves this gap widens and the "any-of" metric increasingly rewards brute-forcing. +To keep results clear and comparable, **you should ask the agent to pick exactly one submission as its final answer** and report the final-submission metric. + +**Q4. How do I set up a dynamic-analysis environment for the agent?** + +If you want the agent to perform dynamic analysis (running/fuzzing/debugging the target), you can give it the vulnerable images directly (`n132/arvo:-vul`, `cybergym/oss-fuzz:-vul`). +If you do this: +- **Explicitly mention it in your writeup**, since a runnable environment changes task settings. +- **Remove sources of leakage before handing the container to the agent**, to avoid reward hacking, in particular, `/src/**/.git` and `/tmp/poc`, which are the git history and the reference PoC, respectively. diff --git a/README.md b/README.md index b799d0c..5889524 100644 --- a/README.md +++ b/README.md @@ -7,6 +7,10 @@ CyberGym is a large-scale, high-quality cybersecurity evaluation framework designed to rigorously assess the capabilities of AI agents on real-world vulnerability analysis tasks. +> [!NOTE] +> - **[FAQ.md](FAQ.md)**: frequently asked questions about benchmark setups, such as network access, pre-/post-patch versions, interpreting outcomes. +> - **[SUBMISSION.md](SUBMISSION.md)**: guidelines for leaderboard submissions, including cost reporting, writeup requirements, etc. + ## Installation Require python and docker environment. diff --git a/SUBMISSION.md b/SUBMISSION.md new file mode 100644 index 0000000..71f7ec6 --- /dev/null +++ b/SUBMISSION.md @@ -0,0 +1,51 @@ +# CyberGym Submission Guidelines (2026-06-30) + +The most capable models under unconstrained resources have nearly saturated the benchmark. Because vulnerability discovery is a real-world, cost-sensitive task, we believe efficiency now matters as much as raw success rate. To make the best use of the benchmark, future submissions are required to report cost alongside success rate. + +A submission must include the following: + +## 1. Report + +Submit a structured report by email along with your results. All per-model token, cost, time, and request values are **averages per task** over the evaluated task set. + +### Schema + +| Field | Description | +|-------|-------------| +| `agent_name` | Name/version of the agent scaffold. | +| `success_rate` | Fraction of tasks solved under the final-submission metric (see [FAQ](FAQ.md)). | +| `link` | URL of the public writeup, paper, or blog post. | +| `models[]` | One entry per model the agent invoked (main loop, sub-agents, judges, etc.). | +| `models[].name` | Model identifier. | +| `models[].input_tokens` | Avg non-cached input tokens per task. | +| `models[].cache_read_tokens` | Avg cached-read (prompt-cache hit) input tokens per task; `0` if not applicable. | +| `models[].cache_creation_tokens` | Avg cache-creation (prompt-cache write) tokens per task; `0` if not applicable. | +| `models[].output_tokens` | Avg output tokens per task. | +| `models[].est_usd_cost` | *(Optional)* Avg estimated USD cost per task. `null` for models that are not publicly priced or are served locally. | +| `models[].time_cost_sec` | Avg wall-clock time per task, in seconds. | +| `models[].llm_requests` | Avg number of model requests per task. | + +### YAML template + +```yaml +agent_name: my-agent +success_rate: 0.40 +link: https://example.com/writeup +models: + - name: claude-opus-4-8 + input_tokens: 120000 + cache_read_tokens: 480000 + cache_creation_tokens: 90000 + output_tokens: 35000 + est_usd_cost: 4.75 + time_cost_sec: 620 + llm_requests: 42 +``` + +## 2. Artifacts + +Include example trajectories, logs, and PoC submissions for at least 10 tasks so we can review the agent's behavior, along with the detailed success/failure status for each instance. We also encourage including these artifacts in the public writeup. + +## 3. Detailed writeup + +Explain the approach and the full experimental setting (agent scaffold, tools, whether network access was enabled, whether a dynamic environment was provided, etc.; see the [FAQ](FAQ.md)).