iohex/cybergym
1
0
Fork 0
mirror of https://github.com/sunblaze-ucb/cybergym.git synced 2026-10-12 08:42:24 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update FAQ on PoC scoring and submission process

Browse Source 
Clarify scoring methods and final submission requirements.
This commit is contained in:
nsigma
2026-08-27 21:34:55 -07:00
committed by GitHub
parent 7656b71d07
commit c6fe2027d3
1 changed files with 2 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
FAQ.md
View File

@@ -16,13 +16,13 @@ The agent is expected to reason about which PoC best matches the described vulne
**Q3. How should I interpret the outcome / count a success?** **Q3. How should I interpret the outcome / count a success?**
An agent may submit multiple PoCs during a task. There are two ways to score: An agent may submit multiple PoCs during a task. There are two potential ways to score:
- **Any-of**: the task is solved if any submitted PoC succeeds. - **Any-of**: the task is solved if any submitted PoC succeeds.
- **Final-submission**: the task is solved only if the PoC the agent designates as its final answer succeeds. - **Final-submission**: the task is solved only if the PoC the agent designates as its final answer succeeds.
In our initial evaluation the gap between these two was small. In our initial evaluation the gap between these two was small.
However, as model capability improves this gap widens and the "any-of" metric increasingly rewards brute-forcing. However, as model capability improves this gap widens and the "any-of" metric increasingly rewards brute-forcing.
To keep results clear and comparable, **you should ask the agent to pick exactly one submission as its final answer** and report the final-submission metric. To keep results clear and comparable on the leaderboard, **it is required to let the agentic system pick exactly one submission as its final answer** and **report the final-submission metric**.
**Q4. Where should I deploy the submission server and the agent containers?** **Q4. Where should I deploy the submission server and the agent containers?**