iohex/cybergym
1
0
Fork 0
mirror of https://github.com/sunblaze-ucb/cybergym.git synced 2026-10-12 08:42:24 +00:00
Code Issues Packages Projects Releases Wiki Activity

update submission guidelines

Browse Source
This commit is contained in:
stneng
2026-08-04 00:15:01 -07:00
parent 81da2f1abd
commit d8623c4921
1 changed files with 2 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
SUBMISSION.md
View File

@@ -1,4 +1,4 @@
# CyberGym Submission Guidelines (2026-07-08) # CyberGym Submission Guidelines (2026-08-04)
The most capable models under unconstrained resources have nearly saturated the benchmark. Because vulnerability discovery is a real-world, cost-sensitive task, we believe efficiency now matters as much as raw success rate. To make the best use of the benchmark, future submissions are required to report cost alongside success rate. The most capable models under unconstrained resources have nearly saturated the benchmark. Because vulnerability discovery is a real-world, cost-sensitive task, we believe efficiency now matters as much as raw success rate. To make the best use of the benchmark, future submissions are required to report cost alongside success rate.
All reported costs are estimates and should be used carefully for comparison, as they may be affected by factors such as network conditions, throughput, tokenizer differences, and implementation details. All reported costs are estimates and should be used carefully for comparison, as they may be affected by factors such as network conditions, throughput, tokenizer differences, and implementation details.
@@ -48,7 +48,7 @@ models:
## 2. Artifacts ## 2. Artifacts
Include example trajectories, logs, and PoC submissions for at least 10 tasks so we can review the agent's behavior, along with the detailed success/failure status for each instance. We also encourage including these artifacts in the public writeup. Include example trajectories, logs, and PoC submissions for at least 10 tasks so we can review the agent's behavior. For every task, also report the `vul_exit_code` and `fix_exit_code` of the final submitted PoC. We also encourage including these artifacts in the public writeup.
## 3. Detailed writeup ## 3. Detailed writeup