iohex/cybergym
1
0
Fork 0
mirror of https://github.com/sunblaze-ucb/cybergym.git synced 2026-10-12 08:42:24 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update submission guidelines to include cost reporting

Browse Source 
Added a note about cost estimates for submissions.
This commit is contained in:
nsigma
2026-07-02 23:27:41 -07:00
committed by GitHub
parent f4269d7235
commit 97cf5b0a91
1 changed files with 1 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
SUBMISSION.md
View File

@@ -1,6 +1,7 @@
# CyberGym Submission Guidelines (2026-06-30) # CyberGym Submission Guidelines (2026-06-30)
The most capable models under unconstrained resources have nearly saturated the benchmark. Because vulnerability discovery is a real-world, cost-sensitive task, we believe efficiency now matters as much as raw success rate. To make the best use of the benchmark, future submissions are required to report cost alongside success rate. The most capable models under unconstrained resources have nearly saturated the benchmark. Because vulnerability discovery is a real-world, cost-sensitive task, we believe efficiency now matters as much as raw success rate. To make the best use of the benchmark, future submissions are required to report cost alongside success rate.
All reported costs are estimates and should be used carefully for comparison, as they may be affected by factors such as network conditions, throughput, tokenizer differences, and implementation details.
A submission must include the following: A submission must include the following: