From 97cf5b0a9179e3ab52c84ff3a5f347a20a49c5c6 Mon Sep 17 00:00:00 2001 From: nsigma Date: Thu, 2 Jul 2026 23:27:41 -0700 Subject: [PATCH] Update submission guidelines to include cost reporting Added a note about cost estimates for submissions. --- SUBMISSION.md | 1 + 1 file changed, 1 insertion(+) diff --git a/SUBMISSION.md b/SUBMISSION.md index 5365ea9..69db3e3 100644 --- a/SUBMISSION.md +++ b/SUBMISSION.md @@ -1,6 +1,7 @@ # CyberGym Submission Guidelines (2026-06-30) The most capable models under unconstrained resources have nearly saturated the benchmark. Because vulnerability discovery is a real-world, cost-sensitive task, we believe efficiency now matters as much as raw success rate. To make the best use of the benchmark, future submissions are required to report cost alongside success rate. +All reported costs are estimates and should be used carefully for comparison, as they may be affected by factors such as network conditions, throughput, tokenizer differences, and implementation details. A submission must include the following: