diff --git a/FAQ.md b/FAQ.md index 560f742..d9a9d5a 100644 --- a/FAQ.md +++ b/FAQ.md @@ -24,7 +24,16 @@ In our initial evaluation the gap between these two was small. However, as model capability improves this gap widens and the "any-of" metric increasingly rewards brute-forcing. To keep results clear and comparable, **you should ask the agent to pick exactly one submission as its final answer** and report the final-submission metric. -**Q4. How do I set up a dynamic-analysis environment for the agent?** +**Q4. Where should I deploy the submission server and the agent containers?** + +**Locally, on a host you control, never exposed to the public internet.** CyberGym is a local evaluation harness, not a hosted service: +- The `/submit-vul` endpoint executes uploaded PoCs inside the target containers, so anyone who can reach the port can run arbitrary input against real unpatched vulnerabilities. +- The API key shipped in the README / `ServerConfig` is a **public placeholder**. Override it with `CYBERGYM_API_KEY`, but do not treat that as making the server safe to expose. +- Bind the server to the **gateway of the Docker network your agents run on** (`python3 -m cybergym.firewall status` reports it as `host_gateway`), not to `0.0.0.0`. That address is reachable from the agent containers and the host, but is not routable from outside the machine. See [Pick the bind address](README.md#pick-the-bind-address). + +If the agent and the server must be on different machines, put them on a private network or VPN — not on a public URL or a forwarded port. + +**Q5. How do I set up a dynamic-analysis environment for the agent?** If you want the agent to perform dynamic analysis (running/fuzzing/debugging the target), you can give it the vulnerable images directly (`n132/arvo:-vul`, `cybergym/oss-fuzz:-vul`). If you do this: diff --git a/README.md b/README.md index 5889524..3968796 100644 --- a/README.md +++ b/README.md @@ -8,9 +8,12 @@ CyberGym is a large-scale, high-quality cybersecurity evaluation framework designed to rigorously assess the capabilities of AI agents on real-world vulnerability analysis tasks. > [!NOTE] -> - **[FAQ.md](FAQ.md)**: frequently asked questions about benchmark setups, such as network access, pre-/post-patch versions, interpreting outcomes. +> - **[FAQ.md](FAQ.md)**: frequently asked questions about benchmark setups, such as network access, local deployment, pre-/post-patch versions, interpreting outcomes. > - **[SUBMISSION.md](SUBMISSION.md)**: guidelines for leaderboard submissions, including cost reporting, writeup requirements, etc. +> [!WARNING] +> **Deploy everything locally. Do not expose any part of CyberGym to the public internet.** + ## Installation Require python and docker environment. @@ -62,12 +65,38 @@ python scripts/server_data/download_subset.py ## Evaluation + +The server only needs to be reachable from the agent containers and from the host. `0.0.0.0` binds every interface, which on a machine with a public IP exposes a partly unauthenticated service to the internet. + +### Pick the bind address + +The right address is the **gateway of the Docker network your agent containers run on** — that address is a host-side interface reachable from those containers and from the host, but not routable from outside the machine. + +```bash +# If you use the firewall (agents on the isolated `cybergym-internal` network): +HOST=$(docker network inspect cybergym-internal -f '{{(index .IPAM.Config 0).Gateway}}') + +# If agents run on the default bridge instead (no firewall): +HOST=$(docker network inspect bridge -f '{{(index .IPAM.Config 0).Gateway}}') + +# If the agent runs directly on the host with no container at all: +HOST=127.0.0.1 + +echo $HOST # e.g. 192.168.48.1 for cybergym-internal, 172.17.0.1 for the default bridge +``` + +Notes: +- The two gateways are **not interchangeable**. `cybergym-internal` is created with `internal=True`, so containers on it have no route to the default bridge — a server bound to `172.17.0.1` is unreachable from firewalled agents. +- Docker assigns the `cybergym-internal` subnet when the network is created, so the gateway is host-specific. Query it rather than hardcoding it. `python3 -m cybergym.firewall start` and `status` both print it as `host_gateway`. +- `FirewallProxyManager.start()` already adds this gateway to `NO_PROXY` and to the proxy's IP allowlist, so agent traffic to the server bypasses Squid. +- Use the same `$HOST` value in `--server` when generating tasks and verifying PoCs below. + Start the PoC submission server: ```bash PORT=8666 # port of the server POC_SAVE_DIR=./server_poc # dir to save the pocs python3 -m cybergym.server \ - --host 0.0.0.0 --port $PORT --mask_map_path mask_map.json \ + --host $HOST --port $PORT --mask_map_path mask_map.json \ --log_dir $POC_SAVE_DIR --db_path $POC_SAVE_DIR/poc.db ``` @@ -77,7 +106,7 @@ PORT=8666 # port of the server POC_SAVE_DIR=./server_poc # dir to save the pocs CYBERGYM_SERVER_DATA_DIR=./cybergym-server-data python3 -m cybergym.server \ - --host 0.0.0.0 --port $PORT --mask_map_path mask_map.json \ + --host $HOST --port $PORT --mask_map_path mask_map.json \ --log_dir $POC_SAVE_DIR --db_path $POC_SAVE_DIR/poc.db \ --binary_dir $CYBERGYM_SERVER_DATA_DIR ``` @@ -85,7 +114,7 @@ python3 -m cybergym.server \ Test: ```bash # generate the task -SERVER_IP= # server ip +SERVER_IP=$HOST # server ip — same bind address chosen above SERVER_PORT=8666 # server port TASK_ID='arvo:10400' OUT_DIR=./cybergym_tmp @@ -115,6 +144,8 @@ bash $OUT_DIR/submit.sh $OUT_DIR/poc After running the agent, you can get the `agent_id` from the `logs/args.json`. You can verify the PoCs submitted by: ```bash +# NOTE: this is the public default placeholder key. Set your own via CYBERGYM_API_KEY +# on both the server and the client. It is not a substitute for keeping the server private. export CYBERGYM_API_KEY=cybergym-030a0cd7-5908-4862-8ab9-91f2bfc7b56d python3 scripts/verify_agent_result.py \ --server http://$SERVER_IP:$SERVER_PORT \ @@ -142,6 +173,8 @@ python3 -m cybergym.firewall stop-all # stop proxy and remove network Agent containers must be started on the `cybergym-internal` network and use the proxy env vars (see `ProxyManager.env_vars()`). The default allowlist is at `src/cybergym/firewall/default_allowlist.txt`. +Note that the firewall only filters **outbound** traffic from agent containers. It does nothing about inbound reachability of the submission server — that is still up to how you bind and firewall the host. + See `python3 -m cybergym.firewall --help` for more usage. ### Example Agents diff --git a/src/cybergym/firewall/default_allowlist.txt b/src/cybergym/firewall/default_allowlist.txt index fb76274..ca587c8 100644 --- a/src/cybergym/firewall/default_allowlist.txt +++ b/src/cybergym/firewall/default_allowlist.txt @@ -2,22 +2,7 @@ # Uses Squid dstdomain format: leading dot matches the domain and all subdomains. # Lines starting with # are comments. -# apt install -archive.ubuntu.com -security.ubuntu.com -ppa.launchpadcontent.net -api.launchpad.net -launchpad.net -old-releases.ubuntu.com - -# pip install -pypi.org -files.pythonhosted.org -pypi.python.org -bootstrap.pypa.io - # llm apis api.openai.com api.anthropic.com generativelanguage.googleapis.com -api.together.xyz diff --git a/src/cybergym/firewall/proxy.py b/src/cybergym/firewall/proxy.py index edc706c..a0fd992 100644 --- a/src/cybergym/firewall/proxy.py +++ b/src/cybergym/firewall/proxy.py @@ -45,9 +45,8 @@ import tarfile import time from pathlib import Path -from docker.errors import APIError, NotFound - import docker +from docker.errors import APIError, NotFound logger = logging.getLogger(__name__) @@ -266,9 +265,16 @@ class FirewallProxyManager: try: net = self._client.networks.get(self.network_name) net.reload() + try: + gateway = self.host_gateway + except RuntimeError: + gateway = None info["network"] = { "name": self.network_name, "internal": net.attrs.get("Internal", False), + # Bind the submission server to this address so agent containers + # on the internal network can reach it without exposing it publicly. + "host_gateway": gateway, "containers": [c.name for c in net.containers], } except NotFound: