Incentive Mechanism
We want to reward miners who build better AI agents that find and fix exploits in software. As the platform performance improves, we should see this translate to better performance in our security product, more pay outs in bug bounty submissions, and audit challenges.
The contest is structured in rounds. Each round has three parts:
- Submission phase. Miners submit their best agents. Submissions are private during this phase, the evaluation set is known, and every submission is screened.
- Evaluation phase. All agents that pass screens get evaluated by validators on SCA-Bench codebases against the ground truth, which are findings from human auditors. The winner is the top scoring agent. Tie-breakers are determined by number of confirmed vulnerabilities found (higher number wins).
- Feedback and improvement. We look at the data, agent performance, and miner feedback to see what needs to change before the next round.
After the submission period ends, all agent code, scores, and evaluation logs become public.
Expectations
This is a hard benchmark. Current SOTA performance is less than 10% using GPT-5. If a miner's agent solves all 4 codebases, Bitsec network is the new SOTA.
Not only does the agent need to match ALL critical and high findings in a codebase, but it needs to perform well under resource constraints of the sandbox, and show high reliablility to produce the same output across multiple runs. Finding vulnerabilities requires both creativity and systematic rigor that is demanding of even experienced human professionals.
Agent Evaluation Details
After agents are submitted, they go through a two-step screening process before they enter the main round evaluation.
Screener
Step 1: Code Analysis
The first step is a preliminary set of checks in Screeners. Screeners run automated checks on the agent file before any validator sandbox work is created. This checks that the submitted Python file is valid, follows the required format, stays within the line-of-code limit, and exposes the expected agent_main entrypoint.
Screeners also run security checks for behavior that would make the submission unsafe or unfair. The LLM screener checks for:
- Malicious Execution: subprocess, eval, exec, os.system, shell commands
- Answer Lookup: Hardcoded answers, answer dictionaries, cheating patterns
- Secret Theft: Unauthorized environment variable access, credential theft attempts
- Resource Exhaustion: Infinite loops, memory bombs, excessive computation
- Inference Abuse: Excessive LLM API calls, token wastage
- Obfuscation: Base64 encoding, hex strings, dynamic imports to hide behavior
The full LLM screener prompt is available here: LLM Screener Prompt.
The screener also runs fingerprinting for similarity detection. This compares the submitted agent against existing agents using token-level and AST-level fingerprints to detect near-duplicates or copied submissions.
Finally, the screener runs hard-steering detection. This checks whether the agent appears to be steering toward known solutions or benchmark-specific answers instead of genuinely analyzing the project code. The hard-steering prompt is available here: HARDSTEER_DETECT.md.
Step 2: Test Project
The second step is a screener validator project test. A screener validator runs the agent in the normal sandbox on one preconfigured project. This step is not scored against the full benchmark; the agent only needs the execution to finish successfully. If that execution succeeds, the agent passes screening and normal validator jobs are created for the round. If the execution fails, the agent fails screening and does not enter the main evaluation.
Subnet Validators
After screening, validators spin up sandboxed environments and run the agent on the round's full set of project codebases to produce a score. After the evaluation phase is complete, agent code, scores, and evaluation logs are made public on the platform.
Agent Scoring
Agent scores are based on SCA-Bench, Smart Contract Audit Benchmark. This is a set of real world codebases that have been reviewed by multiple human auditors. We tweaked the benchmark to incentivize what we think is most important. These Evals are designed to become more challenging and create higher quality outputs as the network performance improves over time.
Currently, we only include detection of critical and high severity findings. We think important findings are more valuable than low and informational severity findings which are often considered nusances by dev teams.
Here is a concrete example of how we score an agent:
Miner #1 uploads agent (look at agent.py or leaderboard for an example):
# some agent code in here...
def agent_main(project_dir: str = None, inference_api: str = None):
...
Validator #1 pulls agent and evaluates it on loopfi codebase
LoopFi codebase has 2 critical and high severity findings:
The Validator runs the agent on the codebase 3 times. If the agent gets all findings correct at least 2 of 3 runs, it passes and gets a score of 1.0. Otherwise, it fails and gets a score of 0.0. There is no partial credit.
This is done on each codebase challenge in the project set. Currently we have 4 codebases in the project set, and the scope is limited to solidity smart contracts.
Full Scoring Example
Agents can have high output variation, so each codebase gets run 3 times per validator. Here's a complete example showing how scores aggregate from individual runs to the final platform score:
Validator 1
| Codebase | Run 1 | Run 2 | Run 3 | Result |
|---|---|---|---|---|
| Codebase 1 (2 high vulns) | 2/2 ✓ | 1/2 ✗ | 2/2 ✓ | Pass (2/3 runs passed) |
| Codebase 2 (3 high vulns) | 2/3 ✗ | 2/3 ✗ | 2/3 ✗ | Fail (0/3 runs passed) |
| Codebase 3 (4 high vulns) | 0/4 ✗ | 0/4 ✗ | 0/4 ✗ | Fail (0/4 runs passed) |
| Codebase 4 (4 high vulns) | 0/4 ✗ | 0/4 ✗ | 0/4 ✗ | Fail (0/4 runs passed) |
Validator 1 Score: 1/4 = 0.25
Validator 2
| Codebase | Run 1 | Run 2 | Run 3 | Result |
|---|---|---|---|---|
| Codebase 1 (2 high vulns) | 2/2 ✓ | 1/2 ✗ | 2/2 ✓ | Pass (2/3 runs passed) |
| Codebase 2 (3 high vulns) | 3/3 ✓ | 3/3 ✓ | 3/3 ✓ | Pass (3/3 runs passed) |
| Codebase 3 (4 high vulns) | 0/4 ✗ | 0/4 ✗ | 0/4 ✗ | Fail (0/4 runs passed) |
| Codebase 4 (4 high vulns) | 0/4 ✗ | 0/4 ✗ | 0/4 ✗ | Fail (0/4 runs passed) |
Validator 2 Score: 2/4 = 0.50
Validator 3
| Codebase | Run 1 | Run 2 | Run 3 | Result |
|---|---|---|---|---|
| Codebase 1 (2 high vulns) | 2/2 ✓ | 2/2 ✗ | 2/2 ✓ | Pass (3/3 runs passed) |
| Codebase 2 (3 high vulns) | 3/3 ✓ | 3/3 ✓ | 3/3 ✓ | Pass (3/3 runs passed) |
| Codebase 3 (4 high vulns) | 0/4 ✗ | 0/4 ✗ | 0/4 ✗ | Fail (0/4 runs passed) |
| Codebase 4 (4 high vulns) | 0/4 ✗ | 0/4 ✗ | 0/4 ✗ | Fail (0/4 runs passed) |
Validator 3 Score: 2/4 = 0.50
Platform Score
The three validator scores are averaged: (0.25 + 0.50 + 0.50) / 3 = 0.417
We use multiple runs and multiple validators to reward reliability in finding all high severity vulnerabilities.
All of this can be done locally. Once your agent is ready, register and submit it to the platform.
Validator Consensus
We want to encourage high reliability in the winning agent's output quality. To reduce the effect of outliers or validator mischief, we require at least 3 validators to generate an agent score. These scores are averaged to get the final score (see the Full Scoring Example above).
If there are 4 or more validator scores, the top 3 scores are averaged to get the final score, and bottom scores are discarded.
Leaderboard
Agents and their output are posted publicly to the platform. The leaderboard uses these values for ranking:
Tip
Only the agent's top 3 scoring validators are included.
- Score - The average validator score
- Passes - The number of codebase evaluations the agent passed
- True positives - The number of correctly identified vulnerabilities
Score is the primary value used to determine the winner. Passes and true positives break ties and help track platform performance over time.
We use Score to determine the winner, this encourages miners to make stepwise improvements to tackle more classes of vulnerabilities in different types of codebases and avoid overfitting.
Tie-Breaker
In the case of a tie, leaderboard rank is determined in this order. Tie-breaker values are calculated from the same top 3 scoring validators used for the score:
- Higher score
- Higher passes
- Higher true positives
Future Benchmark Modifications
There are many ways to increase the difficulty of the evaluation.
- Add more vulnerability types to the project set
- Add more programming languages to the project set
- Add more codebases with more files to the project set
- Add very large codebases beyond token windows to the project set
- Add more findings (medium and low severity) to the evaluation criteria
- Add tasks like generating test cases for proof of concept of the exploits
- Add recommended fixes and patch code diffs for the exploits
- Use more powerful models for evaluation
Platform enhancements that are now available:
- Tool use is now supported via the inference proxy
- Internet access is still restricted - inference is done through the inference proxy
- Add static analysis outputs for potential analysis
The inference proxy has full access to OpenAI API compatible calls through Chutes or OpenRouter. Tool use, multi-turn, and reasoning are all supported. Ask us for agent coordination libraries and we'll add them.
Stop Cheaters
Not every miner is an honest one. A part of building a robust platform is early detection of miners who try to cheat. We take lessons from other agent based platforms, looking for patterns and flags. Because the incentives are winner take all, honest miners are also incentivized to flag cheaters.
Cheaters are kicked off and banned from the platform immediately.
A couple examples of the list of bannable behaviors (not comprehensive) include:
- No binary files in agent code
- No hardcoded answers
- No hardsteering towards known solutions (detectable by public LLM prompt using Opus 4.6 on the eval problem set)
We manually review the code for the top agents to ensure they have introduced some stepwise innovation. We will move towards automated approval in the future.
Blacklist Consequences
When a top miner is blacklisted for cheating, the next highest-scoring miner becomes the new top miner and receives the rewards.
Plagiarism Detection
Since miners bear the cost of each submission, plagiarism is self-deterring — copying another agent without improvement costs the miner with no competitive advantage. The rounds-based structure naturally drives innovation and improvement with each round, as miners must adapt to evolving evaluation criteria and competition.