At OnlyDust, we’ve seen it all—after two years supporting 450+ OSS projects, 8,000+ contributors, and distributing over $10 million in grants manually.
We have iterated a lot on manual allocation (bounties, jury, etc). Our latest system to date was the allocation of money by the intermediary of 300 maintainers, reviewed by hundreds of contributors. It was a massive time investment and not even worth it. Some challenges persisted (article incoming).
But we were sitting on a lot of data. 10,000’s of contributions, labeled in $ by maintainers, using their own budget. This label was a fair proxy of the value created by contributors, according to the project's maintainer.
A gold mine? We thought so.
So we tried something. We used this data to create our first OSS agent rewarding contributors.
An agent to reward them all. And in the OSS, bind them.
Greg, our CEO, probably under drugs.
Maybe, it was possible to evaluate the true value of each open-source contribution at scale.
This article will give insights we have collected along this path. So far, we have allocated $1m with this agent.
But if we could use AI to evaluate quality contributions, why stop there?
1 - How We Allocate Money with an autonomous agent
Distributing Money to Contributors
Instead of rewarding contributors based on promises or popularity, we decided to reward real impact— what actually get's shipped and matters. Meaning, the tangible value each pull request (PR) delivers.
So we built an agent that looks at each pull request (PR) and figures out how valuable it really is. Our AI-based agent combines advanced language models, specialized prompts, and project-specific data to assess the complexity, quality, and scope of every contribution.
Contributors get paid every month, based on what they actually delivered. Not who they know. Not how loud they are. No guesswork. No bias.
1. Collect Data
We start by gathering detailed information from each PR made on GitHub: code changes, the type of work (feature, refactor, or documentation), time spent, quality of back-and-forth with maintainers, linked issues, and more.
It gave us a decent picture how much effort the contributor invested into the PR and how impactful the changes might be.
2. AI Evaluation
Then our agent dives in. It uses the latest LLMs to understand how complex or challenging the work is, how well the contributor addressed maintainer feedback, and whether the PR solves a critical bug or adds a key feature.
It can even pick up on subtle stuff—like code quality, architectural thoughtfulness, and robust testing practices.
3. Scoring Algorithm
After the AI completes its initial review, we calculate a single score using a multi-layered formula. Rather than relying on a simple sum, we incorporate factors like work complexity, impact on the project, and delivery quality in a more nuanced way. An example of our (simplified) formula looks like this:
- \alpha_2 \times \log \bigl(1 + \text{delivery-quality}\bigr)
- \alpha_3 \times \text{maintainer-communication}
- \alpha_4 \times \text{reviews}$
- We then apply an additional transformation to reward standout contributions and ensure trivial work isn’t overcompensated. We are still iterating a lot each month for the exact formula.
- Work Complexity and Impact Factor: We emphasize genuinely demanding tasks that solve important problems, raising complexity to a power and multiplying by an impact factor to highlight critical contributions.
- Logarithmic Delivery Quality: Responsiveness and thoroughness are crucial, but we use a logarithmic scale to prevent minor differences at higher quality levels from overshadowing significant PRs.
- Maintainer Communication: We reward contributors who maintain strong communication and alignment with project goals, reflecting the intangible benefits of smooth collaboration.
- Reviews: Excessive back-and-forth on major issues reduces the final score, motivating contributors to submit well-prepared PRs from the start.
4. Monthly Distribution
At the end of each month, we finalize payouts for every qualifying PR based on its overall score.
PRs with minimal impact or trivial changes earn little or no reward, encouraging contributors to make meaningful contributions.
5. Less Work For Maintainers. More Work For Us
Maintainers don’t have to assign rewards anymore. They just make sure PRs meet technical standards.
That’s huge. It saves time, avoids awkward conversations, and puts the focus back where it belongs: the code. Contributors know their payout reflects the work, not relationships.
Meanwhile, we’re constantly reviewing scores, gathering feedback, and tuning the model. If something feels off, we step in and adjust. And every edge case helps us make the system better.
Many maintainers loved it – "This actually simplifies a lot of things for maintainers."
Skeptics were converted – "Wasn’t convinced at the beginning but gg team! This saves us [a lot] of our time."
Contributors adapted fast – "I actually like this new way. People will contribute more since they’ll see bigger payments for bigger contributions."
2 – What Are the Safeguards?
Just because it's automated, doesn't mean we leave contributors or maintainers stranded. We've introduced some safeguards:
- Appeals Welcome: If a contributor feels the AI’s score is off, they can appeal. This triggers a short review by maintainers and an OnlyDust moderator. If we see that the value score is not fair, we adjust the reward and add this specific case for the next improvement iteration of the distribution algorithm.
- Maintainer Discretion: Maintainers have access to their discretionary project’s budget to manually reward special cases or critical but hard-to-measure tasks (like community support or multi-week refactoring).
These checks and balances ensure the system stays fair, transparent, and adaptable to real-world edge cases.
3 – What’s Next?
We’ve already allocated over $1M through the agent, and we’re just getting started.
As we sharpen our prompts and collect more data, our system will become more capable at recognizing and scoring contributions. We will continue to scale our monthly budget, which already averages around $500k—and we anticipate that figure will rise if more sponsors and foundations want to distribute their rewards with this AI system.
Here’s what we’re building next.
3.1 – Smarter Contributor Scoring
We're getting better at understanding what makes a contribution valuable—not just at the PR level, but across time and projects.
By aggregating metrics like historical PR complexity, reviewer feedback quality, and impact on long-term project health, we can offer a rich, multidimensional ranking of contributors.
- Multi-Project Data Aggregation: We track performance across multiple repositories and ecosystems, creating a universal contributor profile. Contributors who work with diverse technologies or solve tough architectural problems receive bigger recognition for their broader skill set and impact.
- Advanced Scoring Algorithms: Our models are trained from real-world examples on what maintainers actually value. As those preferences evolve, so does the scoring to reflect community standards.
3.2 – AI Matching Engine
“Good first issues” are nice, but we can do better.
Our upcoming AI-based matching engine will leverage deep embeddings and semantic analysis to pair contributors with tasks that align not just with their skill sets but also with their growth ambitions and preferred work style.
- Technical Stack Analysis
- By analyzing contributors’ public code repositories (e.g., GitHub, GitLab), our system builds a technical fingerprint of each user (languages, frameworks, libraries). It then cross-references this fingerprint with issues that require similar expertise, reducing the time spent searching or trial-and-error.
- Contextual Complexity Matching
- Instead of only matching “C++ devs with C++ tasks,” we look at the historical complexity of issues solved, lines of code contributed, and patterns of problem-solving. This ensures that experienced contributors tackle high-impact tasks, while newcomers are guided to simpler or more mentorship-focused tasks.
- Dynamic Recommendations
- The AI engine can push real-time suggestions via integrations. If a new bug is tagged “urgent” and aligns perfectly with your skill profile, you’ll get a direct notification.
3.3 – Maintainer Tools
Maintainers are superheroes. But even heroes get tired.
Our future AI agents aim to ease everything from PR review to contributor onboarding, reducing drudgery and letting maintainers focus on high-level architecture and community building.
- Automated Contributor Vetting
- Our platform compiles a reputation score for new contributors by analyzing prior merged PRs, historical code quality, and even social proof from community interactions. Maintainers can see at a glance whether someone is likely to stick around or vanish after a quick fix.
- Intelligent Code Review Assistance
- Using large language models fine-tuned on open-source data, our agent can highlight potential bugs, security vulnerabilities, or performance bottlenecks. It can also offer context-aware suggestions—for instance, linking a line of code to a relevant style guide or architectural decision made weeks earlier.
- Adaptive Task Assignment
- When a repository gets a new issue or feature request, the AI agent can automatically assign it to the most suitable contributor—factoring in availability, expertise, and past performance. Maintainers only step in for final approval or edge cases.
It’s all about saving mental energy for the stuff only humans can do: mentorship, vision, hard calls.
4 – Two Years, $11M, and Counting
What started as a crazy idea—"can we pay devs fairly using AI?"—is now real. $11 million distributed. 450+ projects. Thousands of contributors. And it’s only getting sharper.
Our agent now handles 94% of pull requests on its own. The rest get reviewed manually. And with each new month, the system gets better at spotting what matters.
We’re not here to replace humans. We’re here to get the boring stuff out of the way, so open source can be what it was always meant to be: collaborative, fair, and focused on building the future.
- No more reliance on hype or personal branding.
- No more manual guesswork for maintainers.
- No more scattered, inefficient funding.
Join Us in Building the Future of Open Source
Whether you’re a maintainer, a contributor, or a sponsor—this is your invitation to try something new.
Our AI-driven platform is live. It’s working. And it’s getting better with every pull request.
- Check out our open issues.
- Sign up to get your project accelerated.
- Follow us to stay updated on our next features.
Open source is evolving. Let’s build what’s next—together.