Generate
Claude “echobots” with different personas and business models propose ideas for one brief. The default is 12 bots; real runs in the documentation tested up to 10.
PROJECT 02 / AI Workflows & Decision Support
Turn a rough idea into a shortlist of competing approaches — judged in a pairwise tournament, attacked by AI critics, and decided by you.
2026
Design, development & evaluation
Conceptual idea-evaluation workflow
THE PROJECT AT A GLANCE
Claude agents generate and critique alternatives; Jev filters and ranks them. Evaluation led to a blind human final rather than treating the AI ranking as the answer.
Code available on request
PROJECT EXPLAINER · FULL HD
2 min 57 sec · 1080p · Narration · English captions available
FROM A SAVED RUN

FROM A REAL IDEAJOUST RUN
Original report from a five-bot run about an Australian grants newsletter. The saved summary reports no single winner and a top group of three.
01 / OBJECTIVE
Turn one rough business idea or research question into competing approaches, explicit objections, and a shortlist the user can compare and decide for themselves.
02 / MY CONTRIBUTION
I designed, developed, and evaluated IdeaJoust as a Python command-line tool and Claude Code plugin. I built the generation, filtering, ranking, critique, and reporting pipeline, then evaluated the judges against blind human preferences and redesigned the final decision around the user.
The evaluation used my own labels. I kept the predictions and analysis plan fixed before collecting the confirmation labels, and compared statistical models using cross-validation. The findings describe this small study rather than general user preferences.
03 / TECHNICAL APPROACH
Claude “echobots” with different personas and business models propose ideas for one brief. The default is 12 bots; real runs in the documentation tested up to 10.
Jev removes duplicates and constraint-breakers. Both checks were validated with planted controls, with one recorded control run each.
Jev judges pairs in both presentation orders. IdeaJoust fits a Bradley–Terry ranking and resamples the recorded answers to measure how decisive the result is.
Claude critics write the strongest objection to each finalist, making assumptions and weaknesses visible before a decision.
In a blind interactive final, the user ranks the top 5 in at most 7 comparisons. Disagreements with the AI ranking trigger a devil’s-advocate check.
04 / EVALUATION
An apparent 76% judge–human agreement did not replicate in the pre-registered confirmation test. The primary judge scored 60% versus 47% for the baseline (n = 30, McNemar p = 0.29). The secondary holistic judge scored 63% versus 47% (p = 0.18). Neither difference was statistically significant.
An exploratory comparison of seven models and baselines included Bradley–Terry, Thurstone, Bayesian logistic regression, and stacking. No model built on Jev data demonstrated lower prediction error than a constant 50/50 baseline with an interval excluding zero. The human’s own choices were consistent and predicted their other choices on the same brief 81% of the time (48/59); that is a different, within-brief task.
I redesigned the tool so AI generates, filters, and critiques while the human makes the final taste call. The interactive final has been tested with scripted answers; real user performance has not yet been measured.
05 / OUTCOMES
A small, single-labeller evaluation. Nothing here shows that winning ideas succeed in the real world.
Want to talk about this project?
Get in touch