All projects

PROJECT 02 / AI Workflows & Decision Support

IdeaJoust

Turn a rough idea into a shortlist of competing approaches — judged in a pairwise tournament, attacked by AI critics, and decided by you.

WHEN

2026

CONTRIBUTION

Design, development & evaluation

TECHNOLOGIES
PythonClaude CodeJev APISQLiteHTML
AI IDEA EXPLORATION02
ONE ROUGH IDEASeveral ways forward
Frugal founderDomain expertContrarian
COMPARE · CRITIQUE · SHORTLISTA tournament of ideas
Jev
Pairwise judging
Claude critics
Strong objections

Conceptual idea-evaluation workflow

THE PROJECT AT A GLANCE

AI explores. You make the final taste call.

Claude agents generate and critique alternatives; Jev filters and ranks them. Evaluation led to a blind human final rather than treating the AI ranking as the answer.

Blind human labels
63
Statistical models compared
7
Documented automated tests
383
Comparisons in the human final
≤ 7

01 / OBJECTIVE

What the project set out to do

Turn one rough business idea or research question into competing approaches, explicit objections, and a shortlist the user can compare and decide for themselves.

02 / MY CONTRIBUTION

My part in the work

I designed, developed, and evaluated IdeaJoust as a Python command-line tool and Claude Code plugin. I built the generation, filtering, ranking, critique, and reporting pipeline, then evaluated the judges against blind human preferences and redesigned the final decision around the user.

The evaluation used my own labels. I kept the predictions and analysis plan fixed before collecting the confirmation labels, and compared statistical models using cross-validation. The findings describe this small study rather than general user preferences.

03 / TECHNICAL APPROACH

How it came together

01

Generate

Claude “echobots” with different personas and business models propose ideas for one brief. The default is 12 bots; real runs in the documentation tested up to 10.

02

Filter

Jev removes duplicates and constraint-breakers. Both checks were validated with planted controls, with one recorded control run each.

03

Rank

Jev judges pairs in both presentation orders. IdeaJoust fits a Bradley–Terry ranking and resamples the recorded answers to measure how decisive the result is.

04

Critique

Claude critics write the strongest objection to each finalist, making assumptions and weaknesses visible before a decision.

05

Decide

In a blind interactive final, the user ranks the top 5 in at most 7 comparisons. Disagreements with the AI ranking trigger a devil’s-advocate check.

04 / EVALUATION

What the evaluation showed

The apparent gain did not replicate

An apparent 76% judge–human agreement did not replicate in the pre-registered confirmation test. The primary judge scored 60% versus 47% for the baseline (n = 30, McNemar p = 0.29). The secondary holistic judge scored 63% versus 47% (p = 0.18). Neither difference was statistically significant.

More modelling did not establish better predictions

An exploratory comparison of seven models and baselines included Bradley–Terry, Thurstone, Bayesian logistic regression, and stacking. No model built on Jev data demonstrated lower prediction error than a constant 50/50 baseline with an interval excluding zero. The human’s own choices were consistent and predicted their other choices on the same brief 81% of the time (48/59); that is a different, within-brief task.

The result changed the design

I redesigned the tool so AI generates, filters, and critiques while the human makes the final taste call. The interactive final has been tested with scripted answers; real user performance has not yet been measured.

05 / OUTCOMES

What the work produced

  • Built an inspectable, resumable workflow with saved model responses, offline reports, and a tournament replay.
  • Used blind labels, a pre-registered confirmation test, and cross-validation to test the judges rather than treating their ranking as ground truth.
  • Added a blind human final that ranks five candidates in at most seven comparisons.

A small, single-labeller evaluation. Nothing here shows that winning ideas succeed in the real world.

UP NEXT / PROJECT 03

TraceLens

View case study

Want to talk about this project?

Get in touch