"The best feedback I got was someone telling me my numbers were wrong"

Let's start at the beginning. Where did ProspectAI come from?

It came from wanting to learn agentic AI properly. I work in enterprise AI, and I kept reading about multi-agent systems without having built one end to end. Reading about a pattern and shipping it are different things.

I picked investment research as the domain for a specific reason. It is a problem where you cannot fool yourself for long. If you build a chatbot, "it feels good" is an acceptable result. If you build something that picks stocks, the market tells you if you were right. I wanted a domain with an external scoreboard.

Why six agents? That sounds like a lot for what could be one prompt.

The obvious way to build it is one agent: here is a stock, analyze it, tell me if it is a buy. It works in a demo and falls apart in practice.

That single prompt has to hold macro context, technical signals, fundamentals and a final decision all at once. When the output is wrong, you cannot tell which part failed. When it invents a number, nothing catches it. And the model talks itself into a conclusion, because there is nobody in the loop to disagree.

So I split it. A Market Analyst sets the macro and sector context. That context feeds a Technical Analyst and a Fundamental Analyst. A Draft Strategist turns those views into a candidate portfolio. Then a Critic attacks the draft, and a Final Strategist revises it.

It is not about having more agents. It is about having clear boundaries between them.

You have said the most important rule in the system is that LLMs never do arithmetic. Why so strict?

Because a model that can hallucinate a stock thesis can also hallucinate a performance metric. And the second one is far more dangerous, because it is the number you use to decide whether the first one was any good.

So no figure that reaches the portfolio comes from a language model. Not the composite score, not the position size, not the stop loss, not the risk reward ratio. All of it is computed by deterministic Python. The model decides which signals matter and why. The math is not its job.

That boundary also turned out to be the answer to the most common criticism I get, which is "why would you let an AI pick stocks when you cannot audit its training bias?" It is a fair concern. But training bias can color the narrative, and it cannot silently bias the picks, because the picks come out of the deterministic layer.

Tell me about the Critic. It seems to be the part people ask about most.

It is my favorite component and it does zero analysis.

It gets a fresh context and one instruction: attack the draft. Find the missing catalyst, the contradictory signal, the setup where the risk does not justify the reward. It does not rebuild the portfolio. It produces structured revision directives, and the Final Strategist applies them.

People ask why I do not just tell one agent to double check its own work. Because a model reviewing its own output in the same context tends to agree with itself. A separate role, with a fresh context and an adversarial instruction, actually pushes back.

You have been sharing the project publicly for months. What has the feedback been like?

Uncomfortable and extremely useful, in that order.

I want to be clear about this because it is the real story of the project. The improvements that mattered did not come from me sitting alone with the code. They came from people on Reddit and other communities telling me, in public, that something I had built was wrong. Almost none of it was hostile. Most of it was people who knew more than me about a specific thing, taking the time to say so.

Can you give an example?

The biggest one changed the whole system.

Someone challenged my performance numbers. I was comparing my signals against the S&P 500 measured from a fixed date. His point was that this is not a fair comparison, because my signals were generated on different dates across a two month window. A signal from May 24 has no business being compared to an index return measured from April 23. Different windows, different market weather.

He was right. So I rebuilt the evaluation layer to compare every signal against the benchmark over its own trigger to current window.

And then I took it further than he asked, because once I started thinking about fair comparisons I realised the S&P was the wrong benchmark entirely. Comparing a healthcare pick to the S&P tells you very little. The honest comparison is against the healthcare sector itself. So now every signal is measured against its own sector ETF.

What did that reveal?

Two things, and they went in opposite directions.

Healthcare looked like a clear win on raw return. It was positive, and I would have posted it. But the healthcare sector had run harder than my picks over the same period, so against the sector ETF I was actually behind. I was standing on an escalator and taking credit for the climb.

Then technology showed the opposite. Raw return was negative, which looks like a failure. But the tech sector had fallen further, so the picks beat their sector by a couple of points. A real win hiding inside a negative number.

That second case is why the exercise was worth doing. A metric that only makes your numbers smaller is not rigor, it is pessimism. One that moves them in both directions is measuring something real.

Any other feedback that changed the product?

Two more that I think about often.

A pharma investor told me that financial indicators alone are not enough in healthcare. He was right, and it explained the healthcare gap. My fundamental analyst was asking the same questions for every company in every sector: valuation ratios, financial health, dividend yield. But every sector has its own drivers. In healthcare it might be the strength of the drug pipeline. In semiconductors, the inventory cycle. In energy, the commodity price. Generic ratios do not capture that. Sector specific inputs are now on the roadmap.

The other one I caught myself. My dashboard once displayed a card reading one hundred percent win rate. It was two signals. Two. A hundred percent on a sample of two is a coin landing heads twice. I nearly published it. Now the evaluation layer refuses to report a metric below a real sample size. A dashboard showing confident numbers on tiny samples is not a feature, it is a bug.

Where do the numbers stand today?

The signals are currently ahead of the S&P by around five points across the full set, over matched windows.

But I would rather people looked at the number underneath that one. When you strip out sector allocation and measure only selection, meaning did the picks beat the sectors they came from, the edge is smaller. Positive, but smaller. Part of what looks like skill against the index is really just having been in the right sectors.

And one sector carries a large share of it. That is a finding, not a verdict, but it is the sort of thing you only learn if you build the instrument that can see it.

I should be clear on the caveats, because they matter more than the headline. This is paper traded on public data. Positions are open and marked to market, so the numbers move daily. Equal weight, no stop losses applied, and a single short market regime. It is nowhere near a track record.

What would you tell someone building something similar?

Build the evaluation layer earlier than you think you need it. I spent months on agent architecture and treated measurement as something to bolt on at the end. It should have been part of the design from day one, because until you have it, your system will confidently tell you it is working and you will have no way to check.

And publish the version of your numbers that makes you look worse. It is uncomfortable, and it is the only reason anyone should believe the ones that make you look good.

What is next?

Sector specific fundamental analysis, so the system stops asking pharma companies semiconductor questions. Multi sector portfolios, which is an interesting architectural problem around parallel execution. And more data, because the honest answer to "does this work" is still that the sample is too small to say.