Let me just come clean - I am a lousy individual stock investor. I repeatedly sell when I should be buying (too much pain) and I frequently end up buying because of FOMO.
Once I had this self-awareness, I invested only in S&P 500 and NASDAQ focused ETFs to keep my life simple. However, I have been aspiring to build an AI agent that would take the emotion out and recommend buy or sell for me. I call him Peter.
Peter reads a company’s fundamentals and recommends the next step: buy, sell, trim (sell part of a position), or hold. Now, he is still in early development, and his outputs have been quite promising. However, there have been days when Peter came back and gave two different verdicts on the same company from the same underlying numbers - buy in the morning, hold in the evening, from the exact same file. It is a nice problem to explore if I am just experimenting. But if I am going to put my hard earned money behind what Peter recommends, I better get a consistent recommendation.
So I spent a few days looking into this problem. I realized that the fix here was more than a better prompt. I need to build two key pieces for my agent - a skill to force the output into a deterministic shape, and an eval to check that output and hold Peter to it. (A skill is just a written set of instructions the agent follows; an eval is a test that grades what the agent produced.)
In this article, I will share the meat of how I built both, and done properly this works for any agent, not just mine.
Where the variance actually comes from
We all know that LLMs are non-deterministic by nature. But that is not really where my variance was coming from. My input file - the one Peter reads for each company - already had the facts: the company’s edge, the bull and bear cases, the price levels that should trigger an action. However, that file did NOT spell out the steps to get from those facts to the final call. I had left that step to the model’s judgment, and I realized that an LLM tends to reason inconsistently on the same fact.
So, to bring determinism in, I decided to take the decision OUT of the model’s head and put it INTO the file as something executable. My reasoning - the model should simply look up the answer, not invent it.
Move the decision into Skills
Here is the shift. Instead of asking Peter “what do you think,” I wrote the decision down as an ordered checklist and told him to just follow it. He reads top to bottom and stops at the first rule that matches. The rules then run in order, first match wins.
For example, if the company thesis shows as broken, sell. If a trim trigger has fired, trim. I built seven rows of simple if-else, with a “hold” at the bottom for everything else. Now, I don’t have any instruction that says “use your best judgment.” The judgment already happened when I wrote the decision ladder. I brainstormed with my agent separately to develop this decision ladder.
Spell out the exact instructions within your Skills
This one bit me hardest. On any non-buy verdict, one field has to print a fixed phrase. Early on I wrote “print the fixed phrase from section 9.4.” A cold run read that literally and typed “9.4” into the field. It failed the test two times out of five. The fix was almost dumb: codify your Skill file to actually write the actual words you want, right there. I learned that you need to be extremely specific in your Skills file.
Stripped to the mechanics, the block within my Skills file looks like this:
DECISION LADDER (top to bottom, FIRST match wins):
broken -> SELL
trim trigger fired -> TRIM
buy trigger + intact -> BUY
nothing fired -> HOLD
OUTPUT (print exactly):
VERDICT: one bare word from the row above
ORDER: if not BUY, print exactly: none
Write the test before you trust it
You can only get so far by being extremely specific within your Skills. You also need to validate that the LLM is interpreting those instructions consistently. So I wrote an Eval, with the single most important rule in mind: the model’s answers can vary, but you need to build a grading system that enforces consistency and determinism.
I call my Eval the grader, and I designed it such that it never calls an LLM. It simply reads the fields Peter printed, applies fixed rules, and says pass or fail. This way, no matter how many times I run the system, I get consistent answers, identical down to the character. This is what I wanted after all.
My simple rule was that the thing measuring determinism has to be deterministic itself, otherwise you are just grading noise.
I also implemented two key enhancements to my Evals. First, I wrote the Eval such that a Buy or Sell recommendation still goes through when the price moves within a small rounding tolerance, so I don’t fail two identical calls over a decimal. Second, I ran each test case five times - a fresh agent each run, with no memory of the previous agents - and the overall Eval qualifies as a pass only when at least four of the five agents report a consistent recommendation.
The kill test: four out of five, or redesign
Here is the part that was important for me. Before building anything, I set the goal: if the Skill does not lift the recommendation consistently to at least four out of five, then my mechanism is wrong - stop and redesign, do NOT lengthen the prompt. The lazy fix for a wandering agent is always to bolt on another paragraph of instruction. It feels like progress and buys you nothing. When one case failed two out of five, I did not add more instructions to the Skills file. I found the “9.4” leak, fixed the structure, and it went to five out of five. Every future file of that shape is deterministic now, for free.
One more honest one. During a batch, a few of my test agents cheerfully reported “file written and verified” with nothing actually on disk. The agent had hallucinated its own success. If I had trusted the self-report, I would have graded ghosts. So verify against the real system, never against the agent’s word for it.
What you can take from this
Everyone building on top of AI talks about Evals. I didn’t know much about Evals before this exercise. It is simple - write the test to reflect the outcome you want before you ask the LLM to build anything, and then point it at the real outcome, not at “did it run.”
My real question was never “does the file parse” - it was “does the same input produce the same decision,” and I could not answer that until I had a proper Eval system to measure it. I learned the importance of building the grader in the same pass as the feature.
Here is how I would summarize the whole thing:
You need to build Skills so you can shape the output that the LLM generated.
You need to fine-tune this Skill to be extremely specific so you can get consistency.
You need to build an Eval so you can consistently validate the outputs the LLM generated. Without an Eval, you have no idea how accurately the LLM executed the instructions you wrote within your Skill.