Let's talk Bitcoin (and sometimes Rust & AI)

I ran a little exercise to give the following 5 AI Agents the same prompt and see how they would fare, in my opinion, in completing the task:

• Claude: Sonnet 5 Medium from Antropic

• ChatGPT: GPT-5.5 from OpenAI

• Gemini: Gemini 3.5 Flash from Google

• Qwen: Qwen 3.7-plus from Alibaba Cloud via Auto Efficient model from Kilo Code

• Hy3 from Tencent via Hy3 Free model from Kilo Code

Except for the Qwen model, which used some Kilo credits (<$0.02), all the models are free to use; there are no subscription plans at all. I was signed in as a registered user for all the agents.

Summary (if you do not want to read the whole post)

Gave each model a short exercise to produce some Rust code and see how each performs.

In the prompt, I asked the models to ask clarifying questions if there is any doubt; do not make any assumptions. Did any model ask, yes 2 did:

• ChatGPT asked whether the Gregorian calendar leap year should be implemented.

  • Gregorian calendar

• Kilo Code Auto Efficient Qwen 3.7-plus asked whether modulo (%) can be used and also whether it should just provide the code or provide the code and execute.

  • Yes, % can be used; just provide the code.

  • Note: There was no need to ask whether modulo can be used; this is an operator just like + , - , / or *

All the models produced decent code, but only 3 used the #[cfg(test)] test harness that Rust provides:

• CharGPT

• Claude

• Kilo Code Hy3

The other 2, Gemini and Kilo Code Auto Efficient Qwen 3.7-plus, include some test code as part of the main() function with println!() output.

Even though only Kilo Code Efficient Qwen 3.7-plus asked whether % can be used, all except Kilo Code Hy3 used it. Kilo Code Hy3 implemented a remainder() function to determine a remainder of 0 after division.

If I have to pick a model based on the prompt, then I would go with either of the Kilo Code models since its output is very detailed and it explains it’s thinking and what it was going to do.

So what was the goal here?

It was to determine:

• How well does the model “listen” to the prompt, did they get the task right?

• Were any follow-up questions asked to clarify something the model did not understand?

• Did the model make any assumptions?

• How accurate was the model in solving the task?

Note, I was not interested this time at all in the cost and/or time that it took the model to solve the task.

Before we look at the result, let’s see how they. stack up on Kilo Bench:

• Used the data from Qwen 3.7-max (slightly more expensive)

Prompt

Here is the prompt I gave each of the models.

Create a Rust function from first principles that returns TRUE or FALSE for a given number whether it is a leap year. Use no crates, dependencies or built-in standard traits. Do not read, write or reference any local files; just provide the code as text in a code block.

Also, create tests and test the function for the following set of numbers: 1, 4, 2023, 2024, 2025, 2048, 1900, 2100, 1600, 2000

Do not make any assumptions, and if in doubt, then ask me.

Detail

• ChatGPT

  • What did it get right?

    • Asked clarifying question(s)

    • Made use of Rust test harness

    • Implement the ask, leap year function correctly.

  • What did it get wrong?

    • Gave no explanation or plan on what it is going to do.

    • Test was created as 1 big test instead of 10 individual ones.

  • Personal Rating: 7/10

• Claude

  • What did it get right?

    • Test cases was split so that one can run each individualy.

    • Gave an explanation of what it did not use.

    • Implement the ask, leap year function correctly.

  • What did it get wrong?

    • Made assumption on what calendar to use

    • Was very brief with no plan

  • Personal Rating: 6/10

• Gemini

  • What did it get right?

    • Implement the ask, leap year function correctly.

  • What did it get wrong?

    • No test case, test was in the main() function

    • Assume the calendar to use.

    • Again very brief with no plan.

  • Personal rating: 5/10

• Kilo Code Auto Efficient: Qwen 3.7-plus

  • What did it get right?

    • Asked clarifying questions

    • Implement the ask, leap year function correctly.

    • Was very detailed on what it was going to do, almost too detailed but I prefer it.

  • What did it get wrong?

    • Did not use the Rust test harness, tests. was just some comparisons and println!() output.

  • Personal rating: 8/10

• Kilo Code Hy3

  • What did it get right?

    • Use test harness, although only 1 test case.

    • As is common for Kilo Code, the output was very detailed and explained exactly what it was going to do.

  • What did it get wrong?

    • Would have preferred individual tests for the 10 tests.

    • Should have used modulo (%) operator instead of creating a remainder() function.

    • Assumed the calendar to use.

  • Personal rating: 8/10

See the References below for the raw output from each model.

Conclusion

All the models basically did what was asked of them. The Kilo Code models were very detailed and explained what it was going to do very well.

In a real-life situation, I would ask the model to first plan what it would do and then based on an agreed plan, implement that plan. The implemention/coding model might be totally different from the planning model. Models perform better or worse depending on what they are asked to do whether it is just some simple question, a detailed plan or a coding implementation based on an agreed plan.

References

Here is the raw data from each of the models:

• Kilo Code: Hy3

• ChatGPT: GPT-5.5

• Claude: Sonnet 5

• Gemini: 3.5 Flash

• Kilo Code Auto Efficient: Qwen 3.7-plus