I just pulled my token usage for the last five months, and I know for February I used about two billion tokens a month. Now I'm at 4.5 billion a month with >85% cache hits. This is across Claude, Gemini, Codex and local models. This is across multiple accounts, including monthly and API accesses. My total token cost summed across the last 5 months has been in the low four figures (not including GPUs and electricity). It's completely possible to do this without hitting weekly limits.

I'm a consultant who charges a premium. My rate's high enough that I don't pass these charges on to my clients: it's just a cost of doing business (for now). My clients get the benefits of a lower quote and more work completed at a higher quality. I'm always trying to get more done in less time, and LLMs are a huge enabler for that. That said, if used incorrectly, LLMs can be a huge time sink. The key is to have a well thought out process for using them.

I've been doing consulting work since 2019, and I've always been splitting my time between multiple clients. Prior to that I ran several engineering teams at different organizations. I've gotten extremely organized, so task switching is less costly. This organization has been a huge enabler for me to leverage LLMs to do work while I'm busy doing something else.

While LLMs can be extremely useful for debugging and interactive pair programming, they frequently hallucinate. Most developers who use agents regularly have to deal with a very confidently wrong agent. This gets very expensive in terms of developer time and can reduce or remove benefits of using LLMs. The thing that unlocked higher productivity was giving the agents tools they can use to check their work.

What's Correct

In the hardware world, it can be hard to define "correct" for a given project. It's even harder to get an agent to understand it. If we provide agents with better tools, they are more likely to produce correct results. Since most frontier models have been trained with lots of standard software development quality tools, we need to leverage these. Ideal firmware projects set up to leverage agents already have:

  • solid documentation
  • CI
  • unit testing
  • static analysis
  • hardware-in-the-loop regression testing

These tools enable agents to do consistent meaningful work. Hallucinations still happen with ALL models. So we need some ground truth to help catch these.

In cases where no ground truth is available, I'll have one frontier model call another. I've had a ton of luck with having Claude Opus call gemini cli and codex for a review before going forward. Again, actual testing is better, but in lieu of that, having the reviews occur saves ME time and reduces hallucinations.

Context management is the key. The longer the context, the more likely hallucinations will occur. Often times we are faced with a complex series of tasks we need completed. It's critical that I plan out phases that the limited context can execute. For example, I'll have a context read a datasheet and summarize it into a markdown with an index file. Then a planning or reviewing agent can use that markdown, without burning a ton of tokens trying to parse/understand a PDF.

If you're living in claude code or codex, remember your agent's context will fill up quickly. Leveraging subagents can help but a custom wrapper around claude code/codex/gemini cli is the real superpower. Most of my harnesses are written in Python -- I'm really familiar with it, and frontier models are GREAT at writing Python.

Get More Done

If you're just starting out with agents and struggling to get value, I'd suggest explicitly planning separately from executing. Basically start a session with claude code, codex or gemini cli and have it plan out the tasks you want to accomplish. Then have it write a markdown file with the tasks. Do a quick review of that plan. Clear your session (or restart your agent) and have it read the markdown file and execute the tasks, checking them off as it goes. If it completes only the first few tasks and then gets stuck, restart your agent and give it the same set of tasks, but only tell it to do one task at a time. This gives you a chance to see if the agent is capable of completing the task, and if it isn't, you can try a different model or approach.

I'd also suggest going with the best frontier model you can afford for testing. I'd start with Opus (not Sonnet) or Codex (not Spark) at least for your planning session.

If you want to jump to the deep end, Maestro is a great tool to help you orchestrate multiple agents (it has a planning wizard and a way to automatically execute). It's VERY powerful, but it is completely insecure. I run it on a separate locked-down computer without access to sensitive data.

Token Savings

Context management isn't just about avoiding hallucinations -- it also controls your token usage. The larger the context, the more tokens are used. Let's say you have a session with a context of 800k and ask it to read in a new file, it costs 800k+tool usage1. So breaking down a complex task into smaller tasks is critical, but if tasks are too small, they may incur more token overhead to get started (increasing your token usage). So for each type of workload it's critical to find the right balance of task size and context size.

For cost (token) savings, I've used Claude Opus to plan out several tasks in Markdown and then feed those tasks to lower cost models (which I've been running locally). I've had a ton of success with some local models using Opencode. Then my only token costs are the initial planning and the final review. The local models are great at executing tasks, but they don't have the context to plan or review. This is a huge cost savings. To be honest if you have a good local model and a lot of VRAM, you may get away with a $20 a month subscription and only use the frontier models for planning and reviewing. I'll write about my local model experiments later.

Footnotes

  1. This could be cached, but it's still not free. ↩