Blog · By Al Del Degan, CTO and Co-Founder

How Claude and I Built an AI Meeting Assistant Through Pure Trial and Error

TL;DR: My CEO wanted her meetings to magically turn into CRM records. Over a couple of months, Claude and I built (and partially broke, and rebuilt) a fully local AI meeting assistant. First on a Raspberry Pi 5, then a Mac menu bar app, and finally a two-machine setup with an NVIDIA Jetson and a 12-year-old desktop in my basement. Everything runs on our own hardware. No audio ever leaves the network. It failed constantly along the way, and every failure made the final system better.

The second half of this is the part I did not expect to write: what happened in the month after it worked, including a correct fix that sat deployed and completely inactive for three weeks without anyone noticing.

The idea

Rea, our CEO at New Idea Machine, has a lot of meetings. After each one, the same chore: create the contact in the CRM, log the company, write up notes, create the follow-up tasks. She wanted to just talk and have the system do the rest.

The requirement that made this interesting: everything had to stay local. No OpenAI, no cloud transcription, no meeting audio leaving our network. Ever.

So the pipeline we needed was: record a meeting, transcribe it, have a local LLM pull out the contacts, companies, tasks, and a summary, write it all to our CRM, and tell Rea it's done.

Simple, right?

Attempt 1: The Raspberry Pi 5

We started with a Pi 5 on Rea's desk. Wake-word detection with Vosk ("start recording" / "stop recording"), transcription with Whisper, extraction with a 3B-parameter Llama model through Ollama, a Node.js middleware to dedupe contacts and call our CRM API, and Piper TTS to talk back through a little speaker.

And it worked. End to end. I was thrilled.

Then I tested it with a made-up contact named "Herbert Gerblebottom" and the transcript came back with "BIRTHGIRL BUDDY."

That's the honest ceiling of small hardware. The models tiny enough to run on a Pi can't handle proper nouns, which is a problem when the entire point is extracting people's names. The LLM was worse: give it a garbled transcript and it would confidently hallucinate an entire board meeting that never happened, complete with invented deadlines.

Lessons kept: the architecture was right. The hardware wasn't.

(Also, Rea vetoed the talking speaker, which is how Telegram became the system's voice. That turned out to be one of the best decisions in the whole project.)

Attempt 2: The Mac menu bar app

New realization: most of Rea's meetings are Zoom/Teams calls anyway. So why not capture the audio directly on her MacBook? Claude and I built a little menu bar app. Click to record, using BlackHole to tap the call audio, with a bigger 8B model running locally on Apple Silicon.

Better transcription. One-click install script. And one fatal flaw: routing audio through a virtual device on macOS breaks the volume keys. Rea uses her volume keys. Showstopper. Not a technical failure, a product failure. The kind you only find by putting things in front of the actual user.

Attempt 3: The Jetson (a.k.a. "did I just buy an expensive microphone?")

Round three: an NVIDIA Jetson Orin Nano on the desk for GPU transcription, and (this was the fun part) the LLM work moved to a 12-year-old i7 desktop in my basement that had been sitting idle since a previous failed project. Its hostname is herbert-gerblebottom-ai. The fake contact lives on.

The division of labour: the Jetson records and transcribes; the basement box runs the LLM extraction, stores every transcript in Postgres (with vector embeddings, for future meeting-intelligence features), writes to the CRM, and sends Telegram updates.

Then the wall. A security patch in NVIDIA's JetPack caps how much GPU memory a single allocation can grab, and the standard NeMo speech framework is so heavy (~180 Python packages, ~6 GB resident) that transcription failed constantly. We tried FP16 tricks. We tried retries. At one point we fell back to CPU-only and I asked Claude, honestly, whether the Jetson had been a waste of money.

The plan was a painful TensorRT model conversion. Then Claude went researching and found nano-parakeet, a community reimplementation of NVIDIA's own Parakeet speech model in pure PyTorch with five dependencies instead of 180. Same model, tiny footprint.

Even that took four rounds of debugging, and each failure moved one layer deeper. That's how you know you're actually progressing:

Full-precision model load: out of memory
Half-precision load worked, but its warmup crashed on CPU
Skip the warmup and the model is on the GPU! ...then transcribing a whole 23-minute file blew up the STFT
Chunk the audio. 3-minute chunks still failed, because attention memory grows quadratically with chunk length
60-second chunks: 24 chunks at ~1.1 seconds each. A 23-minute meeting transcribed in about 40 seconds, on-device, with proper nouns intact.

The Jetson was never the problem. The software wrapped around the model was.

That 60-second figure did not survive contact with uptime, which is a detail I left out of the original version of this article. After the box has been running for hours, the GPU's memory gets fragmented, and the allocator needs contiguous blocks. The same 60-second chunk that worked perfectly on a fresh boot starts failing. So the default came down to 30 seconds, and any chunk that still fails gets split in half recursively, down to a floor of 8 seconds. Fragmentation can shrink how much fits at once, but it can no longer stop a transcription outright. It just gets more granular under pressure.

The failures that became features

Here's the part I'm most proud of, and it wasn't in the original plan at all: every real-world failure got converted into a permanent safety net.

The very first meeting Rea recorded on the new system failed to transcribe. But by then we'd added a rule: a failed recording is saved to disk, never deleted, with a Telegram alert and a /retry command. Her meeting was fully recovered a few hours later after we fixed the root cause. Zero data lost, on day one.
The basement box hard-froze under load. It turned out my old abandoned email-analysis project was still running and cooking the CPU. Found it in the logs, killed it, gained a healthier server.
Long meetings overflowed the LLM's context window mid-answer. Fixed, plus the server now automatically re-queues any meeting stranded by a crash or restart.
The voice wake words false-triggered during a real call (the system started and stopped recordings on its own, during a live client meeting). We turned voice off entirely. Rea now uses two Telegram buttons: 🔴 Start, ⏹️ Stop. Deterministic beats clever.
She has back-to-back meetings, so we decoupled recording from processing. She can start the next meeting while the previous one is still being transcribed in a background queue.

And the data-quality piece that makes this usable in a real business: nothing goes into the CRM without human approval. After each meeting, Telegram shows the extracted contacts and tasks with Approve / Edit / Discard buttons. Edit opens a little web form where you can fix names, delete hallucinated tasks, and rewrite the summary before anything is written. The AI proposes; a human disposes.

The month after it worked

The Jetson section ends with "it works," which is where most build write-ups stop. The month that followed taught me more than the build did, because the failures stopped being about hardware and started being about reality.

Meetings that never ended

Rea forgot to stop a recording. Obvious in hindsight, invisible until it happens.

Now, after 10 minutes with no speech, Telegram asks whether the meeting is over. Yes stops it, No keeps it going, and so does simply talking again. It never auto-stops on silence alone, because a quiet meeting is still a meeting. For the recording nobody remembers at all, there is a hard 3-hour limit that stops and processes it normally.

A smaller cousin: recordings that were nothing but silence were being filed as failures, cluttering the recovery folder with meetings that had never happened. Now they are recognised and discarded.

The 1.5-hour meeting

69,000 characters against a 16,000-token context window. Ollama silently truncated the prompt and the model produced confident garbage.

The fix was structural rather than a bigger number: split the transcript into segments, extract each one, merge the contacts, then make one final small call to fuse the section summaries into a coherent whole. Every limit gets hit eventually. Raising it just moves the meeting that breaks it.

The extraction quality war

This is where most of the last month went, and none of it was glamorous.

"Bob introduced us" created a contact named Bob. The prompt asked for people mentioned. It should have asked for people participating. Somebody named in passing is not someone you met.
Rea and Al kept appearing as their own CRM contacts, and prospects kept getting "New Idea Machine" listed as their employer. The prompt said to exclude anyone from "Rea's company" without ever naming the company, so the model could not apply the rule. Meanwhile New Idea Machine is the most-spoken company name in every transcript, which made it the obvious guess whenever the model was unsure.
The same person came back as a brand new contact every meeting, under whatever spelling the recorder produced that day. Fixed by capturing an email during review and deduping on that before falling back to name matching.

The pattern across all three: the model was doing exactly what it was told. The instructions were wrong.

The fix that ran for three weeks without ever running

On July 22 we expanded the internal-names list with every variant the recorder actually produces. Correct fix, shipped, deployed.

On August 11 Rea reported the same problem, still happening.

The list lives in code as a default. The basement box sets that same variable in its own config file, with a narrower list, and a config value overrides a code default. So the fix had been sitting in the repo, live on the box, and completely inert, for three weeks. Nobody had reason to doubt it: the code was right, the deploy was right, and the behaviour never changed.

I only found it by reading the config on the machine instead of the code in the repo. That is now written into the operations manual in the bluntest language I could manage, because it is the kind of thing that costs you a month twice.

Thirty minutes to say "fetch failed"

Then extraction started failing outright, with a Telegram alert reading exactly ⚠️ Extraction failed: fetch failed and nothing else. It recurred on every restart.

Claude's first three theories were all wrong: a runaway model filling its context, a crashed Ollama, a stale connection. Each one died on contact with a measurement. Ollama had been running without a restart since July 9. The transcript's two extraction calls took 116 seconds each and returned perfectly clean JSON when run by hand. The model was never involved.

What gave it away was the clock. The service died at 30 minutes and 3 seconds after starting, which is the HTTP client's timeout to the millisecond. Three separate defects were stacked in the error path:

The retry loop did not cover the network call. The request sat one line outside the try block, so the loop only ever retried JSON parsing failures. A dropped socket, the single most retryable thing that can happen, got zero retries.
The real error was thrown away. Node reports every transport failure as the bare string "fetch failed" and hides the actual reason on a cause property that the code never read. Connection refused, socket reset and timeout all looked identical in the log.
The timeout was 30 minutes, six times the slowest real call, so a single wedged request burned half an hour before anyone learned anything, and then gave up permanently.

Three small mistakes, none of which would matter alone, arranged so that the system took the longest possible time to tell us the least possible information.

Rhea, Ria, Rea

The very next meeting created a CRM contact for "Rhea Hailley." The list had Rea, Ria and Ria Hailley. It did not have Rhea.

That is unwinnable by listing. The recorder invents new spellings faster than anyone can enumerate them, and each one is discovered only after it has already reached the CRM. So matching moved from exact strings to edit distance, with a budget that scales with length: names of four characters or fewer still match exactly, because two characters of slack on "Rea" would swallow half the phone book, while longer names get one or two characters of tolerance. Al and Ria behave exactly as before. Rhea, Hailey and whatever comes next are caught without a code change.

Quotes that were real but meaningless

To stop invented tasks, we made the model quote the transcript line where each commitment was made. Tasks without a quote get flagged with a warning symbol in the review message, never silently dropped, because a model that quietly stops producing quotes would otherwise erase every task in every meeting.

Then a task arrived carrying this as its evidence:

"Um, so I think that answer is the same to the as as we were gonna do with the UK guys, like we Lucia they build it for."

Which supports nothing at all. Checking that the field was present had only taught the model to fill it in. So quotes are now checked against the actual transcript, tolerant of the model tidying punctuation and capitalisation, and testing that against a real transcript immediately caught two bugs in the checker itself: a naive substring search matched inside words, so "the" verified against any transcript by hiding inside "there," and a one-word quote was being accepted as evidence at all.

The honest limit, which is written into the docs so nobody misremembers it later: this proves the quote is real, not that it supports the task. The garbage quote above was genuine transcript text. It would still pass today. A human reads these before anything is saved, and that remains the only real check.

The feature we talked ourselves out of

The one complaint none of this fixes is attribution. The transcript has no speaker labels, so when the model says who said what, it is guessing. The proper fix is speaker diarization, and I was ready to build it.

Claude's case against was better than my case for. It addresses only half the complaint, since getting the meaning wrong is at least as common as getting the speaker wrong, and diarization does nothing for meaning. The field that would consume it is not even used: tasks are always assigned to Rea regardless. And it is the highest-risk change available to the most fragile box in the system, whose GPU setup is seven interlocking constraints held together with a warning comment.

So we measured the cheap fix instead: tell the model to leave a statement unattributed rather than guess a name. Knowing what not to build turned out to be the most valuable thing that came out of that conversation.

What working with Claude was actually like

This wasn't "type a prompt, receive an app." It was weeks of genuine back-and-forth. Some honest observations:

Claude was wrong plenty of times. The first wake-word fix made false triggers worse. The first Jetson approach (NeMo) was a dead end. What mattered was that each wrong answer came with instrumentation to figure out why it was wrong.
My pushback mattered. "Did we just buy an expensive microphone?" and "do we have blinders on?" both redirected the whole approach. The best moments were when I challenged the plan and Claude went off, did fresh research, and came back with "actually, there's a better path." That's how we found nano-parakeet instead of losing a week to TensorRT.
Everything got documented as we went. A cheat sheet for managing both boxes, an operations manual, and every hard-won fix written down with its root cause. Future me will not have to rediscover why 60-second chunks matter.
Real users break things instantly. Rea's first day surfaced more issues than all my testing. The system's best features (saved recordings, retry, the approval gate, buttons instead of voice) all exist because something failed in front of the person who actually uses it.

Three more, learned the hard way in the month after I thought we were finished:

Measurement beats reasoning, and it is not close. Chasing "fetch failed," Claude produced three confident, plausible, entirely wrong explanations. Every one of them died the moment we actually timed the calls. The answer came from a file timestamp showing 30 minutes and 3 seconds. Now the first move on a mystery is to measure something, not to explain it.
Verify the state you are working from. Claude once wrote a substantial fix against a local checkout that was nine commits behind, which meant reimplementing work that already existed and nearly reverting three live features. Thrown away entirely. The check that would have prevented it takes two seconds.
A fix is not done when it is correct. It is done when you have watched it change the behaviour on the actual machine. The three-week silent failure was a correct fix, properly committed, successfully deployed, and completely inactive.

Where it landed

Today: Rea taps a button on Telegram, has her meeting, taps stop. About a minute later the transcript is done, on the GPU, entirely on our own network. A few minutes after that, she and I both get a summary with the extracted contacts and tasks, and either of us approves, edits, or discards it. Approved records land in the CRM tagged and status-flagged as AI-captured. Every transcript is archived in Postgres with embeddings, waiting for the next phase: mining months of prospect conversations for objections, patterns, and product insight.

Total cloud AI spend: $0. Meeting audio that left our network: none.

A month of real use later, the shape is the same and the edges are very different. Forgotten recordings stop themselves. Meetings of any length get segmented and reassembled. Internal names are matched by similarity rather than by an ever-growing list. Every extracted task has to point at the line of transcript it came from, and that line gets checked against the transcript before Rea ever sees it. None of that was designed. All of it is scar tissue.

The thing I'd tell anyone attempting something like this: the goal isn't a system that never fails. It's a system where failure costs you nothing. Every recording survives, every error explains itself, and every fix is permanent. We didn't build that on purpose. We built it by failing in front of each other, repeatedly, and refusing to lose the same way twice.

The corollary took me another month to learn. A system where failure is cheap will happily fail quietly, for weeks, while looking healthy from every angle you happen to be checking. So the last thing that goes into every fix now is the question of how it would announce itself if it stopped working.

Built with: NVIDIA Jetson Orin Nano (nano-parakeet / Parakeet-TDT 0.6B, FP16, chunked), a 2013-era i7 running Ollama (Gemma 3 4B) + PostgreSQL/pgvector, Node.js, Python, Vosk (retired), Telegram, and a lot of stubbornness. Mine and Claude's.

Al Del Degan is the CTO and co-founder of New Idea Machine. The agent in this story is Hermes Ear, the same one Rea writes about in AI Agents for Business: What Two Real Builds Taught Me.