Self-Hosted Speech To Text For Omi

An essential element of understanding yourself, is being completely aware of what you have said.

Being human, we forget, misremember, and misinterpret conversations, which makes a secondary "witness" to our converations so important. Bonus points if the witness has perfect hearing and memory, can be completely objective, doesn't have cognitive biases, and is an expert in interpreting what was both said and reasonably understood by everyone in the conversation. And yes, wearables with AI seems to check all these boxes. The singularity is upon us :)

AI Wearables

I wear an Omi pendant which uses a speech-to-text (STT) service hosted by Omi. The service is not expensive, but it is also not zero, costing ~$30 a month. I also have a decent Windows AI machine which sits idle most of the time. Running my STT on that machine will give me two benefits: Saving $30 a month, and more privacy as my audio flows from my Omi to my smartphone, to my own STT server.

We are in the early days of self-hosted AI so these setup instructions aren't for the normal end-user. But, I expect over the next two years for self-hosted services like this to be easily used by regular consumers.

Our Home AI Machine

I'm a Mac guy and haven't spent too much time on Windows machines in literally decades. But... Windows dominates the desktop operating system market with over 62% share, and will be a major player in home AI systems. I should familiarize myself with the state of the industry and start tracking its progress towards a consumer friendly experience.

Towards this goal of understanding Windows more, we purchased a Minisforum AI X1 Pro-370 with 64GB of DDR5 RAM and 512GB of SSD. This system uses an AMD Ryzen HX 370 with a Radeon 890M GPU. Not top of the line, but the price point was great: ~$1100 (yes, this is during Rampocalypse). This is half the cost of a Mac Mini while still having unified memory and a pretty fast GPU. I'm more interested in learning about what the normal consumers will buy, and not the power users.

First Round: Manually Installing Whisper As A Webhook

I started off the old-fashioned way: Google, read, download, install, debug, repeat.

It looked something like this:

Make sure Python is installed
Start PowerShell as an Administrator: Press Win + R Type 'powershell' (But don't press enter) Type Ctrl + Shift + Enter Click 'Yes'

Make sure Python 3.10.(6-11) is installed

winget install Python.Python.3.10 --version 3.10.10

Run the PowerShell based Whisper installer
```
iex (irm whisper.tc.ht)
```

Ugh, not working - search again, found a pre-built installer

Install pre-built Whisper for AMD

https://github.com/lemonade-sdk/whisper.cpp-amd

Ughhh, pre-builts are not building on github... so none available

While I could eventually solve this, and I did spin up a bit on Windows, I sas moveing much slower than I am used to with AI assisted programming. The path is obvious... fire up Cursor.

Second Round: Cursor/AI Driven Problem Solving

I'm a big fan of Cursor, and I'm sure all the other popular agentic development environments would do a great job too.

My new process:

Install Cursor on the Windows machine (to give it full access to the computer)
Open a new workspace and create a docs/ folder
Tell a (Cursor) agent that I want a self-hosted Whisper on this machine and I want it to provide a public webhook protected by an API key for my Omi to use. But don't implement yet - write a plan.
Review plan that Cursor made
Ask Cursor to implement the plan.

Boom! We are off to the races as Cursor takes care of all the details I was struggling with earlier

The Much Less Painful Path Forward

Setting up the server still took another hour or so. There were more decisions to be made:

Which (free) Dynamic DNS to use
How to port forward from my Eero firewall to the Windows machine
How to keep the Windows machine from sleeping
Which Whisper model to use
Exactly how to configure Omi to communicate with my STT

Each of the above issues was handled quickly within Cursor, but the prompts still required a knowledge of the technologies.

There are still some outstanding issues I'm working on:

How to do speaker recognition
How to summarise conversations (while not on the Omi paid plan)

The Bottom Line

I feel most consumers will use cloud based solutions, so the above journey will be followed by very few. A small number will buy off the shelf solutions, and a very very few will go down the trail I took ;)