Memex — a queryable second brain over my messages
I wanted to ask my own data questions like "when did I last hear from X?" or "what did we say about that lake house?" and get an actual answer. So I built a hybrid retrieval index over my iMessage and Signal history, ran it on the homelab, and wired it into the assistant I already use.
What it does
71,396 messages indexed across iMessage (~69k) and Signal (~2k), covering three years. Queries hit both a BM25 keyword index (FTS5) and a vector index (sqlite-vec, 384-dim embeddings), and results are combined with reciprocal rank fusion.
Smoke tests that convinced me it worked:
- "happy birthday" surfaces actual birthday messages, top-ranked
- "lake house" finds a thread where someone joked their house was a lake during a storm
- a one-word project codename returns both an exact-match casual mention and the deployment thread for that project
What was hard
A bunch of things, in roughly the order I hit them.
Embeddings throughput on the LXC. I started embedding inside the container and capped out at about 5 emb/sec. The Mac mini, doing the same model under the same Python, sustained 50-130 emb/sec. Backfill moved from "this'll take 4 hours" to 33 minutes once I offloaded it.
rsync corrupted the index file in transit. The 167MB SQLite DB
came across the wire with sporadic page corruption. scp clean-copy
worked. I think rsync's delta-merge confused itself against a target
file that was open elsewhere. Whatever — scp is fine.
macOS system Python doesn't load SQLite extensions. sqlite-vec is a loadable extension, and Apple's stock Python rejects them outright. Homebrew Python 3.13 worked. Took a while to figure out the right error to grep for.
SQLite oid alias quirk. When you write SELECT m.oid, ... FROM
messages m, the cursor description shows the column as rowid, not
oid. My code keyed off the alias name, broke silently. Forcing
SELECT m.rowid AS oid, ... fixed it.
Cost
Almost nothing. Local fastembed (BGE-small-en-v1.5), local SQLite, existing LXC. S3 holds JSONL deltas — about 29 MB across the whole backfill. Ongoing is under a penny a month.
I had originally costed this against Bedrock Titan v2 plus pgvector, which would have been a few dollars a month and not awful. But once I realized my personal AWS account had zero Bedrock quota anyway, going local for embeddings turned out cheaper and faster.
How it stays current
Two pollers run on cron, one per source, every minute. They query the local SQLite database for rows newer than a cursor and write a JSONL delta to S3 — no PUT on idle. A flock'd indexer runs every 5 minutes and catches up any unprocessed S3 keys. Steady-state batches are small enough that the LXC's embedding speed is fine.
MCP integration
The assistant on the LXC sees memex_query, memex_recent,
memex_stats, and memex_get as native tools through MCP. So when I
ask it a question that requires personal history, it just queries
this index instead of guessing.
What I'd do differently
The big win was abandoning pgvector on a new LXC for sqlite-vec on the existing one. I'd held a vague belief that "real" vector search required a real vector database. For one user, hundreds of thousands of vectors, and a hybrid-search budget that fits in a single SQLite file, sqlite-vec is the right answer. The simplest thing that could possibly work, did.