# Henrique Cardoso de Faria > Principal Engineer based in Viana do Castelo, Portugal. Specializing in Ruby on Rails, Elixir, and AI integration. ## About Software engineer with over a decade of experience, remote since 2014. Currently at BSPK building AI-powered features for luxury retail clienteling. Also runs HC Digital Services LLC, helping companies integrate AI into their Rails applications. Organized Euruko 2025, Europe's longest-running Ruby conference, in his hometown of Viana do Castelo. ## Skills - **Backend:** Ruby on Rails, Elixir/Phoenix, Python/Django - **AI/LLM:** Agentic systems, RAG pipelines, natural language search, structured LLM outputs (RubyLLM, OpenAI, Groq) - **Frontend:** Turbo/Stimulus (Hotwire), Vue, React, Angular - **Databases:** PostgreSQL, Elasticsearch, Redis, SQLite, MongoDB, Firestore - **Infrastructure:** AWS (S3, Lambda), Docker, Kamal, Sidekiq, RabbitMQ, MQTT - **Languages:** Portuguese (native), English (fluent) ## Experience ### BSPK — Principal Engineer (Jul 2023 – Present, Remote) Clienteling platform for top-tier luxury retail brands. Primary backend engineer building the AI layer from scratch. - **AI Assistant:** Multi-agent system using swarm architecture. Orchestrator routes to specialist sub-agents (Client Intelligence, Tasks & Calendar) with tools for querying shopper data, purchase history, and schedules. Provider-agnostic (OpenAI, Groq, local models). - **Natural Language Search:** LLM-powered system translating plain English into Elasticsearch queries for conversational client book search. - **Behavioral Analytics:** Observability pipeline — API event capture across 270+ endpoints, session materialization, daily metric rollups, behavioral profiling, funnel analysis, and sales outcome correlation. - **Shopify Integration:** Deep ownership of webhook pipeline — order syncing, customer assignment to nearest stores via geocoding, staff member resolution, data quality. - **Platform:** Led Rails upgrades through 8.1. Built PubSub system for real-time mobile updates with Elixir/Phoenix. Tech: Ruby on Rails, Elixir, Phoenix PubSub, PostgreSQL, Elasticsearch, Redis, Sidekiq, RubyLLM, AWS, Tailwind CSS ### BSPK — Senior Software Engineer (Aug 2020 – Jun 2022) First stint. Worked across the Rails API, admin interface, and data ingestion pipelines. Built the PubSub system for real-time mobile updates. ### Vic.ai — Senior Software Engineer, Elixir (Jul 2022 – Jul 2023) AI for real-time, autonomous enterprise accounting with prescriptive intelligence for financial decision-making. ### IndustryCare — Senior Software Engineer (Sep 2019 – May 2020) IoT data pipeline for manufacturing plants. Architected the MQTT message broker consumer for sensor data normalization and distribution. Built with Elixir, Phoenix, RabbitMQ, PostgreSQL. ### Neru — Senior Software Engineer (Jul 2018 – Sep 2019) Payment mobile application. Co-built the first shipped version, then led the growing team. Architected multi-provider payment API abstraction layer (Iugu, Zoop, Wirecard, GetNet). Built with Elixir, Phoenix, Firebase, NativeScript. ### Not In California / ActionNetwork — Software Engineer (Sep 2014 – Dec 2018) Progressive movement platform. Built events, fundraising, forms, search, and background processing systems. Ruby on Rails, MySQL, Redis, Sidekiq, Elasticsearch. ### Negócio Simples — Software Engineer (Apr 2016 – May 2018) Accounting SaaS. Architected Elixir/Phoenix API with Angular frontend for business owners and accountants. ## Community - **Euruko 2025 Organizer** — Organized Europe's longest-running Ruby conference in Viana do Castelo, Portugal. [2025.euruko.org](https://2025.euruko.org) - **hencf.org** — Open source personal website built with Rails 8.1, featuring an AI chat agent powered by RubyLLM and Groq. [github.com/henriquecf/site](https://github.com/henriquecf/site) - **Open Source** — Contributor to ja_serializer, ruby-conferences.github.io, euruko.org, and other community projects. ## HC Digital Services LLC Rails & AI consulting. Helping companies build and improve software — from Rails applications to AI integrations. Services include: - **AI Integration:** Agentic systems, natural language search, structured LLM outputs, RAG pipelines - **Rails Development:** Full-stack Rails — new features, API design, performance, upgrades, refactoring - **System Architecture:** Data pipelines, Elasticsearch, background processing, real-time systems, third-party API integrations ## Education Universidade Federal de Goiás — Software Engineering, Web Development (2012 – 2018) ## Contact - Email: elo.henrique@gmail.com - GitHub: [henriquecf](https://github.com/henriquecf) - LinkedIn: [henriquecardosodefaria](https://www.linkedin.com/in/henriquecardosodefaria) ## Blog Posts ### I Blamed the Ruby Upgrade URL: https://hencf.org/blog/i-blamed-the-ruby-upgrade Published: 2026-06-09 I bumped a large Rails app to Ruby 4.0.1, pushed the branch, and watched CI go red. Five tests failed. Not the same five every time. I'd rerun the job and four of them would pass. Rerun again and a different one would fail. None of them ever failed on my laptop. They only failed on CI, and only sometimes. That intermittence is its own special kind of frustration. A test that fails every time is a bug you can chase. A test that fails one run in three is a test that makes you doubt your own sanity before you doubt your code. You start clicking "re-run failed jobs" and treating green as the truth, because green is the answer you want. And there was an obvious villain right at the top of the diff. I'd just changed the Ruby version. A major version bump is a big, scary, everything-touching change. When tests start failing the moment you make it, of course it's the upgrade. I spent the first hour reading the Ruby 4.0 release notes hunting for the thing that broke my tests. The upgrade didn't break my tests. It just stopped letting them get away with being wrong. ## The failing tests all had the same shape Once I stopped staring at the changelog and started reading the failures, a pattern showed up. Every failing test was a "prove this operation did nothing" assertion. Create a record, grab a timestamp off it, run some code that's supposed to leave that record alone, then assert the timestamp didn't move. ```ruby test "does not change company_preference if filtered_search is already true" do cp = create_company_preference(company: @company, key: :config, value: { filtered_search: true }) updated_at_before = cp.updated_at perform_job assert_equal updated_at_before, cp.reload.updated_at end ``` The idea is sound. The job is supposed to skip this record, so `updated_at` shouldn't change. Capture it before, compare it after. But look at what's on each side of that final comparison. On the left, `updated_at_before` is a Ruby `Time` object, the one Active Record put in memory when the row was created. On the right, `cp.reload.updated_at` is the same column read back out of Postgres. Those two values are supposed to be the same instant. Most of the time they are. Sometimes they're off by a few hundred nanoseconds, and `assert_equal` fails. ## Postgres rounds, Ruby doesn't Ruby's `Time` can hold nanoseconds, nine digits after the decimal point. This is not new in Ruby 4.0. It's been true since Ruby 1.9. Postgres `timestamp` columns store microseconds, six digits. When Active Record writes a row, the in-memory object keeps whatever precision Ruby gave it. When you reload, Postgres hands back its own rounded, six-digit version of the same moment. So `cp.updated_at` straight from memory carries nine digits, and `cp.reload.updated_at` carries six. They're the same instant at two different precisions. Ask `assert_equal` whether nine digits equals six and, roughly one percent of the time, the rounding has nudged the value and the answer is no. This is a well-worn Rails gotcha with GitHub issues going back to Rails 4. I'd just never been bitten by it, and the reason I'd never been bitten is the whole point of the story. ## Why it failed on CI and never on my laptop How often a timestamp carries sub-microsecond digits depends on the system clock. On Linux, the clock tends to hand out high-resolution times, so most timestamps have digits sitting past the microsecond mark, exactly the digits Postgres throws away on the round trip. On macOS, far fewer do. My CI runs on Linux. I develop on a Mac. Same test, same code, same database engine, and the precision that trips the assertion shows up most of the time on one platform and rarely on the other. That is the entire reason it looked like the upgrade did it. The failures lived in CI, the upgrade ran in CI, and I had never once seen these tests fail anywhere else. The Ruby bump was a coincidence of timing. A fresh base image, a clean dependency install, and a handful of CI runs that happened to land on the unlucky one percent. The biggest change in the diff caught the blame for a bug it had nothing to do with. ## The fix is one word Reload before you capture. ```ruby updated_at_before = cp.reload.updated_at ``` That's it. By reloading before reading `updated_at`, I snapshot the value Postgres actually stored instead of the higher-precision one Ruby happened to be holding in memory. Now both sides of the assertion come from the database, at the database's precision, and they match every single time. Every failing precision test got the same edit. A timestamp captured from an in-memory object became a timestamp captured after a reload. The fix is boring. The afternoon it took to convince myself the upgrade wasn't responsible was not. ## The other flake, while I was in there Auditing the time-sensitive tests turned up a second flake hiding in the same neighborhood, and it's a different bug worth knowing about. A cron job flags "stuck" imports: anything still in `processing` after nine hours. The test set up records right at the boundary. ```ruby started_at: 9.hours.ago.utc + 1.second # not stuck, one second to spare started_at: 9.hours.ago.utc - 1.second # stuck, by one second ``` The records were created at setup time. The job computed its nine-hour cutoff later, at the moment it ran. On a fast, quiet machine that gap is nothing. On a loaded CI runner, enough real time passes between building the fixtures and running the job that a record sitting one second inside the boundary drifts to the wrong side of it. The "barely not stuck" record quietly becomes barely stuck, and the assertion flips. The fix was to stop measuring from wall-clock-at-creation and anchor every record to one fixed reference time. ```ruby started_at: @current_time - 9.hours + 1.second ``` Now every record's age is measured from the same instant, and that one-second margin doesn't evaporate while CI is busy doing something else. Different bug, same family as the first one. Both tests trusted that a value read at one moment would still hold at another. Both were rock solid on a fast machine and flaky on a slow, busy one. Time in tests is treacherous in more than one way, and a slow CI box finds all of them. This is the same reason a [test suite that's green on your laptop](/blog/parallel-testing-elasticsearch-rails) can still surprise you the moment it runs somewhere else. ## What the upgrade was actually hiding The fix took minutes. Finding it took the rest of the afternoon, and nearly all of that time went into suspecting the wrong thing. The loudest change in the diff is a magnet for blame. A Ruby major version bump is exactly the kind of change your eye lands on first. It was sitting right there, it touched everything, and it had nothing to do with the actual bug. These tests had been wrong since the day they were written. They passed for years because I develop on a platform that happened to round in my favor, and CI rounded in my favor often enough that I never had a reason to look. The upgrade didn't introduce the flake. It changed where the dice landed often enough that I finally had to. --- ### SQLite in Production: The 124 GB WAL File URL: https://hencf.org/blog/sqlite-124gb-wal-file Published: 2026-06-01 A few weeks ago I noticed one of my servers was almost full. Not "getting full," almost full. The kind of full where the next deploy fails and you find out about it at the worst possible time. This is the fourth time SQLite's defaults have surprised me in production, so I'm starting to recognize the shape of it. The first three were about [disk space and auto_vacuum](/blog/sqlite-auto-vacuum-rails), [Litestream eating Backblaze's free tier](/blog/litestream-backblaze-b2-free-tier), and [replacing all of it with a cron job](/blog/sqlite-backups-the-boring-way). This one is about a write-ahead log file that grew to 124 GB while every safeguard against exactly that was switched on. The villain, it turned out, was my performance monitoring tool. ## 93% and climbing The first command I ran was the obvious one: ``` $ df -h / Filesystem Size Used Avail Use% Mounted on /dev/sda1 197G 175G 15G 93% / ``` 15 GB free on a 197 GB disk. This is a single VPS that hosts a handful of my side projects, all deployed with Kamal, each with its own Docker volume for SQLite databases and Active Storage files. [espirita.club](https://espirita.club) is the busiest of them. My first instinct was wrong, which is worth admitting because it cost me twenty minutes. I assumed it was Docker cruft. Old images, stopped containers, dangling layers. That's the usual suspect when a Kamal host fills up, and pruning is harmless, so I started there: ``` $ docker system df TYPE TOTAL ACTIVE SIZE RECLAIMABLE Images ... ~1.6GB Containers ... 1.597MB Local Volumes ... 150.8GB 278.5kB (0%) ``` That last line is the tell I glossed over the first time. The Docker volumes held 150 GB, and almost none of it was reclaimable. Pruning images and stopped containers freed a little over a gigabyte, far less than I'd hoped, because the old image tags shared base layers with the running ones. The disk barely moved. So the space wasn't Docker overhead. It was data inside a volume. I'd written summary notes to myself claiming the big volume was "just Active Storage uploads," which was a guess dressed up as a fact. The app in question doesn't have anywhere near that much user data. So I went into the volume and actually measured: ``` $ du -h --max-depth=1 /var/lib/docker/volumes/ce_storage/_data/ | sort -rh ``` And there it was: ``` 124G production_pulse.sqlite3-wal 16G production_pulse.sqlite3 248M production_pulse.sqlite3-shm ``` A 16 GB SQLite database with a **124 GB write-ahead log** sitting next to it. The main database file was timestamped from that morning and hadn't grown. The `-wal` file had been written to seconds before I looked. `production_pulse.sqlite3` is the database for [rails_pulse](https://rubygems.org/gems/rails_pulse), a self-hosted performance monitoring gem. It tracks request timings, slow queries, and route performance, and it stores all of that in its own SQLite database, separate from the app's primary data. The "16 GB of telemetry" part is its own conversation. The "124 GB write-ahead log" part is the emergency. ## What a WAL is supposed to do If you've only used Postgres or MySQL, SQLite's write-ahead log is easy to misunderstand, because it looks like a transaction log but behaves like a staging area. In WAL mode, SQLite doesn't write changes directly into the main database file. Instead, every modified page is appended to a separate `-wal` file. Readers see a consistent view by reading the main file plus whatever newer pages exist in the WAL. Writers append. Nobody blocks anybody, which is the whole point: WAL mode is what makes SQLite usable under concurrent reads and writes, and it's why Rails turns it on by default. The WAL isn't meant to grow forever. Periodically, SQLite performs a **checkpoint**: it copies the pages accumulated in the `-wal` file back into the main database, then lets that WAL space be reused. By default this happens automatically. After any commit pushes the WAL past 1000 pages (roughly 4 MB with the default page size), SQLite runs a checkpoint on that connection. There's a catch in that default, and it's the entire story. The automatic checkpoint is a **PASSIVE** checkpoint. A PASSIVE checkpoint copies what it can and never truncates the file. It reclaims the *space inside* the WAL for reuse, but the file on disk stays whatever size it grew to. More importantly, a checkpoint can only copy frames that sit before the oldest active reader. If some connection is holding an old read snapshot, the checkpoint stops at that reader's position and leaves everything after it in place. So there are two ways a WAL file balloons. Either nothing ever truncates it, or a long-lived reader keeps pinning the checkpoint so it can never catch up to the writes. I had both. ## The checkpoint that did nothing Before I understood any of that, I tried the thing you'd try: force a checkpoint that truncates. The `TRUNCATE` variant checkpoints everything it can and then shrinks the `-wal` file back to zero bytes. I ran it inside the running container, against the live database: ``` $ docker exec ce-web-... \ sqlite3 /rails/storage/production_pulse.sqlite3 \ "PRAGMA wal_checkpoint(TRUNCATE);" 1|32254510|731851 ``` That output is three numbers, and they tell you exactly why nothing happened. `wal_checkpoint` returns `busy | log | checkpointed`. The first column is the busy flag: `1` means the checkpoint could not finish because another connection was in the way. The second is the size of the WAL in pages. The third is how many pages were actually checkpointed. So: **busy**, a WAL of 32,254,510 pages, of which only 731,851 got moved. Thirty-two million pages at 4 KB each is about 128 GB, which matches the 124 GB on disk. The checkpoint moved a couple percent of it and gave up, because the web container had a live connection holding the pulse database open. The WAL didn't shrink. `df` ticked down by a few gigabytes from the partial flush, from 93% to 90%, and that was it. This is the part that's genuinely counterintuitive if you come from a server database. In Postgres, you tell the database to do something and it does it. In SQLite, the database is a library running inside your application's processes, and a "checkpoint" is constrained by every other connection those processes are holding. A long-running Rails app with a connection pool open against that pulse database was, by simply existing, preventing the WAL from ever being reclaimed. Forcing a checkpoint from a second connection couldn't override the first one's read position. ## Every safeguard was on, and that was the problem Here's what makes this one worth writing about rather than just fixing and forgetting. rails_pulse is not careless about disk. The configuration I had in place was, on paper, exactly what you'd want from a tool that writes a lot of rows: ```ruby RailsPulse.configure do |config| config.archiving_enabled = true config.full_retention_period = 2.weeks config.max_table_records = { rails_pulse_requests: 10_000, rails_pulse_operations: 50_000, rails_pulse_routes: 1_000, rails_pulse_queries: 500 } end ``` Two-week retention. Hard caps on row counts per table. A cleanup job running every night and a summary job running every hour. I'd even added an initializer that turned on incremental `auto_vacuum` on the pulse database specifically, with a comment to my future self explaining that SQLite won't reclaim deleted space otherwise. I'd read my own [earlier post](/blog/sqlite-auto-vacuum-rails) and applied its lesson. None of it took effect, and the reason is the WAL. Retention works by deleting rows. Row caps work by deleting rows. `auto_vacuum` reclaims pages freed by deletions. But every one of those operations is a write, and in WAL mode a write goes into the `-wal` file first and only lands in the main database at checkpoint time. If the WAL never checkpoints, the deletes never actually shrink the main database, the freed pages auto_vacuum is supposed to reclaim never make it back, and the delete operations themselves pile up as more pages in the WAL. The cleanup job ran every night and, as far as the file on disk was concerned, made things worse each time. So I had a monitoring tool diligently generating cleanup writes, those writes feeding a WAL that couldn't checkpoint because the app held it open, and the WAL growing without bound until it was eight times the size of the database it was logging. The tool I'd installed to watch for performance problems was the performance problem. ## The fix, and why the order mattered The naive fix is to delete the giant `-wal` file and move on. Don't do that while the application is running. Deleting a WAL out from under an open SQLite connection can corrupt the database, because the connection still believes those committed pages exist in the WAL and haven't been checkpointed into the main file yet. And on Linux, deleting a file that a process still has open doesn't even free the space. The inode sticks around until the process closes the handle, so you'd get corruption risk and no disk back. I could have tried to fix it in place: schedule a recurring `TRUNCATE` checkpoint, or recycle the connection pool so no reader stays pinned, or move the pulse data off SQLite entirely. But sitting there at 90% disk, I had to decide whether this tool was earning its place at all, and the honest answer was no. I almost never opened the dashboard. On a single VPS running a few side projects, the operational risk of an unbounded-growth failure mode was worth more attention than the monitoring data was saving me. The right move wasn't to fix the checkpoint. It was to remove the thing. That made the ordering clean. The only thing holding the WAL open was the running container's connection to the pulse database. So: 1. Remove rails_pulse from the application. The gem, the separate `pulse` database definition in `database.yml`, the engine mount in `routes.rb`, the recurring jobs, the initializer, the schema files. 2. Deploy. Once the new container boots without any rails_pulse code, nothing opens a connection to the pulse database, and nothing holds the WAL. 3. *Then* delete the orphaned files on the host. Removing the gem touched a fair amount of config but no real logic, since the gem is self-contained. After `bundle install` and a grep to confirm there were no lingering references, I committed it on a branch, opened a PR for my own records, merged, and deployed. With the new container up and verified, I checked that nothing held the files open before touching them. `lsof` wasn't installed on the host, so I used `fuser`: ``` $ fuser /var/lib/docker/volumes/ce_storage/_data/production_pulse.sqlite3* $ ``` No output means no process. Safe to delete: ``` $ rm -v /var/lib/docker/volumes/ce_storage/_data/production_pulse.sqlite3 \ /var/lib/docker/volumes/ce_storage/_data/production_pulse.sqlite3-shm \ /var/lib/docker/volumes/ce_storage/_data/production_pulse.sqlite3-wal ``` And the payoff: ``` $ df -h / Filesystem Size Used Avail Use% Mounted on /dev/sda1 197G 31G 159G 17% / ``` From 93% to 17%. Around 144 GB reclaimed, almost all of it from one app's monitoring database. Both apps on the box returned 200 on their health checks, and that was the end of it. ## What I actually think about this I want to be careful not to turn one incident into a sweeping verdict. rails_pulse is a genuinely nice tool, the failure was a configuration interaction and not a bug, and on a server with proper WAL checkpoint hygiene it would have been fine. If you're running it and reading this, the lesson isn't "rip it out," it's "make sure something truncates that WAL, and watch the file." But the decision I made for my own setup was to stop running in-app APM on these boxes, and I'd make it again. Here's the reasoning, since that's the part worth taking away. A performance monitor that lives inside your app and writes to a database on the same disk is, structurally, a second high-churn workload competing with the thing you're trying to observe. On a big setup with a dedicated metrics store, that's fine, that's the whole architecture. On a single VPS running side projects, it means I've doubled my SQLite operational surface to gain dashboards I check once a month. The math doesn't work. The monitoring was costing me more risk than the outages it was supposed to help me catch. What I lean on instead is deliberately boring. Request logs are already there and already structured. [Solid Errors](https://github.com/fractaledmind/solid_errors) catches the exceptions that actually matter, in a table small enough that it never causes this class of problem. When I want to know why something is slow, I'd rather reach for a one-off query or a flamegraph during an investigation than pay a continuous tax to have the data pre-collected. For an app with a few active users, the slow paths announce themselves. I don't need a constant feed to find them. There's a broader pattern across all four of these SQLite posts, and it's not "SQLite is fragile." It's that SQLite gives you a database with no operator. Postgres has a process whose entire job is to vacuum, checkpoint, and manage space in the background, tuned by people who think about nothing else. With SQLite in production you've quietly taken that job, and the defaults assume a workload that may not be yours. auto_vacuum is off. Checkpoints are passive and never truncate. A long-lived connection will pin a WAL forever and nothing warns you. Each of these is reasonable in isolation and each one has bitten me once I ran a workload the default didn't anticipate. The thing I keep relearning is that the failure is always silent until the disk is full. There's no log line that says "your WAL hasn't checkpointed in three weeks." The file just grows, every safeguard you configured quietly feeds it, and the first signal you get is a number on `df` that's too high to ignore. So now `df -h` and the size of every `-wal` file on the box are on the short list of things I glance at before I trust that everything's fine. It's a cheap habit, and it would have turned this from an emergency into a Tuesday. --- ### Off Heroku: The Playbook URL: https://hencf.org/blog/off-heroku-the-playbook Published: 2026-05-12 A couple of weeks ago I wrote about [migrating BSPK off Heroku](/blog/i-migrated-bspk-off-heroku). That post was the narrative version: what it felt like to drive a multi-month infrastructure project without writing much code, what "agentic engineering" looked like in practice. It stayed away from the actual configs and the bugs because the audience was different. This is the other post. The one for the ops people sitting on a Heroku bill they're tired of paying. What we moved to, why, and the awkward bugs we hit along the way. Less narrative, more playbook. ## What moved, and to what The shape of the migration: - Application servers: Heroku dynos → EC2 hosts via Kamal 2 - Provisioning: hand-rolled bootstrap script → Terraform - Container registry: Heroku registry → ECR - Postgres: Heroku Postgres → PlanetScale Postgres - Redis: Heroku Redis → Upstash - Elasticsearch: Bonsai → Elastic Cloud - Elixir Phoenix PubSub service: Gigalixir → Kamal alongside the main app - CDN: Heroku's edge → CloudFront in front of Rails assets - DNS: managed internally → Route 53 with weighted records for cutover - Private networking between hosts: nothing → Tailscale - Secrets: Heroku config vars → 1Password as the source of truth, pulled at deploy - Postgres maintenance: we couldn't install non-trusted extensions → pg_squeeze for table bloat AWS was a constraint our CEO requested, not a preference. On Hetzner, Vultr, or almost any cheap VPS provider, the cost reduction would have been larger. We priced it both ways before starting. The 60%+ reduction we landed on is the conservative version of this move. ## Kamal and Terraform as the deployment unit The new stack runs on EC2 (m7a.large amd64 in production right now), deployed with Kamal 2. Kamal handles the application container, the kamal-proxy in front of it, the SSH-based rollouts, and the accessory containers like Caddy where we use them. The first version of the AWS bootstrap was a shell script. It worked, but it described the world implicitly: you ran it, you got a host. There was no canonical source of "this is what production looks like." We rewrote it as Terraform later in the project once the shape stabilized. The README in `terraform/` is the two-step cutover playbook now. Anyone with credentials can plan and apply. One thing that surprised me: Mac ARM Docker builds against an amd64 production target were painfully slow. The fix was repurposing dev2 (a 32GB VPS we already used for development) as Kamal's remote builder. Now builds run on Linux amd64 directly, the cache stays hot between deploys, and CI deploys finish faster than Heroku's git-push pipeline ever did. The Kamal config for it is one block: ```yaml builder: remote: ssh://deploy@dev2.internal cache: type: registry options: mode: max ``` The `mode: max` cache option is what makes the cache durable across deploys. The default mode only stores the final layer, which defeats the point. For registry credentials, we wired `KAMAL_REGISTRY_PASSWORD` to fall back to the GitHub Actions token when running in CI. Locally it pulls from 1Password. Same Kamal config, two environments, no per-environment branching. ## Secrets without vendor lock-in Heroku's config vars are convenient. They're also a one-way mirror: you can edit them in the dashboard, but there's no canonical source you can diff or audit outside Heroku itself. We made 1Password that source. Every production secret lives in a single vault. Kamal pulls them at deploy time using `kamal secrets`. The `.kamal/secrets` file looks like a shell script that exports each variable, sourced from 1Password through their CLI: ```bash DATABASE_URL=$(op read "op://Production/PostgreSQL/url") SECRET_KEY_BASE=$(op read "op://Production/Rails/secret_key_base") PLANETSCALE_DATABASE_URL=$(op read "op://Production/PlanetScale/url") # ... and so on ``` The dev2 builder and the production hosts both have the 1Password CLI installed and authenticated via service accounts. No `.env` files anywhere. The secrets exist as 1Password items, get pulled at deploy, and live in process memory. This setup bit us once. `ENCRYPTION_SERVICE_SALT` is a multi-character value that includes characters the shell wants to interpret. Kamal's secret pipeline double-escaped it on the way through, and the running app crashed trying to decrypt a value it had written itself. The fix was wrapping the secret in single quotes inside `.kamal/secrets` so the shell didn't re-interpret the escape sequences. Obvious in retrospect. Not obvious when half the requests are 500s and the other half are fine because they don't hit any encrypted attributes. ## The encrypted attributes hazard This was the bug I spent the most time on, and it's the one most people doing a Rails-version-plus-host swap will hit. Rails encrypted attributes derive their key from a digest of the master key plus a salt. Old rows in our database were written with a SHA1-derived digest. The Rails version we were about to ship in production defaulted to SHA256. If I flipped the setting, every existing encrypted value would have become unreadable. If I left it on SHA1, the app would log deprecation warnings and break in a future Rails version. The fix is a small dance: read with fallback, write with the new digest. On decrypt, try SHA256 first, fall back to SHA1 if that fails, and re-encrypt the value with SHA256 the next time the record gets saved. Existing data heals itself as records get touched. New writes are always SHA256. Nothing breaks at the cutover, and over time the SHA1 footprint shrinks toward zero. I'm not pasting the exact config because Rails encryption internals are version-specific and the code that's correct as I write this might not be correct when you read this. The pattern is what matters: read with fallback, write with the new digest, let activity drain the legacy values. Invisible if you do it right. Catastrophic if you don't. ## TLS for hundreds of tenant domains BSPK is multi-tenant. Each customer gets one or more subdomains under our platform domain, plus the option of custom domains pointed at us. On Heroku, ACM and the platform's edge handled TLS invisibly. On EC2, we owned it. I [wrote up the Caddy on-demand TLS setup](/blog/multi-tenant-ssl-caddy-kamal) when I first built it. Caddy sits in front of kamal-proxy, issues certificates per-domain on the first HTTPS request, and validates each domain against a Rails endpoint that checks the database. It's been running since early in the migration. The bug worth mentioning that didn't make it into the original Caddy post: a Caddy boot loop on the new EC2 host during a dry-run cutover. The proxy crashed on startup, restarted, crashed again. Logs said `failed to load TLS config` and nothing else useful. The TLS block in the Caddyfile was deriving from a `{$TLS_ENABLED}` env var that wasn't reaching the accessory because Kamal's env block didn't include it. Either passing the variable through or hardcoding `tls_enabled true` would have worked. Hardcoding was simpler and that's what we shipped. Boot loop gone in five minutes. ## The managed services Application data moved to managed services we don't operate. I'm going to write a separate, deeper post about Postgres → PlanetScale because that piece has its own decisions worth unpacking. The short version here: **Postgres → PlanetScale.** We kept the Postgres dialect, didn't switch to MySQL on Vitess. The cutover was a config flip thanks to a small change in how Rails picks the database URL: prefer `PLANETSCALE_DATABASE_URL` if present, fall back to `DATABASE_URL` otherwise. We could deploy the new wiring well before the actual data cutover and verify both code paths. The piece I want to call out, because it's the biggest operational change of the whole migration, is the replica architecture. PlanetScale fronts the primary with replicas, and most schema operations route through the replicas without locking the primary in a way that causes user-visible downtime. On Heroku Postgres, a long-running `ALTER TABLE` on a hot table was the kind of thing you scheduled for a Sunday at 3 AM with a maintenance window. On PlanetScale, most of those operations run live. We've shipped column additions, index builds, and constraint changes during business hours without anybody noticing. That changes how we think about schema work entirely. The old "save it for the next maintenance window" instinct stops applying, and the bottleneck moves from "when can we afford the downtime" to "is this change actually safe." **Redis → Upstash.** Boring in a good way. One URL change, Sidekiq picked it up, our cache and queue moved over. **Elasticsearch → Elastic Cloud.** I had help on this one from a coworker who knows ES better than I do. Index aliases made the cutover painless: replicate into the new cluster, swap the alias, the app doesn't notice. The pattern across all three: keep the connection string indirection, set up the new destination, replicate or seed, then flip the env var. The app code doesn't change. ## The Elixir service came along We have a small Phoenix app that runs PubSub between front-end clients and the Rails monolith. It used to live on Gigalixir, which is a fine Heroku-shaped host for Elixir. It now lives on the same EC2 fleet as the main app, deployed with Kamal as its own destination. This was a smaller move than the Rails migration but worth mentioning because it's exactly the same pattern. The Elixir release is a Docker image. Kamal pushes it to ECR, pulls it on the hosts, swaps containers behind kamal-proxy. The Phoenix endpoint reads its database and Redis URLs from the same 1Password-backed secrets file. Consolidating two deployment pipelines into one is its own form of cost reduction. ## Cutover day The actual cutover was anticlimactic, which was the goal. Route 53 has weighted records: you can point an A or CNAME at multiple destinations and split traffic by weight. We added the new EC2 elastic IPs to the same record names that pointed at Heroku, with weight 0 to start. The new infrastructure was live, the app was deployed, the database was replicating, but no production traffic was hitting it. Then we ramped weights. 1%, then 10%, then 50%, then 100%. At each step we watched dashboards, error rates, and a smoke test endpoint that exercises the critical paths. If something went wrong, dropping the weight back to 0 reverted the traffic to Heroku within the DNS TTL. CI runs that same smoke test on every deploy now. It hits a handful of endpoints across the major surfaces (auth, client search, the Elasticsearch-backed shopper finder, the AI assistant tools, the Stripe webhook handler) and asserts on response codes and a couple of expected body shapes. If any of them fail, the deploy is marked failed even if the rollout itself succeeded. I cut DNS to 100% on the EC2 fleet on a weekday afternoon. The Heroku side stayed warm for another day in case we needed it. We didn't. ## What we got that we didn't have on Heroku The migration was framed as a cost reduction, and it was. The operational gains are what I notice day to day. `pg_squeeze` runs in the database itself and reclaims bloat from heavily-updated tables on a schedule. Heroku Postgres didn't allow non-trusted extensions, so we had no way to do this in place. We had tables that had grown well past their actual size because of long-running update patterns. pg_squeeze undid that. It also failed to bootstrap initially on a malformed schedule literal, and the fix was a one-character correction. I would not have caught that without a real Postgres shell. We had `pghero` before the migration and it came along to the new stack. What changed is that I trust the slow-query list more now. On Heroku Postgres, the long tail of slow queries was partly a function of shared-host noisy neighbors and a buffer cache we didn't control. On PlanetScale, the slow-query list is closer to "queries that are actually slow because the SQL is bad," which is the version of that signal I want. CloudWatch holds 30 days of application and access logs with searchable retention. We had less of both on Heroku. I rarely need to search logs, but when I do, it's there. Deploys are faster. Heroku's git-push pipeline took two to four minutes per deploy depending on slug compilation. Kamal with the dev2 remote builder and registry cache pulls layers in seconds. Deploys are now bound by the rolling restart, not the build. And the box is visible. Sometimes the right debugging tool is `kamal console`, sometimes it's `ssh` and `htop`. Heroku didn't let us do either. ## What's still in flight The PlanetScale read replica goes in this week. We sized the primary for write-plus-read load and want to push the read traffic to a replica so the primary can scale further on write throughput alone. I want to move more runtime config out of secrets and into the database where it can be tenant-scoped. Most of what's in `.kamal/secrets` belongs there. Some of it (feature flags, rate limit defaults) doesn't belong as a platform-wide environment variable in the first place. And the team angle is unsolved. I drove this migration mostly solo because I was the one with the right Claude Code setup and the most context on the stack. That's not a great long-term equilibrium. The next move is documenting our Kamal and Terraform conventions clearly enough that someone else on the team can drive the next change of this size. The migration itself is done. The next round of "things we couldn't do on Heroku" is just starting. --- ### I Migrated BSPK Off Heroku in Two and a Half Months. I Barely Typed. URL: https://hencf.org/blog/i-migrated-bspk-off-heroku Published: 2026-04-30 I rewrote my homepage tagline last week. It used to say *I build things with Ruby, Elixir, and a healthy obsession with AI*. Now it says: *I'm an agentic engineer. I orchestrate AI agents to ship production software*. That's a stronger claim than I would have made even three months ago. So I should probably explain what I've been doing that makes it true. Over the last two and a half months, mostly solo, I migrated BSPK's entire production stack off Heroku to AWS. BSPK is a unified clienteling and commerce platform used by top-tier luxury retail brands. It runs on Ruby on Rails with an Elixir/Phoenix PubSub service alongside it. We had been on Heroku for years. The cutover happened without downtime. The new infrastructure costs more than 60% less. We kept shipping product features at the same pace throughout, including a new waitlist data model, per-SA reporting, and a bunch of smaller things that don't fit a tagline but pay the bills. I didn't write much of the code that did this. ## What actually moved A migration of this size has a lot of pieces. The short version, for people skimming: - Heroku to AWS EC2, deployed with Kamal 2 and provisioned with Terraform - Postgres to PlanetScale, with the read replica being added this week - Redis to Upstash - Elasticsearch to Elastic Cloud (this one I had help on, the rest I drove solo) - An Elixir Phoenix PubSub service moved off Gigalixir onto Kamal too - Heroku CDN to CloudFront in front of the Rails assets - Route 53 weighted DNS to do the cutover gradually - Tailscale for private networking between hosts - 1Password as the secrets backend, with Kamal pulling them at deploy - pg_squeeze and pghero for the Postgres maintenance and visibility we never had on Heroku - A 32GB VPS we already used for development became the Kamal remote builder, which made CI deploys faster than they ever were on Heroku - CloudWatch monitoring, 30-day log retention, post-deploy smoke tests in CI - A two-step cutover playbook documented in the terraform README so anyone could run it Plus the not-so-glamorous things: ghostscript installed in the runtime image so Paperclip's PDF processing kept working, a fix for `ENCRYPTION_SERVICE_SALT` being double-escaped by Kamal, a TextEncryptor SHA1/SHA256 dance to keep encrypted values readable across the cutover window. Migrations always have these. They're not what I want to write about today. Why AWS specifically? Because our CEO asked for AWS. On almost any cheap VPS provider, the cost reduction would have been bigger. AWS was a constraint, not a preference. I priced it both ways before starting. ## What "directing agents" actually meant Most of my time was spent reading and deciding, not typing. Before any non-trivial change, I had Claude produce a plan. Not a vague plan. A plan with the specific files it intended to touch, the commands it would run, and the things that could break. I read it. I edited it. Sometimes I threw it out and asked for a different approach. Then the agent executed. That sounds slow. It isn't. The agent reads the codebase faster than I do, drafts the plan in less time than it takes me to make coffee, and executes it while I'm doing something else. The bottleneck shifts from typing speed to decision quality. I ran agents in parallel a lot. Two or three [worktrees](/blog/parallel-claude-code-git-worktrees), each with its own Claude session, each working on a different feature or part of the migration. I'm not sure I could have moved this fast on the migration without parallel worktrees. Heroku to AWS isn't one project. It's a few dozen small projects, most of them blocking on something else, some of them parallelizable. Documentation stopped being optional. The `CLAUDE.md` files in our repos got opinionated. I wrote down our Solid Queue conventions, our test fixture approach, the boring shape of our controllers, the gotchas you'd otherwise have to know to avoid stepping on. The agents read those docs every session, alongside the [hooks and slash commands](/blog/claude-code-hooks-commands-skills) I'd set up to keep that context fresh. Keeping it all current was now load-bearing work, which it always was — we just used to pretend the tribal knowledge in our heads was good enough. The test suite is the contract now. When the agent ships more code than I can read line-by-line, I have to trust the tests to catch what I miss. We [migrated from RSpec to Minitest](/blog/rspec-to-minitest-migration) during this stretch, partly because Minitest is faster and partly because I wanted a less mocking-friendly culture. Mocked tests pass while production breaks. I want tests that fail when the thing fails. Reading PRs is a different skill now. I review fewer lines, but I read them differently. I'm looking at the shape of the change, the intent, the failure modes, not the syntax. The syntax is fine. The agent passes RuboCop. The question is whether the agent understood what we wanted, and that's not a question RuboCop can answer. ## A small moment that made it click The clearest moment for me was a [Caddy](/blog/multi-tenant-ssl-caddy-kamal) boot loop on the new EC2 host one evening during a dry-run cutover. The proxy was crashing on startup, restarting, crashing again. I had nothing useful to go on except a nondescript error in the logs. I described what was happening to Claude and pasted the logs. It read them, asked me one question about how the TLS was configured, then proposed hardcoding `tls_enabled true` instead of letting it derive from the environment. I read the diff, agreed, applied it. Boot loop gone. The fix took five minutes. Six months ago I would have spent forty minutes on the same problem. Not because I'm bad at debugging, but because Claude was already three steps into the documentation while I was still parsing the stack trace. The leverage isn't "the AI knows things I don't." It does, sometimes. The leverage is that the AI is faster at the boring parts of investigation, willing to try things in parallel with me thinking about what they mean, and it doesn't get tired. I'm faster at the part where I decide which proposed thing matches what we actually want. ## What surprised me I didn't get faster at writing code. I got faster at making decisions. Most of my time on the migration was spent reading: reading proposed plans, reading proposed diffs, reading documentation the agent was citing, reading our own architecture decisions to remember why a thing was the way it was. The typing was almost incidental. The cost of context went down. Adding a new feature used to involve the warm-up tax of reloading a piece of the codebase into my head. With agents that already have the entire repository in their working memory and re-read it every session, that tax mostly disappears. I can context-switch between two features without paying the price I used to pay. The cost of clarity went up. When I'm vague, the agent ships vague code. When I describe what I want in three sentences instead of one, I get something I don't have to redo. The skill of writing a tight description of an intended change has become more valuable than the skill of writing the change itself. That's a strange sentence to type. I'm still bad at handing off the keyboard sometimes. There are small surgical changes where I know exactly what I want and describing it would take longer than just doing it. I've stopped feeling guilty about typing those myself. The point isn't to never type code. The point is to type only when typing is genuinely the fastest path. ## What I'm still figuring out Trust calibration is the unsolved problem. Some areas of the codebase I let the agent ship into with minimal review because the tests are good and the surface area is small. Other areas I read every line because the blast radius of a wrong change is too large. I don't have a clean rule for which is which yet. I have intuition, and I'm wrong about my intuition more than I'd like. Making a team agentic is harder than making myself agentic. I've done the personal version. The collective version, where everyone on a team is operating this way and the codebase reflects that, is something I'm just starting to explore. Architecture docs help. Conventions help. There's still a layer of "how does the team know what good agent work looks like" that I don't think anyone has figured out yet. The label might not last. *Agentic engineer* is a useful phrase right now because it points at something specific that *senior software engineer* doesn't quite cover. In a year or two, maybe everyone is doing this and the label dissolves back into *engineer*. That would be fine. I'm not attached to the label. I'm attached to the work. ## What's next I want to write a separate post that's just the migration: the Kamal configs, the Terraform modules, the gotchas, the cutover playbook in detail. That post is for ops people sitting on a Heroku bill they're tired of paying. This one wasn't really for them. This one was for the people watching the agentic engineering conversation and wondering if anyone is actually shipping production work with it. I am. If you want the longer story of how I got here, [From Autocomplete to Autonomy](/blog/from-autocomplete-to-autonomy) is where I first wrote about the shift. I've also got plenty I haven't figured out yet — the team angle, the trust calibration, the question of what happens to this label in a year. I'll keep writing as I figure those out. --- ### I Built an LLM Wiki for My Kid URL: https://hencf.org/blog/llm-wiki-for-my-kid Published: 2026-04-28 My kid keeps asking me about Toy Story. Specifically, about Andy's dad. Where is he? Why don't you ever see him? Is he dead? I told him I'd write him a book about it. I knew there were fan theories out there, but I didn't know any of them well enough to write something honest. So before the book, I'd need to actually learn the material. And I'd just spent a month [building an LLM wiki](/blog/karpathy-llm-wiki-claude-code) on a completely unrelated topic. The pattern was sitting right there. So I pointed Claude Code at the Toy Story 0 mythology and built a second wiki. This post is about what changed and what stayed the same. ## The corpus The first wiki was hundreds of books from a single tradition, all long-form text chunked from PDFs. The Toy Story corpus looks nothing like that. There's a couple dozen YouTube videos (Super Carlin Brothers, Mike Mozart's two-hour livestream, a few others), two dozen articles in English and Portuguese, some forum posts, a handful of tweets. Most of the videos are an hour-plus of two people talking with each other, which means the YouTube transcripts are the densest and most contradictory part of the source material. That mismatch matters. A 19th-century treatise on reincarnation has a thesis, a structure, and arguments. A two-hour rambling YouTube livestream has facts buried in tangents and three different versions of the same scene depending on what minute you're listening to. Same Karpathy LLM Wiki pattern, but the schema had to handle it. ## What stayed the same Three layers, identical to the first wiki: 1. `raw/` — every source captured in its original form, never edited. YouTube transcripts go in as VTT plus a cleaned-up Markdown version. Articles get pulled into Markdown with the URL preserved. Tweets get screenshotted text. Once captured, none of this gets touched. 2. `wiki/` — the LLM-owned layer, in Portuguese (the kid reads Portuguese, not English, and the sources are mixed languages anyway). Pages have YAML frontmatter, wikilinks, citations into `raw/`. 3. `CLAUDE.md` — the schema. Page types, citation format, language rules, ingestion workflow, lint checks. The schema is the part that does the heavy lifting. Every session I run, no matter how long ago the previous one was, follows the same conventions because they're written down and Claude reads them first. The "read everything" rule from the spiritist wiki carried over. I had to write it as an explicit note in CLAUDE.md, otherwise the agent skims and writes confident, generic prose from training data instead of grounded summaries from the actual sources. Nothing about that has changed in the past month. ## What changed Three things, and all of them mattered more than I expected. ### Citation format for mixed sources A book citation can be `[^kardec-le-q166]`: author, work, question number. A YouTube citation needs a timestamp, because the source is two hours long and a claim made at minute 4 has different weight than a claim made during a tangent at minute 95. So the schema defines `[^scb-bbmzuoBC1Rs@04:12]`: source slug, video ID, timestamp. Article slugs, tweet IDs, and forum post URLs each get their own conventions. Every claim in the wiki points to a specific second of a specific source. ### Theories as first-class pages The spiritist wiki had concept pages and book pages. The Toy Story wiki has those too, but it also has a `teorias/` directory. Each fan theory is a page with a `canon: fan-theory | debunked | speculation | meta` frontmatter field. There's a comparison matrix page that lays out the competing theories side by side: Mike Mozart's polio version, Jon Negroni's divorce version, Andrew Stanton's official "complete and utter fake news" debunk, plus a few weaker ones. When two theories contradict each other, the wiki tracks both with citations and marks one as ADOPTED. I picked Mozart's. His version makes way more sense to me. Joe Ranft, one of Pixar's original story guys, told it to him in person, and the visual evidence (cowboy decoration in the house, the mother saying "I knew this would happen one day," the way Andy inherits Woody at the deathbed) lines up with a polio death story from the 1950s. Negroni's divorce theory was reverse-engineered from a Buzz-as-stepfather metaphor that Stanton himself called nonsense. I kept Negroni's theory in the wiki, because it's part of the cultural history of these fan theories, but I marked it "historical context, not adopted" and removed the evidence reinterpretations from the main entity pages. That kind of editorial position is something a wiki can do that RAG can't. Retrieval finds chunks that match your query. It can't tell you "this claim is contested" or "this source is a debunk of that other source." The wiki makes those relationships explicit. ### Source errors get pinned, not silently absorbed One Brazilian article (Omelete) said Andy's father married Molly. Molly is Andy's younger sister, not his mother. I caught it on the first read and the wiki now has a warning on the relevant page: "⚠ erro em Omelete: Molly é filha, não esposa." If I ever come back and re-ingest that article, the same warning is there. The error is tracked, not corrected by deletion. Anyone reading the wiki can see that one source got it wrong, what the right answer is, and which other sources contradict it. ## The three things that surprised me I had built a wiki before. I knew the pattern would work. What I didn't know was how the second one would feel different. ### Connections I didn't see coming The most striking moment was Emma Jean. The wiki had a page for Andy's paternal grandmother, initially unnamed. Fan theories called her "the grandmother" or "the woman in the photographs." Then I ingested a Super Carlin Brothers video about Al's Toy Barn, and one of the brothers mentioned a postcard visible in two different Pixar movies: in Andy's house in *Toy Story*, and on Carl Fredricksen's mantel in *Up*. Same postcard. Same handwriting. Signed "Emma Jean." Pete Docter, the *Up* director, has confirmed that Emma Jean was a friend of Carl's from before he met Ellie. Same postcard, same handwriting, signed "Emma Jean." The wiki linked the two films through a single character that nobody had a page for. I didn't ask Claude to make that connection. It emerged when the same name showed up in two source files and the entity page got created to hold it. This kind of cross-source link happens because the LLM is reading every source and updating an existing graph. RAG would have surfaced one of those mentions to one of my queries. It wouldn't have made the connection. ### The LLM composes, doesn't just collect The Toy Story wiki has a page on Al McWhiggin, the toy collector from *Toy Story 2*. Most of what's on that page didn't exist in any single source. The page is composed from fragments across half a dozen sources: Al's mother dying when he was young, his father running a farm, a childhood story about him trying to steal Andy's father's Woody, a parallel Mike Mozart draws between himself and Al as compulsive collectors who both lost a parent young. The Al page synthesizes all of that into a coherent backstory: only child, raised on a farm, mother died when he was young, father compensated by indulging him with toys, grew up obsessed with collecting, eventually tried to acquire the rare Woody he'd coveted as a kid. No single source contains that paragraph. The wiki composed it from fragments across half a dozen sources, with citations pointing to which fact came from where. Reading the page, you can trace any claim back to the second of the YouTube video it came from. I never wrote a prompt that said "compose a backstory for Al." I gave the schema, the citation rules, and the read-everything rule, and the composition emerged. ### Structured data unlocks the next step This is the surprise I want to dwell on, because it's the reason the book exists. A wiki page with consistent frontmatter, citations to specific timestamps, explicit relationships, and a canon field is structured data. Once the wiki was solid, writing the book got easier in ways I hadn't predicted. I asked Claude for chapter outlines. It pulled from the events directory and ordered them chronologically: the cereal letter in the late 1950s, the polio diagnosis, the surviving toys, the move to Seattle, the marriage, Andy's birth, the death, the inheritance. The events all had dates in their frontmatter, so the ordering was deterministic. I asked for a list of every visual detail that would need to be illustrated. Claude pulled from the conceitos directory: cowboy decoration in the house, the backwards N on Woody's boot, the photographs in Andy's room that were really his father's, the chest in the attic, Hidden City Cafe. Each one with a citation to where the detail came from, so I could verify before drawing. I asked for chapters in the voice of a children's book in Portuguese, grounded in the Mozart theory and avoiding any of the rejected theories. The wiki's `canon` field made this filterable: only adopt facts from pages marked as adopted in the matrix; ignore `canon: debunked` and `canon: speculation` content. The drafts came back grounded. By the time I was generating illustrations with Stitch, the prompts I was giving were almost copy-paste from the wiki entity pages. "Andy's father, around eight years old, in a 1950s American kitchen, holding a Woody doll, with cowboy-themed wallpaper in the background." Every visual element traced back to a specific source. The book is in EPUB beta now, with images. He has it. There's probably a second story coming from the same wiki, maybe about Mr. Potato Head's family or one of the other characters with a hinted-at backstory. I haven't decided yet. ## Two wikis later The two projects look nothing alike. The first is hundreds of books of religious doctrine compiled into a public reference site for adults. The second is a few dozen videos and articles about a Pixar fan theory, compiled into research material for a children's book aimed at one specific reader. The sources, languages, and scales have nothing in common. What survives is the schema and the discipline of reading every source. I think the LLM Wiki pattern is going to get used in places I haven't thought of yet. The two cases I have so far don't have much in common except those two things. That's probably the part that matters. Last week he asked me what really happened to the old space ranger toy that Buzz replaced. There's no page on that yet. --- ### Turbo Ate My Redirect URL: https://hencf.org/blog/turbo-ate-my-redirect Published: 2026-04-17 A customer wanted to connect their Stripe account. They clicked the button, saw nothing happen, clicked again, and eventually emailed support. Solid Errors had been quietly collecting the fallout: a string of `RecordNotUnique` exceptions on `StripeConnector`, one per extra click. The database was rejecting duplicate rows because `stripe_connectors` has a uniqueness constraint on `organization_id`. The first click, the one that should have worked, was nowhere in the error log. It had completed successfully. No error, no 500, no Sentry ping. Just a button that looked like it did nothing. The redirect to `connect.stripe.com` was never happening. ## Tracing it backward I opened Claude Code and started by staring at the controller. ```ruby def create account = Stripe::Account.create(type: "standard") connector = current_organization.create_stripe_connector!( account_id: account.id ) account_link = Stripe::AccountLink.create( account: connector.account_id, return_url: return_url, refresh_url: refresh_url, type: "account_onboarding" ) redirect_to account_link.url, allow_other_host: true end ``` It looked fine. `allow_other_host: true` was already in place. `Stripe::AccountLink.create` was generating a valid URL, and production logs confirmed that much. `redirect_to` was being called. Rails was sending back a 302 with the right `Location` header. The redirect response was leaving the server. It just wasn't arriving at the browser. Claude suggested checking whether the button was Turbo-driven. I went to look. ```erb <%= button_to "Conectar Stripe", admin_stripe_connect_path, method: :post, class: "btn btn-primary btn-sm" %> ``` That's when it clicked. `button_to` generates a form. Any form in a Turbo-enabled app gets submitted via `fetch` instead of a real browser navigation. Turbo's `fetch` follows redirects internally. When the redirect points to your own domain, Turbo loads the response into the page. When it points to a different origin, Turbo can't use the response. The browser returns an opaque cross-origin response, and Turbo stops. Quietly. No error in the console. No warning. The user sees a button that looked like it did nothing. ## The silent failure is the real problem If Turbo logged a warning, I would have caught this in development. If it threw an error the user could see, support would have flagged it the first day. If anything had been raised at all, Solid Errors would have picked it up. Instead, the failure mode is "the button doesn't work." Users click again. Some of them double-click out of habit. Each click creates a new `Stripe::Account` on Stripe's side (a real, permanent account) and attempts to insert a new `StripeConnector` row locally. The first one succeeds. The rest fail on the uniqueness constraint. The errors in Solid Errors are a symptom of users frustratedly mashing the button, not the original bug. The original bug left no trace at all. ## Three things had to line up Once I understood the failure mode, I could see that removing any one of three things would have prevented the bug. Not the fix. The bug. **Turbo's default behavior.** In Rails 7+, `form_with` and `button_to` produce Turbo-driven forms by default. That's the right default for most forms. It's the wrong default for every form whose redirect crosses origins, and the framework has no way to know which is which. There's no warning, no lint rule, no dev-mode message saying "this redirect crossed origins and got dropped." The default is convenient everywhere it works and silent everywhere it doesn't. **Claude didn't flag it.** I wrote the Stripe Connect flow with [Claude Code](/blog/from-autocomplete-to-autonomy) and got working code on the first try. It really was working code, for a definition of "working" that includes posting to the right endpoint, generating the right Stripe URL, and receiving a valid redirect from Rails. What it didn't include was actually landing the user on Stripe. The model has enough Rails and Hotwire training data to use Turbo correctly on the happy path. It doesn't have enough to preemptively think "this form redirects off-origin, I should disable Turbo." That's a senior-level piece of lore the training data doesn't highlight, because blog posts mostly cover the happy path. **My create wasn't idempotent.** Even if Turbo had followed the redirect correctly, a user who clicked fast enough could have fired two requests before the first finished. The controller called `Stripe::Account.create` unconditionally and then `create_stripe_connector!` unconditionally. Two clicks, two real Stripe accounts, two attempted rows. That one was on me. A button that creates an external resource should short-circuit if the resource already exists. The fix was three changes: ```ruby def create connector = current_organization.stripe_connector unless connector account = Stripe::Account.create(type: "standard") connector = current_organization.create_stripe_connector!( account_id: account.id ) end # ...generate account link and redirect end ``` Plus `data: { turbo: false }` on the button. Plus a fresh appreciation for how invisible this category of bug can be. ## The audit Once I knew what to look for, I opened every view in the app and checked. Any form whose controller redirects cross-origin. Any link that navigates cross-origin. Not just Stripe. The app is multi-tenant. Organizations live on subdomains (`centro-1.espirita.club`, `estudantesdoevangelho.espirita.club`) while the platform lives on the base domain. Every login crosses origins: a user logs in on the platform, and the server redirects them to their subdomain admin. Every logout crosses the other way. Every org switch. Every "view all organizations" link in the admin nav. Public checkout forms redirect to Stripe. The customer portal link redirects to Stripe. The donation flow redirects to Stripe. All of them were Turbo-enabled. All of them had been "working" in the same way Stripe Connect was "working": the redirect reached Turbo, Turbo couldn't follow it, the user saw nothing. Some paths happened to still land the user somewhere useful because a full-page reload was triggered elsewhere in the flow. Others hung. By the end of the audit, `data-turbo: false` was annotating close to a dozen views: login forms, logout buttons, org picker links, admin layout nav, Stripe Connect buttons, Stripe Checkout subscribe buttons, customer portal links, donation forms, and the public checkout form. Every one of those is a place where the app is telling the browser: this navigation is leaving our origin, please handle it the old-fashioned way. Turbo doesn't get to help here. ## Tests I wrote next The scariest part of this bug is that it's nearly invisible in development. I develop on `localhost` with subdomains like `estudantesdoevangelho.localhost`. Those are technically cross-origin from Turbo's perspective, and they do exhibit the broken behavior. But in development, I'm rapidly switching pages, restarting the server, doing full reloads. I don't sit with any one silent redirect long enough to notice. In production, a user clicks once, sees nothing, clicks again, emails me. So I wrote integration tests for every cross-domain redirect in the app. Platform login to subdomain admin. Subdomain logout to platform root. Admin login for non-members to the platform org picker. Email confirmation to subdomain. I also added E2E coverage for external Stripe redirects so that if someone (me, Claude, a future contributor) accidentally removes `data-turbo: false`, the test breaks before production does. The tests don't directly assert on Turbo behavior. They assert on the redirect location the server returns. Combined with the annotations in the views, the combination is enough to catch regressions. ## What I keep thinking about The thing that bothers me about this bug isn't that it happened. Every Rails and Hotwire app has to learn this lesson once. What bothers me is how invisible it was. Rails teaches you to trust the framework. Turbo teaches you to trust that your forms submit, your redirects follow, and the user ends up where your controller told them to go. That's a good default. I like Hotwire. I'm not writing hand-rolled JavaScript for navigation again. But the guarantee Turbo makes is implicit: your navigation will work as long as it stays on your origin. And origin is a property the controller decides at runtime, not a property of the form at render time. The framework can't check it when the view is rendered. It doesn't even know the form is about to redirect cross-origin, because that decision hasn't happened yet. The only entity who knows is the developer writing `redirect_to account_link.url`, and that developer has to have internalized "if this might leave my origin, the form that triggered it must not be Turbo-driven." That's a lot of load to put on the developer, and the penalty for forgetting is a bug that doesn't show up in tests, doesn't show up in logs, doesn't show up in error tracking, and only shows up when a user writes in. I've added the annotations. I've written the tests. I know the pattern now. But I'm going to forget it once a year, and the LLM I write with is going to keep producing Turbo-driven forms pointed at off-origin redirects, and the bug will still be invisible when it happens. The only structural fix I've found is: every cross-domain redirect in your app needs a test, and those tests need to run in CI. Everything else is vibes. --- ### From RAG to Wiki: What 600 Books Taught Me About Knowledge Compilation URL: https://hencf.org/blog/karpathy-llm-wiki-claude-code Published: 2026-04-14 Four days after Karpathy posted his LLM Wiki gist, I started building one. Not because I needed another project, but because I had the perfect test case: 600 books already chunked and embedded, sitting in a Rails app I was building for a different purpose. I was using them for RAG. Chunked retrieval, semantic search, on-demand answers. The wiki approach proposed the opposite: read every source, compile the knowledge once, and let it compound. Six days later, I had 679 interlinked pages, over 6,000 cross-references, and an answer to a question I'd been avoiding: the wiki is better. Not a little better. Fundamentally different. ## The Gist Karpathy posted [the idea](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f) on April 3. It got something like 15 million views and 5,000 stars in a few days, for good reason. The core argument: stop using LLMs as search engines over your documents. Instead, have them read your sources and compile a persistent, cross-referenced knowledge base. Three layers: raw sources (immutable), wiki pages (LLM-owned), and a schema file that defines the structure. The part that clicked for me was about compounding. RAG re-derives answers from scratch every query. The wiki processes each source once, integrates it into a growing knowledge graph, and that understanding persists and builds. Karpathy's line: "The knowledge is compiled once and then kept current, not re-derived on every query." I had 600 books on Spiritist doctrine — a 150-year tradition with deeply interconnected literature. The source material was already chunked as JSON files from a Rails app I was building. It was almost too convenient. So I pointed Claude Code at the books and told it to build a wiki. ## The First Attempt Was Garbage Claude Code finished surprisingly fast. Too fast. When I checked the output, only 3 of 34 initial books had actually been read. The rest were skimmed — a few chunks sampled, the rest ignored. The wiki pages were fluent, well-structured, and almost entirely generic. Claude was writing from its training data, not from the actual book content. The pages could have been written without ever opening the source files. This was the most important lesson of the entire project. An LLM that skims produces confident, plausible summaries that look like real synthesis until you check them against the source. If you don't check, you'll never know the difference. The writing quality is identical. The grounding is completely absent. ## The Rule: Read Everything The fix was simple and expensive. Every single chunk of every single book. No shortcuts. I wrote it into the project's CLAUDE.md in bold: "Always read every single chunk of the book before creating wiki pages. A shallow skim produces generic pages based on training knowledge; a full read produces grounded pages with specific citations, quotes, and insights unique to each source." For a 300-chunk book (roughly 150,000 tokens), this meant reading in batches of 30-50 chunks, taking notes on key passages, then writing the wiki pages. For Kardec's five foundational works alone, that was over 2,000 chunks, roughly 1.3 million tokens of dense 19th-century Portuguese. It was slow. The context window filled up and reset over 15 times during the first 34 books. But the output was completely different. Instead of "Reincarnation is a key concept in spiritist doctrine," I got pages citing specific question numbers, quoting exact passages, and tracing how the same concept evolves across books written decades apart. ## What a "Read" Actually Looks Like I built a small set of Ruby scripts around the process. `bin/process_pdf` takes a PDF or DOC file, extracts text, and chunks it at roughly 500 tokens with overlap. `bin/dump_book` outputs the chunks as human-readable text with chapter grouping and page references, and supports `--from N --to M` for batch reading. The workflow for each book: 1. `bin/dump_book --stats` to see the structure (chapters, chunk count) 2. Read the entire book in batches of 30-50 chunks, covering everything 3. Create the book page with specific chapter references, quotes, and insights 4. Concept audit: identify every concept the book enriches, update each existing concept page with a new section and citations, create new pages if the book introduces uncovered themes 5. Update entity pages, topic pages, infrastructure files 6. Quality check: no broken wikilinks, all frontmatter valid, concept pages enriched The CLAUDE.md schema file defines all of this in 330 lines. Page types, frontmatter formats, filename conventions, wikilink rules, language rules (English for filenames and code, Portuguese for content), and a post-ingest checklist. Without it, each session would invent its own structure. With it, book #183 follows the exact same format as book #1. ## The Compounding Effect This is the part I didn't expect. When Claude Code reads the first book about reincarnation (Kardec's *The Spirits' Book*), it creates a concept page with a definition and primary source citations. Clean, straightforward, one source. When it reads the second book (*The Gospel According to Spiritism*), it doesn't create a new page. It goes back to the existing reincarnation page and adds a section about how this book expands the concept with exegetical arguments from scripture, a moral framework that wasn't in the first book. By the third book (*Missionaries of the Light*, a spirit narrative), the reincarnation page gains a section about mechanics: spiritual planning committees, the construction of the perispirit, the fertilization process as described by a spirit observer. This isn't in the codification. It's narrative expansion from a completely different genre of writing. By the time the wiki has processed 36 books that mention reincarnation, the concept page is over 800 lines long, with citations from philosophical treatises, spirit narratives, mediumistic poetry, and historical analyses. It traces how understanding deepens across authors, genres, and decades. No single book contains this view. No search query could assemble it. The reincarnation page now lists 36 primary sources, each with specific chapter and page references. The charity page cites 58 sources. Mediumship, 32. These aren't just lists. Each source entry comes with a section explaining what that specific book contributes to the concept that the others don't. Every new book creates dozens of new connections. Not just to the concept pages it directly addresses, but to other books, other entities, other historical periods. The wiki doesn't grow linearly. It grows combinatorially. And you can see it. ## Architecture: Three Layers The project has two repositories. The content engine (Claude Code reading books, maintaining interlinked Markdown), and a Rails app that serves the wiki as a website. ### Sources (Immutable) 599 books chunked as compressed JSON files. Each chunk has content, page numbers, chapter, section, position, and token count. These files are the raw material. Claude never modifies them. ### Wiki (LLM-Owned) 679 Markdown pages with YAML frontmatter, organized by type: 539 book pages, 58 concept pages, 32 topics, 25 people pages, 14 entity pages, and 8 collection pages. The whole directory is a valid Obsidian vault. Open it in Obsidian and you get graph view, backlinks, and full wikilink navigation for free. No setup required. The concept pages are the crown jewels. Each covers a doctrinal concept synthesized across every source that touches it. They start with the authoritative definition from the foundational texts, then layer on how the concept expands across narratives, philosophy, and practical guides. Every factual claim cites a specific book, chapter, and location. ### Schema (CLAUDE.md) This is the constitution. It defines six page types with their frontmatter schemas, the full ingestion workflow, processing order (foundational texts first, then narrative expansions, then secondary authors), filename conventions (no accents, use aliases), and quality checks. The schema is what makes the output consistent across hundreds of books processed in dozens of sessions over days. Without it, you get entropy: inconsistent formats, missed cross-references, drifting conventions. With it, every session follows the same methodology regardless of context window resets. ## Processing Order Matters You can't build a knowledge graph randomly. You need the foundational definitions before the narrative expansions. I processed Kardec's five codification works first. These established authoritative definitions for every core concept. Then the André Luiz series (13 books of spirit narratives expanding the theory with detailed descriptions). Then Emmanuel's historical works, then biographical works, then devotionals. This ordering meant that when Claude read a narrative about reincarnation mechanics in book #40, the concept page already had a solid definition from book #1. The narrative detail was added as an enrichment, not as the primary source. Later, the wiki expanded beyond a single author to include European spiritist researchers (Gabriel Delanne, Ernesto Bozzano) and Brazilian philosophers (J. Herculano Pires). Each new author brought a different perspective and writing style, but the concept pages absorbed their contributions the same way: find what's unique, cite it, and connect it to what's already there. ## The Concept Debt Crisis Halfway through the initial 34 books, I noticed a problem. Claude was creating solid book pages but skipping concept enrichment. Each book got a nice standalone summary, but the concept pages (reincarnation, obsession, mediumship) weren't being updated with citations from new sources. The book pages were islands. The wiki's core value proposition is the connections between them. I called it out and formalized a post-ingest checklist: every book must update 3-5 concept pages with specific citations. No exceptions. If a book touches reincarnation, the reincarnation page gets a new section with what this specific book adds that the others don't. Paying down the concept debt from those first 34 books required parallel Claude Code agents, four at a time, each working on different concept pages. The pattern stuck. For the remaining 150 deep reads, concept enrichment was built into the workflow from the start. ## Scaling with Parallel Agents After proving the pipeline worked with the first 34 books, I scaled it. I downloaded over 500 additional PDFs from multiple sources. The approach shifted from sequential processing to parallel agent batches: up to 13 Claude Code agents running simultaneously in [isolated git worktrees](/blog/parallel-claude-code-git-worktrees), each deep-reading and ingesting its assigned book. This wasn't without friction. Agents hit rate limits and died mid-read. Some merged conflicting changes to the same concept page. I throttled to 2-3 agents at a time and the throughput stabilized. 183 books were deeply read with full concept enrichment. The remaining books got lighter coverage (book pages with structure and themes, but not the deep cross-referencing). The plan is to keep going. ## The Rails App The Markdown wiki works great in Obsidian. But I wanted it on the web, searchable, with a knowledge graph you can explore. I built a Rails 8 app in parallel with the content ingestion. SQLite for everything, Tailwind for styling, deployed with Kamal to [wiki.espirita.club](https://wiki.espirita.club). ### Hybrid Search Search combines SQLite FTS5 for keyword matching (BM25 relevance scoring, title matches weighted 10x) with vector search via sqlite-vec for semantic similarity. Results merge using Reciprocal Rank Fusion. You can search for a Portuguese term and find concept pages that discuss it under a different name. ### Knowledge Graph A D3.js force-directed graph renders the entire wiki as a visual network: 679 nodes, over 6,000 edges, color-coded by page type. Zoom, filter by type, click any node to navigate. Each page also has a neighborhood graph showing its immediate connections. This is where the compounding effect becomes visual. Concept pages sit at the center with dense clusters of connections. Book pages radiate outward, linked to the concepts they address. You can see at a glance which concepts are most deeply covered and which books are most interconnected. ### AI Sprinkles, Not a Chatbot I originally built a full chat page with RAG-powered Q&A. Then I deleted it. Small AI features sprinkled throughout the wiki turned out to be more useful: an AI summary on every page, enhanced search for question-like queries, deep search across source chunks, contextual Q&A scoped to the current book, passage lookup in concept page sidebars, and concept comparison. Each uses the same vector search + LLM pipeline, scoped to wherever the user already is. ### Content Pipeline The first version loaded all Markdown files into memory at boot. That caused a stack overflow during Rails initialization. I moved everything to SQLite with a `wiki:sync` rake task that parses frontmatter, renders Markdown, extracts wikilinks as a join table, and builds the FTS5 index. The content engine writes Markdown, the Rails app reads it. Clean separation. ## RAG vs. Wiki I'm [still building the RAG system](/blog/rag-without-leaving-rails). It has its uses for answering specific factual questions, finding relevant passages, powering conversational interfaces. But after building the wiki, I see it differently. RAG retrieves. It finds chunks semantically similar to your question and pastes them into a prompt. The LLM synthesizes an answer on the fly, every time. Ask the same question tomorrow, it does the same work again. Nothing accumulates. The wiki compiles. It processes each source once, integrates it into a growing structure, and never needs to re-derive that understanding. When you ask "how is reincarnation treated across these 600 books," RAG gives you a handful of relevant chunks from maybe 5-10 books. The wiki gives you an 850-line synthesis across 36 books, with citations to specific chapters and questions. The difference is like asking a librarian to find relevant passages versus asking a scholar who's read every book in the collection to write a literature review. Both are useful. They're not the same thing. The wiki also surfaces connections that no query would find. Nobody searches for "how does the description of reincarnation mechanics in a 1945 spirit narrative relate to the exegetical arguments in an 1864 theological treatise." But when Claude reads both books and enriches the same concept page, that connection exists. It's browsable. It compounds with every new source. ## What Domain Experts Think I built this partly as an experiment. I wasn't sure if people who actually study this material would find it valuable or dismiss it as a shallow imitation. The reaction from people who've spent years reading these books has been the strongest validation. They're finding cross-references they hadn't made, connections between a concept in a devotional work and a passage in a philosophical treatise from fifty years earlier, traced through the wiki's citations. The wiki isn't replacing their reading. It's making the relationships between what they've read visible. ## The Scholar Model There's a mental model that made this project click: Claude Code as a scholar, not a search engine. A search engine takes your query and finds matching documents. A scholar reads deeply, builds understanding over time, and produces work that synthesizes sources into something new. The wiki approach treats the LLM as a scholar. The scholar reads the entire book, not a sample. Takes notes on what's unique about each source. Goes back to previous work and revises it in light of new reading. Produces citations. The scholar's understanding compounds across sources in a way that retrieval never will. This only works because of two things: the CLAUDE.md schema (which gives the scholar a consistent methodology across sessions) and the requirement to read everything (which forces grounded output instead of training-data confabulation). Drop either one and you get the garbage I produced on day one. The wiki keeps growing. Each new author, each new book, each new perspective enriches the existing pages with connections that didn't exist before. It's live at [wiki.espirita.club](https://wiki.espirita.club) if you want to see what an LLM-compiled knowledge graph looks like when it's built from 600 books instead of personal notes. --- ### I Rewrote My Rails App in Elixir. AI Brought Me Back. URL: https://hencf.org/blog/elixir-rewrite-ai-brought-me-back Published: 2026-04-10 I have a side project that aggregates YouTube content, extracts metadata with LLMs, and serves it through search and a chat agent. It's a Rails app with PostgreSQL, Solid Queue, and Ollama for local models. Solo developer, single VPS. A few months ago, I decided to rewrite it in Elixir. Three things pulled me toward Phoenix. First, I wanted streaming LLM responses in the chat feature. Token-by-token output, the way ChatGPT does it. LiveView makes this feel natural. You can push tokens to the client as they arrive without setting up SSE or managing WebSocket subscriptions manually. Second, the BEAM's concurrency model. The app runs a bunch of background jobs: crawling YouTube channels, fetching transcripts, extracting metadata, generating embeddings. Elixir handles concurrent work at the runtime level. Third, I just wanted to learn Elixir. It's a personal project. That's a valid reason. I went for a full migration, not a hybrid approach. ## The shared database trick Before writing any Elixir, I made one decision that turned out to be the smartest part of the whole experiment: both apps would share the same PostgreSQL database. The Phoenix app used Ecto, the Rails app used ActiveRecord, both pointing at the same tables. Ecto migrations used `create_if_not_exists` to avoid stepping on existing Rails schema. Any data created in the Elixir version was immediately available in Rails. No migration scripts, no data export. This made the rewrite completely reversible. If Phoenix didn't work out, the data was already home. ## The rebuild I rebuilt the app in Phoenix with Claude Code. Chat LiveView with streaming LLM responses, CMD+K spotlight search, the video platform with subdomain routing, admin screens, the RAG pipeline, Oban workers for the full crawl-to-publish pipeline. I also used the rewrite as a sandbox for things I'd been putting off in Rails: crawled a second YouTube channel, benchmarked six local models for metadata extraction (gemma3:4b [won](/blog/extracting-metadata-local-llms)), and rewrote the chapter detection system with more strategies. The features worked. Tests passed. I had a functional Phoenix app talking to the same database as my Rails app. Then I went back to Rails. ## Why I came back Not because Elixir is bad. Not because Phoenix is lacking. The reason is more specific: writing code with AI was a noticeably worse experience in Elixir than in Ruby. I use Claude Code for [almost everything](/blog/from-autocomplete-to-autonomy). Over 90% of my commits at work are co-authored with Claude. My setup is tuned for it: architecture docs that get loaded as context, [custom hooks](/blog/claude-code-hooks-commands-skills), plan mode for complex features, [worktrees for parallel work](/blog/parallel-claude-code-git-worktrees). When I sit down to build something in Ruby, the workflow is dialed in. I describe what I want, Claude produces working code, and we iterate from there. In Elixir, that flow broke down. The generated code had more errors. It took more rounds of back-and-forth to reach something that actually worked. Tasks that would be one-shot in Ruby required multiple iterations in Elixir, each one fixing something the previous attempt got wrong. The model knows Elixir, but there's a reliability gap compared to Ruby. That gap compounds fast when AI is writing the majority of your code. Part of this is the model's training data. Ruby on Rails is one of the most documented web frameworks in existence. Twenty years of blog posts, Stack Overflow answers, open source projects. Elixir's ecosystem is smaller and younger. The model has less to draw from. Part of it is my own setup. My Claude Code configuration is optimized for Ruby. The architecture docs, the project-specific instructions, the conventions files. Starting fresh in a new language means starting without all that accumulated context. Switching to a new editor and losing all your muscle memory at the same time. The result: development was slower and rougher in Elixir. For a side project where I build in stolen hours, that friction matters. ## The language your AI knows best This is the part I keep thinking about. When I write code by hand, language choice is about the language: its type system, its runtime characteristics, its ecosystem, how it makes me think about problems. I picked Ruby because I think in Ruby. Other people pick Go or Rust or TypeScript for similar reasons. When AI writes most of your code, there's a new variable: how well the AI handles that language. And it's not just about the model. It's about the entire stack of context you've built around your workflow. Your project docs, your architecture notes, your conventions, your hooks, your test infrastructure. Everything that makes the AI effective in your specific codebase. Switching languages means resetting all of that to zero. I'm not saying everyone should use Ruby because Claude is good at Ruby. The gap will narrow as models improve and as Elixir's ecosystem grows. For a brand new project where you'd be building context from scratch in any language, the difference might not matter as much. But for an existing project with a tuned AI workflow, the switching cost is higher than I expected. It's not just "learn a new language." It's "rebuild your entire AI development environment." ## What I brought back The rewrite wasn't wasted time. The shared database meant everything I built in Elixir was already in PostgreSQL when I came back. I cherry-picked three things into Rails. The expanded database: during the Elixir sandbox period, I'd crawled a second YouTube channel and processed thousands of new videos, all sitting in the shared database, ready to go. The metadata extraction approach: I benchmarked six local models during the rewrite and proved that gemma3:4b on Ollama could replace the cloud API. That work transferred directly. And the chapter detection rewrite: more strategies covering more video formats. The logic was different in Elixir (functional vs OOP), but the strategies themselves were portable. Within a day of coming back, I had all three integrated and was shipping new features at the pace I'm used to. ## What stays with me This isn't a "Rails is better than Phoenix" post. Elixir's concurrency model is genuinely impressive. LiveView's approach to real-time is elegant. I can see myself using Elixir for something where concurrency is the core problem, not a side concern. But I can tackle concurrency in Ruby too. Solid Queue handles my background jobs. If I outgrow it, Ruby has async options, and they'll only get better. Rails conventions and ecosystem maturity give me a productivity baseline that's hard to match in a younger framework, especially when AI is doing most of the typing. The streaming LLM feature that originally motivated the switch? Still on my list. It's doable in Rails with Turbo Streams or ActionCable, just with more manual wiring. The fact that it's solvable in Rails, just less elegantly, tells me the motivation was real but not strong enough to justify switching everything else. No regrets on the experiment. I learned some Elixir, got a feel for Phoenix and LiveView, discovered things about my own project I'd been avoiding, and shipped real data work that made my Rails app better when I came back. The shared database trick made it all low-risk. If you're considering a similar experiment, point both apps at the same database. Build in the new stack, see how it feels. If it doesn't work out, you haven't lost anything. The unexpected takeaway was about AI, not languages. When your AI development setup writes most of your code, the quality of that experience in a given language matters more than it used to. Today, Ruby has a clear edge in my workflow. That might change. But I'm not going to fight my own tools to find out. --- ### Extracting Metadata from 40K Videos with Local LLMs URL: https://hencf.org/blog/extracting-metadata-local-llms Published: 2026-04-08 I run a knowledge base of spiritist YouTube videos at [guia.espirita.club](https://guia.espirita.club). The app aggregates content from dozens of channels and currently has over 40,000 videos. Each video needs structured metadata: who's speaking, what topics it covers. That's how users browse by speaker, filter by theme, and find related content. YouTube's API doesn't give you any of this. The information lives in video titles and descriptions, written by hundreds of different channel operators in inconsistent formats. "PALESTRA ESPIRITA - Claudia Piva" has the speaker in the title. Others bury it in the description as "Palestrante: Janice Leal". Some don't mention the speaker at all. This is an LLM job. Read the title and description, extract speaker names and roles, classify into predefined themes. I needed to do it across the entire catalog, and I didn't want to pay for it. ## Starting with Groq's free tier My first approach was Groq's free API with `gpt-oss-20b`, the same setup I [originally used to extract metadata](/blog/llm-extraction-at-scale) for the smaller catalog. The free tier gives you 30 requests per minute, 1,000 per day, 8K tokens per minute, and 200K tokens per day. For the initial batch of 8,000 videos, this worked fine. Each extraction takes a few hundred input tokens (title + description) and returns a small JSON response with speaker names, roles (speaker, interviewer, moderator, medium), and confidence levels. The prompt also handles Portuguese role conventions, where "Palestrante" and "Expositor" both mean speaker, and "Médium" means someone channeling a spiritual entity rather than presenting their own material. At 30 RPM with a 2-second sleep between requests, I could process about 30 videos per minute. The initial catalog took a few days to get through. The problem showed up when the catalog grew. I added more channels and the total hit 40K. Daily imports bring in 50-100 new videos, well within the free tier's limits. But the backlog of unprocessed videos from newly added channels was thousands deep, and at 1,000 requests per day, clearing it would take weeks. I also tried `llama-3.1-8b-instant`, which has a much more generous free tier at 14,400 RPD. But the quality drop was real. It hallucinated speaker names on ambiguous videos and missed topics on devotional content that didn't have an explicit theme in the title. For a catalog where accuracy matters more than speed, the cheaper model wasn't worth it. I did the math on Groq's paid tier: about $4.83 total for the entire backlog (54M input tokens at $0.075/M, 2.6M output tokens at $0.30/M). Cheap, but the free tier already covered daily processing, and I had a server with Ollama running for embeddings. Why not run the extraction locally? ## YouTube blocks your server Before the local model story, there's an architectural constraint that shaped everything. I covered this in detail in [YouTube Blocked My Server in 15 Seconds](/blog/youtube-blocks-your-server) — the short version is below. The app also needs video transcripts for topic classification, which requires understanding what the video is actually about, not just the title. YouTube's transcript endpoint returns CAPTCHA pages or `FAILED_PRECONDITION` errors from cloud IPs. I tried watch page scraping and the protobuf `get_transcript` endpoint. Same result from any VPS IP. The solution was an internal API. The production server exposes authenticated endpoints: ``` GET /internal_api/metadata_extractions/next POST /internal_api/metadata_extractions ``` The first returns the next video that needs processing. The second accepts the result and persists it, returning the next pending video in the same response so the client doesn't need a separate request. A rake task on my Mac loops continuously: fetch the next pending video, run the LLM extraction, POST the result back, repeat until nothing's left. A launchd plist runs this hourly with a PID lockfile to prevent overlapping runs. ```bash # Simplified version of the local processing loop while true; do response=$(curl -s -H "Authorization: Bearer $TOKEN" \ "$SERVER/internal_api/metadata_extractions/next") [ "$response" = "null" ] && break video_id=$(echo "$response" | jq -r '.id') title=$(echo "$response" | jq -r '.title') description=$(echo "$response" | jq -r '.description') result=$(ollama_extract "$title" "$description") curl -s -X POST -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d "{\"video_id\": $video_id, \"extraction\": $result}" \ "$SERVER/internal_api/metadata_extractions" done ``` This pattern turned out to be the best part of the whole architecture. I use it for transcripts, embeddings, and metadata extraction. The server knows what needs processing. Any machine with API access and a local model can work through the queue. If I got a machine with a faster GPU, I'd point it at the same endpoints and processing would speed up with zero server-side changes. The launchd plists on my Mac run hourly. Each one has a PID lockfile so it skips if the previous run is still going. Logs rotate automatically, keeping the last five runs. It's not glamorous infrastructure, but it's been running unattended for weeks. ## The rate limit cascade While still on Groq, I hit a bug worth mentioning because it's a classic Ruby trap. My extraction job had this rescue chain: ```ruby rescue LlmError, StandardError => e video.update!(processing_status: :failed) ``` `RateLimitError` inherits from `StandardError`. When Groq returned a 429, the error was caught by `StandardError` before it could propagate to the job-level retry handler. Videos got marked as permanently failed instead of retried. A separate hourly job scanned for failed videos and re-enqueued them. Within days, Solid Queue had 18,000+ duplicate extraction jobs, all hitting Groq, all getting rate limited, all getting marked as failed, all getting re-enqueued again. The fix was ordering the rescue clauses correctly and adding `on_conflict: :discard` to prevent duplicate jobs. The real signal was noticing a queue that should have had dozens of jobs somehow had 18,000. ## Going local The tipping point came when Groq's free tier was so throttled that zero extractions completed in a day. The daily cap hit early, every subsequent request got a 20-minute `retry-after`, and I was maintaining a pipeline that processed nothing. I already had Ollama on the production server for embeddings (the same setup that powers the [RAG pipeline](/blog/rag-without-leaving-rails)), but the server only has 4GB of VRAM. The embedding model (bge-m3) takes 1.2GB and stays resident, leaving about 2.8GB for a chat model. Most 4B parameter models hover around 2.5-3.5GB, so it's tight. I set up a benchmark: 25 test scenarios covering the hardest edge cases. The trickiest one was the spirit author problem. In spiritist content, "Emmanuel através de Chico Xavier" means Emmanuel is a spirit entity communicating through the medium Chico Xavier. Emmanuel should be tagged as the author, not the speaker. Chico Xavier is the medium, not the presenter either. The actual speaker might be someone else entirely, reading the dictated text aloud. Early prompts got this wrong constantly. The LLM saw the names in the description and listed them all as speakers. Other edge cases: videos with multiple speakers on a panel, devotional content with no speakers at all (just music or prayers), and Portuguese role names where "Palestrante," "Expositor," and "Orador" all mean speaker but "Médium" does not. Six models, same prompt, same 25 test cases: | Model | Size | Quality | Speed | Verdict | |-------|------|---------|-------|---------| | gemma3:4b | 3.3GB | Perfect speakers, clean topics | 1-5s | Best overall | | ministral-3 | 6.0GB | Perfect | 3-8s | Too large for my server | | qwen3.5:2b | 2.7GB | Good speakers, excessive topics | 2-3s | Usable | | llama3.2:3b | 2.0GB | Hallucinated speakers | 1-11s | No | | qwen3.5:4b | 3.4GB | Good, but broken JSON output | 4-6s | Unreliable | | gemma3:270m | 291MB | Missing roles, empty topics | <1s | Too small | llama3.2:3b was the worst offender for hallucination. Given a video titled "Oração para o lar" (Prayer for the home), it invented a speaker name that appeared nowhere in the title or description. qwen3.5:4b produced good extractions but wrapped its JSON output in markdown code fences despite being told not to, which broke parsing on about one in five responses. qwen3.5:2b was usable for speakers but tagged every video with a dozen topics when most videos have two or three. gemma3:4b won clearly. It fit in memory alongside the embedding model, handled the spirit author distinction correctly after prompt tuning, and produced valid JSON consistently. I ran the initial extraction on the server and cleared the backlog in a few days. ## The thinking mode trap When I later upgraded to newer models, I tried qwen3:4b and something was off. Extractions that should take a second were taking 50-100 seconds. The results were correct. The model was just absurdly slow for a 4B parameter model on decent hardware. Qwen3 enables thinking mode by default. The model generates hundreds of reasoning tokens wrapped in `` tags before producing the actual JSON output. For a structured extraction task where the answer is "read the title, return the speaker name," the model was spending 95% of its compute reasoning about how to extract a name from a string. Disabling it through Ollama's API (`think: false`) dropped inference from 50-100 seconds to under one second. Same model, same prompt, same output quality. ```ruby # Ollama native API - disable thinking for structured extraction response = client.chat( model: "qwen3:4b", messages: [{ role: "user", content: prompt }], format: "json", think: false # This is the difference between 100s and <1s ) ``` This isn't documented prominently. If you're running Qwen3 on Ollama for structured extraction and it's inexplicably slow, check if thinking mode is on. For tasks where the model needs to reason through a complex problem, thinking mode helps. For "read this text and return a JSON with names," it's pure overhead. ## JSON reliability with small models One thing I didn't expect: small models are unreliable JSON producers, even with Ollama's `format: "json"` constraint. gemma4:e4b (Gemma 4's efficient 4B variant) produces structurally invalid JSON maybe one in ten requests. Trailing commas before closing braces. `]` where `}` should be. Markdown code fences wrapping the response. The format constraint helps but doesn't eliminate the problem entirely. I added a sanitization step: ```ruby def sanitize_json(text) text = text.gsub(/```json\s*/, "").gsub(/```\s*$/, "") text = text.gsub(/,(\s*[}\]])/, '\1') text = text.gsub(/\](\s*)$/, '}\1') if text.count("{") > text.count("}") text end ``` It catches the three most common failures: markdown wrapping, trailing commas, and mismatched braces. Turned a 90% success rate into something close to 99%. The alternative is a bigger model that produces clean JSON, but that means cloud API calls and rate limits again. ## Where it landed The system settled into a clear split based on task complexity: **Speaker extraction** (title + description only): runs a 4B model locally on my Mac. The input is short, the task is straightforward, small models handle it fine. **Topic classification** (from full transcript, 32 themes): needs a larger model. Small models produce too many JSON errors with long transcripts and don't classify accurately across that many categories. This goes through a cloud API. **Daily processing**: 50-100 new videos per day, handled automatically by a launchd cron job on my Mac hitting the internal API endpoints. **Backfill**: same rake task, just runs longer. I processed the full 40K catalog from my laptop over a few days. Total cost: $0. The Mac is on anyway. Ollama is free. The internal API pattern means any machine can chip away at the queue. ## What I learned about small models A few things surprised me working with 4B models at this scale. First, the known speakers list. I initially passed all 289 known speakers to the model in the prompt so it could match extracted names against them. On some videos, the 4B model just dumped the entire list back as the extraction result, returning 118 speakers for a single talk. It treated the reference list as the answer rather than using it for matching. Removing the list from the prompt and doing the matching in Ruby afterward was both simpler and more reliable. The DB query to find a fuzzy match against known speakers takes microseconds. The model doesn't need to do it. Second, context length matters more than parameter count for topic classification. Speaker extraction works fine with small models because the input is short (title + description, rarely more than 500 tokens). Topic classification needs the full transcript, which can be thousands of tokens. The small models' JSON reliability degrades noticeably with longer inputs. They lose track of the output structure partway through and produce malformed responses. For classification I had to move to a larger model regardless of whether it ran locally or in the cloud. Third, VRAM management on a constrained server is its own puzzle. Ollama loads models on demand and evicts them when memory is tight. With only 4GB total, loading a 3.3GB chat model alongside a 1.2GB embedding model left almost no headroom. I ended up running the embedding model with `keep_alive: -1` (permanently resident) and letting the chat model load and unload per batch. On my Mac with more memory, this isn't an issue. On the server, it meant I couldn't run both tasks concurrently. Eventually I removed Ollama from the production server entirely. Everything AI-related goes through cloud APIs for the web-facing features (Groq for the chat agent), and all batch processing runs on my Mac through the internal API. The server just serves the app. The Mac does the thinking. --- ### SQLite Backups: The Boring Way URL: https://hencf.org/blog/sqlite-backups-the-boring-way Published: 2026-04-07 A month ago I wrote about [tuning Litestream to stay within Backblaze B2's free tier](/blog/litestream-backblaze-b2-free-tier). The post ended with a config that worked: wider sync intervals, disabled retention, B2 lifecycle rules handling cleanup. I shipped it, monitored it for a few days, and called it done. Three weeks later I ripped the whole thing out and replaced it with a cron job. ## Why Litestream was overkill Litestream is built for continuous WAL replication. It watches your SQLite database, streams every write-ahead log segment to object storage, and gives you point-in-time recovery. If your server dies at 2:47 PM, you can restore to 2:46 PM. For applications where that matters, it's excellent. My application is [espirita.club](https://espirita.club), a platform for spiritist organizations. It runs on a single server. If the server dies, I lose whatever happened since the last backup. The question is: how much is acceptable to lose? For a personal SaaS with a few active organizations, the answer turned out to be "a day." Nobody is making irreversible financial transactions. The content is event schedules, blog posts, membership records. If I lost 24 hours of data, I could recover most of it from the organizations themselves. Point-in-time recovery to the minute is solving a problem I don't have. Once I accepted that, the complexity of Litestream stopped making sense. Even after the tuning described in my [previous post](/blog/litestream-backblaze-b2-free-tier), I had a Docker sidecar container running continuously, watching a Docker volume, uploading WAL segments to B2, and silently failing whenever B2's transaction limits got weird. Three rounds of config fixes over a month, each one disabling another background monitoring loop that was burning through B2's Class C transaction cap. The system worked, but it required understanding Litestream's internals to keep it working. The Litestream documentation is great, but it's written for a general S3-compatible storage backend. Backblaze B2's free tier has constraints that the defaults don't account for. Every time I thought I'd found the last hidden source of API calls, another one popped up. L0 retention checks, compaction monitors, validation loops. Each one was individually reasonable and collectively over budget. ## The replacement: two shell scripts and a cron job I wrote two scripts. One backs up databases, the other backs up Active Storage files. The database script uses `VACUUM INTO` to create a clean copy of each SQLite file, then `rclone sync` to upload everything to B2: ```bash #!/bin/bash set -euo pipefail STORAGE_DIR="/var/lib/docker/volumes/ce_storage/_data" BACKUP_DIR="/tmp/ce-db-backups" B2_BUCKET="ce-espirita-backups" DATABASES=( "production.sqlite3" "production_errors.sqlite3" "production_pulse.sqlite3" ) mkdir -p "$BACKUP_DIR" for db in "${DATABASES[@]}"; do src="$STORAGE_DIR/$db" dest="$BACKUP_DIR/$db" if [ ! -f "$src" ]; then echo "$(date -Iseconds) SKIP $db (not found)" continue fi echo "$(date -Iseconds) Backing up $db..." sqlite3 "$src" "VACUUM INTO '$dest';" echo "$(date -Iseconds) OK $db ($(du -h "$dest" | cut -f1))" done echo "$(date -Iseconds) Uploading to b2:$B2_BUCKET/databases/..." rclone sync "$BACKUP_DIR/" "b2:$B2_BUCKET/databases/" --quiet rm -rf "$BACKUP_DIR" echo "$(date -Iseconds) Done." ``` The storage script is even simpler: one `rclone sync` of the Docker volume, excluding SQLite files (those are handled by the other script): ```bash #!/bin/bash set -euo pipefail STORAGE_DIR="/var/lib/docker/volumes/ce_storage/_data" echo "$(date -Iseconds) Syncing Active Storage files..." rclone sync "$STORAGE_DIR/" "b2:ce-espirita-backups/files/" \ --exclude "*.sqlite3*" \ --quiet echo "$(date -Iseconds) Done." ``` Two cron entries on the host: ``` 0 3 * * * /usr/local/bin/backup-databases >> /var/log/backup-databases.log 2>&1 30 3 * * * /usr/local/bin/backup-storage >> /var/log/backup-storage.log 2>&1 ``` That's it. The scripts run once a day on the host, outside Docker entirely, and upload to the same B2 bucket Litestream was using. No Docker sidecar, no Kamal accessory, no WAL watching, no transaction budgets. One thing worth noting: the scripts run on the host, not inside Docker. They access the SQLite files via the Docker volume's filesystem path (`/var/lib/docker/volumes/ce_storage/_data/`). This means they don't depend on the app container being up, they don't need to mount shared volumes, and they work even if the app is in the middle of a deploy. It also means `sqlite3` and `rclone` need to be installed on the host, not in the Docker image. I deliberately skip the cache, queue, and cable databases. They're ephemeral by design. If I lose Solid Cache entries, the cache warms up on its own. If I lose Solid Queue's completed job history, nothing depends on it. The cable database is transient WebSocket state. Only the primary database (user data), Solid Errors (production error history), and Rails Pulse (monitoring data) are worth backing up. ## The `.backup` trap My first version of the script used SQLite's `.backup` command instead of `VACUUM INTO`. It worked on the primary database but failed immediately on the Pulse database: ``` Error: database is locked ``` `.backup` needs an exclusive lock on the source database. If the Rails app has an open connection (which it always does), `.backup` can't acquire the lock and fails. The primary database happened to have no active writes at 3 AM, so it worked. The Pulse database, which Rails writes to on every request for monitoring, was always busy. The SQLite docs mention the locking requirement, but it's easy to miss when you're writing a quick backup script and testing it against a development database with no concurrent connections. `VACUUM INTO` works differently. It reads the database page by page and writes a fresh, compacted copy to the destination path. It doesn't need an exclusive lock on the source. The running application can keep reading and writing while the backup runs. The resulting file is also smaller than the original because `VACUUM INTO` reclaims free pages, similar to running a full `VACUUM` but without modifying the source database. It was added in SQLite 3.27.0 (2019), so any reasonably modern system has it. I hit this bug and fixed it within minutes of deploying the backup scripts. Litestream doesn't have this problem because it reads the WAL file directly, never needing to lock the main database. When you roll your own backups, you have to know about the locking model. The good news is that `VACUUM INTO` is a strictly better option for online backups: it works on busy databases, it compacts the output, and it produces a standalone file that doesn't depend on a WAL for consistency. ## What I lost Real talk: daily backups are worse than continuous replication in one specific way. If the server dies at 2 AM, I lose almost 24 hours of data. With Litestream, I'd lose at most 5 minutes (my tuned sync interval). For my use case, that's an acceptable tradeoff. But I want to be explicit about it. If you're running an e-commerce checkout, a financial ledger, or anything where losing a day of data would be catastrophic, daily cron backups are not the answer. Litestream, or a real database with streaming replication, is what you need. The tradeoff I made is: simpler operations in exchange for a wider recovery window. For a platform where the worst case is re-entering some event schedules, that math works out. Restore is also simpler than with Litestream. With Litestream, restoring means downloading the base snapshot and replaying WAL segments to a specific point in time. You need Litestream installed, you need the config file, and you need to understand the generation/index structure in the bucket. With the cron approach, restoring is one `rclone copy` command: ```bash rclone copy b2:ce-espirita-backups/databases/production.sqlite3 /tmp/restore/ ``` The file you get back is a complete, consistent SQLite database. No WAL replay, no tooling required beyond `rclone` or even just the B2 web console. ## What I gained The biggest win is observability. The cron job either runs and uploads, or it fails and I see it in the log. There's a log file with timestamps, one entry per database, one entry per upload. If something breaks, I know exactly when and which step failed. Litestream's failure mode was the opposite: it kept running, kept retrying, kept consuming resources, and the only way to know backups were broken was to check the B2 dashboard or tail the container logs looking for 403s. The B2 cost management disappeared entirely. `rclone sync` uploads changed files once per day. That's a handful of Class A (upload) transactions and maybe one or two Class C (list) transactions to diff the remote state. Compare that to Litestream's continuous monitoring, which was generating thousands of Class C transactions daily even with the tuned config. I no longer think about B2's daily transaction cap at all. The deployment got simpler too. Litestream ran as a Kamal accessory, a separate Docker container sharing the app's data volume. That meant the Litestream container, its config file, its environment variables, and its volume mount were all coupled to the app's `deploy.yml`. Removing it cleaned up the deploy config, removed two secrets from `.kamal/secrets`, and eliminated a whole category of "did the accessory come up after deploy" debugging. The backup scripts live on the host, installed once via `scp`, and have no relationship with the app's container lifecycle. ## From blog post to Rails PR The Litestream post was the second in what turned into a series about running SQLite in production. The first was about [auto_vacuum](/blog/sqlite-auto-vacuum-rails): SQLite's default is to never shrink database files, and if you're running Solid Cache or Solid Queue, your disk fills up silently. That auto_vacuum post got more traction than I expected. The post hit the front page of a few aggregators and the feedback was consistent: people were surprised this wasn't already a Rails default. Someone suggested I open a PR to make it one. The idea stuck with me. Every Rails developer who deploys SQLite in production will eventually discover that their database files never shrink. They'll google it, find the `auto_vacuum` pragma, add it to `database.yml`, run a one-time `VACUUM`, and move on. That's a well-documented path now. But it shouldn't be a path at all. If the framework knows you're using SQLite, it should set a sensible default. So I cloned the Rails repo and [opened a PR](https://github.com/rails/rails/pull/57076). ## What the PR changes Two things. First, new SQLite databases created by Rails get `auto_vacuum = incremental` set automatically. This has to happen at database creation time because `auto_vacuum` requires a specific internal page format (pointer-map pages) that can only be set on an empty database. By the time `configure_connection` runs and applies your `database.yml` pragmas, the database file already exists and it's too late. My first approach was to add `auto_vacuum` to the `DEFAULT_PRAGMAS` constant that Rails already uses for other SQLite settings like `journal_mode` and `journal_size_limit`. That didn't work. `DEFAULT_PRAGMAS` are applied in `configure_connection`, which runs after the database file is created. By that point, `auto_vacuum` can't be changed without a full `VACUUM`. The pragma has to be set on the raw connection before any tables exist. The implementation hooks into `new_client`, the method that creates the raw `SQLite3::Database` instance. If the database file doesn't exist yet (meaning Rails is about to create it), it sets `auto_vacuum = :incremental` before anything else happens. If you explicitly configure a different `auto_vacuum` value in `database.yml`, your setting wins. ```ruby def new_client(config) database = config[:database].to_s new_database = !database.include?(":memory:") && !File.exist?(database) db = ::SQLite3::Database.new(database, config) if new_database pragmas = config[:pragmas] || {} db.auto_vacuum = pragmas[:auto_vacuum] || pragmas["auto_vacuum"] || :incremental end db end ``` Existing databases are unaffected. If you already have a production SQLite database with `auto_vacuum = none`, this change doesn't touch it. You'd still need to run `VACUUM` once to restructure the file, as described in the [original post](/blog/sqlite-auto-vacuum-rails). Second, a new `db:maintenance:vacuum` rake task. It runs `PRAGMA incremental_vacuum(1000)` (reclaiming about 4 MB of free pages) and `PRAGMA wal_checkpoint(TRUNCATE)` (resetting the WAL file) on every SQLite database in your configuration. For multi-database apps, it generates per-database variants too (`db:maintenance:vacuum:cache`, etc.). The task is designed to be scheduled hourly via Solid Queue: ```yaml # config/recurring.yml maintenance_vacuum: class: MaintenanceVacuumJob every: 1.hour ``` Or run manually after a large data deletion. ## The bugs I found along the way Contributing to Rails means running the full ActiveRecord test suite, not just your own app's tests. That surfaced three issues I never would have found otherwise. The first was simple: an existing test asserted `auto_vacuum = 0` for in-memory databases. That was correct (in-memory databases can't use auto_vacuum), but the assertion was checking the wrong value after my change. Easy fix. The second was more interesting. Setting `auto_vacuum` is a write operation internally, and it crashed on readonly database connections: ``` ActiveRecord::StatementInvalid: SQLite3::ReadOnlyException: attempt to write a readonly database ``` Rails supports readonly SQLite connections via the `readonly: true` option in `database.yml`. My code was trying to set `auto_vacuum` on every new connection, including readonly ones. The fix was checking `@raw_connection.readonly?` before setting any write-only pragmas. I hadn't considered readonly connections at all because I don't use them in my app. The third was about test isolation. `auto_vacuum` is a persistent property of a database file. It gets set when the database is created and stays forever. The Rails test suite reuses database files between test runs, so databases created before my change still had `auto_vacuum = none`. Tests asserting the new default kept seeing the old value. The fix was straightforward: use fresh temp database files for each test that checks the default. But finding it required understanding how `auto_vacuum` persists, which is the same "you have to know how SQLite works internally" problem that led me to write the original blog post. ## The pattern I keep finding the same pattern with SQLite in production on Rails. The default configuration makes reasonable assumptions that fall apart under specific workloads. The fix is usually a pragma change or a small operational script. And the information lives in SQLite documentation that most Rails developers never read, because they came from PostgreSQL where the database handles vacuuming, WAL management, and space reclamation internally. Rails 8 made SQLite a first-class production option, but the ecosystem around it is still catching up. The Solid suite handles the application-level concerns well. The operational concerns, backups, disk management, monitoring, are still on you. The auto_vacuum default change would eliminate one of these for every new SQLite database going forward. The Litestream-to-cron migration eliminated another kind of complexity for my specific situation. Neither is universally correct. Both made my production setup simpler and more predictable. The PR is still open as of this writing. If it lands, future Rails developers using SQLite won't have to discover the disk space trap on their own. If it doesn't, at least the blog post is there. --- *This is the third in an informal series about SQLite in production with Rails. The first covered [the disk space trap](/blog/sqlite-auto-vacuum-rails), the second covered [Litestream and Backblaze B2's free tier](/blog/litestream-backblaze-b2-free-tier).* --- ### I Built Search with LIKE Queries and It's Fine URL: https://hencf.org/blog/search-without-elasticsearch Published: 2026-04-02 Every Rails app eventually needs search. The default instinct is to reach for Elasticsearch, or at least something like pg_search or SQLite FTS5. I built search for [espirita.club](https://espirita.club) using a denormalized table and LIKE queries. It handles ten models, accent-insensitive Portuguese, a Cmd+K modal, and full-page results. No external dependencies, no special database extensions. Here's how it works and why I didn't need anything fancier. ## The decision espirita.club is a multi-tenant platform for spiritist organizations. Each center has its own subdomain with activities, events, posts, pages, documents, and more. Users needed to search across all of it. The content volume per organization is small. A busy center might have a few hundred records total across all models. The entire platform has maybe tens of thousands of searchable records. This is not a scale problem. There's no need for inverted indexes, tokenizers, or a separate search service. (At my day job I work with the [opposite end of that spectrum](/blog/parallel-testing-elasticsearch-rails) — Elasticsearch, parallel test isolation, the works. Not every app needs that.) LIKE queries on a regular table are fast enough when your dataset fits comfortably in memory. SQLite doesn't even break a sweat. So instead of configuring FTS5 with custom tokenizers for Portuguese, I went with the simplest thing: a single denormalized table that every searchable model syncs into. ## The search_entries table The core idea is one table that holds a flattened copy of every searchable record: ```ruby create_table :search_entries do |t| t.references :organization, polymorphic: true, null: false t.references :searchable, polymorphic: true, null: false t.string :title t.text :body t.string :normalized_title t.text :normalized_body t.string :url_path t.string :content_type t.timestamps end add_index :search_entries, [:searchable_type, :searchable_id], unique: true ``` Every searchable record becomes a single row. The polymorphic `searchable` reference points back to the source (an Activity, a Post, whatever). The polymorphic `organization` reference is the tenant. The unique index ensures one entry per source record. The `normalized_title` and `normalized_body` columns came in a second migration, after I hit the accent problem. More on that soon. ## The Searchable concern Each model that needs to appear in search results includes a `Searchable` concern: ```ruby module Searchable extend ActiveSupport::Concern included do has_one :search_entry, as: :searchable, dependent: :destroy after_save :sync_search_entry after_destroy :remove_search_entry end def sync_search_entry if search_visible? upsert_search_entry else remove_search_entry end end private def upsert_search_entry entry = search_entry || build_search_entry entry.assign_attributes( organization: search_organization, title: search_title, body: search_body, url_path: search_url_path, content_type: search_content_type ) entry.save! end def remove_search_entry search_entry&.destroy end def plain_text_from_rich_text(attribute) send(attribute)&.to_plain_text end end ``` The concern defines the protocol. Each model implements five methods: ```ruby class Activity < ApplicationRecord include Searchable def search_title = name def search_body = [plain_text_from_rich_text(:description), category&.name, locations].compact.join("\n") def search_url_path = "/activities/#{slug}" def search_content_type = "activity" def search_organization = center def search_visible? = true end ``` Posts check `published? && organization.present?` in `search_visible?` so drafts don't leak into results. Each model decides what goes into `search_body`. Activities include their category name and locations. Events include their date and venue. The content is whatever makes sense for that model. The `after_save` callback keeps the search entry in sync on every write. No background job, no eventual consistency. The record saves, the search entry updates, done. ## The ActionText problem There's a subtle issue with ActionText. When someone edits an activity's rich-text description, the `Activity` record itself doesn't change. Only the associated `ActionText::RichText` record does. So `Activity`'s `after_save` never fires, and the search entry goes stale. The fix is an initializer that hooks into ActionText saves: ```ruby # config/initializers/action_text_search_sync.rb Rails.application.config.to_prepare do ActionText::RichText.class_eval do after_save :sync_parent_search_entry private def sync_parent_search_entry record.sync_search_entry if record.respond_to?(:sync_search_entry) end end end ``` Without this, every rich-text edit silently leaves the search index stale. It's the kind of bug you don't notice until someone searches for text they just edited and can't find it. ## The query The `SearchEntry` model handles querying: ```ruby class SearchEntry < ApplicationRecord belongs_to :organization, polymorphic: true belongs_to :searchable, polymorphic: true before_save :normalize_text scope :matching, ->(query) { sanitized = "%#{sanitize_sql_like(transliterate(query))}%" where("normalized_title LIKE :q OR normalized_body LIKE :q", q: sanitized) } def self.transliterate(text) ActiveSupport::Inflector.transliterate(text.to_s) end private def normalize_text self.normalized_title = self.class.transliterate(title) self.normalized_body = body.present? ? self.class.transliterate(body) : nil end end ``` The `matching` scope is the entire search engine. Transliterate the query, wrap it in wildcards, run a LIKE against the normalized columns. That's it. No ranking, no relevance scoring, no stemming. Results come back in whatever order the database returns them, which for LIKE queries on SQLite is insertion order. For the dataset size I'm working with, this is fine. If I needed ranking, I'd add a simple priority based on `content_type` (activities before documents, say) or recency. But nobody has asked for it, so I haven't built it. ## The accent wall The first version searched against the raw `title` and `body` columns. It worked great until someone searched for "acao" and got no results, even though there were activities with "ação" in the title. Portuguese is full of diacritics. "ç" for cedilla, tildes on "ã" and "õ", circumflexes on "ê" and "ô", acute accents everywhere. Users type with and without accents depending on their keyboard, their habits, and whether they're on mobile. Search has to handle both. The solution is `ActiveSupport::Inflector.transliterate`, which strips diacritics using ICU transliteration rules. "ação" becomes "acao", "bebê" becomes "bebe", "programação" becomes "programacao". I apply it at two points: 1. **At index time**: the `before_save` callback writes transliterated text into `normalized_title` and `normalized_body`. 2. **At query time**: the `matching` scope transliterates the user's query before running the LIKE. Both sides are normalized, so "ação" in the database matches "acao" in the search box, and vice versa. The raw columns stay intact for display. You search the normalized version but show the original. This took about twenty minutes to implement once I understood the problem. Adding the two normalized columns, writing the callback, updating the scope, and running a rebuild. Compare that to configuring a Portuguese analyzer with stemming rules and stop word lists in Elasticsearch. ## The controller One controller handles both the Cmd+K modal and full-page search: ```ruby module Public class SearchController < PublicController def show @query = params[:q].to_s.strip @results = current_organization.search_entries .matching(@query) .limit(20) if request.xhr? render partial: "public/search/results", locals: { results: @results, query: @query }, layout: false end end end end ``` XHR requests (from the modal) get just the results partial. Regular requests get the full page with layout. The `current_organization` scope means cross-tenant results are architecturally impossible. Every query is already filtered to the current subdomain's data. Ahoy tracking only fires on full-page searches, not on every keystroke in the modal. ## The Cmd+K modal The search modal uses a native HTML `` element and a Stimulus controller: ```javascript import { Controller } from "@hotwired/stimulus" export default class extends Controller { static targets = ["dialog", "input", "results"] static values = { url: String } connect() { this.handleKeydown = this.handleKeydown.bind(this) document.addEventListener("keydown", this.handleKeydown) } disconnect() { document.removeEventListener("keydown", this.handleKeydown) } handleKeydown(event) { if ((event.metaKey || event.ctrlKey) && event.key === "k") { event.preventDefault() this.open() } } open() { this.dialogTarget.showModal() this.inputTarget.focus() this.inputTarget.select() } async search() { const query = this.inputTarget.value.trim() if (query.length < 2) { this.resultsTarget.innerHTML = "" return } clearTimeout(this.timeout) this.timeout = setTimeout(() => this.fetchResults(query), 250) } async fetchResults(query) { const response = await fetch( `${this.urlValue}?q=${encodeURIComponent(query)}`, { headers: { "X-Requested-With": "XMLHttpRequest" } } ) this.resultsTarget.innerHTML = await response.text() } } ``` Cmd+K opens the dialog. Typing debounces at 250ms, then fetches results via XHR. The response HTML drops straight into the results div. No JSON parsing, no client-side rendering. The server returns the same partial it uses for full-page results. The dialog lives in the layout so it's available on every page: ```erb <%= render "layouts/search_modal" %> ``` ## Highlighting on destination pages When a user clicks a search result, the link includes `?q=` in the URL. A separate Stimulus controller on the destination page picks that up: ```javascript connect() { const query = new URLSearchParams(window.location.search).get("q") if (!query) return this.highlightMatches(query) history.replaceState(null, "", window.location.pathname) } ``` It walks the DOM tree, finds text nodes matching the query, wraps them in `` elements, and scrolls the first match into view. Then it cleans the `?q` parameter from the URL via `replaceState` so the browser history stays clean. This is entirely client-side. The server doesn't know about the highlighting. It just renders the page normally. ## The rebuild task For recovery or after schema changes, a rake task rebuilds the entire index: ```ruby namespace :search do task rebuild: :environment do SearchEntry.delete_all searchable_classes = [Center, Federation, Activity, Event, Page, Document, MembershipPlan, AssociationProgram] searchable_classes.each do |klass| count = 0 klass.find_each do |record| record.sync_search_entry count += 1 end puts "Indexed #{count} #{klass.name.pluralize}" end puts "Total search entries: #{SearchEntry.count}" end end ``` `find_each` processes records in batches. `sync_search_entry` respects `search_visible?`, so draft posts and unpublished content get skipped automatically. I've run this exactly twice: once after adding the normalized columns, and once after expanding what activities include in their search body. ## Snippets Search results show a snippet of the matching text. The `SearchEntry` model handles extraction: ```ruby def snippet_for(query) return nil if body.blank? normalized_query = self.class.transliterate(query.to_s.downcase) normalized = self.class.transliterate(body.downcase) index = normalized.index(normalized_query) return body.truncate(160) unless index start = [index - 80, 0].max stop = [index + normalized_query.length + 80, body.length].min snippet = body[start...stop] snippet = "...#{snippet}" if start > 0 snippet = "#{snippet}..." if stop < body.length snippet end ``` The search for the match position runs against the normalized text, but the snippet itself comes from the raw text. So if the user searches "acao", the snippet shows "...programação espiritual e ação social..." with the original accents intact. ## When you should reach for something else This approach works because of a few specific conditions: **Small dataset per query scope.** Each organization has hundreds of records, not millions. LIKE queries with leading wildcards can't use indexes, so they scan the entire scope. At thousands of rows, that's microseconds. At millions, it's a problem. **No ranking requirements.** Results come back unranked. If users expect Google-style relevance ordering, you need TF-IDF or BM25, which means FTS5 or a dedicated search engine. **No fuzzy matching.** LIKE is exact substring matching (after transliteration). "activty" won't match "activity". If you need typo tolerance, you need trigram indexes or a search service with fuzzy support. **Simple tokenization needs.** I'm matching substrings, not words. Searching "prog" matches "programação". For some apps that's a feature. For others, you'd want word-boundary matching, which LIKE doesn't do well. If any of these conditions change, the upgrade path is clear. The `SearchEntry` table stays. The `Searchable` concern stays. The sync callbacks stay. You swap the `matching` scope from LIKE to FTS5 or plug in a search service that reads from the same table. The denormalized architecture is the hard part, and it's already done. For now, LIKE works. It's been in production for weeks, search is fast, users find what they need, and there's exactly zero infrastructure to maintain. Sometimes the boring solution is the right one. --- ### YouTube Blocked My Server in 15 Seconds URL: https://hencf.org/blog/youtube-blocks-your-server Published: 2026-03-31 In the [previous post](/blog/youtube-transcripts-ruby), I built a Ruby client that fetches YouTube transcripts via the InnerTube player API. Clean implementation, no external dependencies, works perfectly. The next step was obvious: run it across 6,000 videos on the server and populate the knowledge base. That step lasted about 15 seconds. ## The bulk run [Guia](https://guia.espirita.club) is a spiritist content platform with a RAG-powered chat. The chat searches through video transcripts, so every public video needs its captions pulled, chunked, and embedded. I had around 6,000 videos queued for transcript fetching, each one its own background job via Solid Queue. I deployed, the jobs started processing, and the queue drained fast. Within 15 seconds the count dropped from 5,853 to 4,371. Most of those were returning "no captions available" immediately, which is expected for livestreams and older content. But the ones that should have returned transcripts were also failing. Every single one. I SSH'd into the server and ran a quick curl against a video I knew had captions: ```bash curl -sI "https://www.youtube.com/watch?v=some_video_id" ``` 302 redirect to `google.com/sorry/index`. YouTube's CAPTCHA page. The server's IP was banned. ## Trying the back door The `TranscriptClient` fetches transcripts in three steps: scrape the watch page for an API key, call InnerTube for a caption URL, download the XML. The ban hit at step one because the watch page itself was returning the CAPTCHA redirect. I figured the fix might be to skip the watch page entirely. The [Ruby Events](https://github.com/rubyevents) project uses a different approach: POST directly to YouTube's `get_transcript` endpoint with protobuf-encoded parameters. No watch page scrape, no API key extraction. I matched their exact request format, including the protobuf encoding and client context: ```ruby uri = URI("https://www.youtube.com/youtubei/v1/get_transcript") req = Net::HTTP::Post.new(uri.request_uri) req["Content-Type"] = "application/json" req.body = { context: { client: { clientName: "WEB", clientVersion: "2.20250101" } }, params: Base64.strict_encode64(protobuf_payload) }.to_json ``` 400 error. `FAILED_PRECONDITION`. I tried adding cookies from a fresh youtube.com visit. Added more headers. Swapped client versions. Every combination returned the same thing. The issue wasn't the endpoint or the request format. YouTube blocks server IPs from all transcript methods. You can't work around it by changing which internal API you call. This is well-documented in open source issue trackers once you know what to search for. The Python `youtube-transcript-api` has [open issues](https://github.com/jdepoix/youtube-transcript-api/issues/303) about cloud IPs getting blocked. ReVanced has similar reports. The consensus is that YouTube fingerprints requests by source IP range and rejects anything that looks like a datacenter. ## The data damage While I was debugging the IP ban, I noticed something worse. The `FetchTranscriptJob` had a simple rescue clause: ```ruby rescue YouTube::TranscriptClient::TranscriptNotAvailable video.update!(status: :no_transcript) ``` When the watch page returned a CAPTCHA redirect instead of HTML, the client couldn't extract the API key and raised `TranscriptNotAvailable`. Technically correct, from the exception's perspective, but semantically wrong. The video wasn't missing captions. The server was blocked. And the job had marked 7,500+ videos as permanently having no transcript, a status that the pipeline treats as final and never retries. The fix was adding a separate exception class: ```ruby class RateLimited < StandardError; end def fetch_api_key(video_id) # ... response = http.request(req) if response.is_a?(Net::HTTPRedirection) && response["location"]&.include?("google.com/sorry") raise RateLimited, "YouTube rate-limited this IP" end # ... end ``` Then resetting all 7,625 wrongly-marked videos back to their previous state. ## Inverting the architecture The server's IP is burned. Proxies are an option, but rotating residential proxies for 6,000 videos felt like building infrastructure to work around a problem that has a simpler solution: my laptop isn't blocked. YouTube doesn't ban residential IPs from normal-volume transcript fetching. I'd been fetching transcripts locally during development without any issues. So instead of figuring out how to make the server fetch from YouTube, I made the server stop trying. The server would become an API that accepts transcripts, and my local machine would do the fetching. The architecture is straightforward: ``` ┌──────────────┐ GET /api/transcripts/next ┌──────────────┐ │ │ ◄─────────────────────────────────── │ │ │ Server │ │ Local Mac │ │ (Kamal) │ POST /api/transcripts │ (launchd) │ │ │ ◄─────────────────────────────────── │ │ └──────────────┘ └──────────────┘ │ ▼ ┌──────────────┐ │ YouTube │ └──────────────┘ ``` The server exposes two endpoints. `GET /api/transcripts/next` returns the next video that needs a transcript. `POST /api/transcripts` accepts the result and returns the next pending video in the same response. That second detail matters: combining "submit result" and "get next work item" into a single request cuts the round trips in half. ## The server side The API uses bearer token auth with a token stored in Rails credentials: ```ruby module Api class BaseController < ActionController::API before_action :authenticate private def authenticate token = request.headers["Authorization"]&.delete_prefix("Bearer ") expected = Rails.application.credentials.transcript_api_token unless token.present? && ActiveSupport::SecurityUtils.secure_compare(token, expected) head :unauthorized end end end end ``` The transcripts controller handles both directions of the pipeline: ```ruby module Api class TranscriptsController < BaseController def next_pending video = Video.where(status: :archive_approved) .where(raw_transcript: nil) .order(recorded_on: :desc) .first if video render json: { id: video.id, video_id: video.video_id, title: video.title } else head :no_content end end def create video = Video.find(params[:id]) if params[:no_transcript] video.update!(status: :no_transcript) else video.update!( raw_transcript: params[:segments], plain_transcript: params[:segments].map { |s| s[:text] }.join(" ") ) end # Return next pending video in the same response next_video = Video.where(status: :archive_approved) .where(raw_transcript: nil) .where.not(id: video.id) .order(recorded_on: :desc) .first if next_video render json: { id: next_video.id, video_id: next_video.video_id, title: next_video.title } else head :no_content end end end end ``` The server-side background jobs that used to fetch transcripts are neutered with an environment variable guard: ```ruby class FetchTranscriptJob < ApplicationJob def perform(video) return unless ENV["FETCH_TRANSCRIPTS_ENABLED"] == "true" # ... end end ``` That variable isn't set in `config/deploy.yml`, so the job is a no-op on the server. The job class still exists because other parts of the codebase reference it, but it never does anything in production. ## The local side A rake task runs the fetch loop: ```ruby # lib/tasks/transcripts.rake namespace :transcripts do task fetch: :environment do lockfile = Rails.root.join("tmp/transcript_fetch.lock") lock_fh = File.open(lockfile, File::RDWR | File::CREAT) unless lock_fh.flock(File::LOCK_EX | File::LOCK_NB) puts "Another instance is running. Exiting." exit 0 end lock_fh.truncate(0) lock_fh.write(Process.pid.to_s) lock_fh.flush api_url = ENV.fetch("TRANSCRIPT_API_URL", "https://guia.espirita.club") token = Rails.application.credentials.transcript_api_token client = YouTube::TranscriptClient.new fetched = 0 no_transcript = 0 # Get first video video = api_get("#{api_url}/api/transcripts/next", token) while video begin segments = client.fetch(video["video_id"]) response = api_post("#{api_url}/api/transcripts", token, { id: video["id"], segments: segments }) fetched += 1 rescue YouTube::TranscriptClient::TranscriptNotAvailable response = api_post("#{api_url}/api/transcripts", token, { id: video["id"], no_transcript: true }) no_transcript += 1 rescue YouTube::TranscriptClient::RateLimited puts "Rate limited. Stopping." break end video = response # POST returns next video end puts "Done. Fetched: #{fetched}, No transcript: #{no_transcript}" end end ``` The `flock` at the top is important. An earlier version used a PID file, which works fine until you `kill -9` the process and the stale PID file blocks all future runs. `flock` is a kernel-level lock that the OS releases when the process dies, regardless of how it dies. The file stays on disk but the lock is gone, so the next run acquires it cleanly. The rate limiting strategy is deliberately simple: if YouTube returns a CAPTCHA redirect, stop. No delays between requests, no exponential backoff, no retries. The next hourly cron run picks up where this one left off. YouTube's ban seems to reset within an hour for residential IPs, so this approach naturally stays under whatever threshold triggers the block. ## Scheduling with launchd On macOS, `launchd` is the right way to schedule recurring tasks. A plist in `~/Library/LaunchAgents/` handles the hourly runs: ```xml Label com.guia.transcript-fetch WorkingDirectory /Users/henrique/code/guia ProgramArguments bin/rails transcripts:fetch StartInterval 3600 StandardOutPath /Users/henrique/code/guia/log/transcripts/launchd.log StandardErrorPath /Users/henrique/code/guia/log/transcripts/launchd.log EnvironmentVariables PATH /Users/henrique/.local/share/mise/shims:/opt/homebrew/bin:/usr/bin:/bin ``` Install and start it: ```bash cp config/launchd/com.guia.transcript-fetch.plist ~/Library/LaunchAgents/ launchctl load ~/Library/LaunchAgents/com.guia.transcript-fetch.plist ``` The PATH in the plist is critical. launchd runs with a minimal environment that doesn't include mise shims or Homebrew's bin directory. Without the explicit PATH, `bin/rails` can't find Ruby. The same pattern handles embeddings. A second plist runs `bin/rails embeddings:generate` hourly, fetching transcript text from the server, running it through Ollama's bge-m3 locally, and posting the resulting vectors back. Local embedding generation runs about 10x faster than on the CPU-only production server, which was a nice side effect of the architectural inversion. ## The pattern The interesting thing about this solution isn't the specific implementation. It's the inversion. The conventional architecture for data pipelines is: server runs jobs, server fetches external data, server processes it. When the external service blocks server IPs, the instinct is to fix the server: add proxies, rotate IPs, add delays. The alternative is to ask who actually has access. My laptop fetches YouTube transcripts without issues. It's not a server. It's not in a datacenter IP range. YouTube doesn't care about it. So instead of making the server pretend to not be a server, I made it stop trying to be the fetcher entirely. The server became the API, and the machine with access became the worker. This applies to any service that rate-limits or blocks datacenter IPs. Social media scrapers, search engine data, any third-party service that distinguishes between "real users" and "servers" by IP reputation. Instead of building increasingly complex server-side workarounds, consider whether you already have a machine that can do the fetching. A local dev machine, an office server on a residential connection, a Raspberry Pi on a home network. Make your production server the receiver, not the fetcher. The first batch run from my laptop processed 10 transcripts in 32 seconds. The hourly cron chips away at the backlog steadily, processing over a thousand videos per run. No proxies, no IP rotation, no clever request timing. Just the right machine doing the fetching. --- ### Litestream and Backblaze B2's Free Tier URL: https://hencf.org/blog/litestream-backblaze-b2-free-tier Published: 2026-03-27 If you're running SQLite in production with Rails 8, there's a good chance you've seen the same recommendation I did: use [Litestream](https://litestream.io/) for continuous WAL replication to [Backblaze B2](https://www.backblaze.com/cloud-storage). Litestream watches your SQLite database, streams write-ahead log segments to object storage, and gives you point-in-time recovery. Backblaze B2 offers 10 GB of free storage and unlimited Class A (upload) transactions. It's the default answer to "how do I back up SQLite in production," and for good reason. I set it up for [Espirita](https://espirita.club), a multi-tenant Rails 8 app running on a single server with Kamal. Followed the Litestream docs, pointed it at a B2 bucket, deployed. Backups started flowing. Everything looked fine. Two days later, they silently stopped. ## The free tier has a cap nobody talks about Backblaze B2's free tier is generous on storage and uploads, but it caps **Class C transactions** at 2,500 per day. Class C means `ListObjectsV2` calls, the S3 API operation that lists objects in a bucket. Litestream uses these to check on its own replicated data: which segments exist, which need compaction, which are old enough to delete. 2,500 sounds like plenty. It isn't. Litestream's default `sync-interval` is 1 second. Every sync cycle, Litestream uploads new WAL segments (Class A, no cap) but also lists existing segments to decide what to compact or clean up. With a 1-second interval, that's potentially 86,400 ListObjectsV2 calls per day from sync alone. My first config used a 1-second interval because that's what the examples show. I was generating roughly 170,000 Class C transactions per day against a 2,500 cap. The obvious fix: increase the sync interval. ```yaml dbs: - path: /data/production.sqlite3 replicas: - type: s3 bucket: my-bucket sync-interval: 5m ``` Five-minute sync reduces the upload cycles to 288 per day. For my app, a 5-minute recovery point objective is fine. If I lose the server, I lose at most 5 minutes of data. Crisis averted. Except it wasn't. ## The second wave: compaction monitors A few days later, I checked the B2 dashboard and saw Class C transactions still exceeding the cap. Not by as much, but still over. The sync interval was 5 minutes. Where were the extra calls coming from? Litestream has a compaction system that merges small WAL segments into larger ones. It runs at three configurable levels, and each level has a monitoring loop that lists objects to decide when compaction is needed. The default intervals are 30 seconds, 5 minutes, and 1 hour. That innermost 30-second loop was generating around 2,880 ListObjectsV2 calls per day all by itself, independent of the sync interval. ```yaml # Widen the compaction intervals levels: - interval: 5m # default: 30s - interval: 1h # default: 5m - interval: 24h # default: 1h ``` I also disabled validation, which periodically downloads and checksums replicated data (generating even more list and get operations): ```yaml dbs: - path: /data/production.sqlite3 replicas: - type: s3 bucket: my-bucket sync-interval: 5m validation-interval: 0s # disable periodic validation ``` Better. But still not enough. ## The third wave: L0 retention checks The transaction count dropped but kept creeping above 2,500. I dug into Litestream's source code and found another source of ListObjectsV2 calls that isn't obvious from the documentation: L0 retention checks. Litestream tracks recently compacted files at "level 0" and periodically checks whether they're old enough to delete. The default check interval is 15 seconds. That's 5,760 ListObjectsV2 calls per day from a single timer, more than double the entire free tier budget. This one was the real culprit. Even after fixing the sync interval and compaction monitors, L0 retention checks alone would have blown past the cap. ```yaml l0-retention: 1h l0-retention-check-interval: 1h # default: 15s ``` ## The death spiral All of this would be manageable if exceeding the cap just meant degraded backups. It doesn't. When you hit B2's Class C cap, B2 returns HTTP 403 for every subsequent ListObjectsV2 call. Litestream interprets 403 as a transient error and retries. Without backoff. Every failed check triggers an immediate retry, which also fails with 403, which triggers another retry. The monitoring loops that were generating thousands of calls per day now generate thousands of calls per *minute*, all of them failing. Your backups aren't just paused. They're stuck in a retry storm that won't clear until the daily transaction counter resets at midnight UTC. Meanwhile, no actual replication is happening because the upload operations depend on the list operations to figure out what needs uploading. I caught this by SSH-ing into the server and tailing the Litestream logs. Wall-to-wall 403 errors, hundreds per second. No alerting, no graceful degradation. The backup process was running, consuming CPU and network, and accomplishing nothing. ## The config that actually works After three rounds of fixes across about a week, here's where I landed: ```yaml # Disable Litestream's built-in retention enforcement. # Use B2 lifecycle rules to clean up old files instead. retention: enabled: false # L0 retention: keep compacted files for 1h, check once per hour. l0-retention: 1h l0-retention-check-interval: 1h # Widen compaction intervals to reduce list operations. levels: - interval: 5m - interval: 1h - interval: 24h dbs: - path: /data/production.sqlite3 replicas: - type: s3 endpoint: s3.us-east-005.backblazeb2.com bucket: my-backup-bucket path: litestream/production force-path-style: true sync-interval: 5m validation-interval: 0s ``` The core idea: turn off everything that generates ListObjectsV2 calls on a tight loop. Sync every 5 minutes. Compact on wider intervals. Don't validate. Don't enforce retention from Litestream's side. That last part, disabling retention, might make you nervous. Without retention enforcement, old WAL segments accumulate in the bucket forever. But B2 has its own lifecycle rules that handle this better. In the B2 dashboard, you can set a lifecycle rule on the bucket to auto-delete files older than N days. This runs on B2's side, costs zero API transactions, and achieves the same outcome without Litestream polling. ## Deploying with Kamal The full Kamal setup runs Litestream as an accessory container sharing the app's data volume: ```yaml # config/deploy.yml accessories: litestream: image: litestream/litestream:latest host: 147.93.13.116 volumes: - "ce_storage:/data" files: - config/litestream.yml:/etc/litestream.yml env: secret: - LITESTREAM_ACCESS_KEY_ID - LITESTREAM_SECRET_ACCESS_KEY clear: LITESTREAM_B2_REGION: us-east-005 LITESTREAM_B2_BUCKET: ce-espirita-backups cmd: replicate -config /etc/litestream.yml ``` The shared volume (`ce_storage:/data`) is important. The Rails app writes to `/data/production.sqlite3` and Litestream reads from the same path. Both containers mount the same Docker volume, so Litestream sees WAL changes as they happen. B2 credentials live in Rails encrypted credentials and get extracted in `.kamal/secrets`: ```bash LITESTREAM_ACCESS_KEY_ID=$(bin/rails credentials:fetch litestream.access_key_id -e production) LITESTREAM_SECRET_ACCESS_KEY=$(bin/rails credentials:fetch litestream.secret_access_key -e production) ``` This avoids `.env` files or hardcoded secrets in the deploy config. The credentials only exist in the encrypted credentials file and in the running container's environment. ## How to check if you're affected If you're running Litestream with Backblaze B2, log into the B2 dashboard and check your daily transaction counts under **Caps & Alerts**. B2 breaks down transactions by class. If your Class C count is anywhere near 2,500, you're close to the cap. You can also tail Litestream's logs and look for 403 errors: ```bash # With Kamal kamal accessory logs litestream --since 1h | grep 403 # With Docker directly docker logs litestream 2>&1 | grep 403 ``` If you see any 403s, you've already hit the cap and your backups are in the retry spiral. Apply the config changes and restart Litestream: ```bash kamal accessory reboot litestream ``` ## What I'd tell someone setting this up today Start with the tuned config, not the defaults. The Litestream documentation is written for general S3-compatible storage, and the defaults assume you're on AWS S3 or another provider without tight transaction caps. B2's free tier is a different environment with different constraints, and the defaults will silently exceed it within hours. Set up B2 lifecycle rules instead of relying on Litestream's retention. It's one less moving part, zero API overhead, and B2 handles it more reliably than a client-side process polling from your server. And monitor. The worst part of this failure mode is the silence. Litestream doesn't expose a health endpoint or a Prometheus metric for replication status. If your backups stop, you find out when you need them or when you happen to check the logs. I added a simple cron job that checks the most recent object timestamp in the B2 bucket and alerts if it's older than 15 minutes. That's not a Litestream feature. It's a `curl` against the B2 API. But it's the only thing standing between "my backups work" and "my backups stopped a week ago and I didn't notice." The SQLite-on-one-server stack is genuinely simpler than running PostgreSQL with managed backups. But "simpler" doesn't mean "nothing can go wrong." It means the failure modes are different, and some of them are quieter than what you're used to. --- *This is a companion to [SQLite in Production: The Disk Space Trap](/blog/sqlite-auto-vacuum-rails), where I covered another silent SQLite issue in Rails: databases that never shrink.* --- ### SQLite in Production: The Disk Space Trap URL: https://hencf.org/blog/sqlite-auto-vacuum-rails Published: 2026-03-26 I run a multi-tenant Rails 8 app on a single server with SQLite. Solid Cache for caching, Solid Queue for background jobs, and a separate database for Action Cable via Solid Cable. The primary database also has an `ahoy_events` table that tracks page views and cleans up after 90 days. All of these churn data. Cache entries expire, jobs complete and get swept, old analytics rows get deleted. Normal stuff. A few weeks ago I was checking disk usage and noticed the Pulse monitoring database had grown to 6.9 GB. I pulled up the actual data size: ```sql SELECT page_count * page_size AS total_bytes, (page_count - freelist_count) * page_size AS used_bytes, freelist_count * page_size AS free_bytes FROM pragma_page_count, pragma_page_size, pragma_freelist_count; ``` 37 MB of actual data. 6.85 GB of empty space SQLite was holding onto. ## Why SQLite doesn't shrink When you delete rows from a SQLite database, the pages those rows occupied get added to an internal freelist. They're marked as available for reuse, but they're never returned to the operating system. The file size stays the same. This is by design. SQLite's default `auto_vacuum` mode is `NONE` (0). The database file is a single file on disk, and shrinking it means rewriting everything after the freed pages. That's expensive, so SQLite doesn't do it unless you ask. If your application mostly grows (inserts outnumber deletes), this doesn't matter much. The free pages get reused by new inserts. But if your application churns data, deleting old rows on a schedule and inserting new ones, you end up with a file that reflects your peak historical data size, not your current data size. Solid Cache is the worst offender. It writes and expires entries constantly. A cache database that holds 500 MB of active entries might have a 5 GB file because it once held 5 GB worth of entries before they expired. ## The three auto_vacuum modes SQLite has three `auto_vacuum` modes, controlled by `PRAGMA auto_vacuum`: **NONE (0)** is the default. Freed pages go to the freelist. The file never shrinks. You can reclaim space manually by running `VACUUM`, which rebuilds the entire database. On a 6.9 GB file, that means writing 6.9 GB of data to a temporary file, then replacing the original. It works, but it locks the database for the duration and requires enough free disk space for the copy. **FULL (1)** reclaims space after every transaction that frees pages. No freelist accumulation, the file stays compact. The cost is extra write I/O on every delete and update, because SQLite has to move pages around to keep the file contiguous. For write-heavy workloads (like, say, a cache or job queue), this adds measurable overhead to every operation. **INCREMENTAL (2)** is the middle ground. Freed pages get tracked (not on the freelist, but in a pointer map), and the file shrinks only when you explicitly run `PRAGMA incremental_vacuum(N)`, which reclaims up to N pages. You control when and how much space gets reclaimed. For Rails applications, INCREMENTAL is the right choice. It avoids the constant overhead of FULL mode, and it gives you a hook to reclaim space on your own schedule, in a background job, during low traffic, whatever makes sense. ## Setting it up in Rails There's a catch. `auto_vacuum` must be set before the first table is created in a database, or you need to run `VACUUM` to restructure the file. This is because FULL and INCREMENTAL modes use a different internal page format (pointer-map pages) that NONE doesn't have. In Rails, you set pragmas in `database.yml`: ```yaml production: primary: <<: *default database: storage/production.sqlite3 pragmas: auto_vacuum: incremental cache: <<: *default database: storage/production_cache.sqlite3 migrations_paths: db/cache_migrate pragmas: auto_vacuum: incremental queue: <<: *default database: storage/production_queue.sqlite3 migrations_paths: db/queue_migrate pragmas: auto_vacuum: incremental cable: <<: *default database: storage/production_cable.sqlite3 migrations_paths: db/cable_migrate pragmas: auto_vacuum: incremental ``` For new databases, that's all you need. Rails will set the pragma before creating any tables, and INCREMENTAL mode is active from the start. For existing databases, you need to run `VACUUM` once to restructure the file. I did this in a migration: ```ruby class EnableIncrementalAutoVacuumOnAllDatabases < ActiveRecord::Migration[8.0] def up # auto_vacuum pragma is already set via database.yml, # but existing databases need a VACUUM to restructure # the file format for incremental mode. execute "VACUUM" end def down # Can't undo a VACUUM, but the pragma change # can be reverted in database.yml end end ``` If your database is large, this migration will take a while and lock the database for the duration. For a 6.9 GB file, it took about 45 seconds on my server. Plan accordingly. If you're running Kamal, this will happen during deploy, so your app will be down for that window. For most SQLite databases in Rails apps, we're talking single-digit seconds. You'll also want a migration for each database that needs it. The cache, queue, and cable databases all get their own migration directories, so create equivalent migrations in each. ## Reclaiming space on a schedule With INCREMENTAL mode active, freed pages accumulate but the file doesn't shrink until you run `PRAGMA incremental_vacuum`. I set up a recurring job to handle this: ```ruby class IncrementalVacuumJob < ApplicationJob def perform # Reclaim up to 1000 pages (~4 MB with default page size) ActiveRecord::Base.connection.execute("PRAGMA incremental_vacuum(1000)") end end ``` Scheduled in `config/recurring.yml` for Solid Queue: ```yaml incremental_vacuum: class: IncrementalVacuumJob every: 1.hour ``` The `1000` argument means "reclaim up to 1000 free pages." With SQLite's default 4 KB page size, that's roughly 4 MB per run. If there are fewer than 1000 free pages, it reclaims whatever is available. If there are none, it's a no-op. You can tune this. If your cache churns heavily, bump it up or run it more frequently. If your primary database barely deletes anything, once a day is fine. The point is that you're in control, and the operation is bounded: it won't lock the database for 45 seconds like a full `VACUUM` would. For the cache and queue databases, you'd run the pragma against those connections specifically: ```ruby class IncrementalVacuumJob < ApplicationJob def perform connections = [ ActiveRecord::Base, SolidCache::Record, SolidQueue::Record ] connections.each do |base| base.connection.execute("PRAGMA incremental_vacuum(1000)") end end end ``` ## How to check your databases right now If you're running SQLite in production with Rails 8, check your current state: ```bash # SSH into your server (Kamal example) kamal console # Check auto_vacuum mode (0 = NONE, 1 = FULL, 2 = INCREMENTAL) ActiveRecord::Base.connection.execute("PRAGMA auto_vacuum").first["auto_vacuum"] # Check how much space is reclaimable result = ActiveRecord::Base.connection.execute(<<~SQL) SELECT page_count * page_size AS total_bytes, freelist_count * page_size AS free_bytes FROM pragma_page_count, pragma_page_size, pragma_freelist_count SQL total = result.first["total_bytes"] free = result.first["free_bytes"] puts "Total: #{total / 1_048_576} MB, Free: #{free / 1_048_576} MB" ``` If `auto_vacuum` returns 0 and `free_bytes` is large relative to `total_bytes`, you've got a database file that's bigger than it needs to be. Check all your databases. The primary database might be fine if you're mostly inserting, but the cache database almost certainly has wasted space. ## Which databases care most **Solid Cache** is the biggest concern. Cache entries are written and expired constantly. Without auto_vacuum, the cache database file will grow to whatever your peak cache size has been and never shrink below that, even if you reduce the cache size or clear it. **Solid Queue** matters if you process a lot of jobs. Completed jobs get swept, but the space stays allocated. If you had a burst of a hundred thousand jobs last month, that space is still reserved on disk. **Solid Cable** is usually small, but if you have active WebSocket traffic, it churns too. **Ahoy or any analytics** that cleans up old data will accumulate dead space proportional to your cleanup cadence. If you keep 90 days of events and delete older ones daily, after a year you've allocated space for a year of events even though you only hold 90 days. **The primary database** is usually fine, since most Rails apps accumulate records rather than deleting them. But if you have any cleanup jobs or soft-delete sweepers, check it too. ## Why this isn't talked about more Most people running SQLite in production with Rails are early adopters. The Solid suite landed in Rails 8, and Rails 8 itself is still relatively new. The combination of "SQLite in production" and "tables that churn data" hasn't been widespread long enough for this to become common knowledge. On top of that, disk space is cheap and the growth is gradual. You don't get an error. Your app doesn't slow down (SQLite reuses free pages efficiently). You just quietly accumulate a database file that's ten or fifty times larger than your actual data. It's the kind of thing you only notice when you're checking disk usage for an unrelated reason, or when your 20 GB VPS runs out of space. The fix is straightforward once you know about it. Set `auto_vacuum: incremental` in `database.yml`, run `VACUUM` once on existing databases, and schedule `PRAGMA incremental_vacuum` to run periodically. That's it. Your database files will reflect your actual data size instead of your historical peak. --- *This is the first post in an informal series about SQLite in production with Rails. The second covered [Litestream and Backblaze B2's free tier](/blog/litestream-backblaze-b2-free-tier), and the third covered [why I eventually replaced Litestream with a cron job](/blog/sqlite-backups-the-boring-way) and the Rails PR that came out of this post.* --- ### Parallel Testing with Elasticsearch in Rails URL: https://hencf.org/blog/parallel-testing-elasticsearch-rails Published: 2026-03-24 This is the third post in a series about [migrating a large Rails app from RSpec to Minitest](/blog/rspec-to-minitest-migration). The [second post](/blog/fixtures-for-real-rails-apps) covered fixture design. This one is about the hardest part of the whole migration: making Elasticsearch tests run in parallel. BSPK has a lot of search. Shoppers, items, notes, tags, feed posts, sales associates. Most of these are backed by Elasticsearch via Searchkick, and they all had specs that indexed data and asserted on search results. When the test suite ran serially, this worked fine. Every test had the index to itself. When I turned on parallel testing with twelve workers, everything broke. ## The problem Minitest's parallel testing gives each worker its own database. Fixtures load into each worker's DB independently, transactions roll back between tests, and there's no cross-contamination. But Elasticsearch isn't a database. It's a shared external service. All twelve workers were hitting the same ES cluster, writing to the same indexes, and reading each other's data. A test in worker 3 would index five shoppers and assert that a search returned exactly five results. Meanwhile, worker 7 had just indexed its own shoppers into the same index. The search returned twelve results. Test fails. The flakiness was maddening because it was timing-dependent. Run the suite once, three failures. Run it again, different failures. Run it a third time, all green. Classic parallel race condition. ## Per-worker index prefixes The fix was straightforward once I understood the problem. Each parallel worker gets its own Elasticsearch index prefix, so their data never overlaps. In `test_helper.rb`: ```ruby ENV["SEARCHKICK_INDEX_PREFIX"] = "test#{ENV.fetch('TEST_ENV_NUMBER', nil)}" parallelize(workers: :number_of_processors) parallelize_setup do |worker| ENV["SEARCHKICK_INDEX_PREFIX"] = "test#{worker}" Searchkick.index_prefix = "test#{worker}" # Clear cached index objects so models pick up the new prefix Searchkick.models.each do |model| model.instance_variable_set(:@searchkick_index, nil) end end ``` Worker 0 writes to `test0_shoppers`, worker 1 writes to `test1_shoppers`, and so on. Same isolation model as the per-worker databases, just applied to Elasticsearch. The cache clearing is important. Searchkick memoizes the index object on each model class. Without clearing it, the model would keep using the prefix from before the fork, and you'd be right back to shared indexes. ## Disabling callbacks globally Searchkick hooks into ActiveRecord callbacks to automatically index records on create, update, and destroy. That's great in production, but in tests it means every fixture load triggers an ES index operation. With thirty fixture files loading into twelve workers simultaneously, that's a lot of unnecessary indexing. I disabled callbacks globally in `test_helper.rb`: ```ruby Searchkick.disable_callbacks ``` Tests that need search behavior opt in explicitly: ```ruby def with_searchkick(&block) Searchkick.callbacks(true, &block) end ``` This way, a model test that checks validations never touches Elasticsearch. Only the tests that actually exercise search pay the indexing cost. ## The safe_reindex pattern Every search test needs to get data into ES before it can assert on results. The naive approach is to call `Model.reindex` and hope for the best. In parallel, "hope for the best" fails about 30% of the time. The pattern I landed on: ```ruby module ElasticsearchTestHelper def safe_reindex(model_class) model_class.instance_variable_set(:@searchkick_index, nil) model_class.reindex(async: false, mode: :inline, refresh: false) model_class.instance_variable_set(:@searchkick_index, nil) end def with_searchkick(&block) Searchkick.callbacks(true, &block) end end ``` Two things to note. First, the `@searchkick_index` cache is cleared both before and after the reindex. Before, so that Searchkick creates a fresh timestamped index with the current worker's prefix. After, so that subsequent calls see the new index name (Searchkick appends a timestamp to each reindex). Second, there's no `index.delete` call. An earlier version had one: ```ruby index = model_class.searchkick_index index.delete if index.exists? model_class.reindex(...) ``` This caused intermittent 404 errors under parallel load. The problem was a race condition: Searchkick's `reindex` already creates a new timestamped index, imports data, swaps the alias, and cleans up old indexes. The explicit delete before reindex was redundant, and under high concurrency, the delete would sometimes hit right as another operation was reading the alias. Removing it fixed the last source of flaky ES failures. I verified this with five consecutive full suite runs: 7,500+ tests each, zero failures. ## The clean-room company The [fixture design post](/blog/fixtures-for-real-rails-apps) mentioned a three-company structure: Vista (primary data), Art Gallery (cross-tenant), and ES Test (clean-room). The clean-room company exists specifically for search tests. ```yaml # Clean-room company for Elasticsearch tests — has NO shoppers, # store_visits, chats, or other records so safe_reindex produces # a known-empty baseline. es_test: name: ES Test Company dns_names: "{es-test.bspk.com}" external_id_str: es_test_company abbreviated_name: ES ``` With matching fixtures for a store, two sales associates, and their accounts. All accessible through helpers: ```ruby def es_company = companies(:es_test) def es_store = stores(:es_test_store) def es_sa1 = sales_associates(:es_test_sa1) def es_sa2 = sales_associates(:es_test_sa2) ``` When a search test starts, it calls `safe_reindex` on the relevant model class. Because the ES Test company has zero child records in fixtures, the initial index is empty. The test then creates exactly the records it needs using inline factory helpers, re-indexes, and asserts on known data. No surprise records from other fixtures. No bleeding from other tests. The test controls the entire search state. ## before_all for expensive setup Some search test classes have heavy setup: creating dozens of records with specific attributes, then reindexing. The shopper finder tests, for example, create shoppers with different names, emails, phone formats, gender values, and contact preferences to exercise every search filter. Running that setup before every test method was adding up. Six test classes were taking three times longer than they needed to because the same twenty records were being created and indexed sixty times. TestProf's `before_all` runs setup once per test class and wraps it in a transaction that persists across all test methods: ```ruby class ShoppersFinderTest < ActiveSupport::TestCase include ElasticsearchTestHelper include InlineFactoryHelpers include BeforeAll before_all(setup_fixtures: true) do @company = es_company @sa = es_sa1 @shopper1 = create_shopper(company: @company, store: es_store, first_name: "Alice", last_name: "Smith", email: "alice@example.com") @shopper2 = create_shopper(company: @company, store: es_store, first_name: "Bob", last_name: "Jones", phone: "+15551234567") # ... 15 more shoppers with specific attributes safe_reindex(ElasticSearch::SearchClient) end setup do @company.reload # reset any mutations from previous test end def test_search_by_name results = SalesAssociate::ShoppersFinder.new(@sa, query: "Alice").results assert_includes results, @shopper1 refute_includes results, @shopper2 end end ``` The `setup_fixtures: true` flag is required in Rails 8 to make fixture data available inside the `before_all` block. The `setup` block calls `.reload` on objects that tests might have mutated (changing a filter, updating an attribute) so each test sees fresh state. The `before_all_helper.rb` also patches Minitest to deactivate the previous class's transaction when switching between test classes in a parallel worker. Without this, the transaction from one `before_all` class could leak into the next class running in the same worker: ```ruby Minitest.singleton_class.prepend(Module.new do def run_one_method(klass, method_name) prev = defined?(@previous_klass) ? @previous_klass : nil if prev && prev != klass && prev.respond_to?(:before_all_executor) prev.before_all_executor&.deactivate! end @previous_klass = klass super end end) ``` This was a fun one to debug. Tests would pass in isolation, pass when running a single file, but fail when running the full suite because an unrelated test class's `before_all` transaction was still open. ## VCR and the body matching problem Some of our search-adjacent code calls LLMs (the natural language search feature translates English queries into Elasticsearch DSL). These HTTP calls are recorded with VCR cassettes. When we went parallel, the cassettes stopped matching. The issue: VCR matches requests by method, URI, and body. The request body includes the system prompt, which includes the full site content (for the AI chat agent). Every time a blog post changed or a new record was added, the body changed, and the cassette didn't match. On top of that, Elasticsearch index names in the request body now included worker-specific prefixes (`test0_shoppers` vs `test1_shoppers`), so the same test recorded on worker 0 wouldn't match when replayed on worker 3. The fix was a custom request matcher that normalizes both problems: ```ruby VCR.configure do |c| c.register_request_matcher :normalized_body do |request_1, request_2| normalize = ->(body) { return body if body.nil? || body.empty? normalized = body.dup # Strip worker-specific ES index prefixes normalized.gsub!(VCR_INDEX_PREFIX_PATTERN, VCR_NORMALIZED_INDEX_NAME) # Strip LLM system prompts that change with content updates normalized.gsub!(/"role"\s*:\s*"(system|developer)".*?(?="role")/, "") normalized } normalize.call(request_1.body) == normalize.call(request_2.body) end end ``` Cassettes are now recorded with normalized bodies, and replayed with the same normalization. The index prefix `test3_shoppers` in a live request matches `test_shoppers` in the cassette. The system prompt with yesterday's blog posts matches the cassette from last week. ## The parallelize(workers: 1) trap The most expensive mistake I made was a subtle one. During the initial migration of finder specs (Phase 7), I added `parallelize(workers: 1)` to every ES-backed test class. My reasoning: these tests are fragile, let's run them serially to avoid issues. What I didn't realize is that Rails' parallelization is all-or-nothing at the suite level. If *any* test class sets `parallelize(workers: 1)`, Rails falls back to running the *entire suite* in a single process. Not just that class. Everything. The suite was running in about 400 seconds. I assumed that was normal for the volume of tests. When I removed all the `parallelize(workers: 1)` overrides and let Rails use all twelve cores, it dropped to 82 seconds. I'd been running the full suite serially for three days without realizing it. The lesson was simple: don't use `parallelize(workers: 1)` on individual classes. Either fix the parallel isolation issue, or if you really need serial execution, use a different mechanism (like TestProf's `before_all` to reduce per-test cost). ## The final numbers After all the ES parallel work landed, the search test files went from being the slowest, flakiest part of the suite to being unremarkable. Fifty-six test files with ES integration, running across twelve parallel workers, consistently green. The standardization commit that applied `safe_reindex` across all fifty-six files actually *removed* about 180 lines of code. The previous patterns (manual delete + reindex, inline callbacks blocks, redundant refresh calls) were all more code *and* less reliable. For the six heaviest test classes, `before_all` cut execution time by roughly 3x. Those classes have a hundred-plus test methods each, and the setup (creating records + reindexing) only runs once. If you're running Elasticsearch tests in a Rails app and they're either slow or flaky, the playbook is: per-worker index prefixes, globally disabled callbacks with opt-in, a clean-room fixture company with zero indexed records, `safe_reindex` without explicit deletes, and `before_all` for the expensive test classes. Every piece solves a specific problem. Skip one and you'll probably find out which one the hard way. --- ### Designing Fixtures for Real Rails Apps URL: https://hencf.org/blog/fixtures-for-real-rails-apps Published: 2026-03-23 In the [first post of this series](/blog/rspec-to-minitest-migration), I described migrating BSPK's entire test suite from RSpec and FactoryBot to Minitest and fixtures. Five days, seventeen phases, and a 5x speedup in CI. But I glossed over the part that took the most thought: designing the fixtures themselves. Fixtures have a bad reputation, and I get it. Most fixture setups I've seen are terrible. Auto-generated YAML files with records named `:one` and `:two`, no coherent relationships between them, and a vague sense that touching any fixture will break something somewhere. Developers reach for FactoryBot because it feels safer to build fresh objects from scratch than to navigate a minefield of shared state. The problem was never fixtures. It was how people designed them. ## Start from real data The first thing I did was *not* write fixtures by hand. I wrote an export script that pulled a curated subset from our development environment. I picked a single company (Vista, one of our dev tenants) and followed its relationships: stores, sales associates, shoppers, items, appointments, lists, tasks. The script sanitized emails to `@example.com`, stripped sensitive fields, and generated YAML using Rails' association label format. This is the part most people skip. They either hand-write fixtures that look nothing like their actual data, or they let Rails scaffold generic ones. Both approaches fail the same way: the test data doesn't represent reality, so the tests don't catch real bugs. Starting from a real dataset gave me fixtures that had the right shape. Real associations, real cardinalities, real edge cases that existed because actual users had created them. Shoppers with multiple addresses. Appointments across different stores. Sales associates with varying permission levels. I didn't have to invent these scenarios; they were already in the data. The export script was about a thousand lines, which sounds like a lot. Most of it was selecting which records to include and handling association labels correctly. It ran once, generated the YAML, and hasn't been touched since. That's the right ratio for infrastructure code: invest upfront, then forget about it. ## Name things like they matter This is the difference between fixtures that help you and fixtures that haunt you: ```yaml # Bad: what is :one? Why does this test use it? one: first_name: MyString last_name: MyString email: MyString company: one # Good: I know exactly who this is russell_winfield: first_name: Russell last_name: Winfield email: russell.winfield@example.com gender: 0 company: vista store: los_angeles ``` Every fixture in our suite has a name that means something. `shoppers(:russell_winfield)` is a male shopper at the LA store. `sales_associates(:jen_wilson)` is the manager. `stores(:saint_honore)` is the Paris location. When I read a test that references these, I know immediately what data it's working with. We went further and built a `FixtureHelpers` module that adds semantic aliases: ```ruby module FixtureHelpers def vista_company = companies(:vista) def la_store = stores(:los_angeles) def manager_sa = sales_associates(:jen_wilson) # Manager of all stores, LA def senior_sa = sales_associates(:yana_bets) # Senior SA, LA def regular_sa = sales_associates(:yauhen_hatsukou) # Regular SA, LA def paris_sa = sales_associates(:jimmy_shan) # SA, Saint-Honoré def male_shopper = shoppers(:russell_winfield) # gender: 0, phone + email def female_shopper = shoppers(:maria_johnson) # gender: 1, phone + whatsapp end ``` The role comments matter. Six months from now, when someone needs a shopper with WhatsApp enabled, they scan the helpers and find `female_shopper` with its comment. No grepping through factory traits. No guessing. ## Design for multi-tenancy BSPK is a multi-tenant app. Every query is scoped to a company. This means test data needs to reflect that boundary, or you'll write tests that accidentally pass because they're pulling records from the wrong tenant. The fixture dataset has three companies: **Vista** is the primary tenant. Most test data lives here: five stores across two regions, a dozen sales associates, fifty-plus shoppers, items, appointments, lists, tasks. When a test needs "a normal scenario," it uses Vista data. **Art Gallery Demo** is the cross-tenant company. It has its own store, its own sales associates, its own shoppers. Any test that verifies tenant isolation creates data in Vista and asserts it doesn't leak into Art Gallery (or vice versa). Having a second tenant in fixtures makes these tests trivial to write. **ES Test** is the clean-room company. Zero child records. It exists specifically for Elasticsearch tests that need a known-empty baseline before indexing test-specific data. More on this in the next post about parallel testing. This three-company structure wasn't in the original export. I added it after the first few phases of migration, when I realized that single-tenant fixtures would leave a whole class of bugs uncovered. ## Edge cases go at the bottom The exported fixtures represent the happy path: real data, real relationships, everything working as expected. But tests also need edge cases. Deleted records, missing contact info, unusual gender values, opt-out flags. I added these as synthetic fixtures at the bottom of the relevant files: ```yaml # --- Synthetic edge-case fixtures (not from export) --- deleted_shopper: first_name: Deleted last_name: Shopper email: deleted@example.com company: vista store: los_angeles is_deleted: true no_contact_shopper: first_name: Ghost last_name: Person company: vista store: los_angeles is_do_not_contact: true sms_contact: false email_contact: false whatsapp_contact: false ``` Separating exported data from synthetic edge cases keeps the fixture file organized. The top section is "the world as it normally looks." The bottom section is "the weird stuff we need to test." A comment separates them. ## Let Rails resolve the foreign keys One of the reasons old fixture setups were fragile was hardcoded IDs. Change an ID in one file, and a dozen other files break. Rails solved this years ago with association labels, but I still see codebases that don't use them. Every fixture in our suite references associations by label, not by ID: ```yaml jen_wilson: first_name: Jennifer last_name: Wilson company: vista store: los_angeles role: manager ``` `company: vista` resolves to whatever ID Rails assigns to the `vista` fixture in `companies.yml`. No hardcoded integers. No cross-file dependencies on specific IDs. If I rename a fixture, I rename the label everywhere and it all still works. The one exception is polymorphic associations. Rails can't resolve labels for polymorphic foreign keys because it doesn't know which table to look in. For those, we use ERB: ```yaml jen_chat_participant: chat: jen_intro_chat participant_type: SalesAssociate participant_id: <%= ActiveRecord::FixtureSet.identify(:jen_wilson) %> ``` `FixtureSet.identify` is deterministic: given a label, it always returns the same integer. So while this is technically a hardcoded ID, it's derived from the label and stays in sync automatically. ## Know when not to use fixtures This is the part that most "fixtures vs factories" debates miss. It's not one or the other. We use both, and the boundary is clear. **Fixtures** are for the shared world. The baseline data that most tests read from but don't modify. Companies, stores, users, products, reference data. Loaded once per parallel worker, transactionally rolled back between tests. Fast, stable, predictable. **Inline factory helpers** are for test-specific scenarios. Data that a test creates, mutates, or destroys. Edge cases that would bloat the fixture files. Records where you need precise control over every attribute. We built an `InlineFactoryHelpers` module with about fifty `create_*` methods. It's FactoryBot without FactoryBot: plain Ruby methods that create records with sensible defaults and keyword arguments for overrides. A shared sequence starting at 10,000 avoids ID collisions with fixture data. ```ruby def create_shopper(company: vista_company, store: la_store, **attrs) seq = next_sequence Shopper.create!( first_name: attrs[:first_name] || "Shopper", last_name: attrs[:last_name] || "#{seq}", email: attrs[:email] || "shopper-#{seq}@example.com", company: company, store: store, **attrs.except(:first_name, :last_name, :email) ) end ``` Notice that the defaults reference fixture helpers (`vista_company`, `la_store`). Inline-created records live in the same world as fixtures. They share the same company, the same stores. There's no parallel universe of factory data that doesn't match anything. Here are the cases where we reach for inline creation instead of fixtures: **Tests that destroy records.** If your test calls `shopper.destroy!`, you can't use a fixture because it'll be gone for the next test (or, with transactional tests, it'll roll back but the association caches might be stale). Create a disposable record instead. **Uniqueness constraint testing.** When you need to verify that creating a duplicate raises an error, you create the record inside the test so you control the exact attributes. **Parameterized edge cases.** One test that needs nineteen shoppers with specific combinations of contact preferences. That's not a fixture scenario; that's a loop with `create_shopper`. **Elasticsearch tests that need isolation.** Some search tests index records and assert on search results. They use the ES Test clean-room company and create all their data inline so the index contains exactly what they expect. About half our test files still use `create_*` methods. That's fine. The goal was never to eliminate all record creation, just to stop building the entire world from scratch in every test. ## The practical difference With FactoryBot, our test setup blocks looked like this: ```ruby let(:company) { create(:company) } let(:store) { create(:store, company: company) } let(:sa) { create(:sales_associate, company: company, store: store) } let(:shopper) { create(:shopper, company: company, store: store) } let(:appointment) { create(:appointment, sales_associate: sa, shopper: shopper) } ``` Five lines of setup to get an appointment. Each `create` hits the database. Each one builds a complete object graph that might not match what's in production. And this was in almost every test file. With fixtures, the same test setup is: ```ruby setup do @sa = manager_sa @shopper = male_shopper @appointment = appointments(:jen_russell_meeting) end ``` Three lines. No database writes. The data is already there, already consistent, already real. The test starts by reading the world, not by building one. Multiply that by hundreds of test files, and you start to understand where the speed comes from. It's not just parallel testing. It's not hitting the database hundreds of times during setup. ## What I'd do differently If I did this again, I'd write the export script earlier. I spent the first few phases with minimal fixtures (just the scaffolded `:one`/`:two` defaults) and converted to real fixtures at Phase 3.5. Those early phases would have been cleaner if the fixtures existed from the start. I'd also add more edge-case fixtures upfront. Most of the synthetic records were added reactively, when a test needed them. Having a "fixture wishlist" from the beginning would have saved some back-and-forth. And I'd start with the three-company structure immediately. The single-tenant setup worked for the first few phases, but as soon as I hit model and service tests, the lack of cross-tenant data became a problem. The core approach, though, I wouldn't change. Export real data, name everything meaningfully, let Rails resolve foreign keys, and use inline creation only when fixtures genuinely don't fit. Fixtures aren't the problem. Careless fixture design is. Next up: [parallel test isolation with Elasticsearch](/blog/parallel-testing-elasticsearch-rails), and why running search tests across twelve workers is harder than it sounds. --- ### RSpec to Minitest: Migrating a Large Rails App in 5 Days URL: https://hencf.org/blog/rspec-to-minitest-migration Published: 2026-03-20 I've been using RSpec on the BSPK codebase for about eight years. It was already there when I joined, and over time we accumulated the usual entourage: FactoryBot, shoulda-matchers, rswag, pundit-matchers, rspec-sidekiq, rspec-json_expectations. The test suite worked fine. It caught bugs. CI was green more often than not. But something had been bothering me for a while. Every time I opened a spec file, I had to mentally parse a DSL before I could think about the actual behavior being tested. `let` blocks scattered across nested `context` groups. `subject` redefined three levels deep. Shared examples that saved typing but hid what was actually being asserted. The tests were correct, but they weren't *clear*. And then there were the factories. Our FactoryBot setup had grown into its own little universe, traits and sequences and transient attributes building objects that looked increasingly different from the data in production. Every time a test failed, the first question was always: is this a real bug, or did the factory build something that would never actually exist? We'd recently finished upgrading to Ruby 4 and Rails 8.1, and the codebase felt fresh. It seemed like the right moment. Not a gradual deprecation. A full migration. RSpec to Minitest, FactoryBot to fixtures, all of it. It took five days. ## The approach I didn't want a big bang rewrite where everything breaks at once and you spend two weeks debugging test infrastructure instead of shipping features. So I set up Minitest alongside RSpec, both frameworks running in CI simultaneously, and migrated in phases. The plan was simple: start at the edges of the codebase (lib specs, validators, helpers), work inward toward the core (models, services, jobs), and finish with the integration layer. Each phase would be a self-contained commit. If something went wrong, I could revert a single phase without touching the rest. Phase 1 was just infrastructure. A `test_helper.rb`, support modules to replace the RSpec ecosystem (auth helpers, Elasticsearch setup, WebMock stubs, VCR config, custom assertions), and a starter set of YAML fixtures. Nothing migrated yet, just laying the foundation. ## Moving through the codebase Phases 2 and 3 knocked out the easy stuff: validators, routing specs, helpers, forms, mailers, and all the lib specs. These were the simplest conversions because they mostly tested pure Ruby objects with minimal database interaction. The RSpec DSL peeled off cleanly. `describe`/`it` became `def test_`, `let` became local variables or `setup` blocks, `expect(x).to eq(y)` became `assert_equal y, x`. Phase 3.5 was where things got interesting. I exported a curated dataset from our development environment into YAML fixtures, covering about two dozen tables. This wasn't a dump of everything. I picked specific records that represented real data relationships: a company with stores, sales associates, shoppers, appointments, items, lists. Named fixtures with meaningful identifiers instead of `:one` and `:two`. This ended up being one of the most valuable parts of the whole migration. More on that in a follow-up post. Phases 4 through 9 were the bulk: agent tools, data import pipeline, rake tasks, finders, services, and jobs. Hundreds of test files, each one a small translation exercise. Most conversions were mechanical, but finders needed special attention because they hit Elasticsearch. I built an `ElasticsearchTestHelper` with a `safe_reindex` method that ensured indexes were fresh before each test without blowing up parallel workers. Phase 10 was models, all 171 of them. This was also where I made a discovery about parallel testing that changed the whole trajectory of the project. ## The parallel testing surprise When I first set up Minitest, I copied a pattern from some of the existing specs: `parallelize(workers: 1)`. Several spec files had disabled parallel execution because of fixture isolation issues, and I carried that forward without questioning it. During Phase 10, I realized this was unnecessary. Minitest's parallel testing gives each worker its own database. Fixtures load into each worker's DB independently. There's no cross-contamination. I removed the `parallelize(workers: 1)` overrides from hundreds of test files and let the suite run with all available processors. The test suite went from around 400 seconds to 82 seconds. Twelve parallel processes, nearly a 5x speedup, from removing a line of code. The irony is that this was always available in theory, but with RSpec and FactoryBot, we'd never been able to use it cleanly. Factories create data at runtime, and the interactions between parallel processes creating overlapping records had been a constant source of flaky tests. Fixtures, loaded once per worker into isolated databases, just worked. ## Replacing the ecosystem One thing I underestimated going in was how much of our test infrastructure was actually just wrappers around RSpec plugins. shoulda-matchers gave us one-liner association and validation tests. Replacing those took about an hour: two small modules (`AssociationAssertions` and `ValidationAssertions`) with methods like `assert_belongs_to` and `assert_validates_presence_of` that checked the actual ActiveRecord reflections and validators. Same coverage, no gem dependency, and I could see exactly what was being tested. Pundit matchers were even simpler. A Pundit policy is just a Ruby object with methods that return booleans. `assert policy.show?` reads better than `it { is_expected.to permit(:show) }`. The rswag migration was the biggest surprise. We had over 120 integration specs using rswag's DSL to generate OpenAPI documentation from tests. The DSL was verbose: `path`, `get`, `response`, `run_test!` blocks nested three levels deep. I stripped all of it and replaced it with plain Minitest integration tests using the `committee` gem for schema validation. Instead of generating OpenAPI docs from tests, we now validate tests against hand-written OpenAPI specs. Schema-first instead of test-first. This turned out to be a strict upgrade. The OpenAPI specs are now the source of truth, they live in version control, and every integration test validates its response against the schema automatically. If the API drifts from the spec, the test fails. ## The fixture question I know fixtures are controversial. The Rails community spent years moving away from them toward FactoryBot, and for understandable reasons: early fixture setups were often a mess of tangled YAML files where changing one record broke twenty tests. The key difference this time was being intentional about fixture design. Instead of generating a fixture for every model and hoping for the best, I curated a dataset that represents a real slice of the application. A company called "vista" with two stores, each with sales associates, shoppers, appointments, and items. Every fixture has a meaningful name. Every relationship makes sense. The result is that when I read a test, I know exactly what data it's working with. `shoppers(:maria)` is a shopper at the "downtown" store. `items(:silk_scarf)` is a product in vista's catalog. No factory magic, no traits, no transient attributes. Just data that looks like what's actually in the database. I'm planning a dedicated post on fixture design because I think it's the thing most people get wrong, and it's the reason fixtures got a bad reputation in the first place. ## Phase 17: deleting RSpec The final phase was the most satisfying. Remove rspec-rails and every plugin that depended on it. Delete the `.rspec` config. Remove the RSpec configuration from `application.rb`. Update CI to run `rails test` instead of `rspec`. Twelve gems removed in one commit. The Gemfile got noticeably shorter. Boot time dropped. The test infrastructure went from a sprawl of DSL-specific configuration to twenty focused Ruby modules that I could read top to bottom. ## After the merge The migration landed on a Friday. The following week was cleanup: fixing a few tests that had been added on other branches while the migration was in flight, adding JUnit reporting for CircleCI's test summary UI, fixing Ruby 4 deprecation warnings that had been hiding under the RSpec output. I also used the momentum to add coverage that hadn't existed before. With fixtures and a clean test infrastructure, writing new tests was fast enough that I added a couple hundred tests for models and concerns that had been under-tested. When writing a test takes thirty seconds instead of two minutes of factory setup, you write more tests. The `committee` integration also expanded. By the end of the week, all 125 integration test files validated their responses against OpenAPI schemas. Every API endpoint now has a contract test whether we planned it that way or not. ## The side effects The speed improvement was the most obvious win, but two other things changed that I didn't fully anticipate. The flaky tests almost completely disappeared. Our RSpec suite had a handful of tests that failed randomly, often enough that we'd added CI retry logic to mask it. Most of the flakiness came from factories creating records that collided across parallel processes, or from test order dependencies hidden by RSpec's lazy `let` evaluation. With fixtures loaded once into isolated per-worker databases and no lazy evaluation hiding state, the randomness just stopped. We removed the CI retry config within a week. The other surprise was how much better Claude Code handles Minitest. I use Claude Code for most of my development work, and it writes Minitest tests noticeably better than it wrote RSpec. That makes sense if you think about it: Minitest tests are just Ruby methods with assertions. There's no DSL to get wrong, no `let`/`subject`/`shared_examples` nesting to misuse, no factory traits to hallucinate. When I ask Claude to add test coverage for a new feature, the output is correct on the first try far more often than it was with RSpec. The tests it generates look like tests I'd write myself, which was rarely true with the RSpec output. That mocking-resistant, agent-friendly test culture ended up being one of the reasons the [Heroku-to-AWS migration](/blog/i-migrated-bspk-off-heroku) was tractable a few weeks later. ## The numbers Before the migration: a slow, serial test suite with heavy boot time, about a dozen testing gems, and factories that were their own maintenance burden. After: the suite runs in under 90 seconds on twelve parallel workers. Thirty YAML fixture files with curated data. Twenty support modules, all plain Ruby. Zero RSpec dependencies. Tests that are less flaky, easier for both humans and AI to write, and read like what they are: assertions about behavior. ## What's coming next This post covers the full arc, but there's a lot more to dig into. Over the next couple of weeks I'll write about: **[Fixture design for real applications.](/blog/fixtures-for-real-rails-apps)** How to structure YAML fixtures for a multi-tenant codebase with complex associations, and why most fixture setups fail. **[Parallel test isolation with Elasticsearch.](/blog/parallel-testing-elasticsearch-rails)** The specific challenges of running search-heavy tests across multiple workers, and the `safe_reindex` pattern that made it reliable. If you're thinking about making this switch, the short version is: it's less scary than it looks, and the payoff is real. The five days of migration work have already paid for themselves in faster CI, simpler debugging, and tests that are genuinely easier to read and write. --- ### Fetching YouTube Transcripts in Ruby URL: https://hencf.org/blog/youtube-transcripts-ruby Published: 2026-03-19 The [RAG pipeline](/blog/rag-without-leaving-rails) searches through thousands of video transcripts. The [metadata extractor](/blog/llm-extraction-at-scale) processes video titles and descriptions. Both assume the data is already in the database. This post is about how it got there. YouTube's Data API v3 has a Captions endpoint, but it's scoped to videos you own. For a knowledge base of public lecture videos across dozens of channels, I needed the same caption text that's already visible to anyone watching the video, just in a programmatic format. There's no official endpoint for that. Libraries like Python's `youtube-transcript-api` have solved this by using YouTube's internal player API directly. I needed the same thing in Ruby, and it took a few attempts to get there. ## The protobuf approach (broken) Several Python libraries (notably `youtube-transcript-api`) used to fetch transcripts by sending protobuf-encoded requests directly to YouTube's internal endpoints. Some Ruby ports attempted the same thing: encode a specific protobuf payload with the video ID and language code, POST it to an internal endpoint, decode the protobuf response. This worked for a while. Then YouTube changed something on their end and the protobuf schema shifted. Every implementation relying on it broke silently. No error messages, just empty responses or 400s. The Python library had to be completely rewritten. The Ruby ports were abandoned. I spent a few hours trying to get this working before realizing the format had changed and nobody had updated the Ruby implementations. ## Parsing the watch page (session-bound URLs) The second attempt was more straightforward. Load the YouTube watch page, parse the embedded `ytInitialPlayerResponse` JSON, and extract the caption track URL from it. This JSON blob contains everything the player needs, including an array of available caption tracks with their download URLs. It works perfectly in a browser. The problem is that the caption URLs in the watch page response are session-bound. If you extract the URL and try to fetch it in a subsequent HTTP call, you get a 403. The URLs expire quickly and are tied to the original request context. I tried extracting the URL and fetching it immediately in a single pipeline. Some worked, most didn't. The expiration window was too short and unreliable. This wasn't going to scale to thousands of videos. ## What actually works: InnerTube with an Android client YouTube's web player communicates with a backend called InnerTube. This is the same API that the Python `youtube-transcript-api` library uses after its rewrite, and the approach the [Ruby Events](https://github.com/rubyevents) project uses for their transcript pipeline. It's well-documented across multiple open source projects and works reliably with the Android client configuration. Three steps: 1. Extract the InnerTube API key from a YouTube watch page 2. Call the InnerTube player endpoint with the Android client context 3. Fetch the caption URL from the response The `INNERTUBE_API_KEY` extracted in step 1 is not a personal API key. It's YouTube's own client key, embedded in every watch page. It doesn't change often and isn't tied to any account. ## The complete client Here's the full implementation. It fits in a single file with no external dependencies beyond Ruby's standard library (`net/http`, `json`, `rexml`): ```ruby require "net/http" require "json" require "rexml/document" module YouTube class TranscriptClient class TranscriptNotAvailable < StandardError; end INNERTUBE_CLIENT = { clientName: "ANDROID", clientVersion: "20.10.38" }.freeze USER_AGENT = "com.google.android.youtube/20.10.38 (Linux; U; Android 11) gzip" def fetch(video_id, language: "pt") api_key = fetch_api_key(video_id) caption_url = fetch_caption_url(video_id, api_key, language) xml = fetch_transcript_xml(caption_url) parse_srv3(xml) end private def fetch_api_key(video_id) uri = URI("https://www.youtube.com/watch?v=#{video_id}") http = build_http(uri) req = Net::HTTP::Get.new(uri.request_uri) req["User-Agent"] = "Mozilla/5.0" html = http.request(req).body match = html.match(/"INNERTUBE_API_KEY":\s*"([a-zA-Z0-9_-]+)"/) raise TranscriptNotAvailable, "Could not extract API key" unless match match[1] end def fetch_caption_url(video_id, api_key, language) uri = URI("https://www.youtube.com/youtubei/v1/player?key=#{api_key}") http = build_http(uri) req = Net::HTTP::Post.new(uri.request_uri) req["Content-Type"] = "application/json" req["User-Agent"] = USER_AGENT req.body = { context: { client: INNERTUBE_CLIENT }, videoId: video_id }.to_json response = http.request(req) raise TranscriptNotAvailable, "InnerTube request failed" unless response.is_a?(Net::HTTPSuccess) data = JSON.parse(response.body) tracks = data.dig("captions", "playerCaptionsTracklistRenderer", "captionTracks") raise TranscriptNotAvailable, "No caption tracks available" if tracks.blank? track = tracks.find { |t| t["languageCode"] == language } || tracks.first track["baseUrl"] end def fetch_transcript_xml(url) uri = URI(url) http = build_http(uri) req = Net::HTTP::Get.new(uri.request_uri) req["User-Agent"] = USER_AGENT response = http.request(req) raise TranscriptNotAvailable, "Transcript fetch failed" unless response.is_a?(Net::HTTPSuccess) raise TranscriptNotAvailable, "Empty transcript response" if response.body.blank? response.body end def parse_srv3(xml) doc = REXML::Document.new(xml) segments = [] doc.elements.each("timedtext/body/p") do |p| start_ms = p.attributes["t"].to_i duration_ms = p.attributes["d"].to_i text = p.elements.collect("s") { |s| s.text.to_s }.join("").strip next if text.blank? segments << { text: text, start_ms: start_ms, end_ms: start_ms + duration_ms } end raise TranscriptNotAvailable, "No transcript segments found" if segments.empty? segments end def build_http(uri) http = Net::HTTP.new(uri.host, uri.port) http.use_ssl = true http.open_timeout = 10 http.read_timeout = 10 http end end end ``` The `fetch` method is the public interface. Pass a video ID, get back an array of timed transcript segments. Language defaults to Portuguese since that's what [Guia](https://guia.tv) needs, but it falls back to whatever caption track is available. YouTube serves captions in a format called srv3. Each `

` element in the XML is a caption segment with `t` (start time in milliseconds) and `d` (duration in milliseconds). The `` elements inside are individual words. The `parse_srv3` method concatenates them into segment-level text and preserves the timing, so the result is an array of hashes like `{ text: "Boa noite, queridos irmãos.", start_ms: 1000, end_ms: 6000 }`. That millisecond-level timing carries through to the chunking pipeline, so when the RAG system retrieves a relevant chunk, it can link directly to the timestamp in the video. ## Storing transcripts Each video stores the transcript in two forms: ```ruby def fetch_transcript! client = YouTube::TranscriptClient.new segments = client.fetch(video_id) update!( raw_transcript: segments, plain_transcript: segments.map { |s| s[:text] }.join(" ") ) end ``` `raw_transcript` is a JSONB column with the full array of segments and their timing data. This feeds the chunking pipeline that creates vector embeddings for RAG search. `plain_transcript` is concatenated text for full-text search and for quickly eyeballing whether a transcript makes sense. Having both columns has a practical benefit: the plain text is fast to scan when debugging. If a transcript looks garbled, I can open the record and read it immediately without parsing JSON. The structured format is what the downstream pipeline actually consumes. ## Running it across thousands of videos A single fetch takes about two seconds: one request for the watch page, one POST to InnerTube, one GET for the caption XML. The bottleneck is network latency, not processing. For batch runs, each video gets its own background job: ```ruby class FetchTranscriptJob < ApplicationJob queue_as :default def perform(video) return unless video.youtube? return if video.raw_transcript.present? video.fetch_transcript! rescue YouTube::TranscriptClient::TranscriptNotAvailable => e Rails.logger.warn "Transcript not available for video #{video.video_id}: #{e.message}" end end ``` Two guard clauses at the top: skip non-YouTube videos (the platform also has Vimeo content), and skip videos that already have transcripts. The `TranscriptNotAvailable` rescue catches videos with no captions at all. Livestreams, very old videos, and some unlisted content don't have auto-generated captions. These failures are expected and just log a warning. For larger batch runs, a rake task wraps the same logic with progress tracking and error counts, running the full pipeline sequentially: fetch transcript, chunk it, generate embeddings, mark as archived. Slow but predictable. I run it locally against specific date ranges when new content gets added, then export the results to compressed seed files for production, as covered in the [extraction post](/blog/llm-extraction-at-scale). ## What can break YouTube changes their internals without notice. The InnerTube approach has been stable for months, but there's no guarantee. The Android client version string might eventually stop working, the API key extraction regex might need updating, or the caption URL format could change. The client has no retry logic. If a fetch fails, it fails. For batch processing, the job layer handles retries via Solid Queue's built-in mechanism. For one-off fetches from the admin panel, you just click the button again. Adding automatic retries with exponential backoff would be more robust, but for a knowledge base that gets updated in batches, "run it again" has been enough. Auto-generated captions aren't always accurate either. YouTube's speech recognition handles Portuguese reasonably well for clear lecture-style content, but it struggles with proper nouns, technical terminology, and speakers with regional accents. The RAG pipeline accounts for this by using semantic search (vector similarity) rather than exact keyword matching, which is more forgiving of transcription errors. The InnerTube API isn't officially documented, so there's always a chance something changes. In practice, it's been stable for months and is the same approach used by widely adopted open source projects. If the client version or endpoint format changes, the fix is usually updating a version string or adjusting a URL. The Ruby Events project deals with the same maintenance burden for their conference video transcripts, so there's a community keeping an eye on it. --- ### What Breaks When You Run an LLM 6,000 Times URL: https://hencf.org/blog/llm-extraction-at-scale Published: 2026-03-17 The [RAG pipeline I built for Guia](/blog/rag-without-leaving-rails) could search through 6,383 YouTube video transcripts by semantic similarity. But the search results weren't useful on their own. A chunk of transcript text and a video title, with no way to know who was speaking or what topic the lecture covered. The metadata I needed was already in the videos. A title like "Estudo do Evangelho Segundo o Espiritismo com Haroldo Dutra Dias" tells you the speaker and the topic, but that information is locked in an unstructured string. I needed it extracted, normalized, and queryable. So I pointed an LLM at each video's title, description, and channel name. One call took about 175 milliseconds. Getting it to work reliably across all 6,383 videos took considerably longer. ## Defining the output RubyLLM has a schema DSL that maps to JSON Schema. You define a Ruby class, and the gem generates the schema sent to the LLM as the `response_format` parameter. The response is guaranteed to match your structure. ```ruby class OutputSchema < RubyLLM::Schema array :speakers, description: "Speakers/participants in the video" do object do string :name, description: "Full name of the speaker" string :role, enum: %w[speaker interviewer moderator], description: "Role in the video" string :confidence, enum: %w[high medium low], description: "Extraction confidence level" end end array :topics, of: :string, description: "Topics from the provided list that match this video" end ``` Two details in this schema saved me significant cleanup later: the `confidence` level lets the LLM express uncertainty instead of guessing, and constraining topics to a provided list prevents the model from inventing categories. More on both below. ## The prompt does the heavy lifting The extraction model is small and fast (Groq's `gpt-oss-20b`), so the prompt needs to be explicit. The system prompt is in Portuguese, since all the content is Portuguese, and includes pattern-matching rules for extracting speaker roles from video descriptions: - "Palestrante:" → speaker - "Entrevistador:" → interviewer - "Direção:" → moderator - A name after "com" in the title is usually the speaker - Abbreviations map to full topic names: "ESE" → "Evangelho Segundo o Espiritismo" The prompt also receives two critical lists: every known speaker (with their alternative names) and every valid topic. ```ruby def build_prompt(video) speakers_list = Speaker.ordered.map do |s| names = [s.name, *s.alternative_names].join(", ") "- #{names}" end <<~PROMPT Video: #{video.title} Canal: #{video.channel.name} Descrição: #{video.description} Palestrantes conhecidos: #{speakers_list.join("\n")} Tópicos disponíveis: #{Topic.ordered.pluck(:name).join(", ")} PROMPT end ``` Passing the known speakers list was the single biggest quality improvement. Without it, the LLM extracted "Haroldo" from one video and "Haroldo Dutra Dias" from another, creating duplicates everywhere. With the list in context, it matches against existing names and returns consistent results. ## The Ollama detour I started with Ollama running locally, the same setup from the [RAG post](/blog/rag-without-leaving-rails) where it handles embeddings. For metadata extraction, I pointed RubyLLM at the local instance and got results that looked reasonable at first. The structured output wasn't actually structured. Ollama's OpenAI-compatible endpoint (`/v1/chat/completions`) silently ignores the `response_format` parameter. No error, no warning. It returns free-form text that sometimes happens to look like valid JSON and sometimes doesn't. RubyLLM routes through this endpoint, so my `OutputSchema` wasn't enforcing anything. I spent a day debugging inconsistent results before realizing schema enforcement wasn't happening at all. Some videos extracted perfectly because the model happened to produce valid JSON. Others returned partial objects or hallucinated fields that didn't match the schema. The native Ollama API (`/api/chat`) does support structured output, but that's not the endpoint RubyLLM uses. Switching to Groq fixed this immediately. Same RubyLLM code, same schema, same prompt. Groq actually enforces the JSON Schema, so every response matches `OutputSchema`. At about 175ms per call versus multiple seconds locally, batch processing became practical too. ## Speaker matching is the hard part The first extraction run across all videos created over 400 speaker records. Only about 280 were actually unique. The rest were duplicates: accent variations ("Dircinéia" vs "Dircineia"), name fragments ("Haroldo" vs "Haroldo Dutra Dias"), title prefixes ("Prof. José" vs "José Carlos"). I added fuzzy matching to the Speaker model: ```ruby class Speaker < ApplicationRecord def self.find_by_name_fuzzy(name) found = where("LOWER(name) = LOWER(?)", name).first return found if found found = where( "EXISTS (SELECT 1 FROM unnest(alternative_names) AS alt WHERE LOWER(alt) = LOWER(?))", name ).first return found if found where( "LOWER(name) LIKE LOWER(?) OR LOWER(?) LIKE '%' || LOWER(name) || '%'", "%#{name}%", name ).first end end ``` The `alternative_names` column is a PostgreSQL text array. When I find duplicates, I merge them: keep one canonical record, move the variants into `alternative_names`, and re-run extraction. On subsequent runs, `find_by_name_fuzzy` matches against both the canonical name and all alternatives. The confidence field from the schema turned out to be essential for data quality. When the LLM isn't sure about a speaker, it returns `confidence: "low"`. The extractor skips these: ```ruby speakers_data.each do |entry| next if entry["confidence"] == "low" speaker = Speaker.find_by_name_fuzzy(entry["name"]) speaker ||= Speaker.create!(name: entry["name"]) video.video_speakers.create!(speaker: speaker, role: entry["role"]) end ``` Low-confidence entries still get stored in the raw `metadata_extraction` JSONB column on the video record. They're available for auditing, but they don't create associations that would pollute search results with uncertain data. ## Topics use the opposite approach Instead of fuzzy matching and creating new records, the extractor only links topics that already exist: ```ruby topics_data.each do |name| topic = Topic.where("LOWER(name) = LOWER(?)", name.strip).first next unless topic video.video_topics.create!(topic: topic) end ``` There are 15 topics, seeded before extraction runs. The LLM receives the full list in the prompt and picks from it. If it returns something that doesn't match, the entry gets dropped. Without this whitelist, the LLM generated dozens of variations: "Mediumship", "Mediunidade", "Studies on Mediumship", "Practical Mediumship." With it, everything maps to one of 15 canonical categories. The constraint works at two levels: the prompt tells the LLM which topics exist, and the code only links topics it finds in the database. ## One job per video The batch architecture is straightforward. A bulk job enqueues individual extraction jobs: ```ruby class ExtractAllVideoMetadataJob < ApplicationJob def perform(scope: :pending) videos = case scope.to_sym when :all then Video.visible else Video.visible.where(metadata_extracted_at: nil) end videos.find_each do |video| ExtractVideoMetadataJob.perform_later(video) end end end ``` `find_each` loads videos in batches of 1,000 to avoid pulling everything into memory. Each job calls the extractor for a single video. Solid Queue manages concurrency, which naturally throttles API calls without explicit rate limiting. The retry logic appends the error message to the prompt on the second attempt: ```ruby def ask_llm(prompt, error_context: nil) if error_context prompt = "#{prompt}\n\n[Previous attempt failed: #{error_context}. Please try again carefully.]" end # LLM call... end ``` On final failure (after one retry), the extractor stores empty arrays and sets `metadata_extracted_at` to the current time. Without that timestamp, a failed video would get re-enqueued on every bulk run, potentially thousands of times. Marking it as "extracted with empty results" breaks the loop. The whole extraction is idempotent. Running it twice on the same video clears old associations and creates fresh ones, so re-running after a prompt fix or a speaker list update is safe. ## Getting extracted data to production Extraction runs in development against a local database with all the transcripts loaded. Production doesn't have Groq access for batch extraction (only for the live chat agent). So I needed a way to ship the extracted metadata to production without re-running the LLM. The solution is compressed seed files: ```ruby # lib/tasks/videos.rake task export_metadata_seeds: :environment do speakers = Speaker.with_videos.order(:name).map do |s| { name: s.name, bio: s.bio, alternative_names: s.alternative_names || [] } end File.open("db/seeds/speakers.marshal.gz", "wb") do |f| gz = Zlib::GzipWriter.new(f) gz.write(Marshal.dump(speakers)) gz.close end # Same pattern for video metadata... end ``` Marshal + Gzip keeps the files compact. 281 speakers compress to about 3 KB. The video metadata (2,675 entries with speaker and topic associations) fits in roughly 35 MB. Both get committed to the repo and loaded during `db:seed` on deploy. No runtime LLM calls needed on the production server. ## What I'd change The implicit rate limiting through Solid Queue concurrency has worked so far, but it's fragile. If I added more queue workers or switched to a provider with tighter limits, I'd start hitting 429s. A simple delay between jobs or explicit concurrency controls would be more resilient. Speaker deduplication was entirely reactive. Run extraction, notice duplicates, merge, repeat. A dedicated resolution step after extraction (cluster similar names, review, then merge) would catch duplicates before they reach the database. The current approach works because the speaker list is fairly stable at around 280 people across a specialized content library. It wouldn't hold up with thousands of unique speakers. Marshal is Ruby-specific and opaque. You can't inspect the seed files or diff them meaningfully in git. JSON Lines with gzip would be slightly larger but debuggable. For data where git diffs matter, that tradeoff is worth making. I expected the LLM call to be the interesting part of this project. Groq returns structured JSON in 175ms and that part just works. Most of my time went into the surrounding code: normalizing speaker names across variations, preventing the model from inventing categories, handling videos with barely any metadata, and shipping results from dev to a production server without LLM access. I'm still merging the occasional duplicate speaker when I notice one in search results. --- ### RAG Without Leaving Rails URL: https://hencf.org/blog/rag-without-leaving-rails Published: 2026-03-16 I wanted to build a chat that could answer questions about a religious knowledge base: 34 books of Christian spiritualist theology and over 6,000 YouTube lecture transcripts. The kind of thing where you ask "what does this tradition teach about the afterlife?" and it pulls the relevant passages from the source texts, cites the chapter and page number, and synthesizes an answer. Every RAG tutorial I found started the same way: Python, LangChain, Pinecone or Weaviate, OpenAI embeddings. A whole separate stack from the Rails app that would actually serve the chat. I didn't want to run two runtimes, maintain two deployment pipelines, or learn a framework just to glue an LLM to a database query. So I built the entire pipeline in Rails. The app is called [Guia](https://guia.tv), and the RAG architecture uses four components: `pdf-reader` for text extraction, Ollama for local embeddings, pgvector for similarity search, and RubyLLM with Groq for chat completions. No Python. No LangChain. No hosted vector database. Deployed on a single server with Kamal. ## The data model The knowledge base has two types of content: books and videos. Both get chunked into embeddable pieces and stored with their vector representations. ```ruby # Books → BookChunks (with embedding) # Videos → VideoChunks (with embedding) ``` Each chunk stores the text content, positional metadata (page numbers for books, timestamps for videos), and a 1024-dimension vector embedding. The schema looks like this: ```ruby create_table "book_chunks" do |t| t.references :book t.text :content t.string :chapter t.string :section t.integer :page_start t.integer :page_end t.integer :position t.integer :tokens_count t.vector :embedding, limit: 1024 end create_table "video_chunks" do |t| t.references :video t.text :content t.integer :time_start # milliseconds t.integer :time_end t.integer :position t.integer :tokens_count t.vector :embedding, limit: 1024 end ``` The `vector` column type comes from pgvector, PostgreSQL's vector similarity extension. You enable it with `enable_extension "vector"` in a migration, and you're done. No separate database, no external service. Your vectors live right next to your data. ## Chunking: the part that actually matters I spent more time on chunking than on any other part of the pipeline. Chunk too large and you dilute the embedding with irrelevant context. Chunk too small and you lose coherence. The sweet spot for this content was around 500 tokens with 50 tokens of overlap between consecutive chunks. ```ruby module ChunkingSupport CHUNK_SIZE = 500 CHUNK_OVERLAP = 50 CHARS_PER_TOKEN = 4 # rough approximation def estimated_tokens(text) (text.length.to_f / CHARS_PER_TOKEN).ceil end end ``` The token estimation is deliberately rough. Counting actual tokens would require running the tokenizer for the embedding model, and the precision doesn't matter here. You're aiming for a ballpark, not an exact count. If your chunks land between 400 and 600 tokens, you're fine. For books, the chunking gets interesting because not all books have the same structure. Some are Q&A dialog format: numbered questions with authored responses. Others use traditional chapters with numbered paragraphs. Others are straight narrative prose. A single chunking strategy doesn't work for all of them. I built a `ChapterDetector` that samples the first 15 pages and picks a strategy: ```ruby module ChapterDetector def self.for(pages) sample = pages.first(15).map { |p| p[:text] }.join("\n") has_capitulo = sample.match?(/CAP[ÍI]TULO\s+[IVXLCDM]+/i) has_questions = sample.scan(/^\d+\.\s/).size >= 5 if has_capitulo && has_questions QAndA.new(section_prefix: "Q.") elsif has_capitulo ChapterBased.new(section_prefix: "§") else Narrative.new end end end ``` The Q&A detector matches chapter headings and numbered questions. The Narrative detector looks for numbered titles, Roman numeral headings, and all-caps section headers. Each strategy returns chapter and section labels that get stored on the chunk, so when the chat references a passage, it can say "Chapter III, Q. 132" instead of just "page 47." This detection runs automatically during processing. Adding a new book doesn't require manual configuration. Drop the PDF in, and the processor figures out which strategy to use. For videos, chunking is simpler. The raw transcript is an array of timed segments from YouTube's caption track. The processor concatenates segments until it hits the token target, records the start and end timestamps, and moves on. Each video chunk knows exactly which moment in the video it came from. ## Embeddings with Ollama The embedding model is [bge-m3](https://huggingface.co/BAAI/bge-m3), a multilingual model that produces 1024-dimension vectors. It runs locally via Ollama. No API calls, no per-token billing, no rate limits. The client is about as simple as it gets: ```ruby class EmbeddingClient OLLAMA_URL = ENV.fetch("OLLAMA_URL", "http://localhost:11434") MODEL = "bge-m3" def generate(text) generate_batch([ text ]).first end def generate_batch(texts) uri = URI("#{OLLAMA_URL}/api/embed") response = Net::HTTP.post( uri, { model: MODEL, input: texts, keep_alive: -1 }.to_json, "Content-Type" => "application/json" ) JSON.parse(response.body).fetch("embeddings") end end ``` Two things worth noting. The `keep_alive: -1` tells Ollama to keep the model loaded in memory permanently. Without this, Ollama unloads the model after 5 minutes of inactivity, and the next request pays a cold-start penalty of several seconds while the model loads back into RAM. And the batch endpoint (`input` as an array) is critical for processing books. Embedding chunks one at a time would take forever. Batching 50 chunks per request makes the whole pipeline practical. Why bge-m3 specifically? It's multilingual (all my content is in Portuguese), it's small enough to run on a CPU without a GPU (the model is about 567MB), and it scores well on retrieval benchmarks. I tried `nomic-embed-text` first, but bge-m3 handled Portuguese diacritics and terminology noticeably better. ## Vector search with pgvector For the ActiveRecord integration, I use the [neighbor](https://github.com/ankane/neighbor) gem. It adds a `has_neighbors` declaration to your models and gives you nearest-neighbor queries: ```ruby class BookChunk < ApplicationRecord belongs_to :book has_neighbors :embedding end class VideoChunk < ApplicationRecord belongs_to :video has_neighbors :embedding end ``` The retrieval layer is a `KnowledgeRetriever` that searches both chunk types and merges results: ```ruby class KnowledgeRetriever DEFAULT_LIMIT = 5 def search(query, limit: DEFAULT_LIMIT, sources: :all) query_embedding = @embedding_client.generate(query) case sources when :books then search_book_chunks(query_embedding, limit) when :videos then search_video_chunks(query_embedding, limit) else search_all(query_embedding, limit) end end private def search_book_chunks(embedding, limit) BookChunk .joins(:book).where(books: { status: :ready }) .nearest_neighbors(:embedding, embedding, distance: "cosine") .first(limit) end def search_video_chunks(embedding, limit) VideoChunk .joins(:video).where(videos: { archive_status: :archived }) .nearest_neighbors(:embedding, embedding, distance: "cosine") .first(limit * 3) .uniq(&:video_id) .first(limit) end def search_all(embedding, limit) books = search_book_chunks(embedding, limit) videos = search_video_chunks(embedding, limit) (books + videos).sort_by(&:neighbor_distance).first(limit) end end ``` The video search has a deduplication step: it fetches 3x the requested limit, then picks the best chunk per video. Without this, a single long lecture would dominate every result set because all its chunks are semantically similar. The `nearest_neighbors` method generates SQL using pgvector's cosine distance operator (`<=>`). The query plan uses an index scan if you've added one: ```ruby add_index :book_chunks, :embedding, using: :hnsw, opclass: :vector_cosine_ops ``` HNSW (Hierarchical Navigable Small World) is an approximate nearest-neighbor index. It trades a tiny amount of recall accuracy for dramatically faster queries. For a few tens of thousands of chunks, the speedup is significant. ## The chat agent With retrieval working, the RAG flow is straightforward: 1. User sends a message 2. Retrieve the top 5 chunks matching the query 3. Format them as context with source attribution 4. Send context + question + conversation history to the LLM 5. Save the response with structured references ```ruby class ChatAgent def respond(conversation) last_message = conversation.messages.where(role: :user).last chunks = @retriever.search(last_message.content) context = format_context(chunks) chat = RubyLLM.chat(model: "openai/gpt-oss-120b") chat.assume_model_exists = true # Replay conversation history conversation.messages.order(:created_at).last(10).each do |msg| chat.add_message(role: msg.role.to_sym, content: msg.content) end response = chat.ask(context + "\n\n" + last_message.content) conversation.messages.create!( role: :assistant, content: response.content, references: build_references(chunks) ) end end ``` The `add_message` call for history replay is important. An earlier version used `chat.ask()` for each historical message, which made actual API calls for every turn. `add_message` just populates the conversation context without hitting the API. The LLM is Groq running `gpt-oss-120b`, accessed through RubyLLM's OpenAI-compatible provider. `assume_model_exists = true` skips model validation, since Groq models aren't in RubyLLM's model registry — same gotcha I covered when [building the chat agent for this site](/blog/building-ai-agent-ruby-llm-groq). The `references` field stores structured JSON: book ID, chapter, page range, or video slug and timestamp. The frontend renders these as clickable links that jump to the exact page or moment in the video. ## Deploying Ollama with Kamal The production setup runs three containers on a single server, all managed by Kamal: ```yaml # config/deploy.yml servers: web: hosts: - <%= Rails.application.credentials.server_ip %> env: secret: - OLLAMA_URL accessories: db: image: ankane/pgvector:latest port: "127.0.0.1:5433:5432" env: secret: - POSTGRES_PASSWORD directories: - guia_data:/var/lib/postgresql/data ollama: image: ollama/ollama:latest port: "127.0.0.1:11434:11434" directories: - guia_ollama:/root/.ollama ``` The web container reaches Ollama at `http://guia-ollama:11434` via Docker's internal network. The `OLLAMA_URL` environment variable makes this configurable per environment. After the first deploy, you need to pull the embedding model into Ollama's persistent volume: ```bash kamal accessory exec ollama --reuse "ollama pull bge-m3" ``` The `--reuse` flag runs the command inside the existing container instead of spinning up a new one. This matters because the Ollama container's entrypoint is the `ollama` binary, so `kamal accessory exec` without `--reuse` would try to run `ollama` as the shell. The model stays in the Docker volume across deploys. You pull it once, and the `keep_alive: -1` setting keeps it loaded in memory. ## What I'd do differently The `estimated_tokens` approximation (dividing character count by 4) works well enough for Portuguese prose, but it underestimates for content with lots of short words or punctuation. If I were starting over, I'd use a proper tokenizer for the embedding model. The `tiktoken_ruby` gem handles this for OpenAI-compatible tokenizers, and the accuracy improvement matters when your chunks are borderline on the size limit. I'd also add hybrid search from the start. Pure vector search sometimes misses exact keyword matches. A user searching for a specific book title or author name gets better results from traditional text search. The app now has both: semantic search via pgvector and ILIKE text search on titles and descriptions, with the results merged. Building this after the fact wasn't hard, but the retriever would have been cleaner if I'd planned for it from the beginning. The pgvector extension worked out of the box. The `ankane/pgvector` Docker image ships PostgreSQL with the extension pre-installed, and the `neighbor` gem makes it feel like any other ActiveRecord query. If you're already running PostgreSQL, you don't need Pinecone. If you're running SQLite, look into `sqlite-vec` for a similar approach. I keep hearing that RAG requires a specialized stack. That you need Python for the ML pipeline, a dedicated vector database for scale, and a framework like LangChain to wire it all together. For my use case (tens of thousands of chunks, single-digit concurrent users), Rails with a few gems handled everything. The PDF extraction, the chunking, the embeddings, the vector search, the chat interface, the deployment. All in one codebase, one language, one deployment target. If your traffic or data volume demands a dedicated vector database or async embedding pipeline, you'll know when you get there. Start with pgvector and see how far it takes you. --- *This post is part of a series on LLM integration in Rails. Next: [What Breaks When You Run an LLM 6,000 Times](/blog/llm-extraction-at-scale), where I extract structured metadata from the 6,383 video transcripts that feed this RAG pipeline.* --- Older posts available at https://hencf.org/blog