Sqlite on API Coding

A Tiny ETL Binary Competes With curl, jq and SQLite in a Cron Job, So Build It That Small

Mon, 05 Oct 2026 00:00:00 +0000

The real competitor is a shell script in a crontab, and it usually looks like this:

curl -s "https://api.example.com/v1/launches?limit=100" \
  | jq -r '.data[] | [.id, .name, .net] | @csv' \
  | sqlite3 -csv launches.db ".import /dev/stdin launches"

It works on the day you write it. Then the API answers 429 and curl pipes an error page into jq. Or the API has a second page, and the script never asks for it. When the job dies halfway, the rerun inserts the same rows again or trips over the primary key, depending on how the table was made. A field that starts arriving as a string goes unnoticed until a chart looks wrong. Retries, backoff, pagination, incremental state, idempotent writes and schema drift: that’s the list, and shell scripts get every item on it wrong in predictable ways. (To be fair, curl --retry covers the first two.)

Keeping Third-Party API Responses in SQLite Gets You a Cache, an Offline Mode and a History

Mon, 05 Oct 2026 00:00:00 +0000

The first version of an API cache is a dictionary with a timeout. The second is Redis holding a JSON string under a key built from the URL. The third gets written after an incident, when somebody needs to know what the weather provider returned on Tuesday and the cache has already replaced it with Wednesday’s answer. Every app that depends on an outside API walks the same path: a cache, then a retry layer, then a debugging log, then a wish that it had kept the old responses.

Stop Reparsing the Same Big JSON Documents: Persist Them as Indexed Binary Instead

Mon, 05 Oct 2026 00:00:00 +0000

A service loads a 40 MB product catalog from disk every time a worker starts. It parses the JSON into objects, builds a map from SKU to product, and answers price lookups until the next deploy. The parse costs seconds of startup. The object tree often costs several times the file’s size in memory, and every worker holds its own copy. A typical request touches two fields of one product. Multiply that by every deploy, every autoscale event and every cold start. Nothing here is broken. The format was built for exchange and it’s being used as a database.