Audit an API Field's Completeness Before You Build a Feature on It

An API’s documentation lists the fields a record can have. It rarely tells you how many records actually have them. Those are different numbers, and on open data APIs the gap is often enormous.

The failure looks like this. You read the schema, spot a field that would make a great feature, build the feature, and then discover the field is populated on four percent of the data. By then the feature is written and the pitch has been made.

A Worked Case

Occurrence records in GBIF can carry an elevation. For anything ecological that is a valuable field, because altitude drives climate and climate drives most of what a species does.

A query for one Andean orchid returned 241 records. Reasonable sample, and the schema promises elevation.

Now count how many actually have it. GBIF supports range filters, and a range filter only matches records where the field exists:

# every record
https://api.gbif.org/v1/occurrence/search?scientificName=Masdevallia%20veitchiana&limit=0
→ "count": 241

# records where elevation exists and falls in a range covering all possible values
https://api.gbif.org/v1/occurrence/search?scientificName=Masdevallia%20veitchiana&elevation=0,6000&limit=0
→ "count": 12

Twelve out of 241. Five percent.

Two calls, both with limit=0 so no result bodies come back, and the feature was dead before a line of code got written.

The Technique

limit=0 is the important part. You want the count and nothing else, which makes these queries cheap enough to run across every field you care about without worrying about pagination or payload size.

The generalisable version:

  1. Get the total for your query with limit=0.
  2. Re-run it with a filter on the field, set to a range wide enough to match any real value.
  3. Divide.

Range filters work for numerics and dates on most well-built APIs. elevation=0,6000, year=1700,2026, depth=0,11000. For a field that is not range-filterable, look for a dedicated presence parameter. GBIF has hasCoordinate=true and hasGeospatialIssue=false for exactly this reason.

Where none of that exists, fetch a sample of a few hundred records and count non-null values locally. Less elegant, still five minutes of work against days of building on sand.

Completeness Is Not Random

The reason this matters more than a simple percentage suggests: missing data is almost never evenly distributed.

In the orchid case, the records with elevation were nearly all preserved herbarium specimens, collected by botanists who wrote altitude in a field notebook. The records without it were overwhelmingly smartphone observations, where the app captured coordinates automatically and the user never typed an elevation.

So the five percent that had the field was not a random five percent. It was systematically older, systematically professional, and systematically biased toward locations botanists visit. Any average computed across it describes expedition history as much as it describes the species.

That pattern repeats everywhere. In a government contracts API, the records with full descriptions are the large awards. In a product catalogue, the items with complete specs are from the bigger vendors. Ask not only how much is missing, but what the present records have in common.

A Cheap Habit

Before designing anything that depends on a field, spend two minutes and two requests establishing what fraction of the data has it. Write the number into whatever you are planning from, because the next person who reads the schema will make the same assumption you did.

The best outcome is that you find out early and pick a different angle. The second best is that you build the feature knowing it applies to five percent of your rows, and you say so in the interface rather than letting a user assume otherwise.

Related