Airgap Check finds the downloads hiding inside an offline AI app
A local inference endpoint can leave tokenizers and embedding models fetching their own supplies. The published tests reveal a useful tool and a smaller proof than the headline might suggest.
Your AI app can point at a local model and still reach outside for the ingredients needed to use it. The inference server is home; a tokenizer or embedding model has apparently gone shopping. Airgap Check tries to expose those first-use downloads before the network disappears.
The project README describes a static scanner plus an optional Docker run. Its observer combines a DNS sinkhole with Python and Node socket hooks. The maker warns that “Only the code paths you exercise are tested.” An idle application can produce a very reassuring silence.
That warning changes how to read the published benchmark. A probe associated with gpt-engineer asks tiktoken for an encoding and fails offline. The mem0 case exercises ChromaDB collection creation and insertion, triggering the default embedding model’s download. Those are useful reproductions of dependency behavior; they don’t establish that every normal workflow reaches those calls.
The benchmark records a Hugging Face configuration-file probe taking about 23.5 seconds to fail. Someone waiting for a document to load sees a spinner. Someone reading the probe command can see which file request was waiting. That is a much better starting point than waving vaguely at the model server.
Hugging Face documents why a cached model can still cause a network request. Its environment-variable reference says a normal download call checks whether a newer version exists. Offline mode skips that request and uses cached files; when the needed file is absent, it raises an error. Turning off the shopping trip does not put dinner in the fridge.
The same documentation says these environment variables are read when the package is imported. Setting them afterward won’t change that configuration. Cache location matters as well: a populated developer cache won’t magically appear inside a different deployed image or under another user account.
Docker’s documentation explains the isolation underneath the test: the none driver leaves only a loopback device. That’s a concrete way to make an external dependency fail during evaluation. It also removes access to a model server outside that container, so a meaningful smoke test needs to account for that deliberately.
A useful offline acceptance test should start from the image people will actually receive, with exactly the assets it contains. Reusing a comfortable development environment answers whether yesterday’s downloads are still there. The deployment question is whether a new machine can do its first useful job without borrowing them.
The README also lists observation limits, including resolver bypasses and runtime coverage. Those limits leave work for a real deployment test. The attractive future for a tool this size is a place in that test: identify the unexpected request, bake in what is required, then repeat the actual user operation with networking unavailable.
The first run still needs its files.