Cloud models keep improving. That does not mean every dataset should leave the building. When contracts, regulation, or customer agreements forbid external processing, local or on-prem inference is a control decision—not a fashion choice. The tradeoff is capability, ops burden, and how fast you can iterate.
Start from the constraint, not the vibe
Ask what must not leave: identifiers, source code, health or financial records, unpublished product data. If the answer is “almost everything,” design for local inference or a private deployment with clear retention and access rules.
If only a subset is restricted, split the workflow. Public or synthetic steps can use stronger cloud models; sensitive steps stay local.
Capability versus control
Frontier cloud models usually win on reasoning, tool use, and multilingual quality. Local open-weight models win on residency, air-gapping, and predictable cost at high volume—once you own the hardware and the ops.
Measure the actual tasks. A model that is “worse” on a leaderboard can be enough for classification, redaction, or retrieval-augmented drafting inside your boundary.
Own evaluation and updates
Local does not mean set-and-forget. You still need an evaluation set, version pins, rollback, and a patch plan when weights or runtimes change. Shadow AI with an unmanaged USB model is not compliance.
Document who can pull new weights, who approves promotion, and how prompts and logs are retained.
Hybrid is often the honest answer
Many teams keep a strong cloud path for non-sensitive work and a local path for restricted inputs. Route by data class, not by preference. The governance note applies here: approved tools, data classes, review for external actions, logging.