Start from the device, not the model
Computer vision projects on constrained hardware usually begin with a model that performs well on a benchmark and a workstation, and then try to make it fit a phone or an embedded board. Most of the difficulty that follows comes from that order. The device sets hard limits — how long an inference may take, how much memory the model may hold, which operations the accelerator can run, how much battery the feature may use and how hot the device may get — and a model chosen before those limits are known will usually break at least one of them.
So the first artefact of the project is not a model but a device budget. Name the lowest-end device the product must support, not the phone on the engineer's desk. Record its accelerator, its memory, and the latency, battery and thermal limits the feature has to live within. Every later decision is then a trade inside that budget, and it can be measured rather than argued about.
A worked budget
Suppose a field-inspection app must flag a defect in a live camera view, and the lowest supported device is a three-year-old mid-range Android phone. The inspector can wait a quarter of a second for a result, and uses the feature for twenty minutes at a stretch. That sets the budget: about 250 milliseconds per result including pre-processing, measured as a sustained figure rather than a peak, and a battery cost that still leaves the phone usable at the end of a working day.
A detector that needs 400 milliseconds on that phone is ruled out before accuracy is discussed. Running it on every third frame, as a quantised model, at a lower input resolution, with the camera conversion moved into native code, might bring it inside the budget — and each of those steps is then checked against per-class accuracy on images from the field, not against the benchmark the model came from.
A reference pipeline
The reference architecture below is the pipeline we start from. Each stage names the decision, the default we reach for first, and the measurement that decides whether the default holds.
Reference architecture
Stage | Decision | Where we start | What to measure |
|---|---|---|---|
Device budget | The lowest supported device and its limits | The oldest device in the field, not the test phone | p95 latency, sustained frame rate, battery per hour |
Architecture | A model family built for mobile | Compact classifiers and small single-stage detectors | Accuracy on field data at the chosen size |
Input | Resolution and crop | The smallest resolution that keeps the target legible | Accuracy against resolution, per class |
Quantisation | Integer weights and activations | Post-training INT8 first; quantisation-aware training if accuracy drops | Accuracy change per class, not just on average |
Compression | Distillation and pruning | Distil into the small model before pruning it | Latency gained against accuracy lost |
Runtime | Framework and accelerator | LiteRT on Android, Core ML on iOS, ONNX Runtime where both are needed | Which operations fall back to the CPU |
Pipeline | Pre- and post-processing | Native code or the accelerator, never an interpreted loop | Time spent outside the model |
Scheduling | How often the model runs | Frame skipping, regions of interest, a cheap first-stage filter | Frames processed per useful result |
Thermals | Sustained performance | Measure after ten minutes of continuous use | Throttled latency and device temperature |
Why integer quantisation comes first
Converting a model's weights and activations from 32-bit floating point to 8-bit integers makes it roughly four times smaller and, on hardware with integer accelerators, substantially faster. Neural processing units and digital signal processors in mid-range devices are built for integer arithmetic, so a floating-point model often cannot use them at all.
Post-training quantisation, which needs only a representative calibration set, is the cheap first step. The accuracy cost is often small, but it is not evenly spread: rare classes and small objects tend to lose the most. Measure the change per class on field data, and move to quantisation-aware training only when post-training quantisation costs accuracy the product cannot afford.
The operations that fall back
An accelerator runs the operations its driver supports and hands the rest back to the CPU. One unsupported layer in the middle of a network can split the model into pieces that shuttle data between processors, and that traffic can cost more than the accelerator saves. Before committing to an architecture, run it through the target runtime and list which operations execute where. Replacing a single exotic activation function or layer type with a supported one is often worth more than any amount of compression.
The runtime landscape also moves. TensorFlow Lite is now LiteRT, and Android's Neural Networks API was deprecated from Android 15, so acceleration on Android increasingly goes through LiteRT's delegates and the vendors' own. Choose the runtime for the devices in the field, and expect to revisit that choice.
Pre-processing is where the time goes
Profiles of slow on-device vision features often show the model taking a minority of the frame time. The rest goes on converting camera frames between colour formats, resizing, normalising and copying buffers — and afterwards on decoding detections and suppressing overlaps. Moving these steps into native code or onto the accelerator, and avoiding copies between them, regularly recovers more latency than a smaller model would.
Heat and battery
A phone that runs a vision model continuously will warm up, and the operating system will slow the processor to protect it. A pipeline that meets its latency target in a thirty-second test can miss it by a wide margin after ten minutes. Measure sustained performance, not peak.
Scheduling does the most for both heat and battery. Most applications do not need every frame: run the model on a fraction of them, restrict it to a region of interest, or put a cheap first-stage filter in front that wakes the full model only when something is there.
Evaluate on field data
Benchmark accuracy describes a benchmark. Devices in the field see motion blur, poor light, dirty lenses, unusual angles and objects the training set never contained. Collect evaluation data from the real environment early, label it, and make it the test that decides every trade in the table above. Keep a held-out set from each site or season the product serves, because conditions differ between them more than any augmentation can imitate. A model that loses a point on a public benchmark but holds up on field data is the better model.
Shipping and updating models
Ship the model as a versioned asset that can be updated separately from the app, with the version recorded against every result it produces. That makes it possible to roll a model forward to a fraction of devices, compare, and roll back without a store release.
Our own work includes machine vision that runs with zero signal, where every one of these constraints applies at once. It is part of our Applied AI practice, and a constraint we meet most in agriculture technology, where devices are rugged, mid-range and far from any network.