Multimodal inference is a systems problem
Multimodal models combine vision and audio encoders, projectors, and language generation, each with different computational needs. Mapping the whole pipeline to one accelerator leaves other compute units idle and adds memory traffic and latency.
On a small battery-powered device, efficient execution determines how much sensing and inference can fit within the energy and memory budget. NanoMind co-designs the hardware, scheduling, and data movement to make local multimodal assistance practical.
Custom hardware and accelerator-aware software
The custom prototype uses an RK3566 system-on-chip with an integrated NPU and GPU, shared LPDDR4x memory, and a power-management unit. NanoMind decomposes the model into independently executable modules, placing vision encoding on the NPU and language decoding on the GPU.
The Token-Aware Buffer Manager (TABM) coordinates a shared ring buffer so the NPU can write embeddings that the GPU consumes directly, enabling zero-copy transfers. Fused low-bit GEMM kernels and battery-aware scheduling support efficient local execution. In low-power mode, an event-driven cascade loads, executes, and releases each module in sequence.
- Schedule vision, projector, language, and audio modules around accelerator strengths.
- Transfer embeddings through shared memory with TABM zero-copy buffers.
- Adapt execution to battery level through continuous and on-demand modes.
Energy-efficient local multimodal inference
The published paper reports 42.3% lower end-to-end energy consumption against the evaluated mainstream edge frameworks and devkits. For LLaVA-OneVision-Qwen2-0.5B with a camera and a 2,000 mAh battery, it estimates nearly 18.8 hours of operation in on-demand low-power mode under low-duty-cycle, event-triggered workloads.
The system breakdown shows how zero-copy buffering reduces memory use and CPU overhead during embedding transfer, while accelerator-specific placement improves vision encoding. Together, the results connect module-level execution decisions to the energy cost of the complete assistant.
From local inference to persistent assistance
NanoMind brings perception and language generation into a compact, self-contained device that operates offline. Its hardware-software design provides an execution platform for assistants that process visual and audio input locally.
Virgile builds on the NanoMind hardware platform for persistent object and face recognition. PalmBench complements this systems work by measuring the quality, resource, and energy trade-offs of compressed language models on mobile hardware.
Read the paper abstract
Large Multimodal Models (LMMs) are inherently modular, comprising vision and audio encoders, a projector, and a language backbone. Yet existing systems execute them monolithically, underutilizing the heterogeneous accelerators (NPUs, GPUs, DSPs) on modern SoCs and inflating end-to-end latency. We present NANOMIND, a hardware-software co-design inference framework that decomposes each LMM into modular bricks - vision, projector, language, and audio - and maps each brick to its best-suited compute units. A Token-Aware Buffer Manager (TABM) enables zero-copy embedding transfer across accelerators on unified-memory SoCs, bypassing CPU bottlenecks. Combined with customized hardware, a battery-aware scheduler, and fused low-bit GEMM kernels, NANOMIND runs entirely on a compact, battery-powered prototype that operates fully offline. NANOMIND reduces end-to-end energy by 42.3% against mainstream edge frameworks and devkits; in its on-demand low-power mode, the prototype runs LLaVA-OneVision-Qwen2-0.5B with a camera for nearly 18.8 hours on a single 2,000 mAh battery.
