Balancing quality and efficiency on mobile devices
Local language models support offline use and keep prompts on the device. Compression makes larger models fit in mobile memory, while deployment choices also shape response time, energy use, and answer quality.
PalmBench studies how model architecture, quantization, inference runtime, and device hardware interact. Measurements on deployed mobile systems connect model choices to the experience and resource costs of local inference.
Automated measurements on mobile hardware
The framework automates prompt execution and profiling across Android phones, iPhones, and Linux edge devices. Experiments use MLC-LLM and llama.cpp on supported platforms with different weight and activation quantization configurations. Profiling draws on Android Debug Bridge and Android GPU Inspector, Xcode Instruments, and edge-device tooling.
Systems measurements cover CPU and GPU activity, memory footprint, inference latency, throughput, and energy use. Question answering, multi-turn tasks, and hallucination and toxicity checks add the output-quality dimensions needed to compare deployment configurations.
- Measure resource use and execution behavior on the actual deployment platform.
- Compare compressed outputs with non-quantized references and task evaluations.
- Include harmful-output checks alongside efficiency and quality measurements.
Model and runtime choices shape the trade-off
Quantization changes memory traffic, GPU execution, power use, and output quality, with distinct behavior across models, runtimes, and platforms. Smaller models often deliver higher throughput in the tested configurations, while architecture and runtime choices also influence execution efficiency.
Answer quality, hallucination, and toxicity vary across quantization methods, including non-monotonic differences between nearby bit widths. The findings make task-level evaluation part of the compression decision alongside memory, latency, and energy measurements.
Choosing mobile model configurations
PalmBench organizes the deployment decision around combinations of model, quantization method, runtime, and hardware. Comparing these combinations across quality and systems metrics helps identify configurations suited to a device's memory and energy budget and an application's tasks.
This measurement-driven approach complements NanoMind's execution design. PalmBench characterizes compressed mobile LLM workloads, while NanoMind coordinates the accelerators and shared memory used by a multimodal pipeline.
Read the paper abstract
Deploying large language models (LLMs) locally on mobile devices is advantageous in scenarios where transmitting data to remote cloud servers is either undesirable due to privacy concerns or impractical due to network connectivity. Recent advancements have facilitated the local deployment of LLMs. However, local deployment also presents challenges, particularly in balancing the (generative) quality, latency, and throughput within the hardware constraints of mobile devices. In this paper, we introduce our lightweight, all-in-one automated benchmarking framework that allows users to evaluate LLMs on mobile devices. We provide a comprehensive benchmark of various popular LLMs with different quantization configurations (both weights and activations), across multiple mobile platforms with varying hardware capabilities. Unlike traditional benchmarks that assess full-scale models on high-end GPU clusters, we focus on evaluating resource efficiency (memory and power consumption) and harmful output for compressed models on mobile devices. Our key observations include: i) differences in energy efficiency and throughput across mobile platforms; ii) the impact of quantization on memory usage, GPU execution time, and power consumption; iii) accuracy and performance degradation of quantized models compared to their non-quantized counterparts; and iv) the frequency of hallucinations and toxic content generated by compressed LLMs on mobile devices.
