Introduction to Phi-3 Mini
Phi-3 Mini is Microsoft's compact yet powerful small language model (SLM) from the Phi-3 family. With only 3.8 billion parameters, it delivers reasoning capabilities that rival much larger models while being small enough to run entirely on-device. This makes it an ideal candidate for mobile applications where latency, privacy, and offline availability matter.
The model was trained on synthetic data and filtered web content using a curriculum approach inspired by textbook learning. Despite its small footprint (around 2.3 GB quantized), Phi-3 Mini scores impressively on benchmarks like MMLU, HumanEval, and common reasoning tasks, often outperforming models twice its size.
Why Run Language Models on Mobile?
Traditionally, language model inference has been offloaded to cloud servers. However, on-device inference offers several compelling advantages that are reshaping how developers build AI-powered applications.
Key Benefits
- Privacy: User data never leaves the device, eliminating concerns about sensitive information being transmitted or stored on remote servers.
- Latency: No network round-trips mean responses can be generated in milliseconds rather than seconds.
- Offline availability: Applications continue to function without an internet connection, critical for travel, remote areas, or unreliable networks.
- Cost savings: No per-token API charges or server infrastructure to maintain.
- Battery-aware design: Modern mobile NPUs are highly optimized for the kind of matrix math LLMs require.
Phi-3 Mini Architecture and Variants
Phi-3 Mini is a decoder-only transformer with a 4K default context window (extendable to 128K with the long-context variant). It uses grouped-query attention and relative position embeddings for efficiency. Microsoft releases several quantized variants optimized for different deployment targets:
phi-3-mini-4k-instruct— Full precision, 4K contextphi-3-mini-4k-instruct-onnx— ONNX format for cross-platform inferencephi-3-mini-128k-instruct— Long context variant- INT4 and INT8 quantized versions for mobile and edge devices
Setting Up Your Environment
For mobile deployment, the most common approach is to use ONNX Runtime, which provides optimized inference across iOS, Android, and Windows devices. Let's walk through the setup for both major mobile platforms.
Prerequisites
- Python 3.9+ for model conversion and quantization
- Android Studio (for Android) or Xcode (for iOS)
- ONNX Runtime mobile packages
- Approximately 4 GB of free disk space for the model files
Downloading and Preparing the Model
First, download the ONNX-optimized Phi-3 Mini model from Hugging Face. Microsoft provides pre-built ONNX packages that include the necessary tokenizer and configuration files.
# Install required Python packages
pip install onnxruntime-genai huggingface-hub
# Download the ONNX model for mobile
python -c "
from huggingface_hub import snapshot_download
snapshot_download(
repo_id='microsoft/Phi-3-mini-4k-instruct-onnx',
local_dir='./phi3-mini-onnx',
allow_patterns=['cpu-int4/*', 'tokenizer*', 'config.json']
)
print('Download complete')
"
The cpu-int4 variant is the most suitable for mobile devices, reducing the model size to roughly 2.3 GB while maintaining most of the model's quality.
Running Phi-3 Mini on Android
Adding Dependencies
In your Android project, add the ONNX Runtime Android dependency to your build.gradle file:
// app/build.gradle.kts
dependencies {
implementation("com.microsoft.onnxruntime:onnxruntime-android:1.17.0")
implementation("com.microsoft.onnxruntime:onnxruntime-extensions-android:0.10.0")
}
Loading the Model
Place the downloaded ONNX model files in your app's assets folder. The following Kotlin code demonstrates how to load and run inference:
import ai.onnxruntime.OnnxEnvironment
import ai.onnxruntime.OrtEnvironment
import ai.onnxruntime.OrtSession
import android.content.Context
class Phi3Engine(context: Context) {
private val environment: OrtEnvironment = OrtEnvironment.getEnvironment()
private var session: OrtSession? = null
init {
val modelBytes = context.assets.open("phi3-mini-int4.onnx").use {
it.readBytes()
}
val options = OrtSession.SessionOptions().apply {
setIntraOpNumThreads(4)
setOptimizationLevel(OrtSession.SessionOptions.OptLevel.ALL_OPT)
}
session = environment.createSession(modelBytes, options)
}
fun generate(prompt: String, maxTokens: Int = 256): String {
val tokens = tokenize(prompt)
val generated = StringBuilder()
for (i in 0 until maxTokens) {
val inputIds = LongArray(tokens.size + generated.length) { idx ->
if (idx < tokens.size) tokens[idx].toLong()
else generatedTokens[idx - tokens.size].toLong()
}
val attentionMask = LongArray(inputIds.size) { 1L }
val inputShape = longArrayOf(1, inputIds.size.toLong())
val inputIdsTensor = OnnxTensor.createTensor(
environment, LongBuffer.wrap(inputIds), inputShape
)
val attentionTensor = OnnxTensor.createTensor(
environment, LongBuffer.wrap(attentionMask), inputShape
)
val inputs = mapOf(
"input_ids" to inputIdsTensor,
"attention_mask" to attentionTensor
)
val outputs = session?.run(inputs)
val logits = outputs?.get(0)?.value as Array<Array<FloatArray>>
val nextToken = argmax(logits[0][logits[0].size - 1])
if (nextToken == eosTokenId) break
generated.append(detokenize(nextToken))
}
return generated.toString()
}
fun close() {
session?.close()
}
}
Integrating with the UI
Because inference is CPU-intensive, always run it on a background thread to avoid blocking the UI thread:
class ChatViewModel(private val phi3: Phi3Engine) : ViewModel() {
private val _response = MutableStateFlow("")
val response: StateFlow<String> = _response
fun sendPrompt(prompt: String) {
viewModelScope.launch(Dispatchers.Default) {
val result = phi3.generate(prompt, maxTokens = 200)
withContext(Dispatchers.Main) {
_response.value = result
}
}
}
}
Running Phi-3 Mini on iOS
Adding the Framework
For iOS, use Swift Package Manager to add ONNX Runtime:
// Package.swift or via Xcode > File > Add Packages
// URL: https://github.com/microsoft/onnxruntime-swift-package-manager
.package(
url: "https://github.com/microsoft/onnxruntime-swift-package-manager",
from: "1.17.0"
)
Swift Implementation
import OnnxRuntimeBindings
import Foundation
class Phi3Engine {
private var env: OrtEnv
private var session: OrtSession
private let tokenizer: Tokenizer
init?(modelPath: String) {
guard let env = try? OrtEnv(loggingLevel: .warning) else { return nil }
self.env = env
let options = try? OrtSessionOptions()
try? options?.setIntraOpNumThreads(4)
try? options?.setGraphOptimizationLevel(.all)
guard let session = try? OrtSession(
env: env,
modelPath: modelPath,
sessionOptions: options
) else { return nil }
self.session = session
self.tokenizer = Tokenizer.loadFromBundle(name: "tokenizer")
}
func generate(prompt: String, maxTokens: Int = 256) -> String {
var tokens = tokenizer.encode(prompt)
var generatedText = ""
for _ in 0..<maxTokens {
let inputIds: [Int64] = tokens.map { Int64($0) }
let attentionMask: [Int64] = Array(repeating: 1, count: inputIds.count)
let shape: [Int64] = [1, Int64(inputIds.count)]
guard let inputIdsTensor = try? OrtValue(
tensorData: MutablePointerArray(inputIds),
elementType: .int64,
shape: shape
) else { continue }
guard let attentionTensor = try? OrtValue(
tensorData: MutablePointerArray(attentionMask),
elementType: .int64,
shape: shape
) else { continue }
let inputs: [String: OrtValue] = [
"input_ids": inputIdsTensor,
"attention_mask": attentionTensor
]
guard let outputs = try? session.run(
inputs: inputs,
outputNames: ["logits"]
) else { continue }
// Extract logits and find next token
let nextToken = extractNextToken(from: outputs["logits"]!)
if nextToken == tokenizer.eosTokenId { break }
tokens.append(nextToken)
generatedText += tokenizer.decode([nextToken])
}
return generatedText
}
}
Using Apple's MLX Framework (Alternative)
For Apple Silicon devices, the MLX framework provides optimized inference that leverages the unified memory architecture and Neural Engine:
import MLX
import MLXLMCommon
@MainActor
class Phi3MLXEngine {
private var model: Phi3Model?
private var tokenizer: Tokenizer?
init() async {
do {
let modelFactory = ModelFactory.shared
let modelContainer = try await modelFactory.loadContainer(
configuration: ModelConfiguration(
id: "mlx-community/Phi-3-mini-4k-instruct-4bit"
)
)
self.model = try await modelContainer.perform { context in
context.model
}
self.tokenizer = try await modelContainer.perform { context in
context.tokenizer
}
} catch {
print("Failed to load model: \(error)")
}
}
func generate(prompt: String) async -> String {
guard let model = model, let tokenizer = tokenizer else { return "" }
let tokens = try? tokenizer.encode(text: prompt)
var generatedTokens: [Int] = []
for _ in 0..<256 {
// Run forward pass
let logits = model.forward(tokens: tokens + generatedTokens)
let nextToken = argmax(logits)
if nextToken == tokenizer.eosTokenId { break }
generatedTokens.append(nextToken)
}
return tokenizer.decode(tokens: generatedTokens)
}
}
Using ONNX Runtime GenAI
Microsoft provides a higher-level library called ONNX Runtime GenAI that abstracts away much of the manual token management. This is the recommended approach for production applications:
# Python example for testing before mobile deployment
import onnxruntime_genai as og
model_path = "./phi3-mini-onnx/cpu-int4"
print("Loading model...")
model = og.Model(model_path)
tokenizer = og.Tokenizer(model)
tokenizer_stream = tokenizer.create_stream()
prompt = "<|user|>Explain quantum computing simply<|end|>\n<|assistant|>"
input_tokens = tokenizer.encode(prompt)
params = og.GeneratorParams(model)
params.set_search_options(max_length=512, temperature=0.7)
params.input_ids = input_tokens
generator = og.Generator(model, params)
print("Generating response...")
while not generator.is_done():
generator.compute_logits()
generator.generate_next_token()
new_token = generator.get_next_tokens()[0]
print(tokenizer_stream.decode(new_token), end="", flush=True)
print("\nDone.")
This same API is available through C bindings on both Android and iOS, making it straightforward to port the logic to mobile platforms.
Performance Optimization
Quantization Strategies
Choosing the right quantization level is critical for mobile performance. Here is a comparison of the available options:
- FP16 (full precision): ~7.6 GB, best quality, too large for most mobile devices
- INT8: ~3.8 GB, minimal quality loss, suitable for high-end devices with 8+ GB RAM
- INT4: ~2.3 GB, slight quality degradation, recommended for most mobile deployments
- INT4 with group quantization: ~2.1 GB, best balance for older devices
Memory Management
Mobile devices have limited RAM, and language models are memory-hungry. Follow these practices to avoid out-of-memory crashes:
// Android: Use memory-mapped file loading instead of loading into RAM
val options = OrtSession.SessionOptions().apply {
setIntraOpNumThreads(4)
setOptimizationLevel(OrtSession.SessionOptions.OptLevel.ALL_OPT)
// Enable memory mapping for large models
setMemoryPatternOptimization(true)
// Use the CPU execution provider with arena allocator
addConfigEntry("session.use_env_allocators", "1")
}
// Free model resources when not in use
override fun onDestroy() {
super.onDestroy()
phi3Engine.close()
System.gc()
}
Threading Considerations
Mobile CPUs typically have a mix of performance and efficiency cores. Configure thread count based on the device:
fun getOptimalThreadCount(): Int {
val cores = Runtime.getRuntime().availableProcessors()
// Use performance cores only, typically half the total
return (cores / 2).coerceIn(2, 6)
}
Best Practices for Mobile LLM Deployment
Prompt Engineering for Small Models
Phi-3 Mini responds best to clear, structured prompts. Use the model's chat template format for optimal results:
<|system|>
You are a helpful assistant. Answer concisely.
<|end|>
<|user|>
What is the capital of France?
<|end|>
<|assistant|>
Caching and State Management
For chat applications, implement KV-cache to avoid recomputing attention over previous tokens. ONNX Runtime supports this through the past key-value states mechanism:
class CachedGenerator {
private var pastKeyValues: Map<String, OnnxTensor>? = null
fun generateStep(inputIds: LongArray): Int {
val inputs = mutableMapOf<String, OnnxTensor>()
inputs["input_ids"] = createTensor(inputIds)
// Include cached key-values if available
pastKeyValues?.forEach { (key, value) ->
inputs["past_$key"] = value
}
val outputs = session.run(inputs)
// Update cache for next iteration
pastKeyValues = extractKeyValues(outputs)
return argmax(outputs.get("logits"))
}
}
Handling Long Conversations
With a 4K context window, long conversations will eventually exceed the limit. Implement a sliding window or summarization strategy:
fun manageContext(messages: List<ChatMessage>, maxTokens: Int = 3500): List<ChatMessage> {
var totalTokens = messages.sumOf { estimateTokens(it.content) }
while (totalTokens > maxTokens && messages.size > 2) {
// Remove oldest non-system message
val removed = messages.firstOrNull { it.role != "system" }
totalTokens -= estimateTokens(removed?.content ?: "")
}
return messages
}
Battery and Thermal Awareness
Continuous LLM inference can drain battery quickly and cause thermal throttling. Monitor device state and adjust accordingly:
// iOS: Check thermal state before heavy inference
import UIKit
func canRunInference() -> Bool {
let thermalState = ProcessInfo.processInfo.thermalState
switch thermalState {
case .nominal, .fair:
return true
case .serious:
return false // Defer or reduce token count
case .critical:
return false
@unknown default:
return true
}
}
Testing and Benchmarking
Before shipping, benchmark your implementation across target devices. Key metrics to track include tokens per second, time to first token, peak memory usage, and battery impact. Create a simple benchmarking harness:
fun benchmark(model: Phi3Engine, prompts: List<String>) {
val results = prompts.map { prompt ->
val startTime = System.nanoTime()
val response = model.generate(prompt, maxTokens = 100)
val elapsed = (System.nanoTime() - startTime) / 1_000_000
val tokenCount = estimateTokens(response)
val tokensPerSecond = (tokenCount.toDouble() / elapsed) * 1000
BenchmarkResult(
prompt = prompt,
elapsedMs = elapsed,
tokensPerSecond = tokensPerSecond,
response = response
)
}
results.forEach { println(it) }
println("Average tokens/sec: ${results.map { it.tokensPerSecond }.average()}")
}
Typical performance on modern flagship devices ranges from 8 to 20 tokens per second with the INT4 quantized model, which provides a responsive user experience for most conversational applications.
Conclusion
Phi-3 Mini represents a significant shift in how developers can approach AI-powered mobile applications. By running a capable language model entirely on-device, you gain privacy, offline capability, and predictable performance without recurring API costs. The combination of ONNX Runtime's cross-platform support and Phi-3's efficient architecture makes it practical to deploy today on both Android and iOS. Start with the INT4 quantized variant, implement proper memory management and threading, and always benchmark on your target devices. As mobile hardware continues to improve with dedicated NPUs and increased RAM, on-device language models will only become more powerful, and Phi-3 Mini gives you a production-ready entry point into this growing ecosystem.