Edge AI on Android

Edge AI on Android

From Intermediate to On-Device Genius in 2 days

Your phone, smarter - vision, chat, logic, all in your pocket.

You already build Android apps. What you don’t have yet is the superpower of making them think. Over two days, you’ll fuse camera pipelines, ML inference, local LLMs, and remote LLMs into real apps.

No slides about linear regression - you’ll wrangle TensorFlow Lite, MediaPipe, quantization, model conversion, prompt engineering, fallbacks, and hybrid AI architectures. The instructor’s decades in industry mean you’ll hear war stories, pro tips, what fails in real life — not just what works in demos.

Learning Outcomes

By the end of the bootcamp, each participant will be able to:

• Integrate TensorFlow Lite (or alternate runtimes) into Android for on-device inference
• Build camera + vision pipelines (real-time image processing) feeding ML models
• Load and inference small LLMs locally (e.g. via MediaPipe LLM Inference API)
• Connect to remote LLMs (OpenAI, etc.), manage prompt, streaming, fallback
• Architect a hybrid AI system: when to run locally, when to call cloud
• Optimize models: quantization, delegates, threading, memory constraints
• Debug, profile, and tune AI parts in real Android apps
• Deliver a mini demo: e.g. camera-based AI + chat feature

Prerequisites (What You Must Already Know)

• Solid Android app dev (Kotlin / Java, Android SDK, lifecycle, concurrency)
• Experience with Android camera APIs (Camera2, CameraX)
• Familiarity with REST APIs, JSON, networking, threading
• Basic understanding of ML/Neural Networks, inference, quantization (helpful, not mandatory)
• Experience using Android Studio, profiling tools

Training Guideline

1. Edge AI Architecture for Android

• Why AI on device matters: latency, privacy, offline, cost
• Hybrid AI patterns: local-first, fallback, caching
• Resource tradeoffs: CPU / GPU / NPU / memory / battery
• Example app architectures: modular inference layers, model management
• Setting up your project scaffolding: two demo apps (vision + chat)

2. TensorFlow Lite (TFLite) & Inference on Android

• Choosing runtimes / frameworks: TFLite, ONNX, nn-API delegates
• Converting models (TensorFlow / Keras → .tflite)
• Quantization: post-training, dynamic, integer, 8-bit / 4-bit
• Loading models, allocating tensors, buffer layout
• Using delegates: GPU delegate, NNAPI delegate, custom delegates
• Multiple inputs / outputs, dynamic shapes
• Profiling inference time, memory usage, warm-up strategies
• Handling errors, model fallback

3. Camera + Vision Pipelines

• Capturing frames: CameraX, ImageAnalysis, preview, YUV → RGB conversion
• Real-time vs batched inference strategies
• Preprocessing: resizing, normalization, cropping, batching
• Overlay UI: bounding boxes, masks, segment overlays
• Example mini features:
  • Object detection (bounding boxes)
  • Pose / keypoints
  • Segmentation / background removal / filters
• Balancing frame rate and inference cost

4. Local LLMs / SLMs on Android

• What is feasible on Android (model size, quantization)
• MediaPipe LLM Inference API for Android (run LLMs on device)
  • Adding com.google.mediapipe:tasks-genai dependency
  • Pushing and loading .task or quantized model files
  • Using generateResponse() and generateResponseAsync()
  • Multimodal prompting: text + image support
• Example small models: Gemma-3 (1B), quantized 4-bit models
• Tokenization, context window, streaming partial results
• Model conversion, LoRA fine-tuning, combining LoRA with base model
• Fallback logic: local → remote when input too large

5. Remote LLM API Integration

• API design: prompt format, streaming, chunking, rate limits
• Secure credential storage: Android keystore, encryption
• Asynchronous calls, cancellations, partial streaming UI
• Prompt engineering for mobile: template, trimming, summarization
• Combining local + remote: caching, merging responses, fallback

6. Hybrid AI Application: Putting It All Together

• App architecture: model manager, inference layer, fallback logic
• Demo app ideas:
  • “Snap & Chat”: user takes a photo, run object detection + caption locally, then ask LLM to expand / generate narrative
  • “On-device Assistant + Cloud Boost”: first try local LLM, if context too large / slow, escalate to remote
• UI flows: progress feedback, graceful fallback, streaming UI elements
• Logging, metrics, versioning, model download over the air

7. Optimization, Profiling & Best Practices

• Android Profiler, Memory Profiler, CPU / GPU traces
• Reducing memory footprint: load / unload models, pooling, reuse buffers
• Threading and batching strategies
• Delegate tuning: NNAPI, GPU, fallback
• Quantization pruning, model splitting
• Heuristics for fallback, caches, summarization
• Edge cases: OOM, overheating, battery mitigation

8. Final Demos, Review & Roadmap

• Participants demo their apps
• Group code review, critique, lessons learned
• Future directions: federated learning, model updates, cross-device inference
• Resources: Hugging Face, MediaPipe, model hubs, conversion tools
• How to package and distribute AI apps on Play Store (asset size, licensing, disclaimers)

NOTE: Versions and dependencies are always changing. As such, the proposed apps, libraries and topics may be modified by the trainer as needed.