Delivering Accurate, Low-Latency Voice-to-Form AI in Real-World Field Conditions
Summary
Salesforce's Field Service Mobile app uses a hybrid on-device and cloud AI system to convert technicians' natural voice input into accurate, structured form data, even in noisy, real-world environments. By keeping speech-to-text on the device and leveraging cloud-based large language models for semantic mapping, the solution balances latency, cost, and privacy while supporting diverse accents and complex form schemas. Field technicians can quickly capture data hands-free without retraining, increasing efficiency and reliability. This approach offers a production-ready model for integrating voice-driven data capture seamlessly into existing workflows at enterprise scale.
Takeaways
- Use a hybrid on-device speech-to-text and cloud LLM architecture for voice data capture.
- Embed semantic metadata in prompts to improve AI field mapping accuracy across schemas.
- Maintain transcription on device to reduce latency, cost, and protect privacy.
- Design voice workflows to be intuitive, with edit and undo controls for field technicians.
- Test voice AI reliability using real-world diverse voices and noise profiles.
In our Engineering Energizers Q&A series, we highlight the engineering minds driving innovation across Salesforce. Today, we feature Rajashree Pimpalkhare, SVP of Software Engineering, Field Service, and the team responsible for voice-to-form data capture in the Field Service Mobile application, which delivers AI-powered mobile experiences to a field workforce supporting hundreds of thousands of active technicians each month. Discover how her team developed a hybrid on-device and cloud architecture to accurately translate unstructured voice input into structured form data at an enterprise scale, ensured reliable performance across various accents and noisy field conditions through real-world voice testing, and managed latency, cost, and privacy by keeping speech-to-text on the device while leveraging cloud LLMs for intelligent field mapping. AI-driven data flow process diagram.