  Multimodal conversational agents are highly desirable because they offer
natural and human-like interaction. However, there is a lack of comprehensive
end-to-end solutions to support collaborative development and benchmarking.
While proprietary systems like GPT-4o and Gemini demonstrating impressive
integration of audio, video, and text with response times of 200-250ms,
challenges remain in balancing latency, accuracy, cost, and data privacy. To
better understand and quantify these issues, we developed OpenOmni, an
open-source, end-to-end pipeline benchmarking tool that integrates advanced
technologies such as Speech-to-Text, Emotion Detection, Retrieval Augmented
Generation, Large Language Models, along with the ability to integrate
customized models. OpenOmni supports local and cloud deployment, ensuring data
privacy and supporting latency and accuracy benchmarking. This flexible
framework allows researchers to customize the pipeline, focusing on real
bottlenecks and facilitating rapid proof-of-concept development. OpenOmni can
significantly enhance applications like indoor assistance for visually impaired
individuals, advancing human-computer interaction. Our demonstration video is
available https://www.youtube.com/watch?v=zaSiT3clWqY, demo is available via
https://openomni.ai4wa.com, code is available via
https://github.com/AI4WA/OpenOmniFramework.
