Research & engineering

Speech-Driven Avatar Systems

Collaborative speech-to-avatar integration, streaming, and GPU deployment at MILab.

MILab and industry collaboration2025–presentNVIDIA Jetson AGX OrinUpdated 15 October 2026

Deployed engineering system. Collaborative work with Niloy; research-manuscript direction under exploration.

From conversational speech to a visual response

At North South University’s Machine Intelligence Lab (MILab), I work on speech-driven avatars through applied research and industry collaboration. My colleague Niloy and I have developed, deployed, and evaluated a custom PersonaPlex–IMTalker pipeline on NVIDIA Jetson AGX Orin. We are now exploring a research-manuscript direction based on this engineering work.

The systems connect conversational speech generation to facial motion and rendering while managing streaming, interruptions, and browser audio/video synchronization. The wider collaboration includes representation adapters, renderer studies and distillation, TensorRT acceleration, and profiling across server GPUs and Jetson hardware. My individual work includes distributed multi-GPU inference pipelines and an investigation of NVIDIA Audio2Face–Three.js integration and ARKit blendshape recovery.

We build on external models and pretrained checkpoints, including Moshi, PersonaPlex, IMTalker, MuseTalk, F5-TTS, Wav2Vec2, and GaussianTalker. Our work concerns how these components are adapted, connected, and evaluated under deployment constraints.

PersonaPlex and Moshi feed a custom hidden-state capture, speech gate, and adapter before an inherited IMTalker generator and renderer produce synchronized browser audio and video.
Figure 1. The collaborative bridge connects PersonaPlex reply representations to IMTalker’s conditioning space and a streaming visual response. The speech and rendering foundations are external models. View full-size figure

Renderer speed and system performance

Accelerating the renderer changed where the system spent its time. In an earlier fixed-identity Orin comparison, two distilled renderers were 19.84× and 27.62× faster than their teacher, shifting the dominant cost toward PersonaPlex. A power-mode configuration error invalidated the absolute timings attributed to 30 W in that study; the relative component comparison remains informative.

Later deployed-pipeline logs report approximately 66–67 ms per processing step on Orin. This is a runtime step measurement, not a measurement of the delay a user experiences between speaking and seeing a synchronized response. Long-session lag and client-side A/V alignment still need systematic evaluation. The current renderer studies also use a single visual identity, so they do not establish generalization across identities.

Earlier Jetson component comparison: faster fixed-identity renderers shift service time toward PersonaPlex; absolute 30 W timings are invalid.
Figure 2. An earlier relative component comparison shows how renderer acceleration moved the bottleneck upstream. Its absolute 30 W timings are invalid; it is separate from the later deployed-pipeline measurements. View full-size figure

Questions for the next phase

Deployment has made timing and evaluation central research questions: which speech information is available soon enough to drive articulation, and how should we measure a renderer that must respond continuously? Matching a teacher’s output does not by itself establish convincing articulation or a usable conversation.

We are developing these questions into a possible manuscript direction. The integration and deployment are completed engineering work; the proposed research claims require further experiments. A separate StreamTalker-GS investigation tested a covariance-transport hypothesis that I set aside after negative findings. Portions of the industry collaboration remain closed source.