Research & engineering

StreamTalker-GS — Speech-Driven 3DGS Experiments

An experimental investigation of affine covariance transport in FLAME-driven Gaussian avatars.

Completed investigationThree tested identitiesNegative findingUpdated 15 October 2026

Completed study. The improvement hypothesis was not supported and has been set aside.

Testing a geometric hypothesis

I investigated whether affine covariance transport could improve FLAME-driven Gaussian avatar rendering. The motivation was geometric: a local deformation can change a Gaussian’s shape in ways that a similarity transform cannot express. For that distinction to help rendering, however, it also needed to relate to the image residual we were trying to reduce.

This investigation grew out of our collaborative speech-avatar systems work. It built on Moshi/Mimi, GaussianTalker, FLAME 2020, Pixel3DMM, HuBERT, and existing Gaussian rasterization methods. I focused on the covariance-transport question and its experimental interpretation; the broader streaming and conditioning work belongs to the collaboration.

Covariance-transport study: geometric differences were measured, residual association was unsupported across three identities, and the causal renderer swap was not executed. — Figure 1. The study measured geometric differences and their relationship to rendering residuals. The association was not supported across the three tested identities, so the proposed renderer intervention was not run. View full-size figure

What the experiments showed

Across three identities and approximately 1.04 million supported observations, affine and similarity transport produced measurable differences in projected covariance. Those differences did not consistently track the tested rendering residual.

On a small screen, scroll the table horizontally or focus it and use the arrow keys.

Measure	Observation across the three identities
Projected covariance difference of at least 0.5 px	31.669%, 37.731%, and 39.493% of supported visible observations
Spearman association with the rendering residual	−0.8971, +0.3212, and −0.1594; none met the study’s required association criterion

Separate analytic-to-rasterizer controls agreed to a maximum radial RMS difference of 5.02×10⁻⁷ px and responded monotonically to filter and overlap changes. These checks gave confidence in the geometric measurements, but did not demonstrate an improvement to avatar rendering.

Why I set the hypothesis aside

The geometric difference alone was insufficient motivation for the next intervention. I did not complete the causal H4 renderer swap or the broader supervision-transfer program, and the experiments established no rendering improvement. I therefore set aside the covariance-transport hypothesis.

The useful lesson was to test the link between a plausible geometric mechanism and the error it is meant to explain before investing in a larger training program. This is a completed experimental investigation with a negative finding.