PNEUMA — Video Memory for Long-Form Question Answering

PNEUMA was my undergraduate thesis, completed in December 2025 as part of a four-person team under Dr. Nabeel Mohammed at North South University. We investigated how a long video could remain accessible through actions, objects, places, audio, and temporal relationships, rather than only through a single summary.

We built an integration and memory system around frozen pretrained perception and language models. My experience with the project led me to ask more precise questions about evidence retention and retrieval: what information survives processing, what can a query recover, and which retrieved evidence actually supports an answer? These questions later informed ERL.

The full-video path processes overlapping three-second chunks with a one-second overlap. Model outputs are normalized into SQL records describing chunks, objects, text, and occurrences. Five FAISS indices represent action, object, spatial, audio, and visual descriptions using projected CLIP text embeddings. The visual-description index therefore retrieves textual descriptions of appearance, rather than directly indexing raw visual features.

A NetworkX graph connects chunks and entities to day, week, and month nodes, with links for temporal succession and entity occurrence. Query interpretation extracts intent and time constraints; a planner combines structured lookup, vector search, and graph traversal as appropriate. Vector results are combined by weighted reciprocal-rank fusion, then filtered and summarized for answer generation. Some structured queries can be answered without the language model.

SQL, vector indices, and the graph share chunk identifiers, but their update and reload paths are not fully coordinated. This matters for whether a memory remains retrievable after a restart. Likewise, a rich textual description is only useful if the original perception output was visually grounded.

We implemented an evaluation framework for long-video question answering, but reliable final aggregate accuracy is not established. I therefore do not report benchmark accuracy for the thesis. Retrieval consistency, persistence across sessions, and visual grounding remain the main limitations to resolve before drawing conclusions about answer quality.

The undergraduate thesis is complete. I am considering architectural renovation and renewed evaluation as separate subsequent work, beginning with more dependable memory retrieval and direct checks that answers are supported by the video.