Project: Using predictive coding networks to detect multimodal phoneme boundaries

[fram-noah]

Title

vowel-audiovis-pcn

Leaders

Noah R. Fram

Collaborators

Miriam Lense
Warren Jones
Camila Alviar

Project description

Many linguistic stimuli and samples are analyzed based on phonemes, or the smallest units of speech that contain meaning. One high-level form of this process involves labeling boundaries of vowels, sounds generated by filtering sound from the larynx without blocking the airflow, and distinguishing them from consonants, which do entail blocking airflow at some point. Human coders use a combination of auditory features, like the organization of vowel formants, and visual features, like mouth shape, to determine vowel boundaries. Characteristics of vowels including their timing are particularly important when analyzing child-directed speech and child-directed song. However, many naturalistic recordings of child-directed communication have occluded or degraded data in either the visual or auditory domain, making it challenging to code vowel boundaries.

This project aims to use predictive coding networks, a neuromorphic form of deep machine learning based on probabilistic inference, to resolve this issue. We will design a multimodal predictive coding network to predict vowel boundaries based on both auditory features and visual characteristics, train it on synchronized audiovisual recordings, and test how well it handles degradation or removal of auditory or visual signals. This will enable behavioral scientists to code a much wider range of ecologically valid child-directed communication signals, and will allow for more precise and reproducible characterization of audiovisual stimuli representing such signals.

Link to project repository/sources

Forthcoming

Concerete goals with specific tasks for Brainhack Vanderbilt 2026

1. Implement a basic multimodal predictive coding algorithm to detect vowel boundaries, capable of handling time-varying auditory and visual inputs, using the pcn package.
2. Train this model on synchronized audiovisual data.
3. Test the model on stimuli with one modality degraded or removed.

Good first issues

1. Issue one: Download the dataset, containing video recordings and existing vowel annotations, and explore it through visualizations.
2. Issue two: Install and start experimenting with the pcn package.
3. Issue three: Research potential audio processing approaches to prepare the dataset for use in pcn.

Skills

• Python: intermediate to advanced, including familiarity with handling audiovisual data. We will be using crossvalidation techniques to train predictive coding network models.
• Signal processing: beginner to intermediate. We will likely be using windowing and mel spectrograms for the audio portion.

Onboarding documentation

Forthcoming

What will participants learn?

Participants will learn:

• Predictive coding network models
• Differences between child-directed and adult-directed verbal communication
• Basic audio signal processing

Public data to use

We will be using recordings of child-directed singing, annotated for vowel boundaries.

Number of collaborators

3

Credit to collaborators

Project collaborators will be included as authors in the publication of this coding tool.

Image

Forthcoming

Project Summary

Use multimodal predictive coding networks to enable detection of vowel boundaries, even when data from one sensory modality is degrade or occluded

Type

method_development

Development status

0_concept_no_content

Topic

machine_learning

Tools

other

Programming language

Python

Modalities

behavioral

Git skills

2_branches_PRs

Anything else?

We will be using the Omni-PCN Python package developed by Nicole Dumont: https://github.com/nsdumont/omni-pcn.

Things to do after the project is submitted and ready to review.

• Add a comment below the main post of your issue saying: Hi @brainhack-vandy/project-monitors my project is ready!