Gesamtkunstvektoren: Perceptual Embeddings for the Performing Arts

Co-sponsored by the Berkeley Institute for Data Science and the School of Information

Deep neural models capable of encoding semantic aspects of multiple perceptual modalities offer exciting new opportunities for computational analyses of cultural expressive forms in the performing arts, particularly those that derive much of their effectiveness from the layering of different forms of media. 

This presentation will detail efforts to expand existing inquiries into the analysis of recorded theater performances via the application of new technologies such as Meta’s Perception Encoder Audiovisual (PE-AV) family of models, which can generate aligned semantic vector embeddings of video, audio, and language modalities singly or jointly. These models raise the possibility of being able to augment computationally the consideration of works through the lens of the Wagnerian Gesamtkunstwerk, the notion that the worth of art can derive from the totality of its interweaved components (though first acknowledging the biases inherent to both the theory and to the AI models). 

Of particular interest is how these models can complement and extend previous inquiries involving AI-based stylometry of pose and action in contemporary theater, as well as prior efforts to annotate manually the degrees of intermediality in the formal sections of specific recorded performances of Japanese Noh plays.