Building a Benchmark for Long-Form Narrative Understanding

Co-sponsored by the Berkeley Institute for Data Science, the School of Information, and the Department of Scandinavian.

Evaluating the long-form reasoning abilities of multimodal language models is difficult in part due to the simple lack of availability of long-form videos — many existing benchmarks for video understanding are comprised of short YouTube clips, contain copyrighted materials (which are then subject to DMCA takedown requests) or contain privately digitized collections that are not publicly shareable. 

I’ll describe ongoing work to create a new benchmark of popular, openly available movies that we can build benchmarks around; this involves identifying measures of popularity going back to the 1920s, assessing public domain status, and creating a suite of narrative understanding questions that we can use to track the capabilities of modern multimodal language models going forward.

Space is limited. Submit the application form to request an invitation.

Apply to attend