Treffer: Learn to understand movies

Movie, where characters would face various situations and perform various behaviors in various scenarios, is a reflection of our real world. It teaches us a lot such as the stories that took place in the past, the culture and custom of a country or a place, the reaction and interaction of humans in different situations, etc. Therefore, to understand movies is to understand our world. It goes not only for humans, but also for an artificial intelligence system. We believe that movie understanding is a good arena for high-level machine intelligence, considering its high complexity and close relation to the real world. What's more, compared to web images and short videos, the hundreds of thousands of movies in history containing rich content and multi-modality information become better nutrition for the data-hungry deep models. ; However, despite the remarkable advances in visual understanding, how to understand a story-based long video with artistic style remains an open question. In this thesis, we aim to facilitate this topic by exploiting movie understanding with learning methods. ; To be specific, we first introduce a holistic dataset for movie understanding named MovieNet and multiple important topics in movie analysis. Then we try to begin with a simple problem, i.e. tag classification, where we take advantage of the trailers to learn visual models that can be applied to movies. And as a human-centric video, characters play an important role in movie analysis, especially for story-based video understanding. So in the following parts, we studied how to recognize the cast in movies effectively and efficiently, including how to train a face model without labels, how to unify different visual cues, how to use temporal features, and how to take advantage of multi-modal information with memory. ; 電影就是我們真實世界的一面鏡子， 在這里， 不同的角色會在不同的場景下做出各種各樣行為來應對變化多端的狀況。電影還會給我們講述歷史故事， 展現文化風俗， 表現人在不同狀態下的反應， 等等。所以理解電影就是理解我們這個世界。對人來說如此， 對人工智能也是這樣。考慮到電影里覆雜的場景和與真實世界的高度相似， 我們相信智能電影分析是一個高水平機器智能的競技場。而且， 相比與圖片和短視頻， 人類歷史上積攢下來的幾十萬部電影， ...