Treffer: Collecting Image Cropping Dataset: A Hybrid System of Machine and Human Intelligence

Image cropping is a common tool that exists in almost any image editor, yet automatic cropping is still a difficult problem in Computer Vision. Since images nowadays can be easily collected through the web, machine learning is a promising approach to solve this problem. However, an image cropping dataset is not yet available and gathering such a large-scale dataset is a non-trivial task. Although a crowdsourcing website such as Mechanical Turk seems to be a solution to this task, image cropping is a sophisticated task that is vulnerable to unreliable annotation; furthermore, collecting a large-scale high-quality dataset through crowdsourcing is expensive. Alternatively, we introduce a system that uses automatic methods and human inputs to generate and evaluate image crops. Our system is a hybrid of machine and human intelligence. Given an image, the hybrid system generates image crops in three steps: identify main objects in the image; automatically generate a set of potential good crops around the identified main objects following principle photographic composition; and assess the generated crops. The second step is automatic while the first and third steps require inputs from the human. We obtain these user inputs by designing an online game. In the user’s perspective, our system is a website where users can access to play games. In our perspective, by letting people play games, we have them annotate the images for us with no cost. The games are carefully designed so that users’ feedbacks are helpful to our main goal. The system is embedded with a quality control model that assesses the user’s accuracy and the quality of the annotation.