Recognızıng Human Interactıons In Stıll Images

Abstract

Recognizing human interactions in still images is quite a challenging problem since compared to videos, there is only a glimpse of interaction in a single image. In this thesis, we explore the role of two components that provide descriptive features upon recognizing interactions in still images: the role of human faces and poses.

As for the role of human faces, we explore whether the facial regions and their spatial configurations contribute to the recognition of interactions. In this respect, our method involves the extraction of several visual features from the facial regions, incorporating scene characteristics and deep features to the recognition. Extracted multiple features are utilized within a discriminative learning framework for recognizing interactions between people. Our designed facial descriptors are based on the observation that relative positions, size, and locations of the faces are likely to be essential for characterizing human interactions. Since there is no available dataset in this relatively new domain, a comprehensive new dataset that includes several images of human interactions is collected. Our experimental results show that faces and scene characteristics contain vital information to recognize interactions between people.

On behalf of exploring the role of human poses upon recognizing interactions, we propose a multi-stream convolutional neural network architecture, which fuses different levels of human pose information to recognize human interactions better. In this context, several pose-based representations are explored. Experimental evaluations in an extended benchmark dataset show that the proposed multi-stream pose Convolutional Neural Network successfully discriminates a wide range of human-human interactions. Moreover, when used in conjunction with the overall context, human poses provide discriminative cues about human-human interactions.