Dataset
We extended the UNICT-VEDI dataset proposed in Ragusa et al.[1] annotating with bounding boxes the presence of 57 different points of interest. We only considered data acquired using the head-mounted Microsoft HoloLens device. The UNICT-VEDI dataset comprises a set of training videos (at least one per point of interest), plus 7 test videos acquired by subjects visiting a cultural site. Each video of the dataset has been temporally labeled to indicate the environment in which the visitor is moving (9 different environments are labeled) and the point of interest observed by the visitor (57 points of interest have been labeled). For each of the 57 points of interest included in the UNICT-VEDI dataset, we annotated approximately 1,000 frames from the provided training videos, for a total of 54,248 frames.
We considered a total of 9 environments and 57 points of interests.
For more details about our dataset go to this page 
.
. Methods
Recognizing the points of interest observed by visitors in a cultural site is the natural next step after visitor localization (Ragusa et al. [1]). We consider three different variants of the pipeline presented in Ragusa et al. [1] and an approach based on an object detector.
57-POI: is the state-of-the-art method proposed in Ragusa et al. [1]. The discrimination component of the method is trained to discriminate between the 57 points of interest. No ``negative'' frames are used for training. The rejection of negatives is performed by the rejection component of Ragusa et al. [1];
57-POI-N: is similar to the 57-POI method, with the addition of a negative class. The discriminator component of the method in Ragusa et al. [1] is trained to discriminate between 57 points of interest plus the ``negative'' class. In this case, negative frames are explicitly used for training. The rejection component of Ragusa et al. [1] is further used to detect and reject more negatives;
9-Classifiers: nine context-specific instances of the method in Ragusa et al. [1] are trained to recognize the points of interest related to the nine different contexts of the UNICT-VEDI dataset (i.e., one classifier per context). Similarly to 57-POI, no negatives are used for training;
Object-based: A YOLOv3 object detector is used to perform the detection and recognition of each of the 57 points of interest. At test time, YOLOv3 returns the coordinates of a set of bounding boxes with the related class scores for each frame. If no bounding box has been predicted in a given frame, we reject the frame and assign it to the ``negative'' class. If multiple bounding boxes are found in a specific frame, we choose the bounding box with the highest class-score and assign its class to the frame. We have chosen the YOLOv3 object detector because it is a state-of-the-art real-time object detector.
Paper
F. Ragusa, A. Furnari, S. Battiato, G. Signorello, G. M. Farinella. Egocentric Point of Interest Recognition in Cultural Sites. In International Conference on Computer Vision Theory and Applications, 2019. Download the paper here.
Acknowledgement
This research is supported by PON MISE – Horizon 2020, Project VEDI - Vision Exploitation for Data Interpretation, Prog. n. F/050457/02/X32 - CUP: B68I17000800008 - COR: 128032, and Piano della Ricerca 2016-2018 linea di Intervento 2 of DMI of the University of Catania.Related Work
This work is related to the following publication:
- F. Ragusa, A. Furnari, S. Battiato, G. Signorello, G. M. Farinella. Egocentric Visitors Localization in Cultural Sites. In Journal on Computing and Cultural Heritage (JOCCH), 2019.
Web Page.
- G. M. Farinella, G. Signorello, A. Furnari, S. Battiato, E. Scuderi, A. Lopes, L. Santo, M. Samarotto, G. P. A. Distefano, D. G. Marano, “Integrated Method With Wearable Kit For Behavioural Analysis And Augmented Vision" (Italian Version: "Metodo Integrato con Kit Indossabile per Analisi Comportamentale e Visione Aumentata"), Patent Application number 102018000009545, filling date: 17/10/2018, Università degli Studi di Catania, Xenia Gestione Documentale S.R.L., IMC Service S.R.L.
- F. Ragusa, L. Guarnera, A. Furnari, S. Battiato, G. Signorello, G. M. Farinella. Localization of Visitors for Cultural Sites Management. In International Conference on Signal Processing and Multimedia Applications (SIGMAP), Porto, Portugal, July 26-28, 2018. Web Page.
- A. Furnari, S. Battiato, G. M. Farinella, Personal-Location-Based Temporal Segmentation of Egocentric Video for Lifelogging Applications, submitted to Journal of Visual Communication and Image Representation Web Page.
- A. Furnari, G. M. Farinella, S. Battiato. 2017. Recognizing Personal Locations From Egocentric Videos. IEEE Transactions on Human-Machine Systems (2017). https://doi.org/10.1109/THMS.2016.2612002 Web Page.
- A. Furnari, G. M. Farinella, S. Battiato, Temporal Segmentation of Egocentric Videos to Highlight Personal Contexts of Interest, International workshop on egocentric perception, interaction and computing (EPIC) in conjunction with ECCV 2016.
People
