Santi Andrea Orlando1, 2, **, Antonino Furnari1, ** and Giovanni Maria Farinella1, 3, **
We present a tool to generate simulated egocentric data starting from a 3D model of a cultural sites, useful to study the problem of Image-Based Localization (IBL) and artworks detection in cultural sites. In particular our focus is on the localization of users wearing an egocentric camera useful to augment the experience of visitors in cultural sites. The contributions of this work are:
A video example of localization system performed in Bellomo dataset is shown below. We performed this localization considering the test set frames. We show the predicted position with/without smoothing filter (best results width 25% trimmed mean filter w=5).
A video example of localization system performed in Stanford dataset is shown below. We performed this localization considering a path from the test set.
We collected the second dataset using the developed Tool V.2 in Unity 3D. The dataset concerns to cultural heritage, in particular the Galleria Regionale Palazzo Bellomo.
It comprises of 4 simulated navigations generated simulating a visit inside the museum. During each navigation the agenta "look at" 5 different observation points
for each artwork in the current room. During the navigation, we acquired at 5 frames per second and for each frame we saved the RGB frame, Semantic Mask, 6DoF camera pose,
ID of the current room. We obtain a large dataset without any manual labeling for both the two tasks (localization and artwork detection).
For more details about our dataset go to this page .
We collected a large dataset of Simulated Egocentric Navigations using the developed Tool in Unity 3D. The dataset comprises 90 paths generated simulating three virtual agents with different heights (1.5 m, 1.6 m, 1.7m), 30 paths for each agent. During the navigation, we acquired at 30 frames per second and for each frame we saved the RGB Map, Depth Map and the 6DoF of the camera. In this way, we obtain a large dataset without any manual labelling. We provide a 3DoF version of the Dataset comprising labels and RGB images.
We considered a total of 90 random paths, 30 paths for each height of the agent [150, 160, 170]cm. Each path is performed by letting the agent reach 21 target points. For more details about our dataset go to this page .
S. A. Orlando, A. Furnari, G. M. Farinella - Egocentric Visitor Localization and Artwork Detectionin Cultural Sites Using Synthetic Data. Pattern Recognition Letters - Special Issue on Pattern Recognition and Artificial Intelligence Techniques for Cultural Heritage, 2020. Download the paper here.
More details on the dataset and qualitative results can be found in the supplementary material associated to the publication.
This research is supported by PON MISE - Horizon 2020, project VEDI - Vision Exploitation for Data Interpretation, PO FESR 2014/2020 – Azione 1.1.5, project VALUE - Visual Analysis for Localization and Understanding of Environments, by DWORD - Xenia Projetti s.r.l. and Piano della Ricerca 2016-2018 lineadi Intervento 2 of DMI, University of Catania. The authors would like to thank Regione Siciliana Assessorato dei Beni Culturali dell’Identità Siciliana - Dipartimento dei Beni Culturali e dell’Identità Siciliana and Polo regionale di Siracusa per i siti culturali - Galleria Regionale di Palazzo Bellomo.