MIN Faculty
Department of Informatics
Knowledge Technology


This webpage is outdated and has moved. Please find the official Knowledge Technology page at:


Gestures and Reference Instructions for Deep robot learning

GRID is a research project developed in the Knowledge Technology group of Hamburg University funded by CAPES between October 2013 and October 2016.

Persons participating in the project

Leading Investigator: Prof. Dr. Stefan Wermter

Associates: Pablo Barros, Dr. Sven Magg, Dr. Cornelius Weber.


Using only speech as main communication channel sometimes is not enough. Misunderstandings during dialogues can occur, especially if the robot does not understand the context. The same problem can occur with gesture-based communication. Sometimes a gesture is not enough to express the command, and a dialogue could be very hard to accomplish through the limitations of gesture signs. One example of these limitations is natural pointing communication, when a human uses gestures and speech to determine directions to a robot.

The multi-modal characteristic of communication with speech and pointing combined is what makes it so hard to achieve. The lack of correlation between these inputs does not give the robot robust information representation. The use of deep neural architectures, inspired by representations in the human brain, could solve this problem through the capability to represent complex sentences composed by gestures and speech. It could also help to understand how the human brain combines audio-visual inputs into one percept.

The use of neurally inspired inspired models allows to encode complex information and to model the learning processes in the human brain. Using this kind of bio-inspired architecture simulates the human brain capacities in robots, and provides a robust solution for the natural pointing communication problem.