Knowledge Engineering for Autonomous Agents


Sep. 2020 - We presented our work at KR2020! Watch the video here.


HanS is the Health and Safety Robot Inspector in KMi , who is aware of our health and safety guidelines and is able to navigate the lab, checking that these are enforced. In this video, HanS is navigating KMi searching for a set of relevant everyday objects.
HanS' ability to spot the presence of relevant objects in a noisy, real-world environment is a crucial pre-requisite for autonomous sense-making. For instance, by learning to recognise that a pile of paper is sitting right next to an electrical appliance, and by combining this evidence with additional background knowledge about these objects (e.g., "paper is flammable" and "electric appliance is a potential ignition source"), HanS is then able to identify a potential fire hazard.

Currently, HanS is able to recognise many objects commonly found in office spaces automatically (without relying on ARTags), after being trained on only a few shots captured from the target environment.

Using common-sense knowledge from ConceptNet, WordNet and Visual Genome, HanS can automatically suggest corrections for objects that were hard to classify with pure Machine Learning.

With as few as 5 training examples for each target object, captured in its real environment, HanS could achieve a cumulative F1 score of 66%, across 25 object classes.
Crucially, introducing common-sense knowledge to improve the object classification led to an additional 2% improvement in HanS' performance.


Gianluca Bardaros Photo Gianluca BardaroResearch Associate
Agnese Chiattis Photo Agnese ChiattiPhD Research Student
Enrico Dagas Photo Enrico DagaSenior Project Officer
Enrico Mottas Photo Enrico MottaProfessor of Knowledge Technologies
SciRoc Logo