Surrey researchers teach AI to spot what you’re sketching

A new way to teach artificial intelligence (AI) to understand human line drawings – even from non-artists – has been developed by a team from the University of Surrey and Stanford University.

illustration of how various programmes identify objects on an amateur sketch

The new model approaches human levels of performance in recognising scene sketches.

Sketching is a powerful language of visual communication. It is sometimes even more expressive and flexible than spoken language.

Developing tools for understanding sketches is a step towards more powerful human-computer interaction and more efficient design workflows. Examples include being able to search for or create images by sketching something. Dr Yulia Gryaditskaya, Lecturer in People Centred Artificial Intelligence

People of all ages and backgrounds use drawings to explore new ideas and communicate. Yet, AI systems have historically struggled to understand sketches.

AI has to be taught how to understand images. Usually, this involves a labour-intensive process of collecting labels for every pixel in the image. The AI then learns from these labels.

Instead, the team taught the AI using a combination of sketches and written descriptions. It learned to group pixels, matching them against one of the categories in a description.

The resulting AI displayed a much richer and more human-like understanding of these drawings than previous approaches. It correctly identified and labelled kites, trees, giraffes and other objects with an 85% accuracy. This outperformed other models which relied on labelled pixels.

As well as identifying objects in a complex scene, it could identify which pen strokes were intended to depict each object. The new method works well with informal sketches drawn by non-artists, as well as drawings of objects it was not explicitly trained on.

Drawing and writing are among the most quintessentially human activities and have long been useful for capturing people's observations and ideas.

This work represents exciting progress towards AI systems that understand the essence of the ideas people are trying to get across, regardless of whether they are using pictures or text. Professor Judith Fan, Assistant Professor of Psychology at Stanford University

The research forms part of Surrey’s Institute for People-Centred AI, where we draw on Surrey's domain expertise across different disciplines encouraging the creation and sharing of innovative ideas and approaches to foster new research directions and collaborations in AI.

The SketchX programme seeks to understand the way we see the world by the way we draw it.

This research is a prime example of how AI can enhance fundamental human activities like sketching.

By understanding rough drawings with near-human accuracy, this technology has immense potential to empower people's natural creativity, regardless of artistic ability - exactly the kind of people-centered AI we aim to advance at the Institute. Professor Yi-Zhe Song, Co-Director Surrey Institute for People-Centred AI; Programme Lead of MSc in Artificial Intelligence