A team from the University of Surrey and Stanford University has devised a novel method for teaching artificial intelligence (AI) to recognize human line drawings, even if they are not created by an artist. In terms of scene sketch recognition, the new model is on par with human performance.
Dr. Yulia Gryaditskaya, Lecturer at Surrey’s Centre for Vision, Speech, and Signal Processing (CVSSP) and Surrey Institute for People-Centered AI (PAI), stated:
“Sketching is a strong form of visual communication. It is oftentimes more expressive and adaptable than spoken language.
“Creating tools for comprehending sketches is a step toward more effective human-computer interaction and design workflows. Examples include the ability to search for or create images by drawing something.
Drawings are used by people of all ages and backgrounds to express themselves and explore new ideas. However, AI systems have traditionally struggled to grasp sketches.
AI has to be taught how to understand images. Usually, this involves a labour-intensive process of collecting labels for every pixel in the image. The AI then learns from these labels.
Instead, the team taught the AI using a combination of sketches and written descriptions. It learned to group pixels, matching them against one of the categories in a description.
The resulting AI displayed a much richer and more human-like understanding of these drawings than previous approaches. It correctly identified and labelled kites, trees, giraffes and other objects with an 85% accuracy. This outperformed other models which relied on labelled pixels.
“Sketching is a strong form of visual communication. It is oftentimes more expressive and adaptable than spoken language.
“Creating tools for comprehending sketches is a step toward more effective human-computer interaction and design workflows. Examples include the ability to search for or create images by drawing something.
Drawings are used by people of all ages and backgrounds to express themselves and explore new ideas. However, AI systems have traditionally struggled to grasp sketches.
“This work represents exciting progress towards AI systems that understand the essence of the ideas people are trying to get across, regardless of whether they are using pictures or text.”
The research is part of Surrey’s Institute for People-Centred AI, namely its SketchX programme. SketchX uses AI to understand how we see the environment through the way we draw it.
Professor Yi-Zhe Song, Co-director of the Institute for People-Centered AI and SketchX leader, stated:
“This study is an excellent example of how AI may improve basic human tasks such as sketching. This technology, by comprehending rough sketches with near-human accuracy, has enormous potential to enhance people’s inherent creativity, regardless of creative talent.”
The results will be presented at the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2024. It takes place in Seattle from June 17 to 21, 2024.
Reference – Researchers teach AI to spot what you’re sketching