The importance of cups

Imagine you are looking at a photo of a group of friends having lunch in a cafe. Thanks to the squishy contents of your skull, you can quickly identify what's important about the photo. Things like: who is present, where they are, what they are doing and what they are feeling. There might be other details that you process on a subconscious level: the time of day, how large the room is, other people that are not part of the friend group, and so on.

An interesting problem has emerged during our 2022 hackathon project. We want to automatically collate images from a group of friends on a trip into themed mini-albums to be enjoyed at a later date. We might want to see albums with titles like "lunchtime", "at the beach" or "animals", depending on what photos have been taken and gathered.

We plan to use Google's rather impressive AI platform 'Cloud Vision' to understand the contents of each image. It does a pretty good job of identifying objects "woman", "laptop", "cup" and so on. It can generate a couple of hundred tags (with varying certainty) after scanning our cafe photo but it doesn't tell us what's important about the photo, from another human's perspective. This leads to the possibility that our new app would put the cafe photo in a mini-album called "Cups" because the AI has no idea that we don't consider cups to be an interesting theme of the day.

So, we now have to think about how to introduce the concept of 'importance' to our app's AI systems. We could create a 'white list' - a list of themes that would generally work as album titles, if they are detected within an image. But we're concerned that we miss opportunities for interesting titles that humans would not have spotted but still enjoy. For example, maybe the AI would create a "Loud shirts" photo album automatically.

This is a fun example of a challenge that emerges during a Hackathons or R&D project. We don't know if we'll overcome it but on the way we'll enjoy discussing the problem and uncovering cool new tech that might help - which is the essence of our work at Gravitywell.

If you'd like to read more about this year's event check out our daily blog.