What is Reference Resolution?
Apple’s research is included in a paper published on Arxiv.org that looks at “Reference Resolution.” According to one respected guide, reference resolution is a way to express the problem a computer (AI) has to “Find out which object is referred to by an expression, thus gradually building a representation of the objects with their features and evolution.”
In other words, the computer must aim to be as effective as human communication and understanding, such as when we use words like “they” or “those” and the person we are speaking with contextually understands what we are trying to say.
The paper offers an example in which someone might ask ReaLM to show nearby pharmacies. The tech presents the list, and the person could say something vague such as “Call the bottom one,” or “Call this number” (if the number is on screen). Existing virtual assistants would struggle with this, but the researchers’ own tech handles these tasks. They even claim their invention can “substantially outperform” Chat GPT4 in some ways, while matching its performance in others.
When the going gets tough, Siri goes pro
“Critically, we demonstrate how entities that are present on the screen can be passed into an LLM [large language model] using a novel textual representation that effectively summarizes the user’s screen while retaining relative spatial positions of these entities,” they wrote.
In other words, you can anticipate highly effective spoken word control of what’s on screen, perhaps augmented by Apple’s existing Voice Over UI — with obvious implications for its visionOS product line.
This is just one of the many nuggets of information to emerge from Apple’s AI development teams as the company prepares to (hopefully) wow developers at WWDC 24. These many clues also describe technology to support task-focused AI at the edge; superior image intelligence; partnerships with LLM providers such as Google Gemini; augmentation to the company’s existing apps, including Xcode; and more.
