Comment Large language models seem poised to evolve from AI chatbots generating synthetic content on your screen to virtual agents that are capable of performing actions on your PC right at your desk.
Instead of answering questions or creating animated stickers, AI will soon be able to follow instructions and help you tick stuff off your to-do list at work. A new wave of AI agent startups are building products that can automate parts of your day-to-day employment. Correctly, one hopes.
Some, like Lindy, are building next-generation personal assistants that CEO Flo Crivello envisions doing all the tedious administrative chores that suck up people’s time. “People are always worried that robots are stealing people’s jobs. I think it’s people who’ve been stealing robots’ jobs,” he said during a presentation at the AI Engineer Summit in San Francisco in October.
I think it’s people who’ve been stealing robots’ jobs
In the future, instead of having to check your calendar and message back and forth with someone to settle on a time and date for a meeting, for example, Lindy’s agents can connect to your calendar and email apps to automatically find a free time slot, and write and send the email asking them to meet. Ideally, it would even add a Zoom link or Google Maps directions to a place too.
Users would communicate with the Lindy chatbot by describing a task for it to do. Behind the scenes, the LLM system would route the instructions to software that calls upon the relevant API needed to execute a particular action. Crivello told us Lindy can connect to a variety of APIs supporting file systems like Google Drive, sales and marketing platforms like HubSpot, as well as sites like LinkedIn.
Other startups like Adept are focused on teaching agents to perform keyboard and mouse moves. It trains its models on visual elements of user interfaces or web browsers so agents can recognize things like text boxes or search buttons. By training it on videos recording people’s screens as they carry out tasks on specific software, it can learn what exactly needs to be typed and where it needs to click to do something such as copying and pasting information into an Excel spreadsheet. The idea being that the system takes care of the boring, repetitive stuff.
In demos, the company has shown its agent extracting data from invoices to automatically fill in forms to file expenses, for example. “Our Northstar is that we’re trying to make an AI teammate for every knowledge worker. We’re working on step one right now, which is you ask Adept how to do any tedious thing that you’ve already done before,” CEO David Luan told The Register.
Adept’s software accepts images and text as input and returns text and actions as output. The tricky part, however, is making it reliable. The agents have to be fine-tuned on the right kind of data that teaches the way to perform a specific task more consistently. Working to automate keyboard and mouse actions is more difficult than hooking up LLMs to APIs.
There are pros and cons to each method, according to Crivello. “APIs are more reliable, but they don’t let you do everything you might want to do,” he said. Not all software can be accessed via an API, so sometimes it’s better for agents to learn how to directly interact with graphical user interfaces. “The advantage of the UI is you can do everything but it’s much harder to automate format; it’s much more brittle,” he added.
Cooperating with your AI colleagues
The idea of an AI copilot that works alongside humans is already becoming mainstream. Microsoft has packaged multiple AI-powered Office 365 tools into one subscription, naming it Copilot for Microsoft 365, while Google is offering similar capabilities across its Workspace apps with Duet AI.
Over time, these tools will become more capable and integrate with various types of software to do more than analyzing reports and drafting emails.
Researchers and analysts are beginning to forecast the impact that AI work companions will have on the workforce and economy. Employers are drawn to the promise that AI will make their employees more productive, meaning they’ll be able to reach goals and hit targets more quickly.
Tedious taskers
A December report from Forrester viewed by El Reg predicts that in the short term, one to three years from now, autonomous workplace assistants (AWAs) will be able automate away easy tasks that take no longer than a few minutes for a human to perform.
“They are simple to deploy and deliver verifiable productivity returns, but they don’t learn, have no context, and follow predetermined patterns. An unattended bot might perform an address update that a human used to do, but little in the work pattern has changed,” the report said.
The first generation of agents won’t affect what knowledge workers do in their jobs much, but they will begin to change how they do some tasks. Some of the easy drudgery work will be offloaded to machines, according to Craig Le Clair, co-author of the report and a principal analyst at Forrester.
“In the short term, AWAs tackle simple automation like accounting and payroll functions or customer self-service,” he told us. “A key distinction between AWAs in the short-term period and those in the future is this focus on tedious, repeatable, and a low-value task, which can be performed by software and results in little residual value or process change. It primarily minimizes costs by extracting lower-paid human hours.”
The next generation of workbots, expected to arrive in the next four to eight years, will be smarter and able to undertake more complex tasks that involve multiple steps, like setting up sales pipelines, generating potential leads, and converting customers. In more technical settings, they could begin to push code to crunch numbers and perform data analysis, the report said. In the future, these agents will begin using other AI tools to help them complete tasks.
“The later AWAs dramatically alter the relationship between humans and automation and give us new ways of working,” Le Clair said. “AWAs provide higher level functions like decision making, physical agility, and conversation. Automation takes on more human-like characteristics, and they are able to understand a goal, not get stuck, and complete a work task. In this sense they become full coworkers. The AWA can consult [generative AI], for example, to handle workflow variation, consult a human or system if needed, and simulate more advanced human traits that present entirely new ways of doing things.”
The most popular commercial LLMs are already beginning to adopt some of these early capabilities. Users can now use Anthropic’s Claude bot in Google Sheets, while OpenAI introduced the idea of connecting GPTs to APIs to teach custom chatbots to carry out tasks.
“Like plugins, actions allow GPTs to integrate external data or interact with the real world,” OpenAI says. “Connect GPTs to databases, plug them into emails, or make them your shopping assistant. For example, you could integrate a travel listings database, connect a user’s email inbox, or facilitate e-commerce orders.”
Anthropic introduced the concept of “tool use” when it announced that its latest LLM, Claude 2.1, could also connect to simple apps and APIs to do things like consulting a calculator to do arithmetic.
“By popular demand, we’ve also added tool use, a new beta feature that allows Claude to integrate with users’ existing processes, products, and APIs,” the company explains. “Claude can now orchestrate across developer-defined functions or APIs, search over web sources, and retrieve information from private knowledge bases. Users can define a set of tools for Claude to use and specify a request. The model will then decide which tool is required to achieve the task and execute an action on their behalf.”
Will we work less or more?
AI may boost productivity, but the technology won’t be good enough to take most jobs in the short term. Adept’s Luan believes it will mean that workers will get to focus on things that require more intelligence and interpersonal skills.
“I think that we’ll spend more time working on higher reasoning tasks that these models can’t do. Stuff that requires real human judgement and in-person touch point, like spending more time with customers,” he said.
Le Clair agreed, saying that agents will impact industries differently. Nurse practitioners can take on more care responsibility aided by AI for decision support, he said, while paralegals will take on more client relationships and advice support assisted by agents that have passed the bar exam and provide legal services at a lower cost than a licensed attorney.
As AI continues to improve, it will destroy some jobs and create new one in the future.
“Unfortunately, the overall number of middle jobs will decline, and move many to the frontline service worker segments where human agility is still at a premium,” Le Clair told us. “The digital elite will be hurt by AWAs that perform research, programming, and some creative tasks, and will have to depend on their human skills and networks to maintain their lifestyles.”
Some believe that it’ll mean humans can work less and pursue their hobbies and interests, while the more pessimistic reckon that workers aided by software will just be pushed to produce more.
Le Clair is the in the first camp. “It will result in shifting more work to AWAs, and reduce overall employment levels. We will be looking at a four-day work week in five years, with growing populations of alternative non-traditional work lifestyles,” he said.
Hopefully he’s right, and we humans can be a little more free. Throughout history, technological breakthroughs powering industrial revolutions have changed the nature of work, but rarely eliminated it completely. ®
