To err is human; forgiveness is divine. But when it comes to autonomous AI “agents” taking on tasks previously handled by humans, how much room for error is there?
In a recent Fortune magazine brainstorming artificial intelligence A roundtable of experts at an event in San Francisco discussed the issue, with insiders sharing how their companies are tackling security and governance — an issue that is transcending more practical challenges like data and computing power. Companies are in an arms race to parachute artificial intelligence agents into their workflows that can handle tasks autonomously with little human oversight. But many face a fundamental paradox that contributes to slow adoption: moving quickly requires believehowever building trust takes a lot of time.
Dev Rishi, general manager of Rubrik Artificial Intelligence, joined the security company last summer after get A photo of his deep learning AI startup Predibase. Over the next four months, he met with executives from 180 companies. He told the Brainstorm AI audience that he used these insights to divide agent AI adoption into four stages. (At a horizontal level, agent adoption refers to companies implementing AI systems that work autonomously rather than respond to prompts.)
From Rishi’s understanding, the four phases he discovered include early experimentation, where companies are working to prototype agents and map out targets they think they can integrate into their workflows. Rishi said the second phase was the trickiest. That’s when companies move their agents from prototypes to formal working production. The third phase involves scaling these autonomous agents across the company. The fourth and final stage – which no one spoke to Rishi – is autonomous artificial intelligence.
Rishi found that about half of the 180 companies were in the experimentation and prototyping stages, while 25% were working to formalize their prototypes. Another 13% are scaling, and the remaining 12% have not launched any AI projects yet. However, Rishi expects big changes ahead: In the next two years, 50% are expected to move to Phase 2, according to the roadmap.
“I think we’re going to see massive adoption very quickly,” Rishi told the audience.
However, he noted that there are significant risks that prevent companies from moving “fast and hard” in accelerating the implementation of AI agents into the workforce. This risk, and the No. 1 barrier to wider deployment of agents, are security and governance, he said. Because of this, companies are working to move from agents used for knowledge retrieval to action-oriented agents.
“Our focus is really on accelerating the AI transformation,” Rishi said. “I think the first risk factor, the first bottleneck, is the risk (itself).”
Integrate agents into the workforce
Kathleen Peters, Office of Chief Innovation Experian The head of product strategy said the slowdown stemmed from not fully understanding the risks when AI agents cross the guardrails the company has put in place and the failsafes needed if that happens.
“If something goes wrong, if hallucinations occur, if the power goes out, what can we rely on,” she questioned. “It’s one of those questions that some executives (depending on the industry) want to understand, ‘How do we feel safe?'”
She noted that figuring this out is different for every company and can be especially tricky for companies in highly regulated industries. Chandhu Nair, senior vice president of data, artificial intelligence and innovation at home improvement retailer Lowe’s, noted that it’s “pretty easy” to set up agents, but people don’t understand what they are: Are they digital employees? Is it labor? How will it fit into the organizational structure?
“It’s almost like hiring a bunch of people without an HR function,” Nair said. “So we have a lot of agents and no way to draw them correctly, and that’s the focus.”
The company has been addressing some of these issues, including who might be responsible if something goes wrong. “It’s hard to trace back,” Nair said.
Experian’s Peters predicts that many of these issues will continue to be discussed in public in the coming years, even as closed-door conversations between boards and senior compliance and strategy committees take place simultaneously.
“I actually think something bad is going to happen,” Peters said. “There will be breaches. There will be agents going rogue in unexpected ways. Those will make for very interesting headlines in the news.”
Peters went on to say that a big bang would attract a lot of attention and reputational risks would be at stake. She said it would force uncomfortable conversations about who is responsible for software and agents, and it could all lead to increased regulation.
“I think it’s going to be part of our overall change management as a society to think about these new ways of working,” Peters said.
Still, there are some concrete examples of how AI can benefit companies when implemented in a way that resonates with employees and customers.
Nair said Lowe’s has so far embedded AI into its operations and has seen widespread adoption and “tangible” return on investment. For example, each of its 250,000 store associates has an agency partner with deep product knowledge in its 100,000 square feet of stores, which sell everything from electrical equipment to paint to plumbing supplies. Many of the new people joining Lowe’s’ workforce are not business people, and agency partners have become by far the “fastest-adopting technology,” Nair said.
“It’s important to find the right use cases that really resonate with customers,” he said. In terms of driving change management in stores, “If the product is good and adds value, adoption will be high.”
Who’s looking at agents?
But he added that change management techniques would have to be different for those working in headquarters, adding to the complexity.
And many enterprises are also stuck in another early stage question, which is whether they should build their own agents or rely on AI capabilities developed by major software vendors.
Rakesh Jain, Executive Director, Cloud and Artificial Intelligence Engineering, Healthcare Systems General Massachusetts Brighamsaid his organization is taking a wait-and-see approach. With major platforms such as salespersonworking days and Immediate service Establishing his own agency may result in layoffs if his organization simultaneously establishes its own agency.
“If there is a gap, then we want to build our own agency,” Jain said. “Otherwise, we will rely on purchasing the agents that the product suppliers are setting up.”
Jain said that in healthcare, human supervision is urgently needed given the high stakes.
“Patient complexity cannot be determined algorithmically,” he said. “Someone has to be involved.” In his experience, agents can speed up decision-making, but humans have to make the final judgment, with doctors verifying everything before taking any action.
Still, Jain sees huge potential as the technology matures. In radiology, for example, agents trained with the expertise of multiple doctors can spot tumors in dense tissue that a single radiologist might miss. But even if an agent is trained by multiple doctors, “you still have to use human judgment,” Jayne said.
There is always the threat of overreach by an agent who is considered a trusted entity. He compared rogue agents to autoimmune diseases, one of the most difficult conditions for doctors to diagnose and treat because the threat is internal. If agents within the system “become corrupt,” he said, “that’s going to cause huge losses that people can’t really quantify.”
Despite the unanswered questions and looming challenges, Risch said there is a path forward. He identified two requirements for building agent trust. First, companies need systems that can give agents confidence that agents are operating within policy guardrails. Second, they need clear policies and procedures to deal with when inevitable problems arise—a policy that is compelling. Additionally, Nair adds three factors for building trust and moving forward wisely: identity and responsibility and understanding who the agent is; assessing the consistency of the quality of each agent’s output; and reviewing postmortems to explain why and when errors occurred.
“Systems make mistakes, just like humans,” Nair said. “But being able to explain and recover is equally important.”