Miles Brundage, a prominent former policy researcher at OpenAI, is launching an institute dedicated to a simple idea: AI companies shouldn’t be allowed to grade their own work.
Today, Brundage officially announced the creation of the Artificial Intelligence Verification and Evaluation Research Institute (AVERI), a new nonprofit organization designed to advance the idea that cutting-edge AI models should be subject to external auditing. AVERI is also working to develop AI auditing standards.
The release coincides with the release of a research paper co-authored by Brundage and more than 30 AI security researchers and governance experts that lays out a detailed framework for how to conduct independent audits of the companies building the world’s most powerful AI systems.
Brundage spent seven years at OpenAI as a policy researcher and consultant on how companies should prepare for the emergence of human-like general artificial intelligence. He will leave the company in October 2024.
“One of the things I learned while working at OpenAI is that companies are figuring out the norms for this kind of thing on their own,” Brundage said. wealth. “No one is forcing them to work with third-party experts to make sure things are safe and secure. They make their own rules. “
This creates risks. Although leading AI labs conduct safety and security testing and publish technical reports on the results of many of these assessments, some of which are conducted with the help of external “red team” organizations, for now consumers, businesses, and governments only have to trust what AI labs say about these tests. No one forces them to conduct these assessments or reports according to any particular criteria.
In other industries, audits are used to provide assurance to the public, including consumers, business partners and, to some extent, regulators, that a product is safe and has been rigorously tested, Brundage said.
“If you go out and buy a vacuum cleaner, you know that the components inside, like the battery, have been tested by an independent laboratory to strict safety standards to make sure it won’t catch fire,” he said.
New agency will drive policy and standards
Brundage said AVERI is interested in policies that encourage AI labs to move to rigorous external audit systems and looking at what the standards for those audits should be, but not in the audits themselves.
“We are a think tank. We are trying to understand and shape this transition,” he said. “We don’t want every Fortune 500 company to be our customer.”
He said existing public accounting, auditing, assurance and testing firms could get into the AI security auditing business, or launch startups to take on the role.
AVERI said it has raised $7.5 million of a $13 million goal to support 14 employees and two years of operations. To date, its funders include Halcyon Futures, Fathom, Coefficient Giving, former Y Combinator president Geoff Ralston, Craig Falls, Good Forever Foundation, Sympatico Ventures and AI Underwriting Company.
The organization said it has also received donations from current and former non-executive employees of cutting-edge artificial intelligence companies. Brundage said “these people know where the bodies are buried” and “want to see more accountability.”
Insurers or investors could mandate AI security audits
Brundage said there may be a variety of mechanisms to encourage AI companies to start hiring independent auditors. One is that large enterprises purchasing AI models may require audits to ensure that the AI models they purchase will perform as promised and do not pose hidden risks.
Insurance companies may also push for the establishment of AI audits. For example, insurance companies offering business continuity insurance to large companies that use AI models for critical business processes may require audits as a condition of coverage. The insurance industry may also require audits for leading AI companies such as OpenAI, Anthropic and Google.
“The insurance industry is definitely growing rapidly,” Brundage said. “We have a lot of conversations with insurance companies.” He noted that one specialized AI insurance company (AI Underwriting Company) has made a donation to AVERI because “they see the value of auditing, checking for compliance with the standards that they have written.”
Brundage said investors may also request an AI security audit to ensure they are not taking on unknown risks. Given that investment firms are now writing millions and billions of dollars in checks to fund AI companies, it makes sense for these investors to demand independent audits of the safety of the products being developed by these fast-growing startups. If any of the leading labs go public (as OpenAI and Anthropic are reportedly preparing to do within the next year or two), the companies could face shareholder lawsuits or SEC prosecution if problems later cause their stock prices to plummet and they fail to hire auditors to assess the risks of their AI models.
Brundage also said regulation or international agreements could force AI labs to hire independent auditors. There are currently no federal regulations on artificial intelligence in the United States, and it’s unclear if any will be enacted. President Donald Trump signed an executive order aimed at cracking down on U.S. states passing their own artificial intelligence regulations. The government says that’s because it believes a single federal standard is easier for businesses to follow than multiple state laws. But while punishing states for enacting AI regulations, the government has yet to come up with its own national standards.
In other regions, however, the groundwork for auditing may already be taking shape. The EU Artificial Intelligence Act that recently came into force does not explicitly require audits of AI companies’ assessment procedures. But its General Artificial Intelligence Code of Practice, a blueprint for how cutting-edge AI labs can comply with the act, does say that labs building models that could pose “systemic risks” need to provide free access to model testing to external evaluators. The bill itself also stipulates that when organizations deploy AI in “high-risk” use cases, such as underwriting loans, determining eligibility for social benefits or determining health care, the AI system must undergo an external “qualification assessment” before being placed on the market. Some have interpreted these parts of the Act and the Code to mean that auditors are required to be independent in nature.
Establish “assurance levels” and find enough qualified auditors
A research paper released with AVERI outlines a comprehensive vision for cutting-edge AI auditing. It proposes a framework of “AI assurance levels” ranging from Level 1 (which involves some third-party testing, but with limited access, similar to the external assessments AI labs currently employ companies to conduct) all the way to Level 4, which would provide adequate “treaty-level” assurance for international agreements on AI safety.
Building a team of qualified AI auditors is inherently difficult. AI auditing requires a combination of technical expertise and governance knowledge that few people possess, and those who do are often attracted by lucrative offers from companies being audited.
Brundage acknowledged the challenge but said it can be overcome. He talks about mixing people with different backgrounds to create a “dream team” with the right skills. “You might have some people from existing auditing firms, plus some people from cybersecurity penetration testing companies, plus some people from AI security nonprofits, and maybe an academic,” he said.
In other industries, from nuclear power to food safety, it is often disasters, or at least close calls, that provide the impetus for standards and independent assessments. Brundage said he hopes that through artificial intelligence, audit infrastructure and norms can be established before a crisis occurs.
“From my perspective, our goal is to get as smoothly and as quickly as possible to a level of scrutiny that is proportional to the actual impact and risk of the technology without overstepping it,” he said.