Sarah Hook, An artificial intelligence researcher and advocate for cheaper artificial intelligence systems that use less computing power is hanging up his banner.
Former vice president of research at artificial intelligence company Cohere, senior Google deep thinkingRaises $50M in seed funding for her new startup Adaption Labs
Hooker and co-founder Sudip Roy (former director of inferential computing) coherenceare trying to create AI systems that use less computing power and have lower running costs than most current leading AI models. They are also targeting models that use a variety of technologies to make them more “adaptable” than most existing models to the individual tasks they are asked to solve. (That’s the startup’s name.)
The round was led by Emergence Capital Partners, with participation from Mozilla Ventures, venture capital firm Fifty Years, Threshold Ventures, Alpha Intelligence Capital, e14 Fund and Neo. San Francisco-based Adaption Labs declined to provide any information on its post-money valuation.
hook told wealth She hopes to create models that can learn continuously without the need for expensive retraining or fine-tuning, or the extensive prompting and context engineering that most enterprises currently use to adapt AI models to their specific use cases.
Creating models that can continuously learn is considered one of the major challenges in the field of artificial intelligence. “This is probably the most important question I’ve ever studied,” Hook said.
Adaptation Labs represents a major bet on the prevailing wisdom in the AI industry that the best way to create more powerful AI models is to scale up the underlying LL.M.s and train them with more data. Even as tech giants pour billions into training on a larger scale, Hook sees diminishing returns from this approach. “Most labs are not going to quadruple the size of their models every year, mainly because we are seeing saturation in the architecture,” she said.
Hook said that the artificial intelligence industry is at a “Do the math” Improvements no longer come from simply building bigger models, but by building systems that can be adapted more easily and cheaply to the task at hand.
Adaption Labs isn’t the only “neolab” (so called because they are a new generation of cutting-edge AI labs following the success of established companies like OpenAI, Anthropic, and Google DeepMind) working on cracking new AI architectures for continuous learning. OpenAI senior researcher Jerry Tworek left the company in recent weeks to found his own startup, Core Automation, and said he is also interested in using new AI methods to create systems that can continuously learn. Former top Google DeepMind researcher David Silver left the tech giant last month Launched a startup called Ineffable Intelligence The focus is on using reinforcement learning – where an AI system learns from the actions it takes rather than static data. In some configurations, this may also result in the AI model being able to continuously learn.
She said Hook’s startup is organizing its work around three “pillars”: Adaptive Data (in which AI systems generate and manipulate the data they need to answer questions on the fly, rather than having to be trained from large static data sets); Adaptive Intelligence (automatically adjusts the amount of computation to spend based on the difficulty of the question); and Adaptive Interfaces (learning from how users interact with the system).
Since her time at Google, Hook has built a reputation in AI circles for pushing back on the “scale is all you need” dogma of many AI researchers. In “The Hardware Lottery,” a widely cited paper published in 2020, she argued that the success or failure of ideas in artificial intelligence often depends on whether they fit neatly into existing hardware, rather than on their inherent merits. Most recently, she authored a research paper called “On the Slow Death of Scaling,” which argued that smaller models with better training techniques can outperform larger models.
At Cohere, she supports the Aya project, which works with 3,000 computer scientists from 119 countries to bring state-of-the-art AI capabilities to dozens of languages where leading-edge models underperform, and does so using relatively compact models. This work shows that creative approaches to data management and training can compensate for raw scale.
One of the ideas being investigated in the Adaptation Lab is so-called “gradient-free learning.” All of today’s artificial intelligence models are extremely large neural networks, containing billions of digital neurons. Traditional neural network training uses a technique called gradient descent, which works a bit like a blindfolded hiker trying to find the lowest point in a valley by taking small steps and trying to feel if he is going downhill. The model makes small adjustments to billions of internal settings called “weights,” which determine how much a given neuron emphasizes input from any other neurons it’s connected to in its own output, and checks after each step to see if it’s closer to the correct answer. This process requires enormous computing power and can take weeks or months. Once the model is trained, these weights are locked in place.
To hone a model for a specific task, users sometimes rely on fine-tuning. This involves further training the model on a smaller, curated dataset (often still consisting of thousands or tens of thousands of examples) and making further adjustments to the model’s weights. Again, it can be expensive, sometimes running into the millions of dollars.
Alternatively, the user can simply try to provide the model with highly specific instructions or prompts about how the model should accomplish the task the user wants the model to undertake. Hook considers this “prompt acrobatics” and points out that whenever a new version of a model is released, the cues often stop working and need to be rewritten.
She said her goal is to “eliminate just-in-time engineering.”
Gradient-free learning sidesteps many problems through fine-tuning and rapid engineering. Rather than adjusting all of a model’s internal weights through expensive training, Adaption Labs’ approach changes how the model behaves in response to queries (what the researchers call “inference time”). The core weights of the model remain the same, but the system can still adapt its behavior to the task at hand.
“How do you update the model without touching the weights?” Hook said. “There’s really interesting innovation in the architecture space that’s leveraging computing in more efficient ways.”
She mentioned several different ways to do this. One is “on-the-fly merging,” in which the system selects from what is essentially a library of adapters—usually small models trained individually on small data sets. These adapters then shape the response of the large primary model. The model decides which adapter to use based on questions asked by the user.
Another method is “dynamic decoding”. Decoding refers to how the model selects its output from a range of possible answers. Dynamic decoding changes the probabilities based on the task at hand without changing the model’s underlying weights.
“We’re moving away from it being just a model,” Hook said. “That’s part of the deep concept – it’s based on interaction, and the model should change in real time based on the task content.”
Hook believes that moving to these methods will fundamentally change the economics of AI. “The most expensive computation is the pretraining computation, mainly because it requires a lot of computation and a lot of time. With inference computing, you get a greater benefit[per unit of computing power],” she said.
Adaption’s CTO Roy has deep expertise in making artificial intelligence systems run efficiently. “My co-founder made the GPU run very fast, which was important to us because it had a real-time component,” Hook said.
Hook said Adaption will use the seed round to hire more AI researchers and engineers and hire designers to develop different AI user interfaces, not just the standard “chat bar” used by most AI models.