This week, Sakana Ai, the start-based supported-up-based start-up of the dollars from the VC company, making an exceptional claim. The company says that it has created a AI system, AI engineer engineer, which can accelerate certain ai ai’s training with a factor to 100x.
The only problem is, the system is not working.
User ing x quickly found The sacred system causes a worse model training performance than the largest. According to one userAi Sakana returns 3x – not speed.
What’s wrong? Bug in code, according to a Submit By Lucas Beyer, the technical staff member in the opening.
“Incorrect code in (a) a subtle way,” said Beyer on X. “The fact that he opened benchmarking twice with different results should make her stop and think.”
In a PostMortem is published Friday, Sakana admit that the system found a “lie” (because of the defeat of the system “See rankings-. The same phenomenon has been observed AI trained to play chess gameSee rankings-.
According to Sakana, the system found exploitation of evaluation code used by the company that allows the company using validation for accuracy, between other checks. Sakana says that you have solved the problem, and if you would like to check the claim in the updated material.
“We started making evaluations and Runtime Profauts were stronger to eliminate many loopholes,” Ujan Company Company in the post X. “We are in the process of turning on paper, and the results (…) We apologize for readers. We will provide this work revision quickly, and discuss our study. “
Props to sakana to have an error. But the episode is a good reminder that if claims sounds very much, mainly in AImay.
