X open sources its algorithm in the face of transparency fines and the Grok controversy

In 2023, the website was partially known as Twitter open source its algorithm for the first time. At the time, Tesla billionaire Elon Musk had just recently acquired the platform, and he claimed to be on a mission to restructure the social media platform to make it more transparent.

However, the release of the algorithm’s code was quickly criticized for “transparent theater“With critics noting that it’s “incomplete,” and doesn’t reveal much about the organization’s internal workings, or why the code works.

Now the site (renamed X) has re-opened the algorithm, fulfilling the promise made by Musk last week. “We will create a new 𝕏 algorithm, including all the code used to determine organic posts and recommended ads for users, open source in 7 days,” she said. Musk also promised to provide transparency to the algorithm every four weeks for the future.

At shipment on GitHub on Tuesday, X provided an accessible write-up of the feed-generating code, along with a diagram of how the program works.

What has been revealed isn’t particularly earth-shattering—it does provide a look behind the curtain of algorithms. At diagrams shows that, when sifting through content for a specific user’s feed, the site’s algorithm considers engagement history (what posts have been clicked, etc.) and surveys new posts on the network. It also conducts machine learning-based analysis of “out of network” posts – as in, content from accounts that users don’t necessarily follow – that they believe users might also be interested in.

The algorithm then filters out certain types of posts, including those coming from blocked accounts or those associated with muted keywords, as well as content deemed too abusive or spam-like. Algorithms then rank this content based on what users think will be most interesting. This process takes into account factors like relevance and diversity of content so users don’t just get a bunch of the same posts. The algorithm also considers content based on how likely users are to like, reply, repost, like, or more.

The entire system is based on AI, according to X. A GitHub write-up released Tuesday notes that the system “relies entirely” on the company’s “Grok-based transformer” to “learn the relevance of user engagement sequences.” In other words, Grok sees what you click on and likes and feeds that information to its recommendation system. The paper also notes that there is no “manual feature engineering for content relevance,” meaning that humans do not manually manage how algorithms determine what is relevant. It added that automation “reduces complexity in data pipelines and service infrastructure.”

Why is X revealing all this now? It is not entirely clear. In the past, Musk has claimed that he wanted to make the platform an example of corporate transparency—a theme that continues to this day. In 2023, when the Twitter algorithm was first revealed, Musk said that providing “code transparency” would be “so embarrassing” but would ultimately “lead to a rapid improvement in the quality of recommendations.” He added: “Most importantly, we hope to be trusted.” With the first open-sourcing code, the platform states “a new era of transparency” to Twitter.

Although Musk has talked about transparency, certain aspects of the platform may not be open since Musk took over. When a tech billionaire buys Twitter in 2022, the site is forced to transition from a public company. to private— an evolution that is not usually synonymous with openness. When the site is used to release several transparent reports each yearX does not release its first transparency report until September 2024. In December, X exists was also fined $140 million by European Union regulators who claimed that the site had breached the “obligation of transparency” under the Digital Services Act (DSA), and argued that the site’s verification check mark system had made it more difficult for users to judge the authenticity of certain accounts.

X has also come under pressure over the past month over the way its chatbot, Grok, is used to create and distribute sexual content. California Attorney General’s Office and congressional parliamentarian have both scrutinized the platform in recent weeks, mentioned claims that Grok has been used to create images of naked women and children. As a result, some can see this attraction to open as theater only more.