Reverse Engineering the Feed

The "algorithm"isn't a single magic score. It's a massive, industrial-scale retrieval and ranking pipeline designed to filter the entire world's noise down to a handful of tweets you might actually care about.

Recently, X (formerly Twitter) updated their open-source documentation for the "For You"feed. It's distinct from the 2023 release. I've broken down how it works, what's changed, and mechanically, how you can optimize for it.

The process has two distinct phases: Candidate Sourcing (fetching ~1500 candidates from hundreds of millions) and Ranking (scoring and sorting them).

Thunder (In-Network) is the evolution of Twitter's legacy timeline infrastructure. It functions as a scalable, in-memory graph store specialized for "In-Network"retrieval, consuming high-throughput Kafka streams of post-creation events and guaranteeing sub-millisecond access to recent tweets from everyone you follow.

Phoenix (Out-of-Network) is the discovery engine solving the problem: "Find the 100 best tweets for User A from the 500 million posted today."It uses Vector Similarity Search with a Two-Tower architecture.

The User Tower aggregates your real-time engagement history into a dense vector (e.g., 144 dimensions) that updates in near real-time. The Candidate Tower processes every new tweet through LLMs to extract semantic meaning, coupled with author reputation graphs (PageRank), to produce a static Candidate Embedding.

The system uses HNSW graphs to perform dot-product similarity searches across millions of vectors in milliseconds—it traverses the graph to find vectors most aligned with your current state.

Unlike a standard transformer where tokens attend to all other tokens, here a candidate tweet can attend to the User Context (your history, demographics, cell state) but is mathematically blinded to other candidate tweets via an attention mask.

This ensures Score Stability—if Tweet A scores 0.9, it should score 0.9 regardless of whether it's paired with a viral meme or a boring ad. This also allows for aggressive caching where previously scored candidates can be reused without re-inference.

The neural network outputs a vector of independent probabilities for every possible user action using Multi-Task Learning: P(Like), P(Reply), P(Repost), P(Video_View_50%), P(Report), P(Mute). These feed into a weighted sum where Reply is heavily upweighted (54x a Like) and negative signals like Mute carry massive negative coefficients (-74x to -369x).

One "Not Interested"signal effectively erases the positive signal of dozens of likes. It is a survival game first, a popularity contest second.

Diversity Attenuation: The system runs an explicit Author Diversity Scorer after ranking that penalizes multiple posts from the same author in a single session. Posting 10 times an hour is actively harmful—focus on one high-signal post rather than volume.

Embedding Stability: To be retrieved by Phoenix, your Post_Vector needs to be stable. If you post about coding today, politics tomorrow, and cooking on Sunday, your vector drifts. Consistency tightens your embedding variance—pick a lane to help the ANN search find you.

Weighted Actions: Content that generates discussion outranks content that generates passive agreement, but w_mute is a massive penalty. Avoid rage-bait—it drives replies but also drives mutes and blocks that will blacklist you from future candidate sets.

The Filter Problem: If you use polarizing keywords that many people have on their mute list, you are filtered out before the model even sees your content. Clean vocabulary increases total addressable market.