Reverse Engineering the Feed

The "algorithm" isn't a single magic score. It's a massive, industrial-scale retrieval and ranking pipeline designed to filter the entire world's noise down to a handful of tweets you might actually care about.

Recently, X (formerly Twitter) updated their open-source documentation for the "For You" feed. It's distinct from the 2023 release. I've broken down how it works, what's changed, and mechanically, how you can optimize for it.

The Pipeline Architecture

Every time you pull to refresh, the system executes a complex chain of operations. It doesn't score every tweet in existence (which would be computationally impossible). Instead, it uses a funnel approach:

The process has two distinct phases: Candidate Sourcing (fetching ~1500 candidates from hundreds of millions) and Ranking (scoring and sorting them).

1. Candidate Sourcing: Thunder vs. Phoenix

The system pulls candidates from two primary engines. This is not a simple database query; it is a distributed systems challenge of retrieving relevant items from a pool of hundreds of millions within typical P99 latency constraints (under 200ms).

• Thunder (In-Network)
  Thunder is the evolution of Twitter's legacy timeline infrastructure (replacing the old "fanout-on-write" Redis clusters). It functions as a scalable, in-memory graph store specialized for "In-Network" retrieval.

  • Mechanism: It consumes high-throughput Kafka streams of post-creation events.
  • Storage: Instead of materialized home timelines (which waste RAM for inactive users), Thunder likely uses a hybrid approach: storing "User Active" windows in RAM (Manhattan/Redis-style) and falling back to computed indexes for cold retrieval.
  • Efficiency: It guarantees sub-millisecond access to the most recent tweets of everyone you follow, regardless of graph density (whether you follow 10 or 10,000 people).
• Phoenix (Out-of-Network)
  Phoenix is the "For You" discovery engine. It solves the problem: "Find the 100 best tweets for User A from the 500 million posted today." It does this via Vector Similarity Search using a Two-Tower architecture.

  • The User Tower: Aggregates your real-time engagement history (likes, retweets, dwell time), negative signals, and static demographics into a dense vector (e.g., 144 dimensions). This embedding updates in near real-time.
  • The Candidate Tower: Processes every new tweet through LLMs (BERT-derived) to extract semantic meaning, coupled with author reputation graphs (PageRank), to produce a static Candidate Embedding.
  • Approximate Nearest Neighbor (ANN): The system uses HNSW (Hierarchical Navigable Small World) graphs to perform dot-product similarity searches across millions of vectors in milliseconds. It doesn't check every tweet; it traverses the graph to find the "neighborhood" of vectors most aligned with your current state.

Key Distinction: Thunder is deterministic (you follow them, you see it). Phoenix is probabilistic (mathematical guess of interest).

The Scorer: Candidate Isolation

The ranking stage reduces the ~1500 candidates to the final ~20 items you see. This is done by a heavy-weight 48M parameter Masked Transformer.

Unlike a standard transformer (like GPT-4) where tokens attend to all other tokens, the feed scorer utilizes Candidate Isolation.

• The Mechanism: A candidate tweet can attend to the User Context (your history, demographics, cell state), but it is mathematically blinded to other candidate tweets in the same batch via an attention mask.
• The Why: This ensures Score Stability. If Tweet A scores 0.9, it should score 0.9 regardless of whether it's paired with a viral meme or a boring ad. This also allows for aggressive caching—previously scored candidates can be reused without re-inference.

The Weighted Sum (Multi-Task Learning)

The neural network does not output a single "quality score." Instead, it uses Multi-Task Learning (MTL) to predict a vector of independent probabilities for every possible user action:

These probabilities are fed into a final logistic regression layer:

The Asymmetry of Signals:

• Positive Signals: A Reply is heavily upweighted (e.g., 54x a Like) because it signals time spent and conversation.
• Negative Signals: Mute, Block, and Report carry massive negative coefficients (-74x to -369x).
• Implication: You can hunt for likes all day, but one "Not Interested" signal effectively erases the positive signal of dozens of likes. It is a survival game first, a popularity contest second.

Optimization Strategy

Understanding the architecture reveals the optimization function. Here is the technical breakdown of how to align with the system:

Source: Analysis based on recently released X algorithm documentation and architecture diagrams.