X's Recommendation Algorithm
X's Recommendation Algorithm is a set of services and jobs that are responsible for serving feeds of posts and other content across all X product surfaces (e.g. For You Timeline, Search, Explore, Notifications). For an introduction to how the algorithm works, please refer to our engineering blog.
Architecture
Product surfaces at X are built on a shared set of data, models, and software frameworks. The shared components included in this repository are listed below:
| Type | Component | Description | |------------|------------|------------| | Data | tweetypie | Core service that handles the reading and writing of post data. | | | unified-user-actions | Real-time stream of user actions on X. | | | user-signal-service | Centralized platform to retrieve explicit (e.g. likes, replies) and implicit (e.g. profile visits, tweet clicks) user signals. | | Model | SimClusters | Community detection and sparse embeddings into those communities. | | | TwHIN | Dense knowledge graph embeddings for Users and Posts. | | | trust-and-safety-models | Models for detecting NSFW or abusive content. | | | real-graph | Model to predict the likelihood of an X User interacting with another User. | | | tweepcred | Page-Rank algorithm for calculating X User reputation. | | | recos-injector | Streaming event processor for building input streams for GraphJet based services. | | | graph-feature-service | Serves graph features for a directed pair of users (e.g. how many of User A's following liked posts from User B). | | | topic-social-proof | Identifies topics related to individual posts. | | | | Compute scores between pairs of entities (Users, Posts, etc.) using embedding similarity. | | Software framework | | High performance, machine learning model serving written in Rust. | | | | Software framework for building feeds of content. | | | | Framework for generating aggregate features in batch or real time. | | | | Service to retrieve embeddings (i.e. SimClusers and TwHIN). | | | | Legacy machine learning framework built on TensorFlow v1. |
