Social Networks & Feeds

Design a Real-Time Social Newsfeed & Timeline Architecture (Twitter / X)

Staff / Principal

Design a high-scale social media timeline architecture solving the celebrity fan-out problem using a hybrid push/pull model, Redis Sorted Sets, and Snowflake ID sequencing.

Production Scale: 500M Daily Active Users • 1 Billion Posts/Day • 500k Feed Reads/Sec

Functional Requirements

  • •Users can publish tweets (text, images, media)
  • •Follow/unfollow relationship graph
  • •Home timeline rendering latest posts from followed accounts in real-time
  • •User timeline displaying an individual's post history

Non-Functional Requirements

  • •Sub-100ms timeline rendering latency
  • •High availability (99.99%)
  • •Fast post dissemination (followers see tweets in <3 seconds)

Capacity & Scale Estimation

Daily Active Users (DAU)500 Million users
Daily Post Ingestion1 Billion posts/day (~11,500 posts/sec, 50k peak)
Timeline Read Throughput500,000 timeline queries/sec
Read-to-Write Ratio50 : 1 (Heavily read-intensive)

Core Architectural Components

1Post Ingestion Service

Assigns 64-bit Snowflake ID, persists tweet payload to Cassandra / MySQL, and publishes event to Kafka.

2Hybrid Fan-Out Worker Pool

Evaluates author follower count: pushes to Redis timelines for users with <50k followers; uses Pull on Read for celebrities.

3Timeline Cache (Redis Sorted Sets)

Stores top 800 tweet IDs per active user in a Redis `ZSET` scored by Snowflake timestamp ID for O(log N) slicing.

4Social Graph Service (Neo4j / TAO Distributed Graph)

Manages follow/follower edges with caching in memory for sub-5ms adjacency list traversals.

5Timeline Hydration Service

Fetches raw tweet metadata, user profiles, and media URLs in a single multi-get batch to render final feed JSON.

Architectural FAQs & Interview Deep Dives

What is the 'Celebrity Fan-Out Problem' and how does a Hybrid Fanout architecture solve it?

If an account with 100M followers posts, writing to 100M user timelines (Fan-out-on-Write) would overload Redis. A Hybrid model pushes writes only for standard users, while celebrity posts are merged on-the-fly when followers request their feed (Fan-out-on-Read).

How does Twitter Snowflake generate unique, time-sortable 64-bit IDs without database locks?

Snowflake IDs consist of: 41 bits (millisecond timestamp), 10 bits (machine/worker ID), and 12 bits (per-node sequence number), ensuring global uniqueness and natural chronological sorting.

Why are full post payloads not stored directly inside Redis timeline caches?

To minimize memory overhead. Redis timelines only store 64-bit Tweet IDs in a Sorted Set. A hydration service batch-fetches the actual post text and user avatar from Memcached/Cassandra.

How do you handle inactive users who haven't logged in for 30 days?

Inactive users are excluded from Fan-out-on-Write queues to save compute. When an inactive user logs back in, their timeline is rebuilt on-demand from the social graph.

How are retweets and quote tweets indexed in the timeline?

Retweets insert the original Tweet ID mapped with a Retweet metadata pointer into the follower's Redis Sorted Set.

What database is best suited for storing petabytes of user post history?

Wide-column stores like Apache Cassandra or ScyllaDB partitioned by `UserID` and clustered by `TweetID DESC` for high-throughput sequential writes.