🏗️ Real-World Case Studies
Social Media Feed Architecture
Social Media Feed Architecture at 500 Million User Scale
Building a news feed that fans out posts to millions of followers requires a hybrid push/pull strategy to balance write amplification against read latency for 500 million Daily Active Users (DAU).
Key Concepts
Fanout-on-Write (Push Strategy)
When a user posts, a worker pushes the post ID into Redis ZSET timeline caches of all followers. Perfect for normal users (<10,000 followers) — timeline read latency is ultra-fast O(1).
Fanout-on-Read (Pull Strategy)
For celebrities (>10,000 followers), posts are NOT pushed to avoid write amplification (80 million Redis writes). Instead, follower timeline fetches celebrity posts on-the-fly at read time.
Hybrid Feed Engine
The timeline service reads the pre-computed Redis ZSET (normal users) and merges it with recent posts from followed celebrities using a 2-way k-way merge-sort algorithm in <30ms.
Partitioned Cassandra Storage
Tweet posts live in Apache Cassandra sharded by User_ID. Timelines live in Redis in-memory ZSET (score = UNIX timestamp in milliseconds).
⚡ Interactive Architecture Simulator
Social Feed Fan-out Strategy Trade-off Simulator
Evaluate Push (Write) vs Pull (Read) vs Hybrid Fan-out models for 500M Daily Active Users.
Hybrid Push/Pull Architecture (Recommended)Recommended
Push timeline updates to Redis for normal users (<10k followers). Pull dynamically for celebrities (>10k followers).
Pure Push (Fan-out on Write)High Risk
Every post write fan-outs into every follower timeline in Redis immediately.
Pure Pull (Fan-out on Read)Baseline
Timeline built on-demand when user opens app by querying followings DB.
50,000 writes/s
Throughput Capacity
22%
Peak DB CPU Load
25ms
P99 Response Latency
$$
Infrastructure Cost
Architectural Assessment & Bottleneck Analysis
Production Grade ✓ — Prevents hot-key fan-out explosion when celebrities post while keeping user timeline feeds instant.
Capacity Math & Storage Scale (500M DAU)
| Metric | Calculation | Storage / Scale Impact |
|---|---|---|
| Daily Posts Generated | 100M posts / day (~1,150 posts/sec avg, 5k peak) | Low write throughput requirement for post creation. |
| Daily Read Feeds Generated | 500M DAU × 10 feed refreshes = 5B reads/day | 57,800 Feed Read Requests / sec peak. |
| Timeline Cache Storage | 500M users × 800 cached post IDs × 16 bytes | ~ 6.4 TB RAM across Redis Sharded Cluster |
Architecture Blueprint (Twitter/X Hybrid Pipeline)
Production Code Snippet 1: Redis ZSET Fanout Worker
redis-feed-push.tstypescript
1import Redis from 'ioredis';
2
3const redis = new Redis({ host: 'redis-feed-cluster' });
4
5export interface PostCreatedEvent {
6 postId: string;
7 authorId: string;
8 timestamp: number;
9}
10
11export async function pushPostToFollowersTimeline(event: PostCreatedEvent) {
12 const { postId, authorId, timestamp } = event;
13
14 // 1. Get follower list of author
15 const followers = await redis.smembers(`user:followers:${authorId}`);
16
17 // If author is celebrity with > 10,000 followers, SKIP PUSH!
18 if (followers.length > 10000) {
19 console.log(`Author ${authorId} is Celebrity. Skipped Push fanout.`);
20 return;
21 }
22
23 // 2. Batch push postId into each follower's Redis ZSET timeline
24 const pipeline = redis.pipeline();
25 for (const followerId of followers) {
26 const timelineKey = `timeline:${followerId}`;
27 pipeline.zadd(timelineKey, timestamp, postId);
28 // Keep max 800 items per user timeline to cap RAM usage
29 pipeline.zremrangebyrank(timelineKey, 0, -801);
30 }
31
32 await pipeline.exec();
33}Production Code Snippet 2: Hybrid Timeline Merge Generator
generate-hybrid-feed.tstypescript
1import Redis from 'ioredis';
2
3const redis = new Redis();
4
5export async function generateUserFeed(userId: string, limit: number = 20) {
6 // 1. Fetch pre-computed timeline IDs from Redis ZSET (Push feed for normal users)
7 const pushPostIds = await redis.zrevrange(`timeline:${userId}`, 0, limit - 1);
8
9 // 2. Fetch list of followed celebrities for this user
10 const followedCelebrities = await redis.smembers(`user:followed_celebrities:${userId}`);
11
12 // 3. Pull recent post IDs for each followed celebrity
13 const celebPostIds: string[] = [];
14 for (const celebId of followedCelebrities) {
15 const celebPosts = await redis.zrevrange(`user:outbox:${celebId}`, 0, limit - 1);
16 celebPostIds.push(...celebPosts);
17 }
18
19 // 4. Combine and Sort by Timestamp (Descending)
20 const allPostIds = Array.from(new Set([...pushPostIds, ...celebPostIds]));
21
22 // 5. Fetch full Post objects from Redis Hash or Cassandra
23 const pipeline = redis.pipeline();
24 for (const pid of allPostIds) {
25 pipeline.hgetall(`post:${pid}`);
26 }
27 const rawPosts = await pipeline.exec();
28
29 const posts = rawPosts
30 ?.map(([err, res]) => res as any)
31 .filter(Boolean)
32 .sort((a, b) => parseInt(b.timestamp) - parseInt(a.timestamp))
33 .slice(0, limit);
34
35 return posts;
36}💡
Senior Architect Insight: At 500 million user scale, pure push or pure pull collapses under write amplification or read latency. The hybrid architecture (Push for normal users, Pull for celebrities) balances both worlds, delivering instant sub-30ms timelines.