Design a Chat System

A chat system needs three things working together: persistent connections so messages arrive instantly, a durable message store that fixes the order of each conversation, and a routing layer that delivers each message to every device of every recipient, online or not. The classic design uses WebSockets to stateful chat servers, a message service that assigns per-conversation sequence numbers, a pub/sub layer for fan-out and push notifications for anyone offline.

1. Requirements

Functional: one-to-one and group chat, message history, delivery and read receipts, online presence, push notifications, and sync across a user’s phone, laptop and browser.

Non-functional: low latency (a message should feel instant), no lost messages, a consistent order within each conversation, and high availability. Scope it early: are groups capped at a few hundred members, or can channels hold hundreds of thousands? That single choice changes the fan-out design.

As in any interview, agree requirements and scale before drawing boxes; our system design interview framework walks through that opening.

2. Rough scale

Suppose 50 million daily active users each sending 40 messages a day: 2 billion messages a day, about 23,000 per second on average and perhaps three times that at peak. At around 200 bytes per message with metadata, that’s roughly 400 GB of new data a day before replication. Our guide to back-of-the-envelope estimation shows how to reach numbers like these quickly.

3. Connections: WebSockets

HTTP request-response is a poor fit for messages that arrive unprompted. The usual answer is a WebSocket (RFC 6455): a full-duplex connection that stays open, so the server can push a message the moment it arrives. Long polling and server-sent events are fallbacks.

Connections make chat servers stateful: the server holding a user’s socket is the only one that can push to that device. You need:

• A connection registry mapping each device to the chat server holding its connection, kept in a fast key-value store and updated on connect and disconnect.
• Heartbeats to detect dead connections.
• Graceful draining during deploys, so clients reconnect elsewhere without losing messages.

Put a layer-4 or WebSocket-aware load balancer in front, spreading connections evenly across chat servers.

4. Sending a message

1. The client sends the message over its socket with a client-generated ID.
2. The chat server passes it to the message service, which assigns the next sequence number for that conversation and writes it to the message store.
3. Only after the write succeeds does the server acknowledge the sender, which shows one tick.
4. The message service publishes the message to the pub/sub layer, keyed by conversation.
5. Chat servers holding recipients’ connections receive it and push it down each socket.
6. Recipients’ devices acknowledge delivery; later, a read event produces a read receipt.

The client-generated ID makes retries safe: if the sender resends after a timeout, the store recognises the duplicate. This is the same idea as idempotency keys for APIs.

5. Ordering

Global ordering across all chats is unnecessary and expensive. What users notice is order within a conversation, so give each conversation its own monotonically increasing sequence number, assigned by whichever node owns that conversation. Timestamps alone aren’t enough, because clocks on different servers drift.

6. Storage

Messages are written constantly, read mostly by recency and rarely updated, which suits a wide-column or key-value store partitioned by conversation ID and sorted by sequence number. Fetching “the last 50 messages” becomes one partition read. User profiles, contacts and group membership fit a relational database. As volume grows, shard messages by conversation ID so each conversation lives on one partition.

7. Group chat fan-out

For small groups, fan out on write: look up the members and publish to each one’s devices. For very large channels, that becomes millions of deliveries per message, so switch to fan out on read: store the message once and let clients fetch new messages for the channel when they open it, with pushes only for mentions. The same trade-off appears in news feed design.

8. Presence, offline delivery and sync

• Presence: clients send heartbeats; a user is “online” while heartbeats arrive, with a grace period to avoid flicker. Publish changes only to contacts who are currently looking.
• Offline users: if a recipient has no open connection, hand the message to the notification pipeline; our design for a notification system covers push providers, retries and rate limits.
• Multi-device sync: each device stores the last sequence number it has seen per conversation. On reconnect, it asks for everything newer. The server stays the source of truth.

9. Security

Use TLS everywhere. Many messaging apps also offer end-to-end encryption, where only the participants’ devices hold the keys; the Signal Protocol is the best-known design. End-to-end encryption changes the architecture: servers route and store ciphertext, and features like server-side search must move to the device.

10. Reliability checklist

• Acknowledge only after a durable write.
• Deduplicate by client message ID.
• Replay from the last seen sequence number on reconnect.
• Use a durable log or queue between the message service and fan-out, so a slow consumer can catch up.
• Monitor end-to-end delivery latency, not just server health.

Frequently asked questions

Why use WebSockets instead of polling?

Polling wastes requests when nothing has changed and adds delay when something has. WebSockets keep one connection open so the server can push messages instantly.

How do you keep messages in order?

Assign a sequence number per conversation when the message is stored, and have clients display messages by that number rather than by timestamps.

What happens if a chat server crashes?

Clients reconnect to another server, which updates the connection registry. Because messages were acknowledged only after being stored, each device then syncs anything newer than its last sequence number.