|
When creating a live streaming application like Twitch – one has to use an approach that will not only cover functional and non-functional requirements but also create a detailed architectural design, and plan for scalability and reliability. The following sub-topics provide a thorough analysis of each of the major factors that can exist in the design and execution process.
 Design a Live Streaming App like Twitch
Important Topics for Twitch System Design
- User Authentication and Authorization: Some of the things the users require are registration, authentication, and de-authentication. Applying OAuth for a third-party authentication method such as Google and Facebook makes it secure and easy for users.
- Live Streaming: An important requirement can be to the streamers are able to begin and end the live streams, whereas for the viewers, these streams should be viewed in real-time. The system must include solutions different for resolution and bit rates to balance the network conditions and the devices.
- Chat Functionality: A live chat option is available to the viewers, especially when there are live broadcasts going on. There is that need to make this chat positive and equally make sure that it is controlled and therefore that is why streamers should be provided with moderation tools.
- User Profiles: Every user must possess his or her own page, which could contain, for example, information about the bio, followers, and followings.
- Follow/Subscribe: They should be able to follow and subscribe to the streamer of their choice with features such as the ability to receive notifications on when the streamer goes live, all adding to the interaction.
- Content Storage and Playback: The live stream recording should be saved in the system and accessible on-demand and properly tagged to enable an easy search.
- Monetization: They must work on the paid options such as advertisements, subscriptions, and donations since the streamers will be motivated to produce content.
- Scalability: These require the system to support thousands of concurrent users and be able to grow and shrink to accommodate the many visitors without slowness.
- Performance: Streaming video quality and load distribution to different users are very important to deliver a quality video stream.
- Reliability and Availability: Availability requirements dictate high uptime with low downtime; a common requirement is to have redundant data and preferably back up solutions.
- Security: It is essential to ensure the confidentiality of users’ information and prevent interruptions in the availability of services when using social networks; that is why HTTPS for data transfer and protection from DDoS attacks are needed.
- Usability: The key usability of the application is its GUI or graphic user interface and should ideally be available on web, on mobiles and smart TVs.
- Compliance: Among non-functional requirements, it is obligatory to follow the legislation of data protection including GDPR or CCPA.
Use Case Diagram for Twitch System Design
A use case diagram basically depicts the relationships between the actors (The streamers and the viewers) and the system. The key scenarios for applying the functionality are login, live streaming, chat, profile, subscribe/follow, playback, and monetization.
 Use Case Diagram Below is the traffic and storage estimations for twitch system design:
1. Traffic (Bandwidth) Calculation:
1. Assumptions:
- Peak concurrent viewers: 100,000
- Average video bitrate: 3000 Kbps (or 3 Mbps)
- Audio bitrate: 128 Kbps
2. Calculate Video Traffic:
Total video traffic = Peak viewers × Bitrate per viewer
Total video traffic = 100,000 viewers × 3000 Kbps
Total video traffic = 300,000,000 Kbps
Convert Kbps to Mbps (divide by 1000):
Total video traffic = 300,000 Mbps
So, the total video traffic during peak times would be approximately 300,000 Mbps or 300 Gbps.
3. Calculate Audio Traffic:
Total audio traffic = Peak viewers × Audio bitrate per viewer
Total audio traffic = 100,000 viewers × 128 Kbps
Total audio traffic = 12,800,000 KbpsConvert Kbps to Mbps (divide by 1000):
Total audio traffic = 12,800 Mbps
Therefore, the total audio traffic during peak times would be approximately 12,800 Mbps or 12.8 Gbps.
4. Total Traffic:
Add together the video and audio traffic to get the total bandwidth requirement:
Total traffic = Video traffic + Audio traffic
Total traffic = 300,000 Mbps + 12,800 Mbps
Total traffic = 312,800 MbpsConvert to Gbps (divide by 1000):
Total traffic ≈ 312.8 Gbps
So, the estimated total bandwidth required during peak times would be approximately 312.8 Gbps.
2. Storage Calculation:
1. Assumptions:
- Average stream length: 1 hour
- Average bitrate per stream: 3000 Kbps (or 3 Mbps)
- Number of active streamers: 10,000
- Storage redundancy factor: 2 (to account for backups and redundancy)
2. Calculate Storage per Stream:
Storage per stream = Bitrate per stream × Stream length
Storage per stream = 3000 Kbps × 3600 seconds (1 hour)
Storage per stream = 10,800,000 kilobits (Kb)
Convert Kb to Megabytes (MB) (divide by 8 million):
Storage per stream ≈ 1.35 GB
3. Calculate Total Storage for Active Streamers:
Total storage = Storage per stream × Number of active streamers
Total storage = 1.35 GB/stream × 10,000 streams
Total storage = 13,500 GB
Convert GB to Terabytes (TB):
Total storage ≈ 13.5 TB
4. Account for Redundancy:
Since redundancy factor is 2:
Total storage with redundancy = Total storage × Redundancy factor
Total storage with redundancy = 13.5 TB × 2
Total storage with redundancy ≈ 27 TB
Therefore, the estimated storage required to store the video streams of 10,000 active streamers, considering redundancy, would be approximately 27 terabytes.
High-Level Design for Twitch System Design
 High-Level Design The high-level design encompasses the overall system architecture, including frontend and backend applications:
1. Frontend Services
- Web Application: Provides the main interface for users to interact with the streaming platform via browsers.
- Mobile Application: Offers a convenient way for users to access the platform on smartphones and tablets.
- Smart TV Application: Enables users to watch streams on smart TVs.
2. Backend Services:
The API Gateway Service acts as a central hub that manages and routes all incoming API requests to the appropriate backend services. It provides a single entry point for clients, simplifying the client-side architecture.
- Request Routing: Directs incoming requests to the corresponding service based on the endpoint and HTTP method.
- Load Balancing: Distributes requests across multiple instances of a service to ensure even load distribution and high availability.
- Security: Implements security measures such as rate limiting, IP whitelisting, and token validation to protect backend services.
- Protocol Translation: Converts client requests from one protocol to another if needed (e.g., from HTTP to WebSocket).
- Aggregation: Combines responses from multiple backend services into a single response, reducing the number of client-server interactions.
2. Authentication Service
The Authentication Service is responsible for verifying the identity of users and ensuring that they have the necessary permissions to access specific resources.
- User Registration: Handles new user registrations by storing user credentials securely.
- User Login: Authenticates users by verifying their credentials (e.g., username and password).
- Token Generation: Issues JWT tokens upon successful authentication, which clients use to access other services.
- OAuth Integration: Supports third-party authentication providers like Google and Facebook to allow users to log in using their existing accounts.
- Session Management: Manages user sessions, including token expiration and renewal.
- Authorization: Ensures that users have the necessary permissions to access certain resources based on their roles.
3. Live Streaming Service
The Live Streaming Service manages all aspects of the live streaming process, including initiating, maintaining, and terminating live streams.
- Stream Initialization: Allows streamers to start a live stream by providing necessary metadata (e.g., title, description).
- Stream Encoding: Encodes video data into different formats and bitrates to accommodate various network conditions and devices.
- Stream Distribution: Uses a Content Delivery Network (CDN) to distribute the live stream to viewers in real-time.
- Stream Termination: Allows streamers to end their live stream and ensures that the stream is properly closed and saved.
- DVR Functionality: Supports features like pause, rewind, and fast-forward for live streams.
- Analytics: Tracks and records streaming metrics such as viewer count, duration, and bitrate statistics.
4. Chat Service
The Chat Service facilitates real-time communication between viewers and streamers during live broadcasts.
- Message Handling: Manages the sending and receiving of chat messages in real-time.
- Moderation Tools: Provides streamers with tools to moderate chat, such as muting users, deleting messages, and implementing filters for inappropriate content.
- Chat Persistence: Stores chat messages for playback during recorded streams.
- Scalability: Ensures that the chat service can handle a large number of concurrent users without performance degradation.
- Security: Protects against spam and abuse by implementing rate limiting and user bans.
5. Notification Service
The Notification Service manages the delivery of notifications to users about various events and updates on the platform.
- Event Handling: Listens for events (e.g., a streamer going live, a new follower) and triggers notifications accordingly.
- Notification Delivery: Sends notifications via different channels such as email, push notifications, and in-app messages.
- User Preferences: Allows users to manage their notification preferences, including opting in or out of specific types of notifications.
- Scheduling: Schedules notifications to be sent at appropriate times, avoiding user disruption.
- Analytics: Tracks the delivery and engagement metrics of notifications to optimize future communication.
A Content Delivery Network (CDN) is a distributed network of servers that work together to deliver content (like images, videos, and static files) to users faster and more efficiently. These servers, called edge servers, which are strategically positioned across various geographical locations.
A load balancer is a networking device or software application that distributes and balances the incoming traffic among the servers to provide high availability, efficient utilization of servers, and high performance. Load balancers are highly used in cloud computing domains, data centers, and large-scale web applications where traffic flow needs to be managed.
8. Databases
Store user details, streaming content details, and chat records.
Low-Level Design for Twitch System Design
Below is the low-level design for twitch system design:
.webp) Low-Level Design 1. Authentication Service
- Components:
- API Gateway: Routes requests to appropriate services.
- Authentication Service: Handles user registration, login, and JWT token generation.
- Database Schema:
- Users: Stores user information (user_id, username, password_hash, email, etc.)
- Workflow:
- User registers via frontend.
- Frontend sends data to API Gateway.
- API Gateway routes to Authentication Service.
- Authentication Service stores user data in the database and generates JWT tokens.
2. Streaming Service
- Components:
- Media Server: Handles ingestion and broadcasting of streams (e.g., NGINX with RTMP module).
- Transcoder: Converts video into different resolutions and bitrates for adaptive streaming (e.g., FFmpeg).
- CDN: Distributes video streams globally to reduce latency.
- Database Schema:
- Streams: Stores stream metadata (stream_id, user_id, stream_key, status, etc.)
- Workflow:
- Streamer starts a live stream.
- Media Server ingests the stream.
- Transcoder converts the stream into multiple formats.
- CDN caches and delivers the stream to viewers.
3. Chat Service
- Components:
- WebSocket Server: Handles real-time messaging.
- Message Broker: For distributing chat messages (e.g., RabbitMQ or Kafka).
- Database Schema:
- Messages: Stores chat messages (message_id, user_id, stream_id, content, timestamp).
- Workflow:
- User joins a live stream chat.
- Frontend establishes a WebSocket connection with the Chat Service.
- Chat Service broadcasts messages to all connected users.
4. User Profile Service
- Components:
- User Service: Manages user profiles, follows, and subscriptions.
- Database Schema:
- Profiles: Stores profile information (user_id, bio, avatar_url, etc.)
- Followers: Stores follow relationships (follower_id, followee_id).
- Workflow:
- User updates their profile.
- User Service updates the database with new profile information.
5. Notification Service
- Components:
- Notification Server: Sends notifications (e.g., when a followed streamer goes live).
- Message Queue: Handles the delivery of notifications.
- Database Schema:
- Notifications: Stores notifications (notification_id, user_id, type, content, read_status, timestamp).
- Workflow:
- Event occurs (e.g., streamer goes live).
- Event is pushed to the Message Queue.
- Notification Service processes the event and sends notifications to users.
Database Design for Twitch System Design
Database design involves creating tables for different entities.
- Users Table: It holds basic data of a user such as id, username, password hash, email, profile picture, and bio.
- Streams Table: Stores information within the form of records including id, user_id, title, description, category, start_time, end_time, and status.
- Followers Table: Stores followers and their followees with the respective fields as follower_id and followee_id.
- Messages Table: This table stores the chat messages, parameters include the id, stream_id, user_id, message, and time_stamp.
- Subscriptions Table: Subscribes to other users and keeps information in this table by user_id, streamer_id, start_date, and end_date fields.
 Database Design Microservices and APIs Used in Twitch System Design
1. Authentication API:
- Login:
- Request: POST /api/auth/login
- Payload: { “email”: “[email protected]”, “password”: “password” }
- Response: { “token”: “jwt_token” }
- Register:
- Request: POST /api/auth/register
- Payload: { “name”: “username”, “email”: “[email protected]”, “password”: “password” }
- Response: { “message”: “Registration successful” }
2. User API:
- Get User Profile:
- Request: GET /api/user/{userId}
- Response: { “id”: “userId”, “name”: “username”, “profilePic”: “url”, “bio”: “user bio” }
- Update User Profile:
- Request: PUT /api/user/{userId}
- Payload: { “name”: “new name”, “profilePic”: “new_url”, “bio”: “new bio” }
- Response: { “message”: “Profile updated successfully” }
3. Streaming API:
- Start Stream:
- Request: POST /api/stream/start
- Payload: { “title”: “stream title”, “description”: “stream description” }
- Response: { “streamId”: “streamId”, “url”: “stream_url” }
- End Stream:
- Request: POST /api/stream/end
- Payload: { “streamId”: “streamId” }
- Response: { “message”: “Stream ended successfully” }
4. Chat API:
- Get Chat Messages:
- Request: GET /api/chat/{streamId}
- Response: [{ “userId”: “userId”, “message”: “chat_message”, “timestamp”: “timestamp” }]
- Send Chat Message:
- Request: POST /api/chat/{streamId}
- Payload: { “userId”: “userId”, “message”: “chat_message” }
- Response: { “messageId”: “messageId”, “timestamp”: “timestamp” }
5. Notification API:
- Send Notification:
- Request: POST /api/notification/send
- Payload: { “userId”: “userId”, “message”: “notification message” }
- Response: { “message”: “Notification sent successfully” }
Scalability for Twitch System Design
Knowing the scalability strategies, these can be categorized as: horizontal scaling, load balancing, caching, database sharding and CDNs. This technique of scaling refers to adding more servers to provide the additional resources that are required by the application.
- Load balancers ensure that traffic is well divided in order to be utilized by many servers.
- Amount of data logged by applications mean it reduces load on databases, especially the frequently accessed ones.
- Database partitioning involves the division of a large database into different smaller forms to aid their control.
- CDNs help in making the content delivery faster and by offloading servers, particularly for video streaming applications.
Conclusion
The process of designing an app for live streaming shall follow considerable measures as observed in the Twitch example. These include scalability possibilities, speed, stability, protection from attack, and users’ satisfaction. With microservices architecture and modern technologies, one may establish a very reliable and scalable live streaming for millions of audience.
|
.webp){kind=small}
