💥

My framework Part 2: Scale from ZERO to MILLIONS users

How to scale a site to 1 Million Users?

image
  • 10 Users: Single virtual machine for simplicity.
  • 100 Users: Separate VMs for backend and database.
  • 1,000 Users: Multiple availability zones, serverless for infrequent tasks, monolith architecture, leader-follower database replication.
  • 10,000 Users: Autoscaling, replicate stateless web servers, cache popular reads, load balancer, CDN for static content, 3-tier architecture.
  • 100,000 Users: Microservices, more availability zones, additional caching, containers and Kubernetes.
  • 1 Million Users: Federate and partition database, deploy servers across multiple regions, global load balancer.

Single server setup

To start with something simple, everything is running on a single server. Figure shows the illustration of a single server setup where everything is running on one server: web app, database, cache, etc.

image
image

Database

With the growth of the user base, one server is not enough, and we need multiple servers: one for web/mobile traffic, the other for the database (Figure 1-3). Separating web/mobile traffic (web tier) and database (data tier) servers allows them to be scaled independently.

image

Which databases to use?

SQL if you have lot of queries and joins.

NoSQL if you need low latency, if data is unstructured, massive amount of data. If you need only serialize and deserialize data.

Vertical scaling vs horizontal scaling

Horizontal scaling is more desirable for large scale applications due to the limitations of vertical scaling.

NEW PROBLEM: In the previous design, users are connected to the web server directly. Users will unable to access the website if the web server is offline. In another scenario, if many users access the web server simultaneously and it reaches the web server’s load limit, users generally experience slower response or fail to connect to the server. A load balancer is the best technique to address these problems.

Load balancer

image

If server 1 goes offline, all the traffic will be routed to server 2. This prevents the website from going offline. We will also add a new healthy web server to the server pool to balance the load.

NEW PROBLEM: The current design has one database, so it does not support failover and redundancy. Database replication is a common technique to address those problems. Let us take a look.

Database replication

A master database generally only supports write operations. A slave database gets copies of the data from the master database and only supports read operations. All the data-modifying commands like insert, delete, or update must be sent to the master database. Most applications require a much higher ratio of reads to writes; thus, the number of slave databases in a system is usually larger than the number of master databases. Figure 1-5 shows a master database with multiple slave databases.

image
‣
Advantages of database replication
‣
What if one of the DB nodes go down?
image

Let us take a look at the design:

  • A user gets the IP address of the load balancer from DNS.
  • A user connects the load balancer with this IP address.
  • The HTTP request is routed to either Server 1 or Server 2.
  • A web server reads user data from a slave database.
  • A web server routes any data-modifying operations to the master database. This includes write, update, and delete operations.

It is time to improve the load/response time. This can be done by adding a cache layer and shifting static content (JavaScript/CSS/image/video files) to the content delivery network (CDN).

[Load/response time] Cache

Temporary storage area that stores the result of expensive responses or frequently accessed data in memory so that subsequent requests are served more quickly

‣
Considerations

[Load/response time] Content delivery network (CDN)

image
‣
Considerations
image
  1. Static assets (JS, CSS, images, etc.,) are no longer served by web servers. They are fetched from the CDN for better performance.
  2. The database load is lightened by caching data.

Stateless web tier

Now it is time to consider scaling the web tier horizontally. For this, we need to move state (for instance user session data) out of the web tier. A good practice is to store session data in the persistent storage such as relational database or NoSQL. Each web server in the cluster can access state data from databases. This is called stateless web tier.

A stateful server remembers client data (state) from one request to the next. A stateless server keeps no state information.

image
image
image

NEW PROBLEM: Your website grows rapidly and attracts a significant number of users internationally. To improve availability and provide a better user experience across wider geographical areas, supporting multiple data centers is crucial.

Data centers → geoDNS-routed (DNS LB)

image

Several technical challenges must be resolved to achieve multi-data center setup:

‣
1. Data synchronization
‣
2. Test and deployment, health-check

Message queue

Decoupling makes the message queue a preferred architecture for building a scalable and reliable application. With the message queue, the producer can post a message to the queue when the consumer is unavailable to process it. The consumer can read messages from the queue even when the producer is unavailable.

image

Logging, metrics, automation

‣
Logging
‣
Metrics
‣
Automation
image
  1. The design includes a message queue, which helps to make the system more loosely coupled and failure resilient.
  2. Logging, monitoring, metrics, and automation tools are included.

NEW PROBLEM: As the data grows every day, your database gets more overloaded. It is time to scale the data tier.

Database scaling

image
image
‣
Considerations
image

In Figure 1-23, we shard databases to support rapidly increasing data traffic. At the same time, some of the non-relational functionalities are moved to a NoSQL data store to reduce the database load.

Summary

  • Keep web tier stateless
  • Build redundancy at every tier
  • Cache data as much as you can
  • Support multiple data centers
  • Host static assets in CDN
  • Scale your data tier by sharding
  • Split tiers into individual services
  • Monitor your system and use automation tools
SuperMade with Super