Glossary of System Design Basics

Load Balancing

We’ll cover the following

Benefits of Load Balancing Load Balancing Algorithms Redundant Load
Balancers

Load Balancer (LB) is another critical component of any distributed system. It helps to spread the traffic across a cluster of servers to improve responsiveness and availability of applications, websites or databases. LB also keeps track of the status of all the resources while distributing requests. If a server is not available to take new requests or is not responding or has elevated error rate, LB will stop sending traffic to such a server.

Typically a load balancer sits between the client and the server accepting incoming network and application traffic and distributing the traffic across multiple backend servers using various algorithms. By balancing application requests across multiple servers, a load balancer reduces individual server load and prevents any one application server from becoming a single point of failure, thus improving overall application availability and responsiveness.

负载均衡器（LB）是任何分布式系统的另一个关键组件。它有助于将流量分布在服务器集群上，以提高应用程序、网站或数据库的响应能力和可用性。LB还在分发请求时跟踪所有资源的状态。如果服务器无法接收新请求或没有响应或错误率升高，LB将停止向此类服务器发送流量。

通常，负载均衡器位于客户机和服务器之间，接受传入的网络和应用程序流量，并使用各种算法将流量分布在多个后端服务器上。通过跨多个服务器平衡应用程序请求，负载平衡器可以减少单个服务器的负载，并防止任何一个应用程序服务器成为单一故障点，从而提高应用程序的总体可用性和响应能力。

To utilize full scalability and redundancy, we can try to balance the load at each layer of the system. We can add LBs at three places:

• Between the user and the web server
• Between web servers and an internal platform layer, like application servers or cache servers
• Between internal platform layer and database.
  

Benefits of Load Balancing

Users experience faster, uninterrupted service. Users won’t have to wait for a single struggling server to finish its previous tasks. Instead, their requests are immediately passed on to a more readily available resource.

Service providers experience less downtime and higher throughput. Even a full server failure won’t affect the end user experience as the load balancer will simply route around it to a healthy server.

Load balancing makes it easier for system administrators to handle incoming requests while decreasing wait time for users.

Smart load balancers provide benefits like predictive analytics that determine traffic bottlenecks before they happen. As a result, the smart load balancer gives an organization actionable insights. These are key to automation and can help drive business decisions.

System administrators experience fewer failed or stressed components. Instead of a single device performing a lot of work, load balancing has several devices perform a little bit of work.

用户体验更快、不间断的服务。用户不必等待一台苦苦挣扎的服务器完成之前的任务。相反，他们的请求会立即传递给更容易获得的资源。
服务提供商经历更少的停机时间和更高的吞吐量。即使是一个完整的服务器故障也不会影响最终用户的体验，因为负载平衡器只是将其路由到一个健康的服务器。
负载平衡使系统管理员更容易处理传入请求，同时减少用户的等待时间。
智能负载平衡器提供了一些好处，比如在流量瓶颈发生之前就确定它们的预测分析。因此，智能负载平衡器为组织提供了可操作的见解。这些是自动化的关键，有助于推动业务决策。
系统管理员遇到的故障或压力组件更少。负载平衡不是让单个设备执行大量工作，而是让多个设备执行少量工作。

Load Balancing Algorithms

How does the load balancer choose the backend server?
Load balancers consider two factors before forwarding a request to a backend server. They will first ensure that the server they choose is actually responding appropriately to requests and then use a pre-configured algorithm to select one from the set of healthy servers. We will discuss these algorithms shortly.

Health Checks - Load balancers should only forward traffic to “healthy” backend servers. To monitor the health of a backend server, “health checks” regularly attempt to connect to backend servers to ensure that servers are listening. If a server fails a health check, it is automatically removed from the pool, and traffic will not be forwarded to it until it responds to the health checks again.

There is a variety of load balancing methods, which use different algorithms for different needs.

Least Connection Method — This method directs traffic to the server with the fewest active connections. This approach is quite useful when there are a large number of persistent client connections which are unevenly distributed between the servers.
Least Response Time Method — This algorithm directs traffic to the server with the fewest active connections and the lowest average response time.
Least Bandwidth Method - This method selects the server that is currently serving the least amount of traffic measured in megabits per second (Mbps).
Round Robin Method — This method cycles through a list of servers and sends each new request to the next server. When it reaches the end of the list, it starts over at the beginning. It is most useful when the servers are of equal specification and there are not many persistent connections.
Weighted Round Robin Method — The weighted round-robin scheduling is designed to better handle servers with different processing capacities. Each server is assigned a weight (an integer value that indicates the processing capacity). Servers with higher weights receive new connections before those with less weights and servers with higher weights get more connections than those with less weights.
IP Hash — Under this method, a hash of the IP address of the client is calculated to redirect the request to a server.

负载平衡器如何选择后端服务器？
负载平衡器在将请求转发到后端服务器之前会考虑两个因素。他们将首先确保他们选择的服务器实际上对请求做出了适当的响应，然后使用预先配置的算法从一组正常的服务器中选择一个。我们将很快讨论这些算法。
运行状况检查-负载平衡器应该只将流量转发到“正常”的后端服务器。为了监视后端服务器的运行状况，“运行状况检查”定期尝试连接到后端服务器，以确保服务器正在侦听。如果服务器未通过运行状况检查，它将自动从池中删除，并且在它再次响应运行状况检查之前，不会将流量转发给它。
有各种各样的负载平衡方法，它们根据不同的需要使用不同的算法。
最少连接方法-此方法将流量定向到活动连接最少的服务器。当服务器之间存在大量不均匀分布的持久客户端连接时，这种方法非常有用。
最小响应时间方法-此算法将流量定向到活动连接最少且平均响应时间最低的服务器。
最小带宽方法-此方法选择当前服务于最小流量（以兆位每秒（Mbps）为单位）的服务器。
Round Robin Method—此方法在服务器列表中循环，并将每个新请求发送到下一个服务器。当它到达列表的末尾时，它会从头开始。当服务器的规格相同并且没有太多持久连接时，它最有用。
加权循环法-加权循环调度旨在更好地处理具有不同处理能力的服务器。为每个服务器分配一个权重（表示处理能力的整数值）。权重较高的服务器在权重较低的服务器之前接收新连接，权重较高的服务器比权重较低的服务器获得更多连接。
IP哈希-在此方法下，计算客户端IP地址的哈希，将请求重定向到服务器。

Redundant Load Balancers

The load balancer can be a single point of failure; to overcome this, a second load balancer can be connected to the first to form a cluster. Each LB monitors the health of the other and, since both of them are equally capable of serving traffic and failure detection, in the event the main load balancer fails, the second load balancer takes over.

负载平衡器可以是单点故障；为了克服这一点，可以将第二个负载平衡器连接到第一个负载平衡器以形成集群。每个负载均衡都监视另一个负载均衡的运行状况，由于这两个负载均衡都具有相同的服务流量和故障检测能力，因此在主负载均衡出现故障时，第二个负载均衡将接管。

Following links have some good discussion about load balancers:
[1] What is load balancing
[2] Introduction to architecting systems
[3] Load balancing

Caching

We’ll cover the following

Application server cache Content Delivery (or Distribution) Network
(CDN) Cache Invalidation Cache eviction policies

Load balancing helps you scale horizontally across an ever-increasing number of servers,