The pros and cons of the Throttling architecture pattern

Having a grasp of common architectural patterns is essential to designing software architecture at scale. Using them saves not only time but also ensures a reliable implementation of your design. There’s no need to reinvent the wheel when there’s an architectural pattern available that applies to an architecture you’re developing.

The following is a brief overview of the Throttling architectural pattern.

Understanding the Throttling pattern

The Throttling pattern, also known as Rate Limiting, is one in which a throttle is placed in front of a target service or process to control the rate at which data flow into the target. Throttling is a technique that ensures that the flow of data being sent into a target can be digested at an acceptable rate. Should the target become overwhelmed, the throttle will slow down or even stop calls to the target.

In addition to being used as a defensive measure, the Throttling pattern can be used by consumers of cloud providers as a way to control consumption costs. Consumers can use tools to set a rate limit on calls made to a specific third-party service. This limit is usually defined as calls per second. Thus, clients avoid incurring unanticipated costs.

Pros

• The Throttling pattern prevents catastrophic failure, particularly in terms of a denial-of-service (DDoS) attack. A DDoS is one in which a malevolent actor assaults an endpoint with a high number of calls that obstruct other calls from getting access to that endpoint. A throttle will detect the increase in traffic caused by the attack and react. That reaction might be to block traffic coming in from the malevolent IP address or to slow down the rate of traffic in general.
• The Throttling pattern can be used to control infrastructure expenses, either financial or technical, by enforcing rates of usage of a particular process or service. Setting a throttle ensures that the target does not become operationally overwhelmed. Or, the throttling pattern can be used to avoid incurring unexpected charges from commercial managed services.

Cons

• If a system does not have adequate redundancy mechanisms in place overall performance can degrade when key targets are slowed down due to the threshold imposed by the given throttle.
• Implementation of the throttling pattern will vary according to what’s being throttled. Thus, specific engineering expertise is required. The technique used to throttle messages being emitted to a message broker is much different than the technique used to throttle HTTP calls to a particular brand of web server.
• The throttling pattern is difficult to do as a one-off solution implemented in isolation. Often the Throttling architectural pattern is used in conjunction with the Circuit Breaker pattern in order to maintain service availability. For example, when a target reaches a threshold that requires throttling, a circuit breaker is used to reroute traffic to another target that has identical functionality, thus reducing the load on the first target. Typically implementing this type of throttling behavior requires infrastructure management technologies such as Kubernetes Horizontal Pod Autoscaler or a service mesh solution such as Istio or Linkerd.

Putting it all together

The essential benefit of the Throttling pattern is that it allows a system to control both internal and external traffic that might compromise a system's ability to operate in a safe and predictable manner. Unanticipated bursts in call activity do happen; sometimes by accident; something by nefarious intention. The Throttling pattern mitigates the impact of such bursts.

Client-side throttling will control the rate at which calls are made to commercial services that calculate fees according to usage activity. Thus, the Throttling pattern is used as a cost-control mechanism.

Using the Throttling pattern internally on the server-side protects a system from the DDoS acts from bad actors. Also, internal throttling protects the system from unanticipated burst activity from other internal processes and services. The tradeoff is that when throttling is in effect, the system might slow down and thus overall system performance can be affected. Hence, the Throttling Pattern is often used in conjunction with the Circuit Breaker pattern.