How Glovo is protecting their public APIs with a combination of AWS Edge Services

Modern applications often rely on public APIs to exchange information between trusted clients (such as mobile applications or web browsers) and services. Using a combination of HAQM Web Services (AWS) Edge Services (AWS WAF, AWS Shield Advanced, and HAQM CloudFront) allow Glovo to share how they protect their public-facing APIs from a variety of external threats such as vulnerabilities described in the OWASP Top 10, SQL injection or various forms of Distributed Denial of Service (DDoS) attacks.

AWS WAF is a web application firewall. AWS WAF analyzes incoming requests and provides protection against common web exploits, volumetric attacks, and more sophisticated threats such as account creation fraud or bots that attempt to evade detection.

Shield Advanced is a managed DDoS protection service that safeguards applications that are running behind AWS internet-facing resources. Shield Advanced provides DDoS protection for Layers 3–7. Users with Business or Enterprise support can also engage the AWS Shield Response Team (SRT) 24/7 while responding to sophisticated attacks.

CloudFront securely delivers content with low latency and high performance. It acts as a reverse proxy that serves as a single point of entry for your application globally. Viewers anywhere in the world connect to one of CloudFront’s hundreds of Points of Presence (POPs) nearest to them. CloudFront either responds directly from its cache or forwards the request to your application. Because traffic is spread across CloudFront’s edge locations, it makes it an ideal place to protect against DDoS and other application attacks.

Glovo is a pioneering multi-category app connecting users with businesses and couriers, offering on-demand services from local restaurants, grocers and supermarkets, and high street retail stores. Glovo’s vision is to build the largest online marketplace to give everyone access to anything in their city within minutes. Founded in 2015 in Barcelona, it operates across 23 countries in Europe, Central Asia, and Africa.

Initial public API architecture and challenges

Previously, Glovo used to publish their APIs with an Application Load Balancer (ALB) fronting a fleet of third-party API Gateway instances. These instances route traffic to the appropriate internal service.

The following figure depicts Glovo’s initial API architecture:

Figure 1: Glovo’s initial public API architecture

Although this architecture served its purpose during the initial days of Glovo, it didn’t provide the necessary level of protection and availability for their APIs. The following challenges were noted:

• Difficulty blocking malicious external IPs: although Glovo relied on Network Access Control Lists (NACLs) at the subnet level for blocking external IPs, these proved to be cumbersome and prone to error to maintain at scale.
• Observability: Glovo used a combination of ALB and application logs to identify bad actors. This was inefficient and prolonged incidents as multiple tooling and analysis was needed to identify bad actors.
• Rate-limiting: each API has some basic rate-limiting capabilities. While facing volumetric attacks, the APIs were already overloaded and rate-limiting was inefficient.
• Volumetric attacks: unable to identify and block them in a timely manner.

These challenges became evident while facing some of the most popular attacks used by external bad actors:

• Log in by brute force: a different set of bad actors tried to guess the passwords for some of Glovo’s stakeholders by sending floods of log in requests to Glovo’s log in endpoints.
• Credential stuffing: similar to the previous item, bad actors try to use credentials obtained from known external database leaks or the dark web.
• Business logic abuse: in this case bad actors try to abuse a specific application feature exposed by an API.
• Denial of service: this type of attack has a large blast radius as it could bring down specific critical APIs.

In the next section we observe how Glovo redesigned their architecture to tackle the challenges presented and effectively mitigate/stop the most popular attacks used by bad actors.

AWS Edge Services: keeping attackers away from your APIs

Implementing security at the edge means using the CloudFront network infrastructure to implement distributed protection. In combination with AWS WAF and Shield Advanced, only traffic deemed as legitimate would reach the APIs.

The following figure depicts Glovo’s current API architecture:

Figure 2: Glovo’s new architecture based on AWS Edge Services

CloudFront configuration settings are defined through the concept of distribution, in which you tell CloudFront how you want the content to be delivered. Placing CloudFront in front of your APIs means that clients no longer connect directly to them. They are now routed through CloudFront. It is also a best practice to limit access to your origin through a shared secret (header) and a prefix list containing all the Cloudfront IP address ranges.

The AWS WAF configuration is associated with this CloudFront distribution so that any CloudFront POP serving requests enforces the use of AWS WAF. Shield Advanced is also configured to monitor the CloudFront distribution and perform due diligence while monitoring its traffic.

To manage these services, Glovo uses Terraform as an infrastructure as code (IaC) framework so any configuration change is effectively reviewed, approved and deployed as needed. Configuration changes applied to CloudFront or AWS WAF are transparently propagated to all the CloudFront POPs serving traffic for the APIs.

With the new architecture defined, in the next section we dive deeper into some of the design considerations made by Glovo at the time of using each service.

AWS WAF configuration and design principles

AWS WAF rules are defined in the context of a web ACL and define how to inspect HTTP(S) web requests and the action to take on a request when it matches the inspection criteria. Rules are executed sequentially and their order of processing is configurable. AWS offers a set of AWS Managed Rules curated and maintained by AWS, but you can also create your own rules or use third-party rules available in the AWS Marketplace.

Glovo created their rule structure using a combination of AWS Managed Rules and self-crafted rules.

The following figure depicts how Glovo’s AWS WAF rules are structured with each rounded rectangle representing a rule group. Block/Allow actions aren’t represented:

Figure 3: AWS WAF rule structure defined by Glovo

Every rule is defined with an action, which defines if the rule match is terminating and no other rules are processed (Block/Allow), or rather non-terminating and AWS WAF keeps processing the next rule (count).

As well as actions, each rule is typically labeled by either the AWS Managed Rule itself, by a custom label created by Glovo, or by a combination of both (AWS label + custom label). Labels are useful to mark specific traffic and then apply further logic downstream. They are also visible on AWS WAF logs. Most of the AWS Managed Rules contain their own label.

Business-specific rules

• Trusted IPs: sets of IPs defined by the business that should never be blocked or rate-limited. These IPs are static in nature and include IPs from trusted third-parties or even allowed bots from different providers. These IPs are incorporated into a AWS WAF IP Set for further reference within the AWS WAF rules.
• Malicious IPs: previous incidents tend to yield specific sets of IPs that you would always want to block permanently or for a period of time. These lists of IPs are also produced in joint effort with the SRT team.
• Internal logic: certain API paths should never be exposed or be publicly-accessible.
• Malicious JA3 fingerprints: JA3 fingerprinting is a strong mechanism to identify identical fingerprints on distributed attacks by analyzing AWS WAF logs.

IP reputation rules

• AWS IP reputation list: this is one of the most popular AWS Managed Rules (AWSManagedRulesHAQMIpReputationList) as it contains groups of IPs identified as malicious by HAQM internal threat intelligence. One example of a rule within this group is the AWSManagedIPDDoSList, which contains IP addresses identified as actively engaging in DDoS activities.
• Anonymous IP list: another AWS Managed Rule (AWSManagedRulesAnonymousIpList). In this case, it tries to identify requests that obfuscate viewer identity such as VPNs, proxies, or web hosting providers.

Flagging rules

• Other AWS Managed Rules: Glovo uses a combination of further AWS Managed Rules to flag specific external traffic with labels. At this stage traffic is not blocked and just counted and labeled.
• Internal logic: these rules flag traffic for further processing downstream using custom labels such as the presence of specific HTTP headers or the use of specific paths. Some of the rules add more headers so specific APIs at the origin can apply business logic based on these headers.

Processing and rate-limiting rules

• Block certain traffic: a new set of custom rules use labels and specific request information (headers, cookies, paths, etc.) to rate-limit or block specific requests. Previous labels added in the rules upstream flagged the requests and facilitate applying rate-limits. For example, you might want to rate-limit your log in path for requests coming with IPs labeled in the Anonymous IP list rule.

Red button

• Under attack: in a few occasions, sophisticated distributed attacks needed further analysis to identify their signature. Glovo placed a specific rule at the end of the web ACL to block specific traffic conditions.

Shield Advanced configuration and design principles

The following figure depicts how Shield Advanced is currently configured to protect Glovo’s APIs:

Figure 4: Shield Advanced configuration defined by Glovo

While on-boarding into Shield Advanced, Glovo defined an HAQM Route53 health-check that represents their API health. For general guidance, the health-check could be configured to evaluate the API latency based on a CloudWatch alarm (such as monitoring the Origin Latency metric in CloudFront), specific HTTP responses or any other relevant service indicator. This health-check effectively tells the SRT Team when to engage Glovo if there is an active DDoS targeting Glovo’s APIs. In that case, an email is sent to a shared email list so the on-call engineer at Glovo gets paged and is ready to join a conference call hosted by the SRT Team. The SRT Team assists Glovo in identifying the attack vector and making the appropriate configuration changes in AWS WAF to mitigate it.

As well as being engaged directly by the SRT Team, Shield Advanced released Automatic Application Layer DDoS Mitigation in 2021, allowing the threat intelligence engine behind Shield Advanced to place specific AWS WAF rules in the user’s web ACLs when a DDoS event is detected. To do that, Shield Advanced establishes a traffic baseline into the protected resource (CloudFront in this case) so it can learn about the expected traffic patterns over time. Users can use this rule group in Block/Count mode. Glovo has currently placed it in Count mode and can flip the action of the rule as needed.

Besides these functionalities, Glovo shared further items that are instrumental for them in the way they use Shield Advanced:

• SRT team onboarding/post-mortem analysis after DDoS events: the SRT team provided onboarding guidance during Glovo’s early days using Edge Services by helping them architect their rule structure. Furthermore, after every impactful DDoS event, the SRT team provided a detailed postmortem analysis and recommendations to enhance Glovo’s security posture.
• AWS WAF request fees: when scaling your APIs to handle more than one million requests per minute as Glovo, the cost per request in AWS WAF becomes relevant. Shield Advanced absorbs this cost if using AWS Managed Rules within the categories Baseline, use-case, or IP reputation. It also absorbs the cost for your own created rules. In both cases, the request fee is waived if rules are part of the default 1500 WCU limit per web ACL. This is the pricing page for AWS Shield if you need more details.

In terms of observability, Glovo relies on a combination of AWS WAF, Shield Advanced, and CloudFront metrics to monitor traffic patterns and being able to define alarms such as the percentage of requests being Blocked/Counted per AWS WAF rule or DDoS events detected by Shield Advanced. On the other hand, AWS WAF logs act as a single pane of glass for a detailed view of each request (along with its labels).

Lessons learned by operating AWS Edge Services

Aligned with AWS best practices, these are some remarkable learnings shared by the Glovo team while using AWS Edge Services over the years:

1. Restrict access to your origin: your origin (ALB in this case) should only accept requests from your CloudFront distribution. Restrict access using both a shared secret and an AWS-managed prefix list that contains the CloudFront IP ranges.
2. Count, Analyze, and Block when adding new AWS WAF rules: no one wants to Block legitimate traffic. Upon adding new AWS WAF rules, make sure you always add them in Count mode before analysing the traffic hitting the rule and eventually moving them to Block as needed.
3. Rule versioning: Most of the AWS Managed Rules are versioned. Pin a specific rule version and roll out new versions in a controlled fashion. There is also an HAQM Simple Notification Service (HAQM SNS) topic that announces new versions being released.

Final outcome and next steps

In this post, we have covered how Glovo implemented a combination of AWS Edge Services to protect their public APIs, facilitating the identification and mitigation of external threats while improving the overall API protection and traffic observability.

According to Glovo’s metrics, Shield Advanced detects an average of 12 DDoS events per month (mostly Application-layer events blocked by AWS WAF) with an average of 200k requests per second blocked.

Looking into the future, Glovo is already observing new functionalities and automating some of the rule management:

• About adding new functionalities, Glovo is interested in adding further logic to filter unwanted bot traffic. This bot filtering is achieved by using AWS Bot Control rules, by presenting bots with CAPTCHA/Challenges, or by a combination of both. Although not mandatory, these methodologies typically necessitate integrating with the client SDK (mobile SDKs in the case of Glovo) for assessing the attacker profile. These functionalities aren’t included in the Shield Advanced pricing.
• Another new functionality being evaluated by Glovo is CloudFront VPC Origins, which was announced during re:Invent 2024, and is intended to eliminate the need for applications to be exposed on the public internet by restricting access solely through users’ CloudFront distributions.
• Building automation at the time of updating certain AWS WAF rules. Glovo is evaluating adding some intelligence at the time of analyzing JA3 fingerprints so “bad fingerprints” get automatically populated into the “Malicious JA3 fingerprints” rule group. The Security Automations for AWS WAF is a good starting point for building these solutions.

About the authors