Client-side Rate Limiting

note

The following policy is based on the Quota Scheduler blueprint.

Overview

When interacting with external APIs or services, adhering to the rate limits imposed by the service provider is crucial. Aperture can model these external API rate limits, enabling client-side implementation in a distributed application. This approach not only prevents potential penalties for exceeding the rate limits, but also allows Aperture to prioritize access to the external API, ensuring essential workloads receive a fair share of the API quota.

Configuration

This policy utilizes a quota scheduler to regulate the control point some-external-api, which represents all outgoing requests made to an external API from services within a distributed application. The rate limit is applied per api_key label. The rate limiter interval is set to 1 second, meaning that the token bucket is replenished with 25 tokens every second, up to a maximum of 500 tokens. Lazy sync is enabled on the rate limiter, which allows the rate limit counters on each Agent to sync four times every interval (1 second).

The WFQ Scheduler prioritizes interactive and background requests with respective priorities of 200 and 50, ensuring that interactive calls receive roughly four times the quota share of background requests.

aperturectl values.yaml

# yaml-language-server: $schema=../../../../../../blueprints/policies/quota-scheduler/gen/definitions.json
# Generated values file for policies/quota-scheduler blueprint
# Documentation/Reference for objects and parameters can be found at:
# https://docs.fluxninja.com/reference/blueprints/policies/quota-scheduler
policy:
  policy_name: client-side-rate-limiting
  quota_scheduler:
    bucket_capacity: 500
    fill_amount: 25
    selectors:
      - control_point: some-external-api
    rate_limiter:
      label_key: api_key
      interval: 1s
      lazy_sync:
        enabled: false
        num_sync: 4
    scheduler:
      workloads:
        - label_matcher:
            match_labels:
              call_type: background
          parameters:
            priority: 50
        - label_matcher:
            match_labels:
              call_type: interactive
          parameters:
            priority: 200

Generated Policy

apiVersion: fluxninja.com/v1alpha1
kind: Policy
metadata:
  annotations:
    fluxninja.com/blueprint-name: policies/quota-scheduler
    fluxninja.com/blueprints-uri: local
    fluxninja.com/values:
      '{"policy": {"policy_name": "client-side-rate-limiting",
      "quota_scheduler": {"bucket_capacity": 500, "fill_amount": 25, "rate_limiter":
      {"interval": "1s", "label_key": "api_key", "lazy_sync": {"enabled": false, "num_sync":
      4}}, "scheduler": {"workloads": [{"label_matcher": {"match_labels": {"call_type":
      "background"}}, "parameters": {"priority": 50}}, {"label_matcher": {"match_labels":
      {"call_type": "interactive"}}, "parameters": {"priority": 200}}]}, "selectors":
      [{"control_point": "some-external-api"}]}}}'
  labels:
    fluxninja.com/validate: "true"
  name: client-side-rate-limiting
spec:
  circuit:
    components:
      - flow_control:
          quota_scheduler:
            in_ports:
              bucket_capacity:
                constant_signal:
                  value: 500
              fill_amount:
                constant_signal:
                  value: 25
            rate_limiter:
              interval: 1s
              label_key: api_key
              lazy_sync:
                enabled: false
                num_sync: 4
            scheduler:
              workloads:
                - label_matcher:
                    match_labels:
                      call_type: background
                  parameters:
                    priority: 50
                - label_matcher:
                    match_labels:
                      call_type: interactive
                  parameters:
                    priority: 200
            selectors:
              - control_point: some-external-api
    evaluation_interval: 10s
  resources:
    flow_control:
      classifiers: []

Policy in Action

The quota scheduler successfully ensures that the request rate remains within the specified limits, as seen below, with a steady state of 25 requests per second. Additionally, the workload decisions panel shows that interactive requests receive approximately four times the acceptance rate compared to background requests.

info

Circuit Diagram for this policy.