Skip to main content

Load Balancer Component

The Load Balancer component distributes incoming traffic across multiple API Service instances, simulating how real-world systems scale to handle increased load and ensure high availability.

Overview​

In production systems, a single server can only handle so many requests before it becomes overwhelmed. Load balancers solve this by distributing traffic across multiple servers, ensuring no single server becomes a bottleneck.

Whether you're planning production architecture, preparing for system design discussions, or learning distributed systems, the Load Balancer component helps you model and visualize traffic distribution patterns.

Real-World Example

Think of a restaurant with one cashier vs. multiple cashiers:

  • One cashier (no load balancer): Long line, slow service, frustrated customers
  • Multiple cashiers with a host (load balancer): Host directs customers to available cashiers, faster service, happy customers

The load balancer is the "host" that decides which API Service (cashier) handles each request.

When to Use Load Balancer​

Use Load Balancer when you want to:

  • ✅ Simulate traffic distribution across multiple servers
  • ✅ Demonstrate horizontal scaling concepts
  • ✅ Show how systems handle increased load
  • ✅ Design high-availability architectures
  • ✅ Model redundancy and fault tolerance

You don't need Load Balancer if:

  • ❌ You only have one API Service
  • ❌ You're building a simple proof-of-concept
  • ❌ Your system doesn't need to scale horizontally

How Load Balancer Works​

Basic Flow​

User Request → Load Balancer → API Service #1 → Database
                             → API Service #2 → Database
                             → API Service #3 → Database
  1. User Request sends a request to the Load Balancer
  2. Load Balancer selects an API Service using its routing algorithm
  3. Selected API Service processes the request
  4. Database stores or retrieves data
  5. Response flows back through the same path

Architecture Example​

┌─────────────────┐
│  User Request   │
│   POST /users   │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Load Balancer   │
│    (Path)       │
└────┬─────┬──────┘
     │     │
     │     └──────────┐
     │                │
     ▼                ▼
┌─────────┐      ┌─────────┐
│API Svc 1│      │API Svc 2│
│(Validate│      │(Validate│
│ Email)  │      │ Email)  │
└────┬────┘      └────┬────┘
     │                │
     └────────┬───────┘
              ▼
       ┌──────────┐
       │ Database │
       └──────────┘

Component Configuration​

Routing Strategy​

The Load Balancer uses path-based routing to direct requests to the appropriate API Service instance.

How it works:

  • Each API Service connection is identified by a path segment (e.g., /api1, /api2)
  • The Load Balancer extracts the path from the incoming request endpoint
  • Requests are routed to the API Service matching the path segment
  • If no path matches, the request goes to the first connected API Service

Example routing:

Request endpoint: /api1/users → Routes to API Service #1
Request endpoint: /api2/users → Routes to API Service #2
Request endpoint: /users      → Routes to first API Service (default)
Routing Flexibility

Path-based routing gives you control over which requests go to which servers, making it ideal for:

  • A/B testing - Route specific users to different service versions
  • Gradual rollouts - Direct a percentage of traffic to new features
  • Service specialization - Route different endpoints to specialized servers

Connected Servers​

The number of connected servers is automatically detected based on your connections:

  • 1 connection = Load Balancer forwards to 1 API Service
  • 2 connections = Load Balancer can route between 2 API Services
  • 3+ connections = Load Balancer distributes across multiple API Services
tip

Connect multiple API Services to demonstrate horizontal scaling and traffic distribution patterns.

Setting Up Load Balancer​

Step 1: Add Components​

1. Add User Request component
2. Add Load Balancer component
3. Add multiple API Service components (2 or more recommended)
4. Add Database component

Step 2: Create Connections​

User Request → Load Balancer
Load Balancer → API Service #1
Load Balancer → API Service #2
Load Balancer → API Service #3
API Service #1 → Database
API Service #2 → Database
API Service #3 → Database

Step 3: Configure Request Routing​

Set the User Request endpoint to include the path segment:

Route to API Service #1:

Endpoint: /api1/users
Method: POST
Properties: name=John, email=john@example.com

Route to API Service #2:

Endpoint: /api2/users
Method: POST
Properties: name=Jane, email=jane@example.com

Step 4: Configure API Services (Optional)​

You can give each API Service the same code (realistic) or different code (for demonstration):

Same Code (Production-like):

def process_request(input_data):
    data = input_data.get('data', {})

    return {
        'operation': 'INSERT',
        'columns': [
            {'name': 'name', 'type': 'TEXT'},
            {'name': 'email', 'type': 'TEXT'}
        ],
        'data': {
            'name': data.get('name'),
            'email': data.get('email')
        }
    }

Different Code (For Tracking):

Tag each server's requests to see which one handled the request:

# API Service #1
def process_request(input_data):
    data = input_data.get('data', {})

    return {
        'operation': 'INSERT',
        'columns': [
            {'name': 'server_id', 'type': 'TEXT'},
            {'name': 'name', 'type': 'TEXT'},
            {'name': 'email', 'type': 'TEXT'}
        ],
        'data': {
            'server_id': 'server-1',
            'name': data.get('name'),
            'email': data.get('email')
        }
    }
# API Service #2
def process_request(input_data):
    data = input_data.get('data', {})

    return {
        'operation': 'INSERT',
        'columns': [
            {'name': 'server_id', 'type': 'TEXT'},
            {'name': 'name', 'type': 'TEXT'},
            {'name': 'email', 'type': 'TEXT'}
        ],
        'data': {
            'server_id': 'server-2',
            'name': data.get('name'),
            'email': data.get('email')
        }
    }

Step 5: Run Simulation​

Run your simulation and check the logs to see which API Service handled the request based on the path.

Simulation Behavior​

What You'll See in Logs​

When you run a simulation with a Load Balancer, the execution logs show:

🟡 Load Balancer: LB-1
ℹ️  Routing Strategy: Path-based
ℹ️  Available Servers: 3
ℹ️          [1] API-1
ℹ️          [2] API-2
ℹ️          [3] API-3
ℹ️  Requested path: /api2/users
ℹ️  Status: Routing to matching server...
✅ Routed to: API-2
ℹ️  Flow: LB-1 --> API-2

Visual Feedback​

During simulation:

  1. Load Balancer component highlights when active
  2. Connection to selected API Service highlights
  3. Selected API Service component highlights
  4. Connection to Database highlights

This helps you visualize the request path through your system.

Path-Based Routing Explained​

How Path Matching Works​

The Load Balancer extracts the path segment from your endpoint and routes accordingly:

Endpoint Structure: /<path>/<resource>
Example: /api1/users
         └──┬──┘
         Path segment used for routing

Routing Logic:

  1. Extract path segment from endpoint (e.g., /api1/users → api1)
  2. Find API Service connection matching the path
  3. Route request to that API Service
  4. If no match found, use first connected API Service as default

Routing Examples​

Example 1: Explicit routing

User Request endpoint: /api1/users
Connected API Services:
  - API-1 (connected via "api1" path)
  - API-2 (connected via "api2" path)

Result: Request routes to API-1 ✅

Example 2: Default routing

User Request endpoint: /users
Connected API Services:
  - API-1 (connected via "api1" path)
  - API-2 (connected via "api2" path)

Result: Request routes to API-1 (first in list) ✅

Example 3: Version-based routing

User Request endpoint: /v1/users → Routes to API-1 (legacy version)
User Request endpoint: /v2/users → Routes to API-2 (new version)

Benefits of Path-Based Routing​

Advantages:

  • ✅ Deterministic - Same path always routes to same server
  • ✅ Controllable - You decide which requests go where
  • ✅ Testable - Easy to test specific server instances
  • ✅ Flexible - Supports A/B testing and gradual rollouts
  • ✅ Clear - Routing logic is explicit and visible

Best for:

  • Testing different service versions
  • Implementing feature flags via paths
  • Routing to specialized servers (e.g., read vs. write servers)
  • Gradual migration strategies

Practical Examples​

Example 1: A/B Testing Different Features​

Goal: Route users to different feature versions

Setup:

User Request (/featureA/users) → Load Balancer → API-1 (old feature)
User Request (/featureB/users) → Load Balancer → API-2 (new feature)

API Service #1 (Control):

def process_request(input_data):
    data = input_data.get('data', {})

    return {
        'operation': 'INSERT',
        'columns': [
            {'name': 'feature_version', 'type': 'TEXT'},
            {'name': 'name', 'type': 'TEXT'}
        ],
        'data': {
            'feature_version': 'A',
            'name': data.get('name')
        }
    }

API Service #2 (Treatment):

def process_request(input_data):
    data = input_data.get('data', {})

    return {
        'operation': 'INSERT',
        'columns': [
            {'name': 'feature_version', 'type': 'TEXT'},
            {'name': 'name', 'type': 'TEXT'},
            {'name': 'experimental_field', 'type': 'TEXT'}
        ],
        'data': {
            'feature_version': 'B',
            'name': data.get('name'),
            'experimental_field': 'new_value'
        }
    }

Result: Query database to compare behavior between versions.

Example 2: Read/Write Separation​

Goal: Route reads and writes to different servers

Setup:

GET  /read/users  → Load Balancer → API-1 (read-optimized)
POST /write/users → Load Balancer → API-2 (write-optimized)

Read Server (API-1):

def process_request(input_data):
    return {
        'operation': 'SELECT',
        'table': 'users'
    }

Write Server (API-2):

def process_request(input_data):
    data = input_data.get('data', {})

    return {
        'operation': 'INSERT',
        'columns': [
            {'name': 'name', 'type': 'TEXT'},
            {'name': 'email', 'type': 'TEXT'}
        ],
        'data': data
    }

Example 3: Gradual Service Migration​

Goal: Gradually move traffic from old to new service

Setup:

/v1/users → Load Balancer → API-1 (legacy service)
/v2/users → Load Balancer → API-2 (new service)

Test new service with small traffic percentage, then gradually increase by updating client endpoints from /v1/ to /v2/.

Example 4: High Availability Demonstration​

Goal: Show system continues working when one server fails

Setup:

User Request → Load Balancer
                   ↓ ↓
                API-1  API-2 (disconnected)
                   ↓
                Database

What happens:

  • Requests to /api1/users work normally
  • Requests to /api2/users fall back to first available server
  • System remains operational despite partial failure
Real-World Parallel

This demonstrates how load balancers provide high availability. If one server goes down, you can reconfigure routing or the system can fall back to available servers.

Best Practices​

Whether you're designing production systems or building examples for learning, these practices will help you create clear, maintainable system designs:

  1. Use Descriptive Path Segments - Choose clear paths like /v1, /v2, /read, /write that communicate intent
  2. Document Your Routing Strategy - Add descriptions explaining which paths route where and why
  3. Keep Code Consistent - In production, all instances should run identical code unless doing A/B testing
  4. Test Each Path - Run simulations for each routing path to verify behavior
  5. Start Simple - Begin with 2 servers, then scale up as needed
  6. Plan for Fallbacks - Understand default routing behavior when no path matches

Use Cases​

Professional Architecture & Team Collaboration​

  • Model production traffic routing - Design and document how requests flow through your system at scale
  • Plan service scaling strategies - Test different horizontal scaling approaches before implementation
  • Design high-availability systems - Show redundancy and failover patterns to your team
  • A/B testing architectures - Plan feature rollouts and gradual migrations with path-based routing
  • Read/write separation - Design specialized server configurations for different operations
  • Service versioning - Model how to run multiple API versions simultaneously (v1, v2)
  • Collaborate on scaling decisions - Share working examples with your team to discuss trade-offs

Interview Preparation & Portfolio​

  • Demonstrate scaling knowledge - Show how you'd horizontally scale a production system
  • Explain routing trade-offs - Use concrete examples to discuss different load balancing strategies
  • Present failover scenarios - Illustrate high availability patterns with working simulations
  • Build system design portfolio - Create executable examples of distributed system patterns

Learning & Education​

  • Understand horizontal scaling - See how traffic distribution works across multiple servers
  • Experiment with routing strategies - Test path-based routing, versioning, and feature flags
  • Learn high availability patterns - Explore redundancy and failover mechanisms safely

Status Information​

The Load Balancer component displays:

Type: Load Balancer
Routing: Path-based
Servers: Number of connected API Services (auto-detected)

Integration with Other Components​

With User Request​

Connection:

User Request → Load Balancer

The User Request sends its HTTP request (with path) to the Load Balancer, which routes based on the endpoint path.

With API Services​

Connection:

Load Balancer → API Service #1 (path: /api1)
Load Balancer → API Service #2 (path: /api2)
Load Balancer → API Service #3 (path: /api3)

Each connection represents an available server. Path segments in your endpoints determine which server handles each request.

With Database​

Indirect Connection:

Load Balancer → API Services → Database

The Load Balancer doesn't connect directly to the Database. Instead, it routes to API Services, which then interact with the Database.

Troubleshooting​

"No API Services connected"​

  • Cause: Load Balancer has no outgoing connections
  • Solution: Connect Load Balancer to at least one API Service component

Request not routing to expected server​

  • Cause: Path segment in endpoint doesn't match API Service path
  • Solution: Verify your endpoint path matches the intended API Service path (e.g., /api1/users for api1 path)

All requests go to same server​

  • Cause: Not using different path segments in requests
  • Solution: Use different paths in your User Request endpoint (e.g., /api1/users, /api2/users)

Load Balancer shows 0 servers​

  • Cause: No connections from Load Balancer to API Services
  • Solution: Draw connections from Load Balancer to your API Service components

Can't determine routing behavior​

  • Cause: Endpoint doesn't include path segment
  • Solution: Use format /<path>/<resource> in User Request endpoint, or check logs to see which server was selected

Next Steps​

  • Learn about API Service Component for custom business logic and request processing
  • Explore Database Component for data persistence and querying
  • Review User Request Component for configuring incoming requests
  • Try our Templates to see Load Balancer patterns in production-like scenarios
  • Share your architecture designs with your team for collaborative system planning