System Design Fundamentals

Basic Application Architecture

Developer Perspective

A typical application is composed of:

• Build & Deploy Layer — CI/CD pipeline
• Server Layer — handles incoming requests
• Storage Layer — persists application data (can be external)
• Logging Storage Server — logs all events
• Metric Storage Server — logs all metrics
• Alert Server — alerts when something goes down

User Perspective

Users simply make requests to the server and receive responses.

Scaling

• Vertical Scaling: add more physical resources (CPU, RAM) to a single server
• Horizontal Scaling: add more servers to distribute the load
• Load Balancer: distributes traffic between multiple servers

Design Requirements

How We Handle Data

Metrics for Evaluating Our Design

Availability — measures uptime of a system:

Example:

SLI — Service Level Indicator: measures performance of a system

SLO — Service Level Objective: target goals for SLI (e.g., 99.9% availability)

SLA — Service Level Agreement: contracts with customers for SLO

Reliability — measures the probability of a system failing

Fault Tolerance — measures how much our system can continue working under failure

Redundancy — measures how much our system is replicated

Throughput — measures the number of operations in a time frame:

Latency — measures how long it takes to complete a request

Networks

Basics

Router — sends packages across the network

Basic Protocols

API Design

Client-Server Model

• Client — accesses information provided by a server
• Server — provides resources

RPC — Remote Procedure Call

The RPC model allows a program to execute code in a different machine or address space.

Key concepts:

• Stub — function proxy on both local and remote server
• Marshalling — packaging parameters into a message ready to be sent
• Unmarshalling — unpacking parameters from the received message

HTTP / HTTPS

HTTP is the protocol for client-server communication. TLS/SSL encrypts traffic between client and server.

Methods:

Status Codes:

Cons: HTTP is not ideal for real-time data exchange

WebSockets

WebSockets establish a persistent, full-duplex connection between client and server — unlike REST where each request creates a new connection.

Pros:

• Bi-directional communication
• Great for real-time data (chat, live feeds)
• Supports server push and polling

API Paradigms

REST

• Stateless — server doesn’t maintain session state
• Pagination — page through resources
• Cacheable — responses can be cached
• Resource-identified — resources have unique URIs, exchanged in JSON
• Layered — client and server are decoupled; can talk to replicas transparently

GraphQL

• Uses only POST requests
• Body contains queries for fetching data
• Less cacheable compared to REST

gRPC

• Typically used for server-to-server communication
• Implements RPC with schema-defined data
• Faster than REST
• Supports bi-directional streaming
• Uses exceptions instead of status codes
• Built on HTTP/2

API Design Best Practices

API Contract — defines the structure of your API

Handling API changes:

• Adding parameters → make them optional
• Removing parameters → can break external customers

Pagination — limit the number of returned objects:

GET https://api.example.com/v1/users/:id/tweets?limit=10&offset=0

HTTP Idempotency — making the same request multiple times produces the same effect:

Caching

Basics of Cache

Cache Hit Ratio:

Client-Side Caching

1. Check memory cache first
2. Cache miss → check disk cache
3. Cache miss → make network request

Server-Side Caching Strategies

Read Through

Best for read-heavy systems: CDNs, social media feeds, user profiles.

Cache Aside (Lazy Loading)

Data is stored in cache only when needed. Best for systems with high read-to-write ratio (prices, descriptions, stock status).

Write Around

Only frequently accessed data resides in cache. Best for write-heavy systems where data is not immediately read (logging systems).

Write Through

DB and cache are kept in sync. Best for consistency-critical systems — financial applications, online transaction processing.

Write Back

Writes go to cache first, then asynchronously flushed to the database. Ideal for write-heavy scenarios where immediate consistency isn’t critical (logging, social media feeds).

Strategies Summary

Cache Eviction Policies

Eviction Policies determine which elements are removed when the cache is full.

FIFO (First In, First Out)

Implemented with a queue. The oldest item is evicted first.

LRU (Least Recently Used)

Uses a Hash Map + Doubly Linked List:

• Hash Map: key → pointer to a node in the linked list
• Doubly Linked List: maintains access order (head = most recently accessed)

LFU (Least Frequently Used)

Uses two maps and a frequency tracker:

• key_map: key → node
• freq_map: frequency → doubly linked list of nodes
• min_freq: integer tracking the minimum frequency

            key_map
    ┌────────────────────┐
    │ A → Node(freq=3)   │
    │ B → Node(freq=1)   │
    │ C → Node(freq=1)   │
    │ D → Node(freq=2)   │
    └─────────┬──────────┘
              │
              ▼
          freq_map
┌─────────────────────────────────┐
│  freq = 1 → [HEAD ⇄ C ⇄ B ⇄ TAIL] │
│              ↑MRU   ↑LRU           │
│  freq = 2 → [HEAD ⇄ D ⇄ TAIL]     │
│  freq = 3 → [HEAD ⇄ A ⇄ TAIL]     │
└─────────────────────────────────┘
  min_freq = 1

CDN (Content Delivery Network)

CDN Servers are groups of static content cache servers placed close to end users for low latency.

Push CDN

The origin server pushes data to CDN edge nodes proactively.

Pull CDN

CDN edge nodes pull data from the origin server on cache miss.

Proxies

Proxies are a middle layer between client and server.

Forward Proxy

• Client identity is hidden from the server
• Proxy can cache results and filter traffic
• Protects the client

Reverse Proxy

• Server identity is hidden from the client
• Proxy forwards requests to the correct backend server
• Handles SSL termination and caching
• Protects the servers

SSL Termination

SSL connection ends at the proxy. Internal traffic from proxy to server uses plain HTTP.

SSL Pass Through

All traffic remains encrypted end-to-end. The proxy forwards encrypted traffic without decrypting.

Load Balancer

A load balancer is a reverse proxy that distributes traffic to backend servers.

Round Robin

Requests cycle through servers in order: A → B → C → A → B → C…

Weighted Round Robin

Same circular pattern, but servers with higher weight receive proportionally more traffic.

Least Connections

Routes new requests to the server with the fewest active connections.

User Location

Routes based on the user’s geographic location for lowest latency.

Regular Hashing

Same IP always maps to the same server:

Problem: If a server goes down, all hash mappings need to be recalculated.

Consistent Hashing

Servers and keys are mapped onto a hash ring. Keys route clockwise to the nearest server. When a server is removed, only its keys are redistributed.

Layer 4 vs Layer 7 Load Balancing

Storage

RDBMS (Relational Databases)

SQL is the language to query relational databases. Data is stored in tables on disk.

B+ Tree Index

Relational databases implement indexing using B+ Trees for O(log n) I/O operations.

Key properties:

• Every node can have m children
• Each node has m-1 values
• Data is stored only on leaf nodes
• Leaf level forms a sorted linked list
• Root/internal nodes are used to efficiently locate data on leaf nodes

Table Structure

Data Schema defines how data is structured:

CREATE TABLE People (
    PhoneNumber int PRIMARY KEY,
    Name varchar(100)
);

Foreign Keys link two tables together. Joins combine tables based on conditions.

ACID Properties

NoSQL

BASE Properties

Eventually Consistent vs Strict Consistency

Eventually Consistent: Client writes to the master node and receives ACK immediately. The master node updates slaves asynchronously.

Strictly Consistent: Data is written to the master and immediately copied to all slaves. Client receives ACK only after all slaves confirm.

Replication and Sharding

Synchronous Replication

Master replicates data on slaves at write time. Used when critical consistency is needed.

Asynchronous Replication

Master replicates data on slaves later. Used when consistency can be eventually achieved.

Master-Master Replication

Multiple masters replicate each other and each replicates its own slaves. Ideal for multi-region setups with one master per region.

Sharding

Sharding divides data into partitions based on a shard key.

Use cases: massive, high-traffic systems

• Relational databases — sharding is done at the application level
• NoSQL databases — horizontal scaling per shard

CAP Theorem

Systems can guarantee only two of the three properties at the same time.