Distributed Systems: Principles, Challenges and Architectures

Distributed Systems: Principles, Challenges and Architectures

A distributed system is a collection of independent computers that appears to users as a single coherent system. These computers, often called nodes, work together to achieve common goals, sharing resources and coordinating actions through message passing. Unlike centralized systems where all processing happens on one machine, distributed systems distribute computation and data across multiple networked computers.

Distributed systems power most modern large-scale applications. To understand distributed systems properly, it helps to be familiar with networking fundamentals, client-server model, and concurrency concepts.

┌─────────────────────────────────────────────────────────────────────────┐
│                        Distributed Systems Architecture                   │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│   ┌─────────────┐      ┌─────────────┐      ┌─────────────┐             │
│   │   Node 1    │◄────►│   Node 2    │◄────►│   Node 3    │             │
│   │ (Server A)  │      │ (Server B)  │      │ (Server C)  │             │
│   └─────────────┘      └─────────────┘      └─────────────┘             │
│          │                    │                    │                     │
│          └────────────────────┼────────────────────┘                     │
│                               │                                          │
│                    ┌──────────▼──────────┐                              │
│                    │       Network        │                              │
│                    │   (Message Passing)  │                              │
│                    └─────────────────────┘                              │
│                                                                          │
│   Key Characteristics:                                                   │
│   • Independent nodes                    • No shared memory              │
│   • Single system illusion to users      • Autonomous failures           │
│   • Communication via message passing    • Geographic distribution      │
│                                                                          │
│   Core Goals:                                                            │
│   Scalability ──► Fault Tolerance ──► High Availability                 │
│        │                │                      │                         │
│        ▼                ▼                      ▼                         │
│   Add nodes to      Survive component     Eliminate single points       │
│   handle load       failures              of failure                     │
│                                                                          │
└─────────────────────────────────────────────────────────────────────────┘

What Is a Distributed System?

A distributed system is a software system in which components located on networked computers communicate and coordinate their actions by passing messages. The components interact with each other to achieve a common goal, while the complexity of the distribution is hidden from end users who perceive the system as a single, integrated computing facility.

• Node: An individual computer or processing unit, which can be a physical machine, virtual machine, or container.
• Network: The communication medium connecting nodes, with variable latency, bandwidth, and reliability.
• Message Passing: Communication mechanism where nodes exchange data through messages (no shared memory).
• Transparency: Property that hides distribution complexity, making system appear as a single system.
• Middleware: Software layer facilitating communication and coordination in distributed systems.

Why Distributed Systems Matter

Distributed systems are fundamental to modern computing. They enable applications that no single computer could handle alone.

• Scalability: Add more nodes to handle increased load. Grow from few nodes to thousands without architecture changes.
• Fault Tolerance: Inherent redundancy means failure of one node does not stop entire system.
• High Availability: Design for 99.999% uptime by eliminating single points of failure.
• Performance: Parallel processing across nodes achieves higher throughput than any single machine.
• Geographic Distribution: Placing nodes near users reduces latency for global user base.
• Cost Effectiveness: Many smaller commodity computers cheaper than single large computer.
• Resource Sharing: Enable sharing of data, storage, and computational resources across boundaries.

Types of Distributed Systems

Core Challenges of Distributed Systems

CAP Theorem: In a distributed system, you can have at most two of:

Consistency (C)  ── All nodes see same data at same time
Availability (A) ── Every request receives a response
Partition Tolerance (P) ── System continues despite network partitions

Choose your trade-off:
┌─────────────────────────────────────────────────────────────────────────┐
│ CP System                    │ AP System                                │
│ (Consistent during          │ (Available during partitions)           │
│  partitions)                 │                                          │
│ • May become unavailable    │ • May return stale data                  │
│ • Financial transactions    │ • Social media feeds                     │
│ • Banking systems           │ • Shopping carts                         │
└─────────────────────────────────────────────────────────────────────────┘

In practice, networks always have potential for partitions.
Choose either CP or AP for your use case.

Fallacies of Distributed Computing

1. The network is reliable
   → Networks fail. Packets dropped, connections timeout.

2. Latency is zero
   → Microseconds local vs milliseconds/seconds across networks.

3. Bandwidth is infinite
   → Data transfer takes time and consumes resources.

4. The network is secure
   → Attackers can see, modify, or block messages.

5. Topology doesn't change
   → Nodes join, leave, fail. IP addresses change.

6. There is one administrator
   → Different components may have different owners.

7. Transport cost is zero
   → Sending messages consumes CPU, memory, network.

8. The network is homogeneous
   → Different links have different speeds/reliability.

Key Distributed System Models

Model               Message Delay         Processing      Clock Drift
─────────────────────────────────────────────────────────────────────────────
Synchronous        Known upper bound    Known time      Bounded
Asynchronous       No bounds            No bounds       Unbounded
Partial Synchrony  Unknown bounds,      Bounds exist    Bounded
                   become known later   but unknown

Practical Use:
• Asynchronous – Most real networks, must tolerate
• Partial Synchrony – Practical for consensus algorithms
• Synchronous – Theoretical, rarely matches reality

Consistency Models in Distributed Systems

Pattern            Purpose
─────────────────────────────────────────────────────────────
Leader Election    Single coordinator selection
Quorum             Agreement with subset of nodes
Heartbeat          Failure detection
Gossip             Epidemic information spread
Sharding           Data distribution
Replication        Data redundancy for fault tolerance
Load Balancing     Distribute work across nodes
Rate Limiting      Control request flow

Distributed System Anti-Patterns

• Distributed Monolith: Services tightly coupled, requiring coordinated deployments. Loses all benefits of distribution.
• Starvation of Consensus: Using consensus protocols when simpler approaches work, adding unnecessary overhead.
• Sticky Sessions Assumption: Assuming requests always go to same node. Build stateless services instead.
• Network Is Local Fallacy: Designing as if all communication is local. Test with realistic network conditions.
• Time Assumptions: Using timestamps for ordering without accounting for clock skew. Use logical clocks.
• Retry Storm: Aggressive retry without backoff causing cascading failures. Covered in retry pattern.
• Shared Everything Architecture: Using distributed cache as central coordination point, recreating single point of failure.

Design Questions:
□ Can components fail independently?
□ No single point of failure?
□ Retry logic idempotent?
□ Timeouts configured appropriately?
□ Fallbacks for unavailable dependencies?
□ Monitoring and alerting implemented?
□ Traces across component boundaries?
□ Capacity for adding nodes?
□ Partition tolerance strategy defined?
□ Consistency requirements documented?
□ Failure testing performed regularly?

Distributed System Best Practices

• Design for Failure: Assume any component can fail. Build redundancy, retries, timeouts, and circuit breakers.
• Avoid Distributed Transactions: Use sagas, event-driven patterns, or eventual consistency instead. Covered in saga pattern guide.
• Use Idempotent Operations: Design operations safe to retry. Prevents duplicate execution on network failures.
• Implement Timeouts: Every remote call needs timeout. Propagate deadlines through call chains.
• Instrument Everything: Log correlation IDs, collect metrics, implement distributed tracing.
• Test Realistic Conditions: Test with network latency, packet loss, node failures. Use chaos engineering.
• Prefer Asynchronous Communication: Use message queues and event streams to decouple components.
• Handle Partitions Explicitly: Choose between consistency and availability based on use case.
• Keep Services Stateless: Stateless services scale easily. Store state in databases or caches.
• Monitor Node Health: Implement health checks and heartbeat monitoring. Remove unhealthy nodes automatically.

Real-World Distributed Systems

Type                    Examples                           Use Case
─────────────────────────────────────────────────────────────────────────────
Distributed Databases   Cassandra, CockroachDB, Spanner    Data distribution
Distributed Computing   Hadoop, Spark                      Job distribution
Message Brokers         Kafka, RabbitMQ                    Message distribution
Coordination Services   ZooKeeper, etcd, Consul            Leader election, locks
Blockchain              Bitcoin, Ethereum                  Untrusted consensus

Frequently Asked Questions

1. What is the difference between distributed and parallel systems?
   Parallel systems focus on using multiple processors to solve a single computational problem quickly, with tightly coupled processors and often shared memory. Distributed systems focus on coordinating independent computers that may be geographically separated, with looser coupling and message passing. Parallel systems emphasize performance. Distributed systems emphasize scalability and fault tolerance.
2. What is the CAP theorem and why does it matter?
   CAP theorem states distributed systems cannot simultaneously provide Consistency, Availability, and Partition Tolerance. Since partitions are inevitable, you must choose between consistency and availability when partition occurs. This choice affects system design and behavior during network failures.
3. What is the difference between AP and CP systems?
   AP systems prioritize availability during network partitions, serving potentially stale data rather than refusing requests. CP systems prioritize consistency, refusing requests or becoming unavailable during partitions rather than returning inconsistent data. Choose based on whether stale data or downtime is more acceptable for your use case.
4. When should I use eventual consistency?
   Use eventual consistency when absolute consistency is not required immediately, when high availability and low latency are priorities, and when conflicts can be resolved later. Many real-world systems use eventual consistency for user data, shopping carts, and analytics.
5. What is the difference between a log and a database?
   A log is append-only sequence of records, optimized for sequential writes and reads. A database stores current state, supporting random access and updates. Many systems combine both: logs for replication and databases for state storage.
6. What should I learn next after distributed systems?
   After mastering distributed systems, explore consensus algorithms like Raft, distributed transactions and sagas, distributed database internals, distributed tracing, stream processing, and chaos engineering.