Kleppmann, Martin & Riccomini, Chris. Designing Data-Intensive Applications - The Big Ideas Behind Reliable, Scalable and Maintainable Systems. 2nd Ed. O’Reilly Media, 2026.
Chapter 1 - Trade-Offs in Data Systems Architecture
- Data-intensive vs compute-intensive
- Frontends and backends
- Operational Systems (backend engineers) vs Analytical Systems (business analysts and data scientists)
- Data engineers and Analytics engineers
- A transaction, a point query, Online Transaction Processing (OLTP)
- Online Analytical Processing (OLAP)
- Operational System (OLTP) vs Analytical Systems (OLAP)
- Fixed sql queries in oltp vs arbitrary sql queries in olap
- Tableau, Looker, Microsoft Power BI
- Product Analytics or Real-time analytics - Pinot, Druid, ClickHouse
- A data warehouse
- Undesirable for business analysts and data scientists to query otlp systems, for several reasons,
- Data silos
- Extract-transform-load (ETL),
- Sometimes ELT, transformation is done in the data warehouse, after loading
- ETL for SaaS APIs,
- Data connector services such as Fivetran, Singer, Airbyte
- Hybrid transactional/analytical processing (HTAP)
- Hundreds of seperate operational databases, but usually one data warehouse for an enterprise
- A data warehouse often uses relational data with SQL
- Training an ML model - features, feature engineering
- Python data analysis libraries - pandas, scikit-learn
- Statistical analysis languages - R
- Distributed analytics frameworks - Spark
- A data lake
- File formats - avro, parquet
- Commoditized file storage - object stores
- ETL processes - Data pipelines
- Sushi principle - raw data is better
- DataOps Manifesto
- General Data Protection Regulation (GDPR)
- California Consumer Privacy Act (CCPA)
- Files and relational tables - stream of events, stream processing
- Output of analytical systems made available to operational systems - Reverse ETL
- Machine learning models can be deployed to operational systems - TFX, Kubeflow, MLFlow
- Systems of record (source of truth),
- Derived data systems
- Pros and cons of cloud services
- Cloud native system architecture - an architecture that is designed to take advantage of cloud services
- Examples of self-hosted and cloud native database systems (OLTP and OLAP)
- Remote direct memory access (RDMA) network interfaces
- The key idea of cloud native services - to build upon lower-level cloud services to create higher-level services
- Object storage services - Amazon S3, Azure Blob Service, Cloudflare R2
- Cloud-based analytical database (data warehouse) - Snowflake
- Redundant array of independent disks (RAID)
- Virtual disk storage - Amazon EBS, Azure managed disks, persistent disks in Google Cloud
- Storage (disk) and computation (CPU and RAM) have become somewhat seperated in cloud native services
- Cloud native systems are often multitenant
- database administrators (DBAs) or system administrators (sysadmins)
- DevOps philosophy, Site reliability engineers (SREs)
- Cloud storage replaces fixed-size disks with metered billing
- Distributed vs single-node systems
- A system that involves several machines communicating via a network is called a distributed system. Each of the processes participating in a distributed system is called a node.
- Various reasons for using distributed system,
- inherent distribution
- requests between cloud services
- fault tolerance/high availability
- scalability
- latency
- elasticity
- specialized hardware
- legal compliance
- sustainability
- Problems with distributed systems
- Tracing tools like OpenTelemetry, Zipkin, Jaeger
- Single-node databases - DuckDB, SQLite, KùzuDB
- Service-oriented architecture (SOA)
- Microservices architecture
- Pros and cons of many independent services
- Orchestration frameworks such as kubernetes
- API description standards such as OpenAPI and gRPC
- Microservices are primarily a technical solution to a people problem
- Serverless or function as a service (FaaS)
- Serverless approach is bringing metered billing to code execution
- BigQuery, Kafka
- Cloud computing is not the only way of building large-scale computing systems, an alternative is high-performance computing (HPC), also known as supercomputing
- Differences between HPC and cloud/enterprise datacenter
- Cloud datacenter networks are often based on IP and Ethernet, arranged in Clos topologies to provide high bisection bandwidth
- Supercomputers often use specialized network topologies, such as multidimensional meshes and toruses
- GDPR (General Data Protection Regulation), CCPA, EU AI Act
- ‘Right to be forgotten’
- In general, we store data because we think that its value is greater than the costs of storing it. costs of storage extend beyond the bill - risks of liability and reputational damage if the data were to be leaked - risk of legal costs and fines if incompliant with law
- The principle of data minimization runs counter to the “big data” philosophy of storing lots of data
- Payment Card Industry (PCI) standards, Service Organiza tion Control (SOC) Type 2 standards
- Summary
- understand trade-offs—that is, to recognize that many questions do not have one right answer, but several possibilities that each have pros and cons
- distinction between operational (transaction processing, OLTP) and analytical (OLAP) systems and exploring how they differ, not only in managing different types of data with different access patterns, but also in serving different audiences
- concepts of a data warehouse and data lake, which receive data feeds from operational systems via ETL
- compared cloud services to traditional paradigm of self-hosted software
- Cloud systems are intrinsically distributed, and we briefly examined some of the trade-offs of distributed systems compared to using a single machine
- Finally, we saw that a data system’s architecture is determined not only by the needs of the business deploying the system, but also by privacy regulations that protect the rights of the people whose data is being processed
Chapter 2 - Defining Nonfunctional Requirements
- a
- a
- Reliability, roughly, “continuing to work correctly, even when things go wrong.”
- To be more precise about things going wrong, we will distinguish between faults and failures
- Fault - a particular part of a system stops working correctly. Failure - the system as a whole stops providing the required service to the user - in other words, when it does not meet the SLO
- One hard drive exists and it fails - fault and failure both. If multiple harddrives, one hard drive failing is only a fault from pov of the bigger system, and it may be able to tolerate that fault by having a copy of the data on another hard drive.
- We call a system fault-tolerant if it continues providing the required service to users in spite of certain faults occurring.
- If a system cannot tolerate a certain part becoming faulty, we call that part a single point of failure (SPOF), because a fault in that part escalates to cause the failure of the whole system.
- Fault tolerance is always limited to a certain number of certain types of faults.
- Counterintuitively, in such fault-tolerant systems, it can make sense to increase the rate of faults by triggering them deliberately - for example, by randomly killing individual processes without warning. This is called fault injection. By deliberately inducing faults, you ensure that the fault-tolerance machinery is continually exercised and tested.
- Chaos engineering is a discipline that aims to improve confidence in fault-tolerance mechanisms
- a
- A more powerful machine - more CPU cores, more RAM, more disk space - vertical scaling or scaling up
- You can get parallelism on a single machine by using multiple processes or threads. All the threads belonging to the same process can access the same RAM, and hence this approach is also called a shared-memory architecture.
- The problem with a shared memory approach is that the cost grows faster than linearly
- Another approach is the shared-disk architecture, which uses several machines with independent CPUs and RAM but stores data on an array of disks that is shared among the machines, which are connected via a fast network: network-attached stor age (NAS) or a storage area network (SAN).
- This architecture has traditionally been used for on-premises data warehousing workloads, but contention and the overhead of locking limit the scalability of the shared-disk approach
- By contrast, the shared-nothing architecture (also called horizontal scaling or scaling out) involves a distributed system with multiple nodes, each of which has its own CPUs, RAM, and disks.
- Any coordination between nodes is done at the software level, via a conventional network.
- The advantages of this approach, are that it has the potential to scale linearly
- it can more easily adjust its hardware resources as load increases or decreases
- it can achieve greater fault tolerance by distributing the system across multiple datacenters and regions
- The downsides are that it requires explicit sharding, and incurs all the complexity of distributed systems
- a
Chapter 3 - Data Models and Query Languages
Chapter 4 - Storage and Retrieval
Chapter 5 - Encoding and Evolution
Chapter 6 - Replication
Chapter 7 - Sharding
Chapter 8 - Transactions
Chapter 9 - The Trouble with Distributed Systems
- If you want your system to be reliable in the presence of faults, you have to radically change your mindset and focus on what could go wrong, even though it may be unlikely.
- Anything that can go wrong will go wrong
- An individual computer with good software is usually either fully functional or entirely broken, but not something in between.
- In a distributed system, there may well be some parts of the system that are broken in an unpredictable way, even though other parts of the system are working fine. This is known as a partial failure. The difficulty is that partial failures are nondeterministic: if you try to do anything involving multiple nodes and the network, it may sometimes work and sometimes unpredictably fail.
- If a distributed system can tolerate partial failures, that opens up powerful possibilities - for example, it means we can perform a rolling upgrade, rebooting one node at a time to install software updates while the system as a whole continues working uninterrupted.
- Fault tolerance therefore allows us to make distributed systems more reliable than single-node systems; we can build a reliable system from unreliable components.
- In distributed systems, suspicion, pessimism, and paranoia pay off.
- In past - mainframes - RAID (Redundant Array of Independent Disks)
- the distributed systems we focus on are mostly shared-nothing systems - a bunch of machines connected by a network
- Instead of having redundancy of components within a single machine, shared-nothing systems use replication across separate machines for redundancy
- The internet and most internal networks in datacenters (often Ethernet) are asynchro nous packet networks. In this kind of network, one node can send a message (a packet) to another node, but the network gives no guarantees as to when it will arrive or whether it will arrive at all. If you send a request and expect a response, many things could go wrong
- Your request may have been lost (perhaps someone unplugged a network cable)
- Your request may be waiting in a queue and will be delivered later (perhaps the network or the recipient is overloaded)
- The remote node may have failed (perhaps it crashed or it was powered down)
- The remote node may have temporarily stopped responding (perhaps it is experiencing a long GC pause), but it will start responding again later
- The remote node may have processed your request, but the response has been lost on the network (perhaps a network switch has been misconfigured)
- The remote node may have processed your request, but the response has been delayed and will be delivered later (perhaps the network or your own machine is overloaded).
- If you send a request and don’t get a response, it’s not possible to distinguish whether (a) the request was lost, (b) the remote node is down, or (c) the response was lost
- These issues are indistinguishable in an asynchronous network; the only information you have is that you haven’t received a response yet. If you send a request to another node and don’t receive a response, it is impossible to tell why
- The usual way of handling this issue is a timeout: after some time, you give up waiting and assume that the response is not going to arrive. However, when a timeout occurs, you still don’t know whether the remote node got your request (and if the request is still queued somewhere, it may still be delivered to the recipient, even if you’ve given up on that possibility)
- The limitations of TCP
- Network fauls in practise
- Fault detection
- Timeouts and Unbounded Delays
- Network congestion and queueing
- TDP vs UPD
- Variability of network delays
- Synchronous Versus Asynchronous Networks