The System Architecture Playbook

00Introduction — How I Use These Notes

Architecture is the set of design decisions that are expensive to change. I find the hardest part isn't writing code — it's picking the few decisions that determine whether a system still works five to ten years out.

I organized this note as a progression: mindset (how to think), quality attributes (what to optimize), concrete building blocks (styles, data, scale, resilience, security, observability, delivery), and the operating practices (estimation, decisions, anti-patterns, checklists) I run in design reviews.

How I read it

• Skim the diagrams first. Each section has a canonical diagram; I use them as a visual index when I'm orienting on a problem.
• Use the trade-off tables as decision aids, not as verdicts. I add my own column for the constraints I'm actually under.
• The checklists in §18 are the operational distillation. I print them and run them in design reviews.

01The Architect's Mindset

Before any pattern, the disposition. For me, architecture is a thinking discipline applied under uncertainty, organizational politics, and incomplete information.

1.1 First principles over patterns

Patterns are compressed experience, but they encode the context in which they were discovered. Always decompose to the underlying forces — latency budgets, failure domains, data gravity, team topology — before reaching for a named solution. A pattern applied without its forces is cargo-culting.

1.2 The laws you cannot repeal

1.3 What I try to deliver in an architecture role

• Clarity under ambiguity — turning a vague mandate into bounded problems with explicit trade-offs.
• Written, reviewable decisions — RFCs/ADRs that survive the author leaving.
• Force multiplication — raising the design ceiling of every team they touch, not just shipping their own code.
• Risk pricing — naming what could go catastrophically wrong and what it would cost to insure against it.

02Quality Attributes — What You Are Actually Optimizing

Architecture exists to satisfy non-functional requirements (the "-ilities"). Functional requirements decide what the system does; quality attributes decide whether it is worth running. You cannot maximize all of them — they trade against each other.

2.1 The attribute catalog & how to make each measurable

A quality attribute you cannot measure is a wish. Every attribute below must be expressed as a testable scenario: source → stimulus → environment → response → response measure.

2.2 Availability math you should know cold

Series vs. parallel: dependencies in series multiply (each hard dependency lowers your ceiling — 5 deps at 99.9% ≈ 99.5%). Redundant components in parallel add nines (two 99% replicas ≈ 99.99% if failures are independent). Architect for independent failure domains.

03The Design Process — A Repeatable Method

When a design has to be legible to others — not just workable in my head — I follow a method like the one below. It scales from a one-pager to a platform decision.

3.1 The steps, with the questions to ask at each

1. Frame the problem. What user/business outcome? What are the explicit non-goals? What constraints are hard (compliance, budget, deadline, existing systems)? Most bad designs are answers to the wrong question.
2. Requirements. Separate functional from quality attributes. Rank the top 3 -ilities; you will optimize for these and explicitly compromise the rest.
3. Estimate scale (see §13). Reads vs. writes, peak vs. average, data size, growth rate, latency budget. Numbers convert opinions into engineering.
4. Define contracts. APIs and data schemas first — they are the longest-lived artifacts and the hardest to change once clients depend on them.
5. High-level design. Boxes and arrows: clients, gateways, services, stores, queues, caches. Keep it to one whiteboard.
6. Deep dives. Attack the riskiest part: the hottest path, the largest table, the strongest consistency need, the worst failure mode.
7. Trade-off analysis. For each significant decision, list ≥2 options and why you rejected the alternatives. Write the ADR (§15).
8. Validate. Prototype the risky bit, write fitness functions, run a design review, and load-test assumptions before committing.

04Architecture Styles — The Macro Choice

The style is your top-level structural commitment. It is the most expensive decision to reverse, so it deserves the most rigor. Choose by team size, domain complexity, and required independence of deployment — not by fashion.

4.1 The styles in brief

4.2 Style trade-off matrix

05Communication & Integration

How components talk is as architecturally significant as how they're divided. The core axis: synchronous (request/response, temporal coupling) vs. asynchronous (messaging, temporal decoupling).

5.1 Choosing a protocol

5.2 Patterns that earn their keep

• API Gateway — single entry point for auth, rate limiting, routing, TLS termination, request shaping. Keep business logic out of it.
• Backend-for-Frontend (BFF) — a tailored aggregation layer per client type (web, mobile) to avoid one-size-fits-none APIs.
• Service mesh (sidecar) — push mTLS, retries, timeouts, traffic shaping, and telemetry out of app code into the platform (e.g., Envoy-based meshes).
• Outbox pattern — write the business change and the "to-publish" event in one local transaction, then relay — the cure for dual-write inconsistency.
• Async request-reply — for long operations: return 202 Accepted + a status/polling URL or callback, rather than holding a connection.

06Data Architecture — Where Systems Truly Age

Stateless tiers are easy; state is where the hard, irreversible decisions live. Data has gravity: it is heavy to move, expensive to reshape, and the schema outlives the code. Spend disproportionate care here.

6.1 Choosing a store by access pattern, not by hype

6.2 Scaling data: replication and partitioning

• Replication (leader–follower) scales reads & gives failover, but introduces replication lag → read-your-writes anomalies. Mitigate with read-from-leader for the writer, or sticky sessions.
• Sharding scales writes/storage. The hardest part is the shard key: it must spread load evenly and avoid hot spots and cross-shard transactions. Bad shard keys are nearly impossible to change later.
• Consistent hashing minimizes data movement when nodes are added/removed — the standard for distributed caches and stores.

6.3 Advanced data patterns

07Scalability Strategy

Scalability is keeping cost-per-unit roughly flat as load grows by orders of magnitude. The toolkit is small and well-understood; the art is applying it before — not during — the incident.

7.1 The scaling toolkit

• Statelessness — push session/state to a shared store so any node can serve any request; the precondition for horizontal scale and autoscaling.
• Horizontal > vertical — add commodity nodes (linear-ish, fault-tolerant) rather than bigger boxes (hard ceiling, single point of failure). Vertical buys time; horizontal buys headroom.
• Caching at every layer (browser → CDN → app → distributed cache → DB buffer). The cheapest performance you'll ever buy — and the richest source of staleness bugs.
• Load balancing — distribute work; remove unhealthy nodes; enable zero-downtime deploys.
• Async offload — move anything not needed for the response (emails, thumbnails, indexing, analytics) onto a queue.
• Read/write split — replicas for reads, primary for writes; CQRS when shapes diverge.

7.2 Caching strategy & the hard parts

7.3 Back-pressure & load shedding

A scalable system must protect itself when demand exceeds capacity. Back-pressure propagates "slow down" upstream (bounded queues, flow control). Load shedding deliberately rejects low-priority work (with 429/503 + retry-after) to keep the core healthy. Rate limiting (token/leaky bucket) caps per-client demand. Failing fast and partial beats failing slow and total.

08Reliability & Resilience Engineering

At scale, failure is not an exception — it is the steady state. Disks die, networks partition, dependencies time out, deploys regress. Resilience is designing so that partial failure stays partial and the system degrades gracefully instead of collapsing.

8.1 The resilience pattern library

8.2 RTO, RPO, and the spectrum of DR

RTO (Recovery Time Objective) = how long until service is restored. RPO (Recovery Point Objective) = how much data loss is acceptable. These business-set numbers dictate the (rising) cost of your DR strategy:

8.3 Operating reliability: SRE practices

• SLI / SLO / error budgets — measure the indicator, set the objective, and spend the budget: when you're within budget, ship fast; when you're burning it, freeze features and fix reliability. This aligns velocity and stability with one number.
• Chaos engineering — deliberately inject failure (kill nodes, add latency, partition) in controlled experiments to validate resilience before reality does it for you.
• Blameless postmortems — treat incidents as system failures, not human failures; the output is action items that change the system, not blame.
• Game days — rehearse failover and incident response so muscle memory exists when it counts.

09Distributed Systems Foundations

The moment your system spans more than one machine, a new physics applies. The "fallacies of distributed computing" — the network is reliable, latency is zero, bandwidth is infinite, topology is stable — are all false, and assuming otherwise produces the subtlest, most expensive bugs.

9.1 Consistency models — the spectrum, not the binary

9.2 Consensus, ordering, and time

• Consensus (Raft, Paxos) — how a cluster agrees on a value/leader despite failures. You rarely implement it; you use systems built on it (etcd, ZooKeeper, consensus-backed DBs). Know that it requires a majority quorum and cannot make progress without one.
• Logical clocks (Lamport, vector clocks) — order events without synchronized wall clocks. Never trust wall-clock ordering across machines — clock skew is real.
• Quorums — with replication factor N, choosing read quorum R and write quorum W such that R + W > N guarantees overlap and thus strong reads; tuning R/W trades latency vs. consistency.

9.3 Delivery guarantees & idempotency

Messaging systems offer at-most-once (may lose), at-least-once (may duplicate), or exactly-once (expensive, often "effectively once" via dedupe). In practice, the robust pattern is: at-least-once delivery + idempotent consumers. Design every side-effecting operation to be safely repeatable — via idempotency keys, natural dedupe keys, or conditional writes. This single discipline prevents a huge class of distributed bugs.

10Security Architecture

Security is a quality attribute that must be designed in, not bolted on. The modern posture is zero trust: never trust the network, always verify identity, and assume breach. Defense in depth means no single control failing should be catastrophic.

10.1 The pillars

10.2 Threat modeling — STRIDE

Systematically ask, per component and data flow, how it could be attacked:

11Observability

You cannot operate what you cannot see. Monitoring tells you whether the system is healthy against known questions; observability lets you ask new questions of production without shipping new code — essential for debugging novel failures in distributed systems.

11.1 The three pillars (and the fourth)

• Metrics — cheap, aggregatable numeric time-series. Track the RED method (Rate, Errors, Duration) for services and USE (Utilization, Saturation, Errors) for resources.
• Logs — structured (JSON), with a correlation/trace ID on every line. Unstructured logs don't scale to investigation.
• Traces — follow a single request across every service hop; the only practical way to find latency and failure in a distributed call graph.
• (Profiles / events) — continuous profiling and high-cardinality wide events round out modern observability.

11.2 Good alerting hygiene

• Every alert must be actionable and urgent; otherwise it's a dashboard, not a page. Alert fatigue is a reliability risk.
• Use multi-window, multi-burn-rate SLO alerts to catch both fast and slow budget burns without noise.
• Every page links to a runbook. The metric-to-action gap is where MTTR goes to die.

12Delivery, Deployment & Platform

Architecture includes how software reaches production. The DORA research is clear: elite performers deploy frequently, with low lead time, low change-failure rate, and fast recovery — and these come from architecture that supports safe, small, independent deploys.

12.1 Deployment strategies

12.2 Platform foundations

• Containers + orchestration (Kubernetes) — package once, run anywhere; declarative scaling, self-healing, rollouts. Powerful but operationally heavy — adopt deliberately.
• Infrastructure as Code (Terraform, etc.) — all infra versioned, reviewed, reproducible. No click-ops in production.
• GitOps — Git as the single source of truth for desired state; a controller continuously reconciles reality to it. Auditable, revertible deployments.
• Platform engineering / Internal Developer Platform — pave golden paths so product teams ship safely without re-deriving infra. Reduces cognitive load; this is where Conway's Law meets DevEx.
• Service mesh — when service count grows, externalize mTLS, retries, traffic policy, and telemetry to the platform.

13Capacity Planning & Back-of-the-Envelope Estimation

I try to turn hand-waving into numbers in minutes. Estimation isn't about precision — it's about getting the order of magnitude right so you choose the right architecture and don't over- or under-build by 100×.

13.1 Latency numbers every engineer should know (approx.)

Takeaway: memory ≫ SSD ≫ disk ≫ cross-region network. Every order of magnitude changes the design. Keep hot data in memory; keep users close to their data.

13.2 A worked estimation (template)

Assume: 100M daily active users, each does 10 reads + 1 write/day.

Reads/day  = 100M × 10 = 1B  → /86,400s ≈ 11,600 RPS avg
Writes/day = 100M × 1  = 100M → ≈ 1,160 WPS avg
Peak factor ≈ 3×  →  ~35k RPS read, ~3.5k WPS write at peak

Storage/write: ~1 KB per record
  100M writes/day × 1 KB = 100 GB/day  → ~36 TB/year (before replication)
  With 3× replication → ~110 TB/year

Cache: keep hot 20% of reads in memory
  working set ≈ (some GB) → size Redis cluster accordingly
Bandwidth (read): 35k RPS × ~1 KB ≈ 35 MB/s egress (plan CDN offload)

This 5-minute exercise immediately tells you: you need a distributed cache, read replicas or sharding, a CDN, and roughly how many app nodes — before writing any code.

13.3 Capacity headroom & cost

• Plan for peak × growth × safety margin, not average. Run hot at ~50–70% utilization so spikes don't tip you over.
• Autoscale on the right signal (queue depth, RPS, p99) — CPU is often a lagging proxy.
• FinOps: track unit economics ($/request, $/tenant). The cheapest-to-build design and the cheapest-to-run design are rarely the same — make the trade explicit.

14Architecting for AI / ML & Modern Workloads

AI-intensive systems are now mainstream, and they bend several classic assumptions: non-deterministic outputs, GPU economics, large-payload latency, prompt/data-driven behavior, and new failure and safety modes. A current playbook must address them.

14.1 What changes when AI is in the loop

14.2 Inference vs. training paths

• Serving/inference is the latency-critical, always-on path: GPU autoscaling (with cold-start mitigation), request batching, streaming, and aggressive caching dominate.
• Training/fine-tuning is a batch, data-pipeline problem: feature stores, reproducible datasets, experiment tracking, and a model registry with lineage.
• MLOps/LLMOps closes the loop: deploy models like code (CI/CD), monitor for drift and quality regression, and enable safe rollback of model/prompt versions.

15Making & Recording Decisions

The artifact I care most about is the written decision. An architecture no one can reconstruct the reasoning behind becomes legacy the day its author leaves. ADRs and RFCs make decisions durable, reviewable, and reversible-with-eyes-open.

15.1 The Architecture Decision Record (ADR)

A short, immutable, append-only document per significant decision. The key sections:

# ADR-0042: Use event-driven sagas for order fulfillment

## Status
Accepted  (supersedes ADR-0017)

## Context
Orders span inventory, payment, and shipping services. We need
"transaction-like" consistency without a distributed 2PC, and the
fulfillment flow must survive partial failures and be auditable.

## Decision
Use a choreographed saga over Kafka with compensating actions.
Each step is idempotent; the outbox pattern guarantees event publish.

## Consequences
+ Services stay decoupled and independently deployable.
+ Full audit trail via the event log.
- Eventual consistency: UI must reflect "pending" states.
- Requires saga monitoring + dead-letter handling.

## Alternatives considered
- 2PC across services: rejected (availability + coupling cost).
- Synchronous orchestration service: rejected (single point of
  coupling, harder to scale steps independently).

15.2 A decision rubric

1. Is this a one-way or two-way door? Calibrate rigor to reversibility.
2. What are we optimizing, and what are we sacrificing? Name the losing -ilities.
3. What's the blast radius if we're wrong? Can we contain/roll it back?
4. What would change our mind? Write the falsifiable condition for revisiting.
5. Who must be in the room? The people who own the consequences.

16Anti-Patterns & Failure Modes to Recognize

Knowing what not to do is half of architecture. These recur across companies and decades; recognizing them early is a superpower.

17A Reference Architecture (Putting It Together)

A concrete, modern reference for a high-scale, multi-region SaaS — annotated with which decisions matter and why. Treat it as a starting skeleton, not a mandate.

17.1 Why each piece is there

18The Playbook — Operational Checklists

The distilled, printable core. Run these in design reviews, readiness reviews, and incident retros.

18.1 One-page heuristics I keep nearby

• Make it work, make it right, make it fast — in that order. Premature optimization and premature distribution are twin sins.
• Boring technology by default. Spend your innovation tokens where they create real differentiation.
• Optimize the bottleneck, measure first. Intuition about performance is usually wrong; profile.
• Design for failure, test for failure. Untested failover is a hope, not a plan.
• Couple loosely, align tightly. Loose technical coupling, tight clarity on contracts and ownership.
• Write the decision down. If it isn't an ADR, it didn't happen.
• Reversibility first. Move fast on two-way doors; deliberate on one-way doors.
• The simplest thing that could possibly work — and no simpler.

19The Canon & Further Reading

Texts and bodies of knowledge I keep returning to. I read them for the reasoning, not the recipes. Annotated citations with DOIs and URLs: §20 References & Sources.

20References & Sources

Annotated bibliography behind the mindset heuristics, quality attributes, design process, architecture styles, integration patterns, data and scale tactics, resilience and distributed-systems foundations, security, observability, delivery, estimation, AI/ML workloads, ADRs, anti-patterns, reference topology, checklists, and canon sections. Section tags (e.g. §04) show where each source is used. Mermaid diagrams, trade-off matrices, and synthesis checklists are my own unless noted.

Scope. Synthesis of textbooks, peer-reviewed papers, industry frameworks, and vendor-neutral standards (May 2026). Cloud-specific numbers, SKUs, and product names change frequently — verify against current vendor docs before production decisions. Security and compliance references are architectural starting points, not audit checklists.

Citations are numbered continuously [1]–[n] within this section.

Architecture mindset, trade-offs & organizational forces (§00, §01, §18)

1. Brooks, F. P., The Mythical Man-Month: Essays on Software Engineering. Addison-Wesley, 1975 (rev. ed. 1995). Essential vs. accidental complexity; why late projects stay late — background for §00 irreversible-decision framing. — §00, §01.
2. Conway, M. E., "How Do Committees Invent?" Datamation, 14(4), 28–31, 1968. Conway's Law — §01 law callout and team–architecture coupling. melconway.com — §01.
3. Skelton, M., & Pais, M., Team Topologies: Organizing Business and Technology Teams for Fast Flow. IT Revolution Press, 2019. Inverse Conway Maneuver and stream-aligned teams — §01 Conway corollary and §04 org-fit guidance. — §01, §04, §19.
4. Bezos, J. (via Amazon leadership principles), "Type 1 vs. Type 2 decisions." One-way vs. two-way door framing widely attributed to Amazon internal doctrine; popularized in tech leadership writing — §01 reversibility heuristic. See also All Things Distributed essays by Werner Vogels — §01, §18.
5. Beck, K., et al., Extreme Programming Explained: Embrace Change (2nd ed.). Addison-Wesley, 2004. YAGNI and incremental design — §01 YAGNI vs. evolvability card. — §01.
6. Richards, M., & Ford, N., Fundamentals of Software Architecture: An Engineering Approach. O'Reilly, 2020. Architecture characteristics, trade-off analysis, and role of the architect — §00 progression and §02 attribute catalog. — §00, §02, §19.

Distributed-systems laws: CAP, PACELC, Amdahl & scalability limits (§01, §09)

1. Brewer, E. A., "CAP Twelve Years Later: How the 'Rules' Have Changed." IEEE Computer, 45(2), 23–29, 2012. CAP theorem refinement — §01 CAP callout and §09 consistency trade-offs. DOI: 10.1109/MC.2012.37 — §01, §09.
2. Abadi, D., "Consistency Tradeoffs in Modern Distributed Database System Design." IEEE Computer, 45(2), 37–42, 2012. PACELC extension — §01 PACELC callout. DOI: 10.1109/MC.2012.39 — §01, §09.
3. Amdahl, G. M., "Validity of the Single Processor Approach to Achieving Large Scale Computing Capabilities." AFIPS Spring Joint Computer Conference, 1967. Serial fraction limits parallel speedup — §01 Amdahl callout. — §01, §07.
4. Gunther, N. J., The Practical Performance Analyst / Universal Scalability Law. Guerrilla-capacity.com, 2008+. Coherency/contention penalties beyond Amdahl — §01 USL callout. perfdynamics.com — §01, §07.
5. Deutsch, P., et al., "Eight Fallacies of Distributed Computing." Sun Microsystems internal list, popularized 1990s. Network unreliability and latency assumptions — §09 distributed foundations lead. — §09.

Quality attributes, scenarios & fitness functions (§02)

1. ISO/IEC 25010:2023, Systems and software engineering — Systems and software Quality Requirements and Evaluation (SQuaRE). ISO standard for quality characteristics (-ilities taxonomy) — §02 attribute catalog. — §02.
2. Bass, L., Clements, P., & Kazman, R., Software Architecture in Practice (4th ed.). Addison-Wesley, 2021. Quality attribute scenarios (stimulus–response–measure) — §02 measurability paragraph. — §02.
3. Ford, N., Parsons, R., & Kua, P., Building Evolutionary Architectures: Automated Governance for Software Teams (2nd ed.). O'Reilly, 2022. Fitness functions and continuous architectural verification — §02 Figure 2.1 feedback loop. — §02, §03.
4. Google SRE Team, Site Reliability Engineering: How Google Runs Production Systems. O'Reilly, 2016. SLI/SLO/error-budget framing — §02 availability math and §08.3 SRE practices. sre.google — §02, §08, §11.
5. Google SRE Team, The Site Reliability Workbook. O'Reilly, 2018. Multi-window multi-burn-rate alerting — §11.2 alerting hygiene. sre.google/workbook — §08, §11.

Design process, diagrams & decision records (§03, §05, §15)

1. Brown, S., The C4 Model for Visualising Software Architecture. c4model.com, 2018+. Context/container/component/code hierarchy — §03 design-process communication. c4model.com — §03, §05, §19.
2. Nygard, M., "Documenting Architecture Decisions." Cognitect blog, 2011. Original ADR format — §15 ADR template and §15 lead. cognitect.com — §15.
3. Thomson, J., "ADR: Architecture Decision Records." GitHub / community templates, 2010s+. Widely adopted ADR practice — §15.1 sections (Status, Context, Decision, Consequences). adr.github.io — §15.
4. Richards, M., & Ford, N., Software Architecture: The Hard Parts. O'Reilly, 2021. Trade-off rubrics for distributed architecture decisions — §15.2 decision rubric and §09 consistency choices. — §09, §15, §19.

Architecture styles, DDD & evolutionary paths (§04, §16, §17)

1. Newman, S., Building Microservices: Designing Fine-Grained Systems (2nd ed.). O'Reilly, 2021. Microservices trade-offs, distributed monolith warning — §04 styles, §04.2 matrix, §16 anti-patterns. — §04, §16, §19.
2. Evans, E., Domain-Driven Design: Tackling Complexity in the Heart of Software. Addison-Wesley, 2003. Bounded contexts and ubiquitous language — §04 modular monolith seams and §06 data ownership. — §04, §06, §19.
3. Vernon, V., Implementing Domain-Driven Design. Addison-Wesley, 2013. Tactical patterns and service boundaries — §04 hexagonal/DDD pairing. — §04, §19.
4. Cockburn, A., "Hexagonal Architecture." Alistair Cockburn, 2005. Ports and adapters — §04 hexagonal card. alistair.cockburn.us — §04.
5. Martin, R. C., Clean Architecture: A Craftsman's Guide to Software Structure and Design. Prentice Hall, 2017. Dependency rule and domain-centric layering — §04 clean/hexagonal card. — §04.
6. Reactive Manifesto. Lightbend et al., 2013. Responsive, resilient, elastic, message-driven systems — §04 event-driven style context. reactivemanifesto.org — §04, §19.
7. Vogels, W., "Building Scalable, Highly Concurrent, and Fault-Tolerant Applications." AWS re:Invent / All Things Distributed, 2018+. Cell-based architecture for blast-radius containment — §04 cell-based card and §17 reference topology. allthingsdistributed.com — §04, §17.
8. Heroku, The Twelve-Factor App. 2011+. Stateless processes, config, logs as streams — §12 platform foundations and §07 stateless scaling. 12factor.net — §07, §12, §19.

Integration, APIs & messaging patterns (§05)

1. Fielding, R. T., Architectural Styles and the Design of Network-based Software Architectures (REST dissertation). UC Irvine, 2000. REST constraints — §05 REST/JSON row. ics.uci.edu — §05.
2. Google, gRPC: A high performance, open source universal RPC framework. Protocol Buffers + HTTP/2 RPC — §05 gRPC row. grpc.io — §05.
3. Facebook / GraphQL Foundation, GraphQL Specification. Client-shaped queries — §05 GraphQL row. spec.graphql.org — §05.
4. Richardson, C., Microservices Patterns: With Examples in Java. Manning, 2018. API Gateway, BFF, messaging, outbox, saga patterns — §05.2 pattern list. — §05, §06.
5. Hohpe, G., & Woolf, B., Enterprise Integration Patterns. Addison-Wesley, 2003. Message routing, publish–subscribe, async integration — §05 sync vs. async axis. — §05.
6. Envoy Proxy / CNCF, service mesh architecture. Sidecar-based mTLS, retries, telemetry — §05 service mesh bullet. envoyproxy.io — §05, §12.
7. Debezium / transactional outbox pattern. Dual-write avoidance via outbox relay — §05 outbox pattern bullet. See also Richardson, Microservices Patterns, ch. 3 — §05, §06.

Data stores, replication, CQRS, event sourcing & sagas (§06)

1. Kleppmann, M., Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems. O'Reilly, 2017. Replication, partitioning, consistency, stream processing — backbone for §06–§09. — §06, §07, §09, §19.
2. DeCandia, G. et al., "Dynamo: Amazon's Highly Available Key-value Store." ACM SIGOPS Operating Systems Review, 41(6), 205–220, 2007. Eventual consistency at scale — §06 store trade-offs and §09 eventual model. DOI: 10.1145/1323293.1294281 — §06, §09, §19.
3. Young, G., "CQRS Documents." CodeBetter / Greg Young, 2010. Command/query responsibility segregation — §06.3 CQRS card. — §06.
4. Young, G., "Event Sourcing." Domain-driven design community, 2010s. Event log as source of truth — §06.3 event sourcing card. — §06.
5. Richardson, C., "Pattern: Saga." microservices.io. Orchestrated/choreographed distributed transactions — §06.3 saga card. microservices.io — §06, §04.
6. Dehghani, Z., "How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh." Martin Fowler blog, 2019; Data Mesh (O'Reilly, 2022). Domain-owned data products — §06.3 data mesh card. martinfowler.com — §06.
7. Karger, D., et al., "Consistent Hashing and Random Trees." STOC, 1997. Minimal remapping on node changes — §06.2 consistent hashing bullet. — §06, §07.

Scalability, caching & web topology (§07)

1. Memcached / Redis documentation and industry practice. Distributed cache tiers — §07 Figure 7.1 and §07.2 caching strategy. See also Kleppmann ch. 3 — §07.
2. CDN best practices (Cloudflare, Akamai, AWS CloudFront docs). Edge caching and origin offload — §07 CDN layer. Vendor docs vary; principle is widely documented — §07.
3. AWS Architecture Center, "Web Application Hosting." Reference scalable web tier patterns — §07 canonical topology. AWS Well-Architected — §07, §17.
4. Dean, J., & Barroso, L. A., "The Tail at Scale." Communications of the ACM, 56(2), 74–80, 2013. Latency outliers at scale — §07 back-pressure and §13 latency context. DOI: 10.1145/2408776.2408794 — §07, §13.

Reliability, resilience patterns & chaos engineering (§08)

1. Nygard, M. T., Release It! Design and Deploy Production-Ready Software (2nd ed.). Pragmatic Bookshelf, 2018. Circuit breaker, bulkhead, timeout, stability patterns — §08 resilience library and §08 Figure 8.1. — §08, §19.
2. Fowler, M., "CircuitBreaker." bliki, 2014. Circuit breaker state machine — §08 circuit breaker row. martinfowler.com — §08.
3. Netflix Technology Blog, Hystrix and resilience engineering posts. Bulkheads, fallbacks, latency tolerance — §08 pattern table. netflixtechblog.com — §08.
4. Beyer, B., et al., Site Reliability Engineering (Chaos Engineering chapter). Controlled failure injection — §08.3 chaos engineering bullet. — §08.
5. AWS, "Disaster Recovery of Workloads on AWS: Recovery in the Cloud." RTO/RPO strategies (backup, pilot light, warm standby, active-active) — §08.2 DR table. AWS DR whitepaper — §08.
6. Allspaw, J., "Blameless PostMortems and a Just Culture." Etsy / Code as Craft, 2012. Incident learning without blame — §08.3 postmortems. — §08.

Consensus, clocks, ordering & delivery semantics (§09)

1. Lamport, L., "Time, Clocks, and the Ordering of Events in a Distributed System." Communications of the ACM, 21(7), 558–565, 1978. Logical clocks — §09.2 ordering bullet. DOI: 10.1145/359545.359563 — §09, §19.
2. Lamport, L., "The Part-Time Parliament." ACM Transactions on Computer Systems, 16(2), 133–169, 1998. Paxos consensus — §09.2 consensus bullet. — §09, §19.
3. Ongaro, D., & Ousterhout, J., "In Search of an Understandable Consensus Algorithm (Raft)." USENIX ATC, 2014. Raft for replicated logs — §09.2 consensus bullet. raft.github.io — §09, §19.
4. Fischer, M. J., Lynch, N. A., & Paterson, M. S., "Impossibility of Distributed Consensus with One Faulty Process." Journal of the ACM, 32(2), 374–382, 1985. FLP impossibility — background for §09 quorum/consensus limits. — §09.
5. Apache Kafka documentation, delivery semantics. At-least-once, at-most-once, idempotent producers — §09.3 delivery guarantees. kafka.apache.org — §09.

Security architecture, zero trust & threat modeling (§10)

1. NIST, Zero Trust Architecture, SP 800-207. U.S. Department of Commerce, 2020. Never trust, always verify — §10 zero-trust lead and Figure 10.1. csrc.nist.gov — §10, §19.
2. OWASP Foundation, OWASP Top Ten. Web application risk categories — §10 shift-left callout and §18 security checklist. owasp.org — §10, §18, §19.
3. Microsoft, "The STRIDE Threat Model." Spoofing, Tampering, Repudiation, Information disclosure, Denial of service, Elevation of privilege — §10.2 STRIDE table. learn.microsoft.com — §10.
4. Hardt, D., et al., RFC 6749: The OAuth 2.0 Authorization Framework. IETF, 2012. OAuth2 flows — §10 identity card. datatracker.ietf.org — §10.
5. OpenID Foundation, OpenID Connect Core 1.0. Identity layer on OAuth2 — §10 OIDC authentication. openid.net — §10.
6. CNCF, Software Supply Chain Best Practices / SLSA framework. Artifact signing, provenance, SBOM — §10 supply-chain card and §12 signed artifacts. slsa.dev — §10, §12, §18.
7. OWASP, Top 10 for Large Language Model Applications. Prompt injection and LLM abuse — §14.1 safety card and §18 LLM input bullet. owasp.org — §14, §18.

Observability: metrics, logs, traces & OpenTelemetry (§11)

1. Sigelman, B., et al., "Dapper, a Large-Scale Distributed Systems Tracing Infrastructure." Google technical report, 2010. Distributed tracing lineage — §11 traces pillar. — §11.
2. Charity Majors, Liz Fong-Jones, & George Miranda, Observability Engineering. O'Reilly, 2022. Observability vs. monitoring — §11 lead and high-cardinality investigation. — §11.
3. OpenTelemetry, specification & documentation. Vendor-neutral instrumentation — §11 Figure 11.1 and OTel principle callout. opentelemetry.io — §11, §18.
4. Wilkes, J., "Site Reliability Engineering: Measuring and Managing Reliability." Google SRE Book, ch. 4. SLI/SLO definitions — §11 SLO/error-budget path. — §08, §11.
5. Google SRE Workbook, "Alerting on SLOs." Multi-burn-rate alerting — §11.2 hygiene bullets. — §11.
6. Elasticsearch / OpenSearch documentation. Log aggregation at scale — §11 logs pillar (structured logging). Vendor-neutral principle in SRE literature — §11.

Delivery, DORA metrics & deployment strategies (§12)

1. Forsgren, N., Humble, J., & Kim, G., Accelerate: The Science of Lean Software and DevOps. IT Revolution Press, 2018. DORA four keys — §12 lead and delivery performance. — §12, §19.
2. DORA / Google Cloud, State of DevOps Reports. Ongoing research on deployment frequency, lead time, change failure rate, MTTR — §12 DORA reference. dora.dev — §12, §19.
3. Humble, J., & Farley, D., Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. Addison-Wesley, 2010. Pipeline design and release safety — §12 CI/CD diagram. — §12.
4. Fowler, M., "Feature Toggles (aka Feature Flags)." Feature flags decouple deploy from release — §12 progressive delivery. martinfowler.com — §12.
5. CNCF, Kubernetes documentation. Container orchestration platform layer — §12.2 platform foundations. kubernetes.io — §12, §17.
6. LaunchDarkly / industry practice on progressive delivery. Canary, blue/green, rolling deployments — §12.1 deployment strategies table. See also Humble & Farley — §12.

Capacity planning, back-of-envelope math & unit economics (§13)

1. Dean, J., "Numbers Everyone Should Know" (latency table). Google slides / Communications of the ACM follow-ons. ns–ms–s hierarchy — §13.1 latency numbers table. Widely reproduced; verify current hardware — §13.
2. Barroso, L. A., Hölzle, U., & Parthasarathy, R., "Web Search for a Planet: The Google Cluster Architecture." IEEE Micro, 23(2), 22–28, 2003. Hyperscale capacity thinking — §13 estimation context. — §13.
3. AWS / Google Cloud pricing calculators and Well-Architected cost optimization pillar. Unit economics and headroom — §13.3 capacity headroom and §18 cost checklist. Vendor-specific — §13, §18.

AI / ML & LLM system architecture (§14)

1. Sculley, D., et al., "Hidden Technical Debt in Machine Learning Systems." NeurIPS, 2015. ML systems complexity beyond models — §14 MLOps loop. papers.nips.cc — §14.
2. Google, Machine Learning Engineering / TFX documentation. Training vs. serving paths, model registry — §14.2 inference vs. training. tensorflow.org/tfx — §14.
3. Chowdhery, A., et al., "PaLM: Scaling Language Modeling with Pathways." Journal of Machine Learning Research, 2023. Large-model serving economics background — §14.1 cost/latency card. — §14.
4. LangChain / semantic caching and model-routing patterns (industry practice). Provider abstraction gateway — §14.1 economics and §14 principle callout. Patterns evolving rapidly — §14.
5. Vector database vendors (pgvector, Pinecone, Milvus documentation). Embedding retrieval — §06 vector store row and §14 retrieval-augmented patterns. — §06, §14.

Anti-patterns, reference topology & operational checklists (§16, §17, §18)

1. Richardson, C., "Anti-pattern: Shared database." microservices.io — §06 shared-database callout and §16 anti-patterns. microservices.io — §06, §16.
2. Richardson, C., "Anti-pattern: Distributed monolith." microservices.io — §04 distributed monolith callout and §16. microservices.io — §04, §16.
3. AWS Well-Architected Framework. Operational excellence, security, reliability, performance, cost pillars — §17 reference architecture and §18 checklists. AWS; see also Azure and GCP equivalents — §17, §18, §19.
4. Google SRE Book & Workbook. Production readiness, alerting, incident response — §18 production-readiness and design-review checklists. — §18.
5. Knuth, D. E., "Structured Programming with go to Statements." ACM Computing Surveys, 6(4), 261–301, 1974. "Premature optimization is the root of all evil" — §18.1 heuristics. DOI: 10.1145/356635.356855 — §18.

Foundational canon cited in §19

1. Kleppmann, M., Designing Data-Intensive Applications. O'Reilly, 2017. — §19 foundational books list. — §19 (+ §06–§09).
2. Richards, M., & Ford, N., Fundamentals of Software Architecture & Software Architecture: The Hard Parts. O'Reilly, 2020–2021. — §19 list. — §19.
3. Newman, S., Building Microservices (2nd ed.). O'Reilly, 2021. — §19 list. — §19.
4. Evans, E., Domain-Driven Design; Vernon, V., Implementing Domain-Driven Design. — §19 list. — §19.
5. Nygard, M., Release It! (2nd ed.). Pragmatic Bookshelf, 2018. — §19 list. — §19.
6. Google SRE Team, Site Reliability Engineering & The Site Reliability Workbook. O'Reilly, 2016–2018. — §19 list. — §19.
7. Forsgren, N., Humble, J., & Kim, G., Accelerate. IT Revolution, 2018. — §19 list. — §19.
8. Skelton, M., & Pais, M., Team Topologies. IT Revolution, 2019. — §19 list. — §19.

Author synthesis

1. Truong, L., System Architecture Playbook — personal working notes. May 2026. Trade-off matrices, Mermaid diagrams, checklists, and synthesis prose. LinhTruong.com — all sections.