AI : Data & Knowledge Engineer
Grounding models in governed data, retrieval you can defend, and answers you can trace

1. Executive Summary

Most of the pain I see in production is data and knowledge, not which LLM badge is on the box. Models churn; what lasts is information that is permissioned, curated, and structured enough to retrieve under scrutiny. AI Data & Knowledge Engineering is the work of building and running that layer—not the slide version, the on-call version.

2. The Role & Responsibilities

The AI Data & Knowledge Engineer sits at the intersection. A software engineer moving into this role trades "deploy a service" for "deploy a data product" — versioned, contract-checked, observable, and consumable by both humans and models.

3. Reference Architecture

The canonical end-to-end pipeline a Software Engineer should be able to draw on a whiteboard:

4. The 2026 Tech Stack (Picks & Trade-offs)

5. Data Engineering Pillar

5.1 The Medallion Pattern (Bronze → Silver → Gold)

5.2 Storage & Format Choices

• Parquet is the default file format. Columnar, compressed, splittable.
• Iceberg / Delta / Hudi provide ACID, time travel, schema evolution, hidden partitioning.
• Use Z-order / liquid clustering on high-cardinality filter columns.
• Always design partition keys around query patterns, not ingest patterns.

5.3 Streaming vs Batch Decision

5.4 The Modern Document Ingestion Pipeline (for RAG)

5.5 Chunking Strategies (ranked, 2026)

1. Structure-aware (headings, sections, tables) — best default.
2. Semantic chunking (embedding-based boundary detection).
3. Parent-document retrieval (embed small, return large).
4. Late-chunking / contextual embedding (embed in context of doc; 2024+ technique).
5. Fixed-size sliding window — only as fallback.

6. Knowledge Engineering Pillar

6.1 The Knowledge Stack

6.2 Ontology Design — Rules of Thumb

• Start from use cases, not the universe. Model what is queried.
• Reuse before invent: schema.org, FIBO (finance), SNOMED (health), GoodRelations (commerce), W3C PROV (lineage).
• Separate T-Box (schema) from A-Box (instances); version T-Box independently.
• Use SHACL for validation, not just OWL for inference.
• Prefer property graphs (LPG) for app speed; RDF when you need standards / federation.

6.3 Entity Resolution Pipeline

6.4 The Semantic Layer

A semantic layer defines metrics, dimensions, and joins once so BI, apps, and LLMs all answer the same question the same way. This is the difference between "ChatGPT for our database" demos and systems that survive an audit.

# dbt Semantic Layer / MetricFlow style
semantic_models:
  - name: orders
    model: ref('fct_orders')
    entities:
      - name: order_id
        type: primary
      - name: customer_id
        type: foreign
    dimensions:
      - name: order_date
        type: time
        type_params: { time_granularity: day }
      - name: status
        type: categorical
    measures:
      - name: revenue
        agg: sum
        expr: gross_amount - discounts

metrics:
  - name: weekly_active_revenue
    type: simple
    type_params:
      measure: revenue
    filter: "{{ Dimension('orders__status') }} = 'completed'"

7. Retrieval & RAG Patterns

7.1 The RAG Maturity Ladder

7.2 Hybrid Retrieval Architecture (L2 — production default)

7.3 GraphRAG (L5)

GraphRAG augments retrieval with a knowledge graph: entities and relationships extracted from the corpus become first-class retrieval keys. Strong for multi-hop questions ("Which suppliers of company X were affected by event Y?") and for producing global summaries that vector search cannot.

7.4 Practical RAG Code Skeleton

from typing import List
import numpy as np

def retrieve(query: str, k: int = 50) -> List[dict]:
    # 1. Rewrite + classify
    rewrites = llm_rewrite(query, n=3)          # multi-query
    # 2. Parallel lexical + dense
    lex  = bm25.search(rewrites, k=k)
    dense = vector.search(embed(rewrites), k=k, filter=acl(user))
    # 3. Reciprocal Rank Fusion
    fused = rrf([lex, dense], k=k)
    # 4. Cross-encoder rerank top-N
    reranked = reranker.rerank(query, fused[:50])[:8]
    return reranked

def answer(query: str) -> dict:
    ctx = retrieve(query)
    packed = pack_context(ctx, max_tokens=6_000, dedup=True)
    out = llm.chat(
        system=SYSTEM_WITH_CITATIONS,
        user=query,
        context=packed,
        tools=[search_kg, lookup_metric],   # optional agentic
    )
    log_trace(query, ctx, out)              # eval/observability
    return out

8. Agentic Data Workflows

Agents make data & knowledge systems active: they plan, call tools, branch, and self-correct. The software-engineer mindset: an agent is a state machine with tool-calls and a budget, not magic.

8.1 Engineering Rules for Production Agents

• Deterministic skeleton, LLM in the joints. Hard-code the graph; let the LLM decide branches.
• Tools are typed APIs with schemas, idempotency keys, retries, and timeouts.
• Budgets everywhere: max steps, max tokens, max latency, max $ — fail loud when exceeded.
• Durable execution (Temporal, Restate, LangGraph-checkpointed) for any agent running > 30s.
• Human-in-the-loop at irreversible actions (writes, payments, deletes).
• Trace everything — every prompt, tool call, retrieval, and response — to a queryable store.

8.2 Text-to-SQL on the Semantic Layer

Don't let LLMs write raw SQL against raw tables. Expose the semantic layer as the tool surface. The LLM picks metrics + dimensions + filters; the semantic layer compiles deterministic SQL. This collapses error rates from ~50% to <5% on real schemas.

9. Data & Knowledge Quality

9.1 Data Contracts

A data contract is a producer-signed schema + SLA + semantics. It is the single biggest cultural shift required to make AI data reliable.

# contract: orders.v3
owner: team-commerce
schema:
  order_id: { type: uuid, pii: false, required: true }
  customer_id: { type: uuid, pii: false, required: true }
  amount_cents: { type: int64, required: true, range: [0, 10_000_000] }
  status: { type: enum, values: [pending, paid, refunded, cancelled] }
  created_at: { type: timestamp_utc, required: true }
freshness_sla: 15m
volume_sla: { min_rows_per_day: 1_000, max_change_pct: 30 }
quality_checks:
  - unique: [order_id]
  - not_null: [order_id, customer_id, amount_cents, created_at]
  - referential: { customer_id: customers.customer_id }
breaking_change_policy: deprecate_30d_then_remove
consumers: [feature_store.orders_v3, kg.order_node, dashboards.revenue]

9.2 The Test Pyramid for Data

9.3 Knowledge Quality

• SHACL shapes enforce KG node/edge constraints (every Order must have exactly one Customer).
• Coverage metrics: % of mentions linked, % of entities with canonical ID, edge density per type.
• Drift detection: new entity types or relations appearing without ontology updates.
• Provenance on every triple (source doc + extractor version + confidence).

10. Evaluation & Observability

10.1 Eval Strategy Matrix

10.2 Offline → Online Loop

10.3 LLM-as-Judge — Use Wisely

• Use rubrics, not vibes. Define what 1–5 means per criterion.
• Pairwise beats absolute scoring. Use pairs against a baseline.
• Audit the judge: 100 human-labeled samples per quarter, track agreement.
• Use a stronger model as judge than the one being judged when possible.

11. MLOps / LLMOps

11.1 The Four Lifecycles

11.2 CI / CD / CT (Continuous Training/Tuning)

• CI: lint, unit, contract, dbt build, eval on a small set, security scan.
• CD: deploy data assets, indexes, prompts, agent graphs behind feature flags.
• CT: schedule retraining / re-embedding / KG rebuilds with quality gates.
• Rollback: every artifact is content-addressed and versioned (datasets, indexes, prompts).

11.3 Versioning Everything

12. Governance, Security, and Cost

12.1 Data Governance Pillars

12.2 PII & RAG — Specific Hazards

• Tag PII at parse time; carry tags through chunking, embedding, and retrieval.
• Enforce ACL before reranking, not after — saves cost and prevents side-channel leaks.
• Redact prompts at the LLM gateway; route sensitive prompts to private / on-prem models.
• Honor right-to-erasure: every PII record must be re-findable by subject ID across lakehouse + vector + KG.

12.3 Compliance Map (2026)

12.4 FinOps for AI Data

• Cache aggressively: semantic cache for repeated questions; prompt cache for system prompts.
• Route by complexity: small model first; escalate on uncertainty (router pattern).
• Right-size embeddings: dim reduction (Matryoshka) cuts storage 4–8× with marginal quality loss.
• Batch and async: never block UX on full agent runs — stream + progressively reveal.
• Track $/query, $/user, $/business-outcome, not just $ total.

13. Implementation Strategy

13.1 Strategic Operating Model

13.2 The 90-Day Plan for a Software Engineer Stepping In

13.3 Decision Records You'll Need

• ADR-001: Lakehouse table format & engine
• ADR-002: Vector store + embedding + reranker stack (with re-embedding policy)
• ADR-003: Chunking strategy and metadata schema
• ADR-004: LLM gateway + model routing + caching policy
• ADR-005: Eval methodology + judge model + cadence
• ADR-006: PII classification + ACL propagation to vector store
• ADR-007: Knowledge graph store + ontology versioning
• ADR-008: Agent durability + human-in-loop boundaries

14. Anti-patterns & Pitfalls

15. Skills Matrix & Career Path

15.1 Core Competencies (T-shape)

15.2 Career Ladder

16. References & Further Reading

Starter reading only; your vendor contracts and internal policies still win. Canonical HTML: LinhTruong.com — Linh Truong.