Building Medallion Architectures Piethein Strengholt

Spis treści książki

• Foreword
• Preface
  • Who Should Read This Book
  • Navigating This Book
  • Conventions Used in This Book
  • Using Code Examples
  • OReilly Online Learning
  • How to Contact Us
  • Acknowledgments
• I. Understanding the Medallion Framework
• 1. The Evolution of Data Architecture
  • What Is a Medallion Architecture?
  • A Brief History of Data Warehouse Architecture
    • OLTP Systems
    • Data Warehouses
    • The Staging Area
    • Inmon Methodology
    • Kimball Methodology
    • Key Takeaways from Traditional Data Warehouses
  • A Brief History of Data Lakes
    • Hadoops Distributed File System
    • MapReduce
    • Apache Hive
      • External and internal tables
      • Hive Metastore
    • Spark Project
    • Moving Forward with Data Lakes
  • A Brief History of Lakehouse Architecture
    • Founders of Spark
    • Emergence of Open Table Formats
    • The Rise of Lakehouse Architectures
    • Medallion Architecture and Its Practical Challenges
  • Conclusion
• 2. Laying the Groundwork
  • Foundational Preconditions
  • Extra Landing Zones
  • Raw Data
  • Batch Processing
  • Real-Time Data Processing
    • Spark Structured Streaming
    • Change Data Feed
    • Change Data Capture
    • Considerations and Learning Resources
  • ETL and Orchestration Tools
  • Managing Delta Tables
    • Z-Ordering
    • V-Ordering
    • Table Partitioning
    • Liquid Clustering
    • Compaction and Optimized Writes
    • DeltaLog
  • Conclusion
• 3. Demystifying the Medallion Architecture
  • The Three-Layered Design
  • Bronze Layer
    • Processing Hierarchy
    • Processing Full Data Loads
    • Processing Incremental Data Loads
    • Data Historization Within the Bronze Layer
    • Schema Evolution and Management
    • MergeSchema and Schema Enforcement
    • Technical Validation Checks
    • Usage and Governance
    • The Bronze Layer in Practice
  • Silver Layer
    • Cleaning Data Activities
    • Designing the Silver Layers Data Model
      • Conforming and renaming columns
      • Denormalization
      • Slowly changing dimensions
      • Surrogate keys
    • Harmonization with Other Sources
    • 3NF and Data Vault
    • Operational Querying and Machine Learning
    • Managing Overlapping Requirements
    • Automation Tasks
    • The Silver Layer in Practice
  • Gold Layer
    • Star Schema
      • Loading the dimension tables
      • Loading the fact tables
      • Optimizing loads
    • Star Schema Design Nuances
    • Curated, Semantic, and Platinum Layers
    • One-Big-Table Design
    • Serving Layer
    • The Gold Layer in Practice
  • Conclusion
• II. Crafting the Medallion Layers
• 4. Building a Medallion Foundation with Microsoft Fabric
  • Our Case Study: Oceanic Airlines
  • Introducing Microsoft Fabric
    • Domains
    • Workspaces and Capacities
    • OneLake
    • Data Engineering with Spark
    • Data Warehousing with T-SQL
    • Other Fabric Workload Types
  • Setting Up the Foundation
    • Setting up Capacities
    • Setting up Domains
    • Setting up Workspaces
    • Creating Lakehouses
    • Capacity Considerations
    • Domain Considerations
    • Workspace Considerations
    • Lakehouse Entities Considerations
    • Storage Account Considerations
  • Conclusion
• 5. Construct the Bronze Layer
  • Building the Data Pipeline
    • Deploying the AdventureWorks Sample Database
    • Set Up an Azure SQL Database Connection
    • Creating a New Data Pipeline
      • Building the ForEach loop
      • Configuring the CopyTable activity
    • Additional Considerations
  • Implementation of Lakehouse Tables
    • Traverse Parquet Files to Managed Delta Tables
    • Using External Tables
    • Updating Tables with MERGE Operations
    • Spark Structured Streaming
      • Example with Azure Event Hubs
    • Using Change Data Capture
    • Navigating Data Handling Techniques
  • Schema Management
    • Create Tables Without Defining Schemas
    • Define Schemas with the DataFrame API
    • SQL DDL Statements
    • YAML or JSON Configurations
    • Metadata-Driven Approach
    • Databricks Auto Loader
    • Third-Party Tools
    • Handling Schema Evolution
  • Conclusion
• 6. Build the Silver Layer
  • Quick Recap
  • Implementation of a Metadata-Driven Approach
    • Implementation of the Metadata Store
    • Implementation of Dynamic Data Validations
    • Improvement Areas
  • Data Cleansing
    • Implementation of Data Cleansing Tasks
    • Data Cleansing Considerations
    • Data Transformation Frameworks and Data Quality Tools
  • Optimization of Query Performance with Denormalization
  • Lightweight Enrichments
  • Data Historization
  • Optimization Jobs
  • Orchestration with Apache AirFlow
  • Final Recommendations
  • Silver-Layer Data as a Product
  • Conclusion
• 7. Streamline the Gold Layer
  • Design of the Gold Layer
    • Transform Data Using a Star Schema
      • Creation of the Gold-layer tables
      • Creation of the dimensional table for address
      • Creation of the dimensional table for customer
      • Creation of the dimensional table for date
      • Creation of the dimensional table for product
      • Creation of the fact table for sales
    • Creation of the Semantic Model
    • Creation of the First Power BI Report
    • Creation of Task Flows
    • Enhancements for Gold-Layer Design
    • Microsoft Fabric in Practice
  • Data Products
    • Introduction to data product guidelines
    • Types of data products
    • Data modeling guidance
    • Governance guidance
  • Data Governance with Microsoft Purview
    • Microsoft Purview Design Considerations
      • Governance domains
      • Collections
      • Microsoft Purview data products
    • Guidance for Medallion Architectures
  • Conclusion
• III. Real-World Case Studies
• 8. Case Study: Data, Analytics and Business Strategy at AP Pension
  • Medallion Architecture
  • Other Considerations
  • Final Recommendations
• 9. Case Study: Amadeus, a Tech Leader in the Travel Industry
  • Medallion Architecture
  • FinOps
  • Data Models
  • Data Contracts
  • Data Governance
• 10. Case Study: Strategic Data Transformation at ZEISS
  • Data Platform Evolution
  • Medallion Architecture
  • Data Products and Sharing
  • Recommendations and Best Practices
• IV. Scaling, Governance, and the Future of Medallion Architectures
• 11. Scaling the Medallion Architecture
  • Decentralization of Data Management
    • Flexibility in Federation
    • Medallion Mesh
    • Number of Medallion Architectures
  • Medallion Inner Architecture Variations
    • Separate Data Product Layers
    • Tailored Medallions Architectures
    • Adaptability of the Bronze Layer
    • Silver Layer Variations
    • Gold Layer Variations
    • Enterprise Data Models
    • Master Data Management
    • Reference Data Management
  • Conclusion
• 12. Medallion Governance and Security
  • Data Governance
    • Governance Within a Medallion Architecture
    • Unity Catalog
    • Medallion Architecture with Unity Catalog
  • Data Contracts
    • Contracts Within a Catalog
    • Contracts Within a Metastore
    • Data Contracts Using YAML Files and GitOps
    • Other Data Contract Specifications
  • Data Security and Access Management
  • Conclusion
• 13. Future Medallion Architectures with Generative AI
  • Unstructured Data Processing
    • Retrieval-Augmented Generation
    • Bronze Layer
    • Silver Layer
    • Gold Layer
  • Integration of LLMs and Medallion Architectures
  • Role of Agents
  • Training and Fine-Tuning LLMs
  • Future of Medallion Architectures
  • Conclusion
• Index

pokaż cały spis treści

ukryj recenzje