Apache Polaris: The Definitive Guide. Enriching Apache Iceberg Data Lakehouses with an Open Source Catalog Alex Merced, Andrew Madson, Tomer Shiran

Dostępność produktu

Produkt nie został jeszcze oceniony pod kątem ułatwień dostępu lub nie podano żadnych informacji o ułatwieniach dostępu lub są one niewystarczające. Prawdopodobnie Wydawca/Dostawca jeszcze nie umożliwił dokonania walidacji produktu lub nie przekazał odpowiednich informacji na temat jego dostępności.

Spis treści książki

• Foreword
• Preface
  • Conventions Used in This Book
  • Using Code Examples
  • OReilly Online Learning
  • How to Contact Us
  • Acknowledgments
• I. Data Lakehouses and Apache Iceberg Fundamentals
• 1. Data Lakehouse and Apache Iceberg
  • Modern Data Challenges
    • The World of Data Warehouses
    • Moving Forward with Data Lakes
      • Hadoops architecture
      • Challenges with data lakes
    • The Cloud Revolution
    • File-Based Analytics with Apache Parquet
      • How Apache Parquet works
      • Challenges with multi-file datasets
  • The Data Lakehouse Solution
    • The Key Benefits of a Data Lakehouse
    • The Path Forward: Data Lakehouse Table Formats
  • The Role of Table Formats
    • The Benefits of Table Formats
    • Existing Table Formats
    • Apache Iceberg
  • What Is Apache Iceberg?
    • Metadata File (metadata.json)
    • Manifest List
    • Manifest Files
    • Data Files
    • Delete Files
  • Conclusion
• 2. The Role of Apache Iceberg Catalogs
  • What Is and Isnt an Apache Iceberg Catalog
    • The Mechanics of Apache Iceberg Catalogs
  • Types of Apache Iceberg Catalogs
    • File-System Catalogs
      • How the Hadoop Catalog works
      • The challenges with file-system catalogs
    • Service Catalogs
      • Hive catalog
      • JDBC catalog
      • Nessie catalog
      • AWS Glue catalog
  • Challenges of Diverse Catalog Options
    • Client-Side Complexity
    • Configuration Challenges
    • Authorization Challenges
    • The Need for a Unified Approach
  • The Apache Iceberg REST Catalog Specification
    • Key Benefits of the REST Catalog Specification
    • The Evolution of REST Catalog Implementations
  • Apache Polaris
    • The Birth of Apache Polaris
    • Polaris: A New Era of Lakehouse Catalogs
    • Conclusion
• II. Apache Polaris
• 3. The Apache Polaris Security Model
  • What Is Polaris?
  • Catalogs
    • Key Features of Polaris Catalogs
    • Benefits of Multi-Catalog Architecture
  • Principals
    • What Are Principals?
    • Managing Principals
    • Principal Lifecycle
  • Catalog Roles
    • Defining Permissions in Catalog Roles
      • Table Privileges
      • View Privileges
      • Namespace Privileges
      • Catalog Privileges
    • Assigning Catalog Roles to Principals
      • Benefits of catalog roles in Polaris
      • Relationship between principal roles and catalog roles
    • Best Practices for Catalog Roles
  • Principal Roles
    • What Are Principal Roles?
    • Benefits of Principal Roles
    • Best Practices for Principal Roles
  • Polaris Security Best Practices
    • Multi-Tenant Environments
    • Cross-Team Collaboration
    • Compliance and Sensitive Data Governance
    • Cloud-Native Deployments
  • Conclusion
• 4. External Catalogs
  • Nessie
    • What Makes Nessie Unique?
    • Why Use Nessie with Polaris?
    • Example: Nessie and Polaris in Action
  • Gravitino
    • What Makes Gravitino Unique?
    • Why Use Gravitino with Polaris?
    • Example: Distributed Metadata Governance
  • Lakekeeper
    • What Makes Lakekeeper Unique?
    • Why Use Lakekeeper with Polaris?
    • Example: Multi-Tenant Metadata Governance
  • AWS Glue
    • Why Use the AWS Glue Catalog?
    • Why Use Glue with Polaris?
    • Example: Hybrid Team Collaboration
  • Conclusion
• 5. Polaris REST API
  • Catalog Operations
    • List Catalogs
    • Create a Catalog
    • Get Catalog Details
    • Update a Catalog
    • Delete a Catalog
  • Principal Operations
    • List Principals
    • Create a Principal
    • Get Principal Details
    • Update a Principal
    • Delete a Principal
    • Rotate Principal Credentials
  • Managing Roles
    • Create a Catalog Role
    • Create a Principal Role
    • List Catalog Roles
    • List Roles Assigned to a Principal
    • List All Principal Roles
    • List Principals Assigned to a Principal Role
    • Get Catalog Roles Mapped to a Principal Role
    • Get Details of a Principal Role
    • Add a Grant to a Catalog Role
    • Revoke a Grant from a Catalog Role
    • Assign a Catalog Role to a Principal Role
    • Assign a Role to a Principal
    • Update a Principal Role
    • Revoke a Role from a Principal
    • Revoke a Catalog Role from a Principal Role
    • Delete a Principal Role
    • Delete a Catalog Role
  • Apache Iceberg REST Catalog Endpoints
    • Configuration API
    • OAuth2 API
    • Table API
      • List Table Identifiers
      • Create a Namespace
      • Load Namespace Properties
      • Check Namespace Existence
      • Drop a Namespace
      • Set or Remove Namespace Properties
      • Create a Table
      • Register a Table
      • Load Table Metadata
      • Commit Updates to a Table
      • Drop a Table
      • Check Table Existence
      • Rename a Table
      • Submit Metrics for a Table
      • Send Table Notifications
      • Commit Updates to Multiple Tables
    • View API
      • List View Identifiers
      • Create a View
      • Load View Metadata
      • Replace a View
      • Drop a View
      • Check View Existence
      • Rename a View
  • Conclusion
• III. Hands-on with Apache Polaris
• 6. Working with Apache Polaris OSS
  • Deploying Locally with Docker
    • Prerequisites
    • Step 1: Clone the Repository
    • Step 2: Configure Environment Variables
    • Step 3: Understand the Docker Compose File
      • Polaris service
      • Spark service
      • Shared network
      • Step 3 summary
    • Step 4: Starting the Environment
    • Step 5: Stopping the Environment
  • Creating Catalogs
    • When to Create a Catalog
      • When to create a new catalog
      • When not to create a new catalog
      • Recommended catalog strategies
      • Long-term considerations
  • Creating Catalog Roles
    • When to Create Catalog Roles
      • Use roles to reflect real-world responsibilities
      • Favor least privilege
      • Reuse and generalize where possible
      • Limit catalog roles by scope and policy
      • Document ownership and purpose
      • Periodically review and rotate
  • Creating Principals
  • Creating Principal Roles
    • When to Create a Principal Role
      • When to create a new principal role
      • Map roles to real organizational units
      • Reuse roles across environments
      • Keep roles manageable
      • Audit and review
  • Assigning the Catalog Role to the Principal Role and Setting Permissions on the Catalog
  • Summary
• 7. Using Apache Polaris with Apache Spark
  • Connecting Your Apache Polaris Catalog to Apache Spark
  • Using Spark Dataframe API with Apache Polaris (Incubating)
    • Creating a Table
    • Querying a Table
    • Updating a Table
    • Deleting Rows
    • Appending Data
    • Reading Metadata Tables
  • Using SparkSQL with Apache Polaris
    • Creating a Table
    • Querying a Table
    • Inserting Data
    • Updating Data
    • Deleting Data
    • Merging Data
    • Reading Metadata Tables
    • Time Travel Queries
  • Using Spark Streaming with Apache Polaris
    • Setting Up Spark Streaming with Polaris
    • Streaming Reads from Polaris
    • Streaming Writes to Polaris
    • Handling Deletes and Overwrites
    • Using Partitioned Tables
    • Maintaining Streaming Tables
  • Conclusion
• 8. Using Apache Polaris with Snowflake
  • Establishing Connectivity Between Snowflake and Polaris
    • Configuring an External Volume
    • Creating a Polaris Catalog Integration
  • Querying Iceberg Tables via Snowflake and Polaris
    • Registering an Existing Polaris Table in Snowflake
    • Querying the External Iceberg Table
      • Automated versus manual metadata refresh
      • Read-only behavior of external tables
      • Querying Polaris metadata from Snowflake
      • Handling permissions and errors
  • Using Snowflake Open Catalog (Managed Polaris)
  • Polaris-Backed Tables vs. Native Snowflake Tables
  • Conclusion
• 9. Using Apache Polaris with Dremio
  • Connecting Dremio to an Apache Polaris Catalog
    • Connecting Polaris Using the REST Catalog Connector
      • Add a new source
      • General settings
      • Advanced Options
      • Advanced options: Catalog credentials
      • Save the source
    • Connecting Snowflakes Open Catalog to Dremio
      • Add a new source
      • General settings
      • Advanced options
      • Reflection refresh settings
      • Metadata settings
      • Privileges
      • Save and validate
      • Updating and deleting a source
    • Why Disable Use Vended Credentials?
  • Using Dremio SQL with Apache Polaris
    • Querying Iceberg Tables via Polaris
    • Querying the Iceberg Metadata Tables
      • Inspecting data files with table_files()
      • Viewing table history with table_history()
      • Getting snapshot metadata with table_snapshot()
      • Listing manifest files with table_manifests()
      • Analyzing partitions with table_partitions()
      • Monitoring table clustering with clustering_information()
    • Creating Tables and CTAS in Polaris via Dremio
      • Creating an empty table
      • Create Table as Select (CTAS)
    • Adding Data from Files to a Table Using Copy Into
      • Practical examples
        • Load all files in a folder (e.g., JSON logs)
        • Load a subset of files using a regex
        • Specify CSV format options and error handling
        • Transform and reorder fields during ingest
      • Supported file formats and options
      • Monitoring and debugging COPY INTO jobs
    • Maintaining Your Iceberg Tables with Dremio
      • Reclaiming storage with VACUUM
        • Remove snapshots selectively
        • Retain most recent snapshots
        • Use default settings
      • Improving query performance with OPTIMIZE
        • Default optimization with bin-packing
        • Tune file sizes and thresholds manually
        • Optimize only specific partitions
        • Rewrite manifest files to control metadata bloat
    • Dremio Automates Optimization
      • Showing table properties
  • Conclusion
• 10. Advanced Polaris Configuration and CLI Management
  • Using the Polaris CLI
    • CLI Structure, Authentication, and Profiles
      • Authentication options
      • Profiles for convenience
      • Connecting to Polaris
    • Managing Entities with the CLI
      • Catalogs
      • Principals
      • Roles: principal and catalog
      • Namespaces
      • Privileges
    • Understanding Realms
      • What is a realm?
      • How realm context works
      • Determining realm using the Polaris-Realm HTTP header (default)
      • Realm-aware authentication and bootstrapping
      • Realm isolation in practice
  • Observability: Metrics, Tracing, and Logging
    • Metrics with Micrometer and Prometheus
      • Accessing metrics
      • Scraping with Prometheus
      • Customizing metric tags
    • Tracing with OpenTelemetry
      • Enabling tracing
      • Customizing trace metadata
      • Troubleshooting trace export
    • Logging and Debugging with Quarkus
      • Basic logging behavior
      • Setting log levels
      • Debugging configuration issues
      • Enabling JSON logs
      • Using MDC for contextual logging
      • Adding custom MDC keys
  • Configuring Polaris for Production
    • Security and Authentication Configuration
      • Configure OAuth2 and token brokers
      • Bootstrap root credentials for a realm
        • Token issuance example
        • Best practices for production
    • Durable Metadata with Metastores
      • Using relational JDBC (recommended)
      • Deprecated: EclipseLink metastore
    • Hardening Defaults and Managing Feature Flags
      • Disable test catalog initialization
      • Restrict supported storage types
      • Enforce credential rotation policies
      • Realm-specific overrides
      • Review logs for flag behavior
    • Scaling, Concurrency, and Rate Limits
      • Task concurrency
      • Rate limiting with token buckets
      • Scaling considerations
    • Finalizing and Verifying Your Production Setup
  • Conclusion
• 11. Looking to the Future of Apache Polaris
  • Managed Polaris
  • The REST Catalog Ecosystem
    • Data Processing Engines
    • Streaming and Ingestion Platforms
    • Other Data-Stack Tools
  • The Apache Polaris Roadmap
    • Generic Table Support
      • API and architecture overview
      • Spark integration
      • Governance and identity model
      • Looking ahead
    • Policy Store
      • Architecture and key capabilities
      • Looking ahead
    • Table Maintenance Framework
      • Metadata-driven maintenance coordination
      • Flexible policy definition
      • Inheritance and governance
      • Integration model with TMS
      • Looking ahead
    • SQL and NoSQL Persistence
      • Deployment flexibility
      • Looking ahead
    • S3-Compatible Storage Support
      • Implementation overview
      • Deployment simplicity
    • Catalog UI
      • Feature scope
      • Looking ahead
    • Federated Catalogs
      • Federation modes and facade types
      • Milestone 1 scope
      • Milestone 2: JIT-creation of entity facades
      • Use cases
      • Looking ahead
      • Catalog Federation summary
    • Federated Role Support
      • Architecture overview
      • SCIM and future synchronization
      • Practical implications
    • Polaris Event Listeners
      • Initial event set
      • Implementation options
      • Looking ahead
    • Unstructured Data in Polaris
      • Architectural overview
      • Volume properties and behavior
      • API design
      • Limitations and looking ahead
  • Conclusion
• Index

pokaż cały spis treści

ukryj recenzje