Databricks Certified Generative AI Engineer Associate Study Guide. Generative AI with Databricks Rajaniesh Kaushikk

Dostępność produktu

Produkt został zweryfikowany pod kątem ułatwień dostępu więcej przeczytasz w deklaracji dostępności.

Spis treści książki

• Foreword
• Preface
  • Who Should Read This Book
  • Conventions Used in This Book
  • Using Code Examples
  • OReilly Online Learning
  • How to Contact Us
  • Acknowledgments
• 1. Exam Details and Resources
  • Prerequisites
  • Learning Objectives
    • Certification Overview and Benefits
      • Purpose and value of the certification
      • Career growth and industry applications
    • Exam Format and Objectives
      • Official exam structure
      • Understanding scored versus unscored questions
      • Registration fee, validity, and recertification
      • Domain weightings and focus areas
      • Breakdown of exam domains
        • Section 1: Design Applications
        • Section 2: Data Preparation
        • Section 3: Application Development
        • Section 4: Assembling and Deploying Applications
        • Section 5: Governance
        • Section 6: Evaluation and Monitoring
  • Key Preparation Strategies
    • Review the Exam Blueprint
    • Gain Hands-On Experience
    • Use Databricks Documentation Effectively
    • Practice with Sample Questions and Mock Exams
    • Practical Exam Tips
  • Setting Up Your Databricks Workspace
    • Step 1: Open the Databricks Website
    • Step 2: Start the Databricks Sign-Up Process
    • Step 3: Authenticate with Your Microsoft Account
    • Step 4: Access the Databricks Account Setup Flow
    • Step 5: Open the Azure Portal to Create the Databricks Workspace
    • Step 6: Configure Workspace Settings on the Basics Tab
    • Step 7: Configure Networking Options
    • Step 8: Review Encryption Settings
    • Step 9: Review Security and Compliance Settings
    • Step 10: (Optional) Review Tags
    • Step 11: Validate and Create the Workspace
    • Step 12: Create a Machine Learning Cluster
    • Step 13: Create a Notebook and Run a Test Command
  • Summary
• 2. Designing Generative AI Applications
  • Prerequisites
  • Learning Objectives
  • Crafting Prompts for Structured Output
    • Real-World Examples of Structured Prompting
    • Strategies for Structured Prompting
    • Using Prompt Templates in Code
    • Prompt Injection Considerations When Using Templates
    • Matching Model Tasks to Use Cases
      • Common LLM task types
      • Use case: Misaligned task type
      • Contrasting use cases: Classification versus extraction
      • Choosing the right model task
      • Best practices for prompt-task matching
      • Aligning prompts to use cases
      • Task type decision matrix
    • Designing Prompts That Elicit Specifically Formatted Responses
      • Why format control matters in real systems
      • Techniques for enforcing specific output formats
      • Example: Enforcing JSON-only output in Databricks
      • Common formatting pitfalls and how to avoid them
      • Example prompts to avoid common formatting pitfalls
      • Handling partial responses and fallback strategies
  • AI Pipeline Composition
    • Selecting Chain Components
      • Core pipeline components
      • Scenario using pipeline components
    • Summarizing Claims with LangChain
    • Template Registries and Reuse
    • Prompt Registries in Practice: An Open Source Example
    • Translating Use Cases into Input/Output Design
    • Input/Output Design Use Case
      • Decompose the task into multiple steps
      • Define the inputs
      • Specify the outputs
      • Map to pipeline components
      • Summary generation with a prompt template
    • Tips for Robust I/O Design
    • Designing Reusable Prompt Templates
    • Identifying Prompt/Response Pairs That Align with a Given Model Task
      • What alignment means in practice
      • Evaluating prompt/response pairs by task type
      • Example: Detecting misalignment in a Databricks pipeline
      • Using aligned prompt/response pairs for evaluation
      • Alignment as a design discipline
  • Multi-Stage Reasoning Tools
    • Tool Ordering for Sequential Logic
    • Planning Agent Chains
    • Agent Planning with LangChain
  • Designing Multi-Agent Systems with Tool Use
    • Scenario: Insurance Claim Processing
    • Key Design Principles of Multi-Agent Systems
      • Define agent roles clearly
      • Use consistent interfaces
      • Support asynchronous or iterative reasoning
      • Use centralized coordination if needed
      • Defining and ordering tools for multi-stage reasoning
      • Scenario: Ordered tools in a LLM reasoning pipeline
      • Ordering tools in agent-based systems
    • Embedding and Model Selection
    • Selecting Embedding Model Context Lengths and LLM Sizes
      • Context length: Why it matters
      • Comparing model sizes
    • Reading Model Cards to Understand Risk and Bias
    • Balancing Cost, Latency, and Quality When Choosing Models
      • Selecting LLMs based on model metadata and model cards
        • Understanding model metadata
        • Where to find model metadata in practice
        • What model cards add beyond metadata
        • Using model cards to assess risk and bias
        • Scenario: Selecting an LLM for a Databricks RAG application
        • Model selection as a repeatable process
  • Summary
  • Practice Questions
    • Answer Key
  • Hands-On Lab: Multi-Agent Workflow with LangChain + OpenAI in Databricks
    • Scenario
    • Objective
• 3. Preparing and Chunking Data for RAG Applications
  • Prerequisites
  • Learning Objectives
  • RAG Pipeline Overview and Where Chunking Fits
  • Document Analysis and Chunking
  • Chunking Strategies and When to Use Them
    • Fixed-Length Chunking
    • Sentence-Level Chunking
    • Paragraph-Based Chunking
    • Sliding Window Chunking
    • Semantic Chunking
    • Controlling Overlap and Granularity
    • Overlapping Chunks for Contextual Flow: Fixed and Dynamic
    • Dynamic Chunking with Semantic Continuity
    • Impact of Chunking on Retrieval
      • Real-world example: Customer support knowledge base
      • Visualizing the trade-off
    • Applying Chunking Strategy Based on Model Constraints
  • Content Filtering and Extraction
    • Removing Redundancy and Noise
    • Real-World Example: Software Documentation Portal
    • Extracting Content from PDFs and Images
    • Converting to Delta Format for Querying
      • Why use Delta Lake in RAG pipelines?
      • Example conversion workflow
      • Real-world example: Legal document indexing
    • Choosing the Correct Python Package for Content Extraction
    • Evaluating Retrieval Quality
      • Precision and recall
      • Reranking: Improving relevance
      • When to use precision, recall, and reranking
  • Using Metrics to Improve Performance
    • Validating Chunking and Filtering Strategies
    • Monitoring and Feedback Loops
      • The role of reranking in retrieval
      • Common reranking approaches
      • When reranking matters most
      • Practical considerations for using reranking
      • Using tools and metrics to evaluate retrieval performance
      • Offline evaluation with labeled queries
      • Connecting metrics to system improvements
      • Common causes of retrieval gaps
    • Corrective Techniques to Resolve Retrieval Gaps
      • Revisiting chunking strategy
      • Aligning embedding models
      • Enhancing the indexing pipeline
      • Query expansion and rewriting
      • Hybrid search techniques
      • Use case: Legal document search engine
    • Data Extraction Tools
      • Tools for PDF and image content extraction
      • Common approaches
        • Text-based PDF parsing
        • Scanned PDF/image OCR
        • Hybrid extraction
      • Writing chunked data to Delta tables
      • Document ingestion use cases
      • Tools for PDF and image content extraction
        • Text-based PDF extraction tools
        • OCR tools
        • Hybrid extraction approaches
        • Choosing the right tool
      • Writing chunked data to Delta tables
        • Why use Delta tables for chunked data
        • Representing chunks as table rows
        • Schema design considerations
        • Common pitfalls to avoid
      • Document ingestion use cases
        • Legal and compliance documents
        • Healthcare and clinical documents
        • Financial and business reports
        • Technical documentation and knowledge bases
        • Choosing an ingestion strategy by use case
      • Identifying missing source documents needed for RAG quality
        • Understanding document coverage gaps
        • Signals indicating missing source documents
        • Practical techniques to identify missing documents
        • Common missing document categories by use case
        • Preventing gaps through ingestion design
  • Summary
  • Practice Questions
    • Answer Key
  • Hands-On Lab: Chunking and Indexing for RAG
    • Scenario
    • Objective
• 4. Building GenAI Applications with Python and LangChain
  • Prerequisites
  • Learning Objectives
  • LangChain Fundamentals
    • Prompt Templates and Chains
    • Prompt Templates
    • Chains
    • Why Prompt Templates and Chains Matter
  • Memory and Context Management
    • Short-Term Context: Managing Recent Interactions
    • Long-Term Context: Persisting Relevant Information
    • Context Injection as a Design Pattern
    • Full-History Context Injection (Pattern)
    • Windowed Context Injection (Pattern)
    • Fact Store or Knowledge-Triple Memory (Pattern)
    • Why Memory Matters
  • LangChain Agents Overview
    • How Agents Work
    • Benefits of Using Agents
    • When a Single Agent Is Not Enough
  • Model and Tool Integration
    • Plugging in Tools (APIs, Databases, Calculators)
    • How Tools Work
      • Example: Connecting a calculator tool
      • Example: Querying a SQL database
      • Example: Calling a web API
      • Why tool integration matters
  • RAG Integration with Chains
    • How Retrieval Works in LangChain
    • Example: Using a Vector Store for Retrieval
    • Why Retrieval Matters
  • Tool Use Cases in Different Industries
    • Healthcare
    • Finance
    • Retail
  • Quality and Safety Mechanisms
    • Common Issues in Model Responses
    • Real-World Impact
  • Prompt Augmentation for Safety
    • Techniques for Prompt Augmentation
    • Example: Safe Summarization
    • Real-World Applications
    • Benefits of Prompt Augmentation
  • Implementing Guardrails
    • Techniques for Implementing Guardrails
    • Example: Guardrails with Output Validation
    • Real-World Applications
    • Comparison: Without Guardrails Versus with Guardrails
  • Qualitatively Assessing LLM Responses (Quality, Safety, and Hallucination Issues)
    • A Practical Evaluation Workflow for LLM Outputs
    • Define Quality in an Exam-Aligned Way
    • Define and Detect Hallucinations
      • How to systematically detect hallucinations during qualitative review
    • Assess Safety and Policy Compliance in Outputs
    • Build a Qualitative Scoring Rubric You Can Reuse
    • A Databricks-Oriented Test Set Strategy
    • Writing Reviewer Notes That Drive Engineering Fixes
    • Common Qualitative Pitfalls and How to Avoid Them
  • Writing Metaprompts to Minimize Hallucinations or Data Leakage
    • What a Metaprompt Isand Isnt
    • Why Metaprompts Matter for Hallucination Control
    • Common Metaprompt Objectives in Databricks Applications
    • Structure of an Effective Metaprompt
    • Preventing Data Leakage with Metaprompts
    • Metaprompts and Prompt Injection Resistance
    • Testing and Iterating on Metaprompts
    • Common Metaprompt Mistakes
  • Agent Frameworks and Multi-Agent Systems
    • Multi-Stage Reasoning Agents
    • Designing Agent Prompt Templates
    • Multi-Agent System Architecture
      • User input
      • Coordinator agent (analyzes query and assigns tasks)
      • Compliance agent (policy checking)
      • Data analysis agent (statistical processing)
      • Domain knowledge agent (factual retrieval)
      • Coordinator agent (synthesizes results)
      • Final response (accurate, compliant, explainable)
  • Creating Tools Needed to Extract Data for Retrieval Tasks
    • Why Tools Are Critical for Retrieval Tasks
    • Tool Categories Commonly Used in RAG Systems
    • Designing a Retrieval Tool: Principles and Constraints
    • Tools and Retrievers: How They Work Together
    • Adding Preprocessing to Extraction Tools
    • Tool Design for Safety and Governance
    • When to Use Tools Versus Prompt Instructions
    • Summary of Tool Creation for Retrieval
  • Summary
  • Practice Questions
    • Answer Key
  • Hands-On Lab: Building a Retrieval-Augmented GenAI App
    • Scenario
    • Objective
• 5. Deploying and Integrating RAG Systems on Databricks
  • Prerequisites
  • Learning Objectives
  • Chain Assembly and Testing
    • PyFunc Model Structure
    • LangChain Chain to Deployable Model
    • Local Versus Hosted Deployment Paths
    • Coding a Simple Chain Based on Requirements
      • Understanding requirements before writing code
      • Mapping requirements to chain components
      • Designing a minimal chain interface
      • Example: Coding a simple RAG chain
      • Testing the chain against requirements
      • Preparing the chain for deployment
    • Coding a PyFunc Model with Pre- and Postprocessing
      • Why PyFunc is used for deployment
      • Understanding pre- and postprocessing
      • Implementing a simple PyFunc model
      • Preparing the PyFunc model for registration
  • Model Deployment Process
    • MLflow Tracking and Registration
    • Deploying to Model Serving Endpoints
    • Access and Version Control
    • Controlling Access to Model Serving Endpoints
      • Why access control matters for Model Serving
      • Identity-based access to endpoints
      • Common permission levels for serving endpoints
      • Access control in RAG applications
      • What not to do: Access control inside the model
      • How access control fits into the deployment workflow
  • Vector Search and Endpoint Optimization
    • Vector Search Fundamentals in Databricks
      • Why Vector Search matters in RAG
      • Core components
      • Retrieval pipeline overview
      • Creating a basic Vector Search index
    • Tuning for Latency and Cost
      • Common bottlenecks and solutions
      • Optimized query flow
      • Real-world example
    • Incorporating Foundation Model APIs
      • Why incorporate foundation models?
      • Common integration patterns
      • Optimized FM call flow
      • Direct invocation in PyFunc
      • Retry and fallback logic
      • Trade-offs in FM integration
    • Identifying Resources Needed to Serve Features in RAG Applications
      • Core resource categories in RAG serving
      • Resources for vector search
      • Resources for Databricks Model Serving
      • Supporting platform resources
      • Balancing performance, cost, and reliability
      • How resource identification fits into RAG deployment
  • Advanced Model Serving
    • Pre- and Postprocessing in PyFunc Models
    • Endpoint Access and Resource Control
    • Batch Inference Workloads with ai_query
  • Summary
  • Practice Questions
    • Answer Key
  • Hands-On Lab: End-to-End RAG System Deployment on Databricks
    • Scenario
    • Objective
• 6. Managing Models with MLflow and Unity Catalog
  • Prerequisites
  • Learning Objectives
  • MLflow Basics and Tracking
    • Tracking Experiments and Parameters
    • Logging Models and Artifacts
    • Reviewing Runs and Metrics
    • Putting It All Together: A Simple Workflow
    • Lifecycle and Governance Mapping
    • Evaluating RAG Model Performance Using MLflow
      • Common automated metrics in RAG evaluation
      • Why RAG evaluation requires a different approach
      • Common evaluation dimensions for RAG systems
      • Logging RAG evaluation metrics in MLflow
      • Reviewing evaluation results across runs
      • From evaluation to lifecycle decisions
  • Model Registration and Deployment
    • Registering Models with Unity Catalog
    • Managing Versions with Aliases and Tags
    • Model Lifecycle with Aliases
    • Model Lifecycle Real-World Case Study: Fraud Detection Model
    • Aliases at a Glance
    • Governance and Access Control
    • Case Study: Governance in Action
  • Operational Best Practices
    • Reproducibility and Documentation
    • Managing Metadata and Model Signatures
    • Archiving and Cleanup Policies
    • Case Study: Fraud Detection with Governance in Action
    • Using Databricks Features to Control LLM Costs
      • Why cost control matters for LLM operations
      • Controlling costs through MLflow experiment discipline
      • Using model versioning to limit unnecessary inference
      • Workspace and endpoint controls for LLM usage
      • Integrating cost awareness into operational practices
  • Summary
  • Practice Questions
    • Answer Key
  • Hands-On Lab: End-to-End Model Management with MLflow and Unity Catalog
    • Scenario
    • Objective
• 7. Responsible GenAI: Guardrails and Governance
  • Prerequisites
  • Learning Objectives
  • Guardrail Techniques
    • Prompt Filtering and Redaction
    • Input Validation and Constraints
    • Rate Limiting and Monitoring
    • Using Masking Techniques to Meet Performance or Safety Objectives
      • Why masking improves safety and performance
      • Choosing a masking pattern
      • Where masking belongs in the request flow
    • Selecting Guardrail Techniques to Protect Against Malicious Inputs
      • Understanding malicious input patterns
      • Mapping threats to guardrail controls
      • Layered guardrail selection
  • Data Governance and Privacy
    • Data Masking Techniques
    • Mitigating Problematic Text
    • Adhering to Licensing Requirements
    • Legal and Licensing Requirements for Data Sources
      • Why licensing matters in generative AI pipelines
      • Common data source categories and constraints
      • License-aware data intake
    • Providing Alternatives for Problematic Text Mitigation (RAG Input)
      • Why substitution is preferable to deletion in RAG
      • Common substitution patterns
      • Substitution-based retrieval flow
  • Risk Management Frameworks
    • Responsible AI Checklists
    • Legal and Ethical Considerations
    • Audit Trails and Transparency
  • Legal and Licensing Considerations
    • Licensing Requirements for External Data
    • Data Source Compliance Strategies
    • Legal Risk Mitigation
  • Summary
  • Practice Questions
    • Answer Key
  • Hands-On Lab: Implementing AI Guardrails
    • Scenario
    • Objective
• 8. Monitoring and Evaluating LLMs in Production Environments
  • Prerequisites
  • Learning Objectives
  • Metrics and Evaluation Criteria
    • Choosing Key Performance Metrics
    • Quantitative Evaluation with MLflow
    • Ground Truth and Human Evaluation
    • Evaluation Dataset Best Practices
    • Case Study: Evaluating Hallucination Risk in a Summarization Endpoint
  • Selecting LLM Size and Architecture Based on Evaluation Metrics
    • Using Evaluation Metrics to Compare Model Sizes
    • Architecture Selection as a Feedback Loop
    • Practical Guidance for Production Decisions
  • Monitoring and Logging
    • Inference Tables and Logs
    • Using Agent Monitoring Tools
    • Anomaly Detection and Alerts
    • Monitoring Dashboard Best Practices
    • Alerting Strategies
    • Case Study: Identifying a Latency Spike in a Customer Support Chatbot
    • Using Inference Logging to Evaluate RAG Performance
      • Evaluating retrieval quality through inference logs
      • Querying inference tables for RAG evaluation
      • Distinguishing hallucination from retrieval failure
      • RAG evaluation as a continuous process
    • Using Agent Monitoring and Inference Tables
      • Role of inference tables in agent-based systems
      • Role of agent monitoring in multi-step workflows
      • Using both tools together for effective diagnosis
      • Exam-relevant decision guidance
  • Cost and Performance Optimization
    • Tracking Token Usage and Latency
    • Resource Allocation Strategies
    • Cost-Saving Recommendations
      • Establishing review cycles
    • Case Study: Reducing Token Costs in an Ecommerce Recommendation Assistant
    • Using Databricks Features to Control GenAI Costs
      • Inference tables as the cost audit trail
      • MLflow evaluation for cost validation
      • Monitoring dashboards and alerts for cost governance
      • Platform features as cost control surfaces
  • Advanced Evaluation Techniques
    • Human Versus Automatic Evaluation
    • Evaluation Versus Monitoring
    • Calibrating Evaluation Metrics
    • Case Study: Calibrating Groundedness Metrics for a RAG-Based Search Assistant
    • Identifying Which Evaluation Judges Require Ground Truth
      • What ground truth means in LLM evaluation
      • Evaluation judges that require ground truth
      • Evaluation judges that do not require ground truth
      • Using retrieved context as a proxy for ground truth
  • Summary
  • Practice Questions
    • Answer Key
  • Hands-On Lab: Evaluating and Monitoring LLM Performance
    • Scenario
    • Objective
• 9. Scaling AI Solutions with Vector Search and Mosaic AI
  • Prerequisites
  • Learning Objectives
  • Vector Search Fundamentals
    • Cosine Similarity, Dot-Product Similarity, and Vector Normalization
      • Cosine similarity
      • Dot-product similarity
      • Why normalization matters
      • Interpreting metric behavior in real retrieval systems
    • Indexing and Similarity Search
    • Creating and Querying Vector Search Index
      • Index creation essentials and best practices
    • Querying a Vector Search Index
    • Querying with Metadata Filters
    • Retrievers and Embedding Models
      • Choosing the k value in Vector Search
      • What k controls
      • Trade-offs when selecting k
      • Practical guidelines for selecting k
      • Testing strategies
    • Choosing the Right Embedding Model: Scenario Decision Guide
    • Quick Scenario-to-Model Mapping
  • Scaling Applications with Mosaic AI
    • Overview of Mosaic AI Architecture
    • Model Serving at Scale
    • Case Study: High-Throughput Retrieval
  • Optimization and Infrastructure
    • Choosing Context Length and Embedding Dimensions
    • Batch Inference Workloads
    • Performance Testing and Profiling
    • Identifying Batch Inference Workloads Suitable for ai_query()
  • Summary
  • Practice Questions
    • Answer Key
  • Hands-On Lab: Building a Scalable Vector Search and Retrieval System
    • Scenario
    • Objective
• 10. Databricks Certified Generative AI Engineer Associate Exam Preparation
  • Prerequisites
  • Learning Objectives
  • Reviewing the Exam Blueprint
    • Recap of All Domains
    • Domain Summaries
      • Generative AI foundations
      • LLM concepts
      • Generative AI workflows
      • Databricks GenAI tools and capabilities
    • High-Priority Topics
  • Practice Strategies
    • Exam Simulation and Practice Tests
      • Practice test simulation using Databricks Model Serving
      • Objective
      • Code
    • Time-Management Tips
  • Final Readiness Checklist
    • Technical Setup
    • Exam-Day Readiness
  • Practice Exam
    • Answer Key
  • Summary
  • Hands-On Lab: End-to-End Generative AI Readiness Assessment
    • Scenario
    • Objective
• Index

pokaż cały spis treści

ukryj recenzje