Explore Databricks — Your Data Learning Hub

Curated by practitioners: the certifications worth earning, the updates worth knowing, and the repositories, articles, tools, and talks actually worth your time — mostly Databricks, with the broader data-stack essentials mixed in.

Latest News

Only the updates that actually change how you build — not an endless feed.

Agent Bricks reaches general availability — Auto-optimized agent building on Mosaic AI — describe the task and Databricks tunes the agent and its evals for you.
Databricks One is now generally available — A simplified, no-notebook workspace for business users to explore dashboards, Genie spaces, and apps.
Lakeflow Connect adds new managed connectors — More first-party ingestion connectors for databases and SaaS sources — no pipeline code required.
Lakeflow Declarative Pipelines reach GA — The evolution of Delta Live Tables: build batch and streaming pipelines declaratively in SQL or Python.
Lakebase — managed, autoscaling Postgres in the lakehouse — A fully-managed Postgres integrated with Unity Catalog, for serving and transactional workloads next to your analytics.
Unity Catalog Metrics — Define governed, reusable metrics once and query them consistently across SQL, dashboards, and notebooks.
Serverless GPU model serving — Scale-to-zero GPU endpoints for low-latency LLM and custom-model inference, billed per use.
Databricks Apps — build and host data apps — Ship interactive data and AI apps directly on Databricks with governed access to lakehouse data.

Certifications

The official Databricks credentials — with the exam guide and a Bronze2Gold practice test for each.

Data Engineer Associate — Core Databricks data engineering: the lakehouse, Delta Lake, ELT with Spark SQL and Python, and production pipelines.
Data Engineer Professional — Advanced pipelines, Lakeflow, testing and deployment, Unity Catalog governance, and performance tuning at scale.
Data Analyst Associate — Databricks SQL, dashboards, and analytics on the lakehouse for BI-focused practitioners.
Machine Learning Associate — MLflow, feature engineering, model training and serving, and AutoML on Databricks.
Machine Learning Professional — Production ML: the feature store, MLOps, model deployment, monitoring, and drift on Databricks.
Generative AI Engineer Associate — RAG, vector search, agents, and the Mosaic AI stack for building generative-AI applications.
Apache Spark Developer Associate — The Spark DataFrame API, transformations, and structured streaming — the foundation for everything else.

The Databricks Platform

Every major capability, explained in a line — from DLT and Genie to Unity Catalog and Mosaic AI. Your map to what each part of Databricks actually does.

Lakeflow Declarative Pipelines (DLT) — Build reliable batch and streaming pipelines declaratively in SQL or Python, with built-in data quality and auto-orchestration. The evolution of Delta Live Tables.
Lakeflow Jobs (Workflows) — The native orchestrator — schedule and chain notebooks, pipelines, SQL, and ML tasks into production workflows with retries and alerts.
Lakeflow Connect — Managed, no-code ingestion connectors that pull from databases and SaaS apps straight into the lakehouse.
Auto Loader — Incrementally and efficiently ingest new files from cloud storage as they arrive, with automatic schema inference and evolution.
Delta Lake — The open table format under the lakehouse — ACID transactions, time travel, and schema evolution on your data files.
Structured Streaming — Process real-time data with the same DataFrame API you use for batch — exactly-once, fault-tolerant streaming on Spark.
Photon Engine — The vectorized, C++ query engine that transparently accelerates SQL and DataFrame workloads for lower cost and latency.
Databricks SQL — Serverless SQL warehouses for fast, low-cost analytics and BI directly on lakehouse data.
AI/BI Genie — Ask questions of your data in plain English — Genie spaces turn natural language into governed, trustworthy SQL answers.
AI/BI Dashboards — Build and share interactive dashboards on the lakehouse, with AI-assisted charting and one governed source of truth.
Unity Catalog Metrics — Define business metrics once and query them consistently across SQL, dashboards, and notebooks.
Mosaic AI Model Serving — Deploy any model — custom, foundation, or external — behind scalable, scale-to-zero REST endpoints with unified governance.
Mosaic AI Vector Search — A managed vector database for similarity search — the retrieval layer for RAG applications, synced from Delta tables.
Mosaic AI Agent Framework — Build, evaluate, and deploy production RAG and tool-using agents, with tracing and quality evals built in.
Foundation Model APIs — Pay-per-token access to leading open and proprietary LLMs from within Databricks — no separate provider to wire up.
MLflow — The open standard for the ML and GenAI lifecycle — experiment tracking, model registry, evaluation, and deployment.
Feature Engineering & Store — Create, share, and serve features governed by Unity Catalog, with consistent values across training and inference.
AutoML — Generate strong baseline models automatically — with the editable notebooks behind every run, so it's a glass box, not a black box.
Databricks Assistant — A context-aware AI copilot in notebooks and the SQL editor — write, explain, fix, and optimize code in place.
Unity Catalog — One governance layer for all data and AI assets — a three-level namespace, fine-grained access, lineage, and auditing.
Delta Sharing — The open protocol for sharing live data across organizations and platforms — no copies, no vendor lock-in.
Clean Rooms — Collaborate on shared datasets with partners in a secure environment — run joint analysis without exposing raw data.
Lakehouse Federation — Query external databases and warehouses in place through Unity Catalog — no ingestion required to start.
Databricks Apps — Build and host interactive data and AI apps directly on Databricks, with governed access to lakehouse data.
Lakebase — A fully-managed, autoscaling Postgres integrated with the lakehouse — OLTP and serving next to your analytics.
Databricks Marketplace — An open exchange for data sets, ML models, notebooks, and solutions — powered by Delta Sharing.
Asset Bundles (DABs) — Define jobs, pipelines, and resources as versioned YAML and deploy them reproducibly across environments.

How-To in Databricks

Practical, do-this-now guides — from vibe-coding an app with the Assistant to building RAG, setting up governance, and shipping to production.

Vibe-code an app with Databricks Assistant — Describe what you want in plain English and let the Assistant scaffold it — then ship it as a Databricks App, no front-end setup required.
Build a RAG chatbot on your own docs — Chunk and embed content into Vector Search, wire it up with the Agent Framework, and serve it behind an endpoint with quality evals.
Set up Unity Catalog from scratch — Create your first catalog and schemas, apply grants, and get lineage and auditing right from day one.
Ship an ELT pipeline with Lakeflow Declarative Pipelines — Author a batch or streaming pipeline declaratively, add data-quality expectations, and let Databricks handle orchestration.
Query your data in plain English with Genie — Create a Genie space over a set of tables, add instructions and sample questions, and let stakeholders self-serve trustworthy answers.
Ingest files continuously with Auto Loader — Stream new files from cloud storage into Delta with automatic schema inference and evolution — a few lines of PySpark.
Track and serve a model with MLflow — Log experiments, register the winner in Unity Catalog, and deploy it to a scale-to-zero serving endpoint.
Deploy to production with Asset Bundles & CI/CD — Define jobs and pipelines as versioned YAML, then promote them across dev, staging, and prod from your CI pipeline.
Tune Spark jobs (and cut your bill) — Read the query plan, lean on Photon, fix skew and small files, and cache with intent — the highest-leverage performance wins.
Call an LLM with Foundation Model APIs — Hit leading open and proprietary models pay-per-token from a notebook or app — no separate provider account to manage.
Share live data across orgs with Delta Sharing — Publish a share and grant a recipient access to always-current data — no copies, across clouds and platforms.
Turn a notebook into a shareable dashboard — Promote your analysis into an AI/BI dashboard with governed data and AI-assisted charts stakeholders can actually use.

Databricks Labs

Open-source projects from Databricks Labs — the tools the field team builds and ships. All under github.com/databrickslabs.

dqx — A declarative data-quality framework for PySpark and Lakeflow pipelines — define checks and quarantine bad rows.
ucx — The toolkit for migrating a workspace to Unity Catalog — assessment, table upgrades, and code linting.
remorph — SQL transpiler and reconciliation toolkit — migrate legacy warehouse SQL onto Databricks and validate it.
dlt-meta — Metadata-driven framework for generating Lakeflow Declarative Pipelines (DLT) at scale from config.
dbldatagen — Generate large, realistic synthetic datasets in Spark for testing, benchmarking, and demos.
overwatch — Observability for your lakehouse — unify cluster, job, and cost telemetry to see what's actually running.
tempo — Time-series utilities for PySpark — as-of joins, resampling, and interpolation over large datasets.
discoverx — Run operations and searches across many Unity Catalog tables at once — great for scans and audits.

Reading Room

Every read in one place — filter to fresh drops, evergreen classics, hands-on guides, or highly technical deep dives.

Understanding LLM Models: What Makes One More Powerful? — A practical, fact-checked guide to parameters, tokens, context windows, training, and inference cost.
Unity Catalog: A Practical Quickstart — The three-level namespace, first catalog and grants, and the pitfalls to avoid.
Databricks Engineering Blog — First-party deep dives on the platform internals, performance, and new capabilities.
The Big Book of Data Engineering — A regularly-refreshed playbook of end-to-end lakehouse patterns, from ingestion to production ELT.
Kimball Group — Dimensional Modeling Design Tips — The canonical archive on dimensional modeling: grain, facts and dimensions, and slowly changing dimensions.
How we structure our dbt projects — The widely-adopted staging → intermediate → marts convention for analytics engineering.
Delta Lake documentation — The open table format under the lakehouse — ACID, time travel, and schema evolution.
Lakehouse: A New Generation of Open Platforms — The founding paper that defines the lakehouse architecture and argues why it supersedes the two-tier data-lake-plus-warehouse stack.
Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores — How Delta's transaction log delivers ACID, time travel, and fast metadata over plain object storage.
Photon: A Fast Query Engine for Lakehouse Systems — The design of Databricks' vectorized C++ engine — why it's fast and how it stays compatible with Spark.
Deep Dive into Spark SQL's Catalyst Optimizer — How Catalyst turns your DataFrame and SQL into optimized physical plans — analysis, logical optimization, and code generation.
Diving Into Delta Lake: Unpacking the Transaction Log — A step-by-step walk through the _delta_log — atomicity, optimistic concurrency, and how checkpoints keep reads fast.

Recommended Books

The references worth owning — the canonical books on Spark, the lakehouse, data engineering, and modeling.

Spark: The Definitive Guide — The comprehensive reference to Spark's APIs and engine, from the creators of the project.
Learning Spark, 2nd Edition — A hands-on introduction to Spark 3, DataFrames, and Structured Streaming — free from Databricks.
Delta Lake: The Definitive Guide — The complete guide to the open table format under the lakehouse — internals and best practices.
Fundamentals of Data Engineering — The data-engineering lifecycle end to end — the best big-picture book in the field.
The Data Warehouse Toolkit — The definitive, platform-agnostic reference on dimensional modeling — star schemas and SCDs.
Designing Data-Intensive Applications — The modern classic on the ideas behind data systems — storage, replication, and consistency.

Learn

Where to build the skills — official paths, docs, and hands-on tutorials.

Databricks Academy — Official self-paced and instructor-led courses, mapped to each certification.
Databricks Documentation — The canonical reference for every feature, API, and configuration.
Get started with PySpark — The official PySpark quickstart — DataFrames, transformations, and SQL.
Databricks Data Engineering Learning Path — A structured route from the lakehouse basics to production pipelines.
Dimensional Modeling Techniques — The reference catalog of star-schema and SCD techniques — the grammar of analytics data models.

Tools & Utilities

What we actually install to build on Databricks — official tooling plus community utilities that make projects easier.

Databricks CLI — Command-line access to the workspace, jobs, and Asset Bundles — script deployments and wire into CI/CD.
Databricks Connect — Run and debug local Python/Scala code against a remote Databricks cluster from your IDE or app.
Databricks VS Code Extension — Run and debug code on Databricks clusters from your local editor, with workspace sync.
Asset Bundles (DABs) — Define jobs, pipelines, and resources as versioned YAML and deploy across environments.
MLflow — Track experiments, package models, and manage the ML/GenAI lifecycle end to end.
dbt for Databricks — Build tested, version-controlled SQL transformations on Databricks SQL with the dbt adapter.
SQLFluff — A configurable SQL linter and auto-formatter with a Databricks/Spark SQL dialect — keep queries clean.
chispa — PySpark test helpers — assert DataFrame and column equality so you can unit-test transformations.

Watch

Talks and demos worth the time — no autoplay, just the good ones.