BigQuery vs Databricks: A Technical Comparison

BigQuery is Google Cloud’s serverless data warehouse. Databricks is a unified lakehouse platform built on Apache Spark. They overlap as the analytical core of a modern data stack but were designed on different premises and optimize for different workload shapes. This post is the engineering-side comparison: how each system is organized, where the bills come from, what each is fast and slow at, and what the cost actually looks like across three representative workloads.

What each is

BigQuery was productized in 2011 from Dremel, the internal query engine Google built to scan petabyte-scale logs. It is SQL-first and serverless: there are no clusters to size, no warm-up time, no auto-scaling policies to tune. A customer never sees the bytes on disk. Compute is allocated in units called slots, drawn from either an on-demand shared pool or a reserved baseline. The system is opinionated about how analytical queries should run, and that opinion delivers fast results out of the box for most analytical SQL.

Databricks was founded in 2013 by the creators of Apache Spark and has evolved from a managed Spark service into a full data platform combining ETL, analytics, ML, and streaming. The defining design choice is that data lives in open formats (Delta Lake on Parquet) inside the customer’s own object storage bucket. Compute runs on customer-provisioned clusters using Spark plus Photon, a vectorized C++ execution engine. The customer owns the bytes; Databricks provides the engine. More responsibility on the user; more architectural freedom in return.

The two systems represent different philosophies. BigQuery is the modern incarnation of the data warehouse — managed, opinionated, fast by default. Databricks is the modern incarnation of the data lake (now the lakehouse) — open formats, customer-owned storage, flexible compute, the same engine usable for SQL, ML, and streaming.

Architecture at a glance

BigQuery’s storage and scheduler hidden behind Google-managed SQL; Databricks’ storage and compute exposed as customer-tunable surfaces.

The structural difference shows up in every other section below. BigQuery hides the storage and the scheduler behind a single SQL interface; Databricks exposes both as customer-tunable surfaces. One trades flexibility for simplicity; the other trades simplicity for flexibility.

Storage

BigQuery storage uses Capacitor, Google’s proprietary columnar format, persisted to Colossus, the distributed file system that also backs Google Search and Gmail. Tables can be partitioned (by ingestion time, date column, or integer range) and clustered (on up to four columns). Time-travel is built in: any query can read a table as it existed up to seven days ago via the FOR SYSTEM_TIME AS OF clause. Storage is invisible to the customer — there is no S3 bucket to look at, no Parquet files to read with another tool. Reading the same data from outside BigQuery requires an export.

Storage cost: $0.020/GB-month for active tables (modified in the last 90 days), auto-transitioning to $0.010/GB-month long-term storage. Storage is rarely the bill that scares finance.

The trade-off is vendor lock-in at the storage layer. To move data to Snowflake or Databricks, the bytes have to be re-exported as Parquet/Avro/CSV and re-ingested on the other side.

Databricks storage is Delta Lake, an open-source table format built on Parquet. Delta tables live in the customer’s S3, ADLS, or GCS bucket, paid for directly to the cloud provider (about $0.023/GB-month on S3 standard tier). Delta layers an ACID transaction log over Parquet, providing time-travel by version, schema enforcement, MERGE statements, and concurrent writers without corrupting state. Tables can be partitioned and Z-ordered (a multi-dimensional clustering technique for skipping data during scans).

Because Parquet is open and Delta is open-source, the same files can be read by Snowflake (via Iceberg interop or external tables), Trino, Athena, DuckDB, Spark in any environment, or anything that speaks Parquet. This is the lakehouse promise: customer-owned storage, vendor-replaceable compute. If you leave Databricks, the data does not move — a different engine points at the same files.

The trade-off is operational responsibility. Customers handle bucket lifecycle policies, VACUUM operations to clean up old Delta versions, OPTIMIZE for file compaction (Delta tables develop small-file pathology over time), and partition pruning hygiene. None are difficult; all require attention.

Compute

BigQuery compute is built around slots. A slot is a unit of computational capacity — Google does not publish the exact specs, but a slot maps roughly to a virtual CPU plus a fixed memory and I/O allocation. Every query is decomposed into stages by Dremel; each stage parallelizes across the slots allocated to the query. Slot allocation is managed by Google’s scheduler. Two billing models:

On-demand — $6.25 per TiB of data scanned per query (US list price, mid-2026; varies by region). Slots are drawn from a shared on-demand pool of up to 2,000 slots per project. Simple to budget against, expensive at high scan volumes.

Editions (reservations) — Commit to a baseline number of slots, pay per slot-hour. Three tiers:

• Standard: $0.04/slot-hour
• Enterprise: $0.06/slot-hour
• Enterprise Plus: $0.10/slot-hour

Higher tiers add features: BigLake, Apache Iceberg native support, advanced encryption, vector search, geographic replication. Editions reservations support autoscaling — set a baseline plus a maximum, the scheduler scales between them.

Past about 100 slot-hours of monthly average compute, Editions usually wins on cost. The crossover depends on workload shape: scan-heavy queries with low slot-hour consumption favor on-demand; long-running transformations with high slot-hour consumption favor Editions.

Concurrency is handled by slot queueing. If demand spikes past reservation capacity, queries either wait or trigger autoscaling.

Databricks compute is cluster-based. The customer explicitly chooses cluster type, instance class, size, and lifecycle. Three cluster categories:

Job Clusters — Ephemeral. Spin up for a specific job, terminate when it finishes. Cheapest per DBU because there is no idle billing. Used for scheduled batch jobs and orchestrated pipelines.

All-Purpose Clusters — Persistent. Stay running until manually terminated or after an inactivity timeout. Most expensive per DBU because they are sized for peak and often idle. Used for interactive notebook work, ad-hoc exploration, ML experimentation.

SQL Warehouses — SQL-optimized clusters with two tiers. Pro requires manual cluster management at a lower per-DBU rate. Serverless has no cluster management to do, faster startup, higher per-DBU rate. Used for BI and analytics SQL workloads.

Billing is in DBUs (Databricks Units). One DBU corresponds to a fixed unit of compute time on a reference instance. DBU consumption depends on the cluster’s instance type, the number of workers, and the time the cluster is running. Rate varies by SKU and tier:

• Jobs Compute: ~$0.07-0.15 per DBU
• All-Purpose: ~$0.40-0.55 per DBU (Premium / Enterprise tiers; Databricks retired the Standard tier on AWS and GCP in October 2025)
• SQL Warehouse Pro: ~$0.22-0.55 per DBU
• SQL Warehouse Serverless: ~$0.70 per DBU

These are AWS list prices for mid-2026; Azure and GCP run within 10-15%.

Photon is Databricks’ vectorized C++ execution engine, enabled per-cluster. For SQL workloads it runs roughly 2-4x faster than open-source Spark. Photon costs 1.5x the base DBU rate but typically reduces total DBUs consumed because queries finish faster. For SQL-heavy workloads Photon usually wins on total cost. For Python-heavy Spark workloads (UDFs, custom ML loops) Photon adds cost without speeding things up.

Concurrency is per-cluster. SQL Warehouses autoscale the number of cluster replicas behind a single endpoint. Job clusters scale within their own configured size. There is no shared slot pool across the platform — every concurrent query is hosted on a specific cluster.

Multi-cloud: BigQuery is GCP-only. Databricks runs on AWS, Azure, and GCP with the same control plane. For organizations already standardized on AWS or Azure, this matters.

Pricing in one table

The fundamental cost-shift between the platforms: BigQuery bills are dominated by data scanned and slot-hours used; Databricks bills are dominated by cluster runtime measured in DBUs. The workload shape decides which is cheaper.

Performance characteristics

Where BigQuery is fast:

• Ad-hoc analytics over very large datasets. Dremel’s whole point was scanning petabytes in seconds.
• BI dashboard refreshes against standardized aggregations.
• Massively parallel queries — BigQuery can pull 2,000 slots without thinking, with no warm-up cost.
• Sparse workload patterns where compute provisioning would be wasteful. Nothing to size, nothing to warm.

Where BigQuery is slow:

• Streaming ingestion at low latency. The Storage Write API has improved, but real-time use cases below sub-second still struggle against the system’s batch-oriented design.
• ML training. BigQuery ML supports a constrained set of model types; serious ML moves to Vertex AI or off-platform.
• Iterative transformations. SQL-only execution constrains Spark-style transformation patterns with feedback loops.
• Single-row operations. BigQuery is columnar-optimized; OLTP patterns do not fit.

Where Databricks is fast:

• ETL and ELT pipelines at scale. Spark’s distributed compute model is purpose-built for transforming large datasets.
• ML training and inference. Native MLflow integration, GPU clusters, Mosaic AI.
• Streaming. Structured Streaming is a first-class API with both micro-batch and continuous processing modes.
• Lakehouse pattern execution. Medallion architecture, Delta Live Tables, CDC, multi-engine read patterns.
• Workloads that benefit from open formats. Anything Iceberg, Parquet, or multi-engine.

Where Databricks is slow:

• Very small ad-hoc queries. Cluster startup latency on SQL Warehouse Serverless has dropped to roughly 10-30 seconds in recent releases — the gap with BigQuery has narrowed but not closed.
• Workloads that do not justify cluster maintenance overhead.
• Spiky concurrent BI traffic. Per-cluster concurrency scaling is less elastic than BigQuery’s slot pool.

A worked cost projection across three workload shapes

Cost comparisons between BigQuery and Databricks are extremely sensitive to workload shape. Three scenarios using the same baseline data show why.

The numbers below are illustrative cost projections from mid-2026 vendor list pricing, not figures from a specific engagement. The $400K-$800K migration cost cited later is the typical mid-market range observed across published case studies and Replatform’s pricing model; your specific workload’s economics will vary.

Baseline workload: A mid-market data team running 20 TB of active warehouse data, 200 dbt models, 30 nightly batch jobs averaging 50 GB scan each, 5 BI dashboards refreshing 10 GB each on the hour during business hours, and 200 ad-hoc analyst queries per day averaging 5 GB scan each.

Scenario A — BI-heavy ad-hoc analytics

Workload pattern: lots of small interactive queries spread across business hours, low batch volume. Total monthly scan around 30 TB. Average concurrent slots required, roughly 150.

BigQuery (Editions Enterprise with autoscaler):

Compute:  150 baseline slots × 11 business hours × 22 weekdays × $0.06
        = $2,178
Storage:  20 TB × $20/TB-month
        = $400
Total   ≈ $2,580/month

Databricks (SQL Warehouse Serverless, Medium):

Compute:  ~20 DBU/hr × 10 hours/day × 22 weekdays × $0.70/DBU
        = $3,080
Storage:  20 TB × $23/TB-month (S3 standard)
        = $460
Total   ≈ $3,540/month

Verdict: BigQuery wins by ~25%. The slot pool’s elasticity is hard to beat for spiky BI traffic with no provisioning lead time.

Scenario B — ETL-heavy predictable batch

Workload pattern: 30 nightly batch jobs scanning 50 GB each, complex transformations with joins and window functions. Light BI workload during the day. Total monthly scan around 80 TB.

BigQuery (Editions Enterprise with autoscaler):

Compute:  Batch  500 slots × 4 hrs nightly × 30 days × $0.06  = $3,600
          BI     ~50 slots × 11 hrs × 22 days × $0.06         = $726
Storage:  20 TB × $20/TB-month                                = $400
Total                                                          ≈ $4,726/month

Databricks (Job Clusters with Photon for batch, SQL Warehouse for BI):

Batch:    30 jobs × 30 days × 8 DBU/hr × 0.5 hrs × $0.15 Photon
        = $540
BI:       ~10 DBU/hr × 10 hrs × 22 days × $0.70 Serverless
        = $1,540
Storage:  20 TB × $23/TB-month
        = $460
Total                                                          ≈ $2,540/month

Verdict: Databricks wins by ~46%. Job clusters are very cheap per DBU when they only run during the work and terminate on completion. Photon acceleration on predictable transformations compounds the advantage.

Scenario C — ML and streaming on top of the lakehouse

Workload pattern: continuous Kafka ingestion, real-time aggregation, ML model training on a weekly cadence, plus the batch and BI workloads from Scenario B.

BigQuery (with Vertex AI for ML, Storage Write API for streaming):

Warehouse compute  (Scenario B level)                          ≈ $4,326
Streaming via Storage Write API + Dataflow workarounds         ≈ $1,500
Vertex AI weekly training                                       ≈ $2,000
Storage                                                         = $400
Total                                                          ≈ $8,226/month

Databricks (Structured Streaming + Job Clusters + MLflow):

Streaming  2 DBU/hr × 24 × 30 × $0.15                          = $216
Batch + BI (Scenario B)                                         ≈ $2,080
ML  100 DBU/wk × $0.40 All-Purpose × 4                         ≈ $160
GPU training compute (when applicable, weekly cluster)         ≈ $400
Storage                                                         = $460
Total                                                          ≈ $3,316/month

Verdict: Databricks wins by ~60%. When the workload extends beyond SQL analytics into streaming and ML, the architectural fit drives the cost.

Workload shape determines which platform is cheaper. Vendor pitch decks almost always show Scenario B or C. Audit your actual query mix before comparing list prices.

The pattern across the three scenarios: workload shape determines which platform is cheaper, and the gap can be 25-60% in either direction. Vendor pitch decks almost always show a Scenario B or C; the real-world workload is often closer to Scenario A. The “we saved 40%” claim that circulates at CFO dinners is almost never reproducible without surfacing which scenario the source team was actually running.

Migration considerations

If the comparison above suggests moving from BigQuery to Databricks (or the reverse), three things shape the migration math:

SQL dialect translation. BigQuery and Databricks SQL diverge on function signatures (DATE_TRUNC argument order, TIMESTAMP_DIFF vs DATEDIFF, table-qualified column reference rules inside WITH clauses), date/time semantics, type coercion behavior, and INFORMATION_SCHEMA shape. Across a 200-model dbt project, expect 8-15% of queries to break on the first translation pass. Most failures are mechanical and translatable with a small set of regex passes; a smaller subset require structural rewrites because the underlying semantics differ.

Storage format change. Moving from Capacitor-on-Colossus to Delta-on-S3 is a one-way architectural decision. The data has to be either re-ingested from the source (Kafka, Fivetran, Airbyte, application database CDC) or exported from BigQuery via the Storage Read API, written to Parquet, and registered as Delta tables. Straightforward for most table shapes; non-trivial for partition-heavy tables or tables that rely on BigQuery’s time-travel for downstream consumers.

Compute model change. Slot-based pricing becomes cluster-based pricing. Workloads that were effectively free under BigQuery’s elastic slot pool (small ad-hoc queries between dashboard refreshes) now need to be routed to a SQL Warehouse or an All-Purpose cluster. Workloads that were expensive under BigQuery (multi-hour transformations) move to cheaper Job Clusters. The cost shape changes; re-routing workloads by type is non-trivial but mechanical.

Total project cost. A typical mid-market BigQuery → Databricks migration covering 200-300 dbt models, a Looker layer, and a Fivetran or Airbyte connector graph runs $400K to $800K all-in over 6-9 months. Engineering hours, parallel-run cost during cutover, validation harness build-out, and BI re-pointing dominate the bill. Realistic payback timelines land at 2-4 years before risk discount; 4-5 years after applying typical migration overrun (30-50% on schedule, 20-40% on cost).

The 40% savings claim that motivates many migrations usually combines three things: a pricing-model fix (on-demand to Editions, or to Databricks Job Clusters from All-Purpose) that would have happened anyway, a source-side tech-debt cleanup performed during translation that would have happened anyway, and an apples-to-oranges SKU comparison. The migration carries the cost of those changes without being the cause of the savings.

The honest verdict

For teams already on BigQuery considering a move, the conditions where the migration math works are specific:

• A real lakehouse architecture pattern, not “we want a lakehouse” framing
• Workloads that include streaming, ML training at scale, or open-format storage requirements (Iceberg, Delta read by other engines)
• A bursty-concurrent compute pattern with deep idle valleys, where BigQuery’s slot reservation model overpays in the valley
• Six to nine months of engineering capacity and a CFO comfortable with a 4-year payback

For new builds, the decision is workload-driven:

• BI-heavy plus ad-hoc analytics: BigQuery
• ETL-heavy plus ML plus streaming: Databricks
• Hybrid (the most common pattern in practice): both — BigQuery for analytics, Databricks for ML and streaming, federated through Unity Catalog external tables or BigQuery Omni for cross-cloud read

For teams unhappy with their bill, the bill is almost never the platform. The bill is the workload pattern, the pricing-model choice, and the technical debt accumulated in the source. Tune those first. The platform comparison only matters once the workload is clean.