How to Use SQL and Python Together in a Single Microsoft Fabric or Databricks Notebook

One of the greatest strengths of Microsoft Fabric and Databricks notebooks is their ability to mix multiple languages—most commonly SQL and Python—in a single, seamless workflow. This unlocks a powerful pattern: use SQL for what it’s best at (querying, filtering, aggregating), and use Python for everything else (transformations, machine learning, visualization, automation).

If you've ever wondered “How exactly do I combine SQL and Python in one notebook?”, this guide walks you through the full story.

Why Mix SQL and Python?

SQL and Python play very different but highly complementary roles:

SQL shines when you need:

• Fast filtering and joins

• Aggregations and summarisation

• OLAP-style queries

• Logical, declarative transformations

Python excels when you need:

• Complex data processing

• Machine learning and AI

• Visualization

• Programmatic logic

Thanks to shared Spark runtimes in both Fabric and Databricks, these two languages can operate on the same data session, making them work together effortlessly.

How to Switch Between SQL and Python in One Notebook

In both Fabric and Databricks, you have two easy methods.

Method 1: Use the Cell Language Picker (UI)

At the top-left of every notebook cell, there’s a language selector.

You can choose:

• Python

• SQL

• PySpark

• Markdown

• Scala (Databricks)

This is the cleanest and most readable way to build multi-language notebooks.

Method 2: Use Magic Commands (%%language)

Instead of changing the cell’s language in the UI, you can override it with a command at the top of the cell.

Run SQL

%%sql

SELECT *

FROM sales

WHERE Amount > 500;

Run Python (PySpark)

%%pyspark

df = spark.read.table("sales")

display(df)

If you already use the language picker, magic commands aren’t required—just another option.

Sharing Data Between SQL and Python

One of the best features of Fabric and Databricks notebooks is that SQL and Python share the same Spark session. That means anything you create or query in one language can immediately be used in the other.

Let’s look at the patterns.

Pattern 1: Python → SQL via Temp Views

Python cell

df = spark.read.table("lakehouse.sales.transactions")

df.createOrReplaceTempView("sales_view")

SQL cell

%%sql

SELECT Region, SUM(Amount) AS TotalAmount

FROM sales_view

GROUP BY Region;

Pattern 2: SQL → Python

SQL cell

%%sql

CREATE OR REPLACE TEMP VIEW high_sales AS

SELECT *

FROM lakehouse.sales.transactions

WHERE Amount > 1000;

Python cell

df = spark.sql("SELECT * FROM high_sales")

display(df)

Putting It Together: An End-to-End Example

Below is a realistic example combining both languages for a full analytics workflow.

Step 1: Load Data in Python

sales = spark.read.table("lakehouse.sales.transactions")

sales.createOrReplaceTempView("sales")

Step 2: Transform with SQL

%%sql

CREATE OR REPLACE TEMP VIEW sales_by_category AS

SELECT category, SUM(amount) AS total

FROM sales

GROUP BY category;

Step 3: Analyse & Visualize in Python

df = spark.sql("SELECT * FROM sales_by_category ORDER BY total DESC")

pdf = df.toPandas()

pdf.plot(kind="bar", x="category", y="total", title="Sales by Category")

Step 4: Apply a Machine Learning Model

from sklearn.linear_model import LinearRegression

model = LinearRegression()

model.fit(pdf[["total"]], pdf["total"])  # demo purpose only

print("Model trained successfully!")

Now you’ve gone from raw data → SQL cleaned data → Python visualisation → Python ML all inside a single notebook.

Best Practices for Mixing SQL and Python

Let SQL handle table queries and aggregations

Don't overuse Python for operations that SQL can do more efficiently.

Use Python for logic-heavy transformations

Loops, ML, visualization, and business rules belong in Python.

Stick to temporary views for sharing data

They make your SQL and Python code clean and consistent.

Use Markdown cells to narrate the notebook

Good documentation turns notebooks into readable stories.

Keep cells short and purposeful

Avoid huge monolithic cells—break up your workflow logically.

Final Thoughts

Being able to use SQL and Python in one notebook is a superpower for modern data engineering and analytics. Whether you're building transformation pipelines, exploring datasets, or training ML models, mixed-language notebooks in both Microsoft Fabric and Databricks help teams move faster and keep everything in the same place.