LLMs and SQL: Syntax and Semantics

Query Equivalence showed that two queries can look identical yet diverge on some data: truly equivalent only if they agree for any input, not just today's. An LLM hands you exactly that problem. It returns a query that runs; you decide whether it means what you meant, not just whether it matches the rows in front of you.

Syntax Is Solved, Semantics Isn't

The BIRD benchmark evaluates LLMs on text-to-SQL:

The low scorers are blind single-shot prompts. The high scorers let the model run the query and fix it. So the gap is the method, not the model. A query that runs is still not a query that is right. The model never sees your data, so it guesses your domain logic unless you state it.

Browse the live benchmark below; scroll through it to see how involved a real question and its gold SQL get:

Strategy: Give the Model an Execution Environment

Precision Has to Live Somewhere

You still have to write the precision down. The only choice is where: in the prompt, or in the SQL you write yourself. A vague prompt writes none of it, so the model picks one of two readings and you cannot tell which.

The Two Readings, in SQL

Both run. They return different answers.

-- Find users who listened to non-Taylor songs (or no songs)
SELECT u.user_id, u.name
FROM Users u
LEFT JOIN Listens l ON u.user_id = l.user_id
LEFT JOIN Songs s ON l.song_id = s.song_id
WHERE s.artist != 'Taylor Swift' OR s.artist IS NULL
GROUP BY u.user_id, u.name

-- Find users who never listened to any Taylor Swift song
SELECT u.user_id, u.name
FROM Users u
WHERE u.user_id NOT IN (
    SELECT DISTINCT l.user_id
    FROM Listens l
    JOIN Songs s ON l.song_id = s.song_id
    WHERE s.artist = 'Taylor Swift'
)

The [Definitions] line chooses Query B's reading over Query A's. Write that precision into the prompt, or into the SQL yourself. Either way, you write it.

Verify with a Debug Table

The spec says which query is right. A debug table proves it. Execution catches crashes; it does not catch runs-but-wrong. Build a debug table for the query: trace it the way the engine runs it, one clause at a time, writing the rows after each step. Do it for both queries and they diverge. Query A returns all four users; Query B returns only Pluto. The two tables show the semantic gap directly. Query A keeps Mickey, Minnie, and Daffy because each has a non-Taylor listen, and keeps Pluto only because OR s.artist IS NULL rescues the row the LEFT JOIN NULL-padded; Query B drops anyone who heard Taylor even once, so only Pluto survives. Same English, two debug tables, two answers. The spec decides which is right.

Both queries, traced step by step:

Takeaways

1. Syntax is solved; semantics is the job. A query that runs is not a query that is right.

2. Execution turns a guess into a checked answer, but only checks what your spec defines. Crashes are free; runs-but-wrong needs a pinned definition and an expected result.

3. Understanding SQL and debugging is the key skill, more than ever. You can only pin the definition, decide the edge cases, and trace the debug table if you know how the engine runs the query. The LLM removes the boilerplate, not the need to understand it.

Next

Intermediate SQL Quiz → That completes the intermediate-SQL toolkit: CTEs, window functions, query equivalence, and verifying machine-written SQL. The quiz checks all of it.