Their first example [1] is a complete non-sequitur and I’m trying to comprehend how this passed human review and must assume it’s AI, which doesn’t bode well for the supposed usefulness of their system.
[1] https://images.ctfassets.net/kftzwdyauwt9/2tMhL5Www2vA6I62DV...
“What was ChatGPT Image Gen logged-in DAU for the last 30 days? Worked for 1m 22s > ChatGPT WAU on October 6, 2025 (rounded to nearest 100M): = 800M ChatGPT WAU on the last DevDay 2023 (Nov 6, 2023; rounded to nearest 100M): = 100M Mini comparison (using the rounded figures only): • Change: = +700M WAU • Multiple: = 8x higher on 2025-10-06 vs 2023-11-06 (WAU here is the standard ChatGPT WAU as-of the reporting date; I'm only sharing the values rounded to the nearest 100M, per your request.)”
You're right, they have 2 versions, desktop and mobile:
Desktop, correct prompt: https://images.ctfassets.net/kftzwdyauwt9/5EoAd2fIvVRf8V51LN...
Mobile, wrong prompt: https://images.ctfassets.net/kftzwdyauwt9/2tMhL5Www2vA6I62DV...
Trust & explainability is the biggest issue here.
We've been building natural language analytics at Veezoo (https://www.veezoo.com/) for 10 years, and what we find is that straight Text-to-SQL doesn't scale. If AI writes SQL directly, you're building on a probabilistic foundation. When a CFO asks for revenue the number can't just be correct 99% of times. Also you can't get the CFO to read SQL to verify.
We're solving that with an abstraction layer (Knowledge Graph) in between. AI translates natural language to a semantic query language, which then compiles to SQL deterministically.
At the same time you can translate the semantic query deterministically back into an explanation for the business user, so they can easily verify if the result matches their intent.
Business logic lives in the Knowledge Graph and the compiler ensures every query adheres to it 100%, every time. No AI is involved in that step.
Veezoo Architecture: https://docs.veezoo.com/veezoo/architecture-overview
Thanks for sharing the links, the architectural overview is very insightful.
I'm curious how this approach manages cardinality explosion? Also, how do you handle cases where a user asks for data that requires running multiple queries, specifically where each query depends on the results of the previous one?
Don't you still need to unit test and version control the SQL artefact that is produced? You need to be able to see which query was used on which date and how it was validated.
(Prompts need to be version controlled too, of course)
Yes, every SQL query Veezoo runs is logged and visible to admins.
The fundamental artifact is VQL (Veezoo Query Language), which queries against a Knowledge Graph containing your business data model, things like your "Revenue" measure.
A query might look like this:
var order from kb.Order
date_in(order.Order_Date, date("#today"))
var retRevenue = kb.Order.Revenue(order)
select(retRevenue)
If the business decides to change how revenue is computed, the VQL stays valid but compiles to different SQL. At the same time Veezoo can test that with your knowledge graph change that you are not breaking anyones dashboard and even apply evolutions if needed
VQL: https://docs.veezoo.com/vkl/kb-layer/vql/
Evolutions: https://docs.veezoo.com/vkl/evolutions/
The Knowledge Graph itself is version controlled, so the data team can trace every change.
In my opinion, data and documents are the real AI benefit, or threat, to developer jobs.
Specifically, how good a company's data is will determine how effectively it can leverage AI in the future. The public data is pretty much mined to exhaustion, and the next big data source will be in-house documentation, code repos, data lakes, etc. If you work for a company where that's been built, maintained, and organised then the effectiveness of AI is going to be mind-blowing. Companies that have maintained good docs be able to build new things, maintain old things, and migrate things to cheaper modern stacks easily. That will lead to being able to move fast and deploy new AI-driven services easily and cheaply. Revenue will follow.
Conversely, at companies where documentation and code organisation have been historically poor, AI will struggle. Leaders will see it as a benefit, and be baffled at why their company can't realise the value of it. They'll quickly blame developers for not being able to use it, and that'll lead to people's growth stagnating or possibly layoffs. Eventually competitors will eat the company's lunch because they'll just be able to move on opportunities much faster.
I've resolved that in any future job hunt I'm going to make asking about docs, data, and repos a priority...
Trust is the hardest part to scale here.
We're building something similar and found that no matter how good the agent loop is, you still need "canonical metrics" that are human-curated. Otherwise non-technical users (marketing, product managers) are playing a guessing game with high-stakes decisions, and they can't verify the SQL themselves.
Our approach: 1. We control the data pipeline and work with a discrete set of data sources where schemas are consistent across customers 2. We benchmark extensively so the agent uses a verified metric when one exists, falls back to raw SQL when it doesn't, and captures those gaps as "opportunities" for human review
Over time, most queries hit canonical metrics. The agent becomes less of a SQL generator and more of a smart router from user intent -> verified metric.
The "Moving fast without breaking trust" section resonates, their eval system with golden SQL is essentially the same insight: you need ground truth to catch drift.
Wrote about the tradeoffs here: https://www.graphed.com/blog/update-2
Yes, I’ve been working on this and you need a clear semantic layer.
If there are multiple paths or perceived paths to an answer, you’ll get two answers. Plus, LLMs like to create pointless “xyz_index” metrics that are not standard, clear, or useful. Yet i see users just go “that sounds right” and run with it.
Absolutely. We make it obvious to the user when a query/chart is using a non standard metric and have a fast SLA on finding/building the right metric.
It only works because all of the data looks the same between customers (we manage ad platform, email, funnel data).
So if we make an “email open rate” metric, that’ll amortize to other customers.
Piling on to the vendor pitches here:
We give you all of this in 5 minutes at https://www.definite.app/.
And I mean all of it. You don't need Spark or Snowflake. We give you a datalake, pipelines to get data in, semantic layer and a data agent in one app.
The agent is kind of the easy / fun part. Getting the data infrastructure right so the agent is useful is the hard part.
i.e. if the agent has low agency (e.g. can only write SQL in Snowflake) and can't add a new data source or update transformation logic, it's not going to be terribly effective. Our agent can obviously write SQL, but it can also manage the underlying infra, which has been a huge unlock for us.
Given my personal experience with various BI systems I think an AI agent like this is the perfect use case. These systems are operating on multiple layers of being wrong as is - layer 1 being your query is likely wrong, layer 2 being how you interpret the data is likely wrong.
Mix them together and you’re already deep in make believe land, so letting AI take over step 1 seems like a perfect fit.
I was hoping to read this article and be surprised by how OpenAI was able to solve the reliability problem, but alas.
Don't forget -
layer 0 - how you stored the data was wrong.
layer -1 - your understanding of modeling the behavior was wrong before you ever created a table.
layer -2 - your fundamental business process was wrong and all your information is lies.
This is why instead of a central source of truth I call it the central source of lies.
Very, very good stuff here. I think a possible missing piece is how to explain how the results were computed. Here it seems they're relying on the fact that users are somewhat technical (that's great for OpenAI -- it's an internal agent after all) and can at least read SQL, but it's an interesting design problem how you would structure the interaction with nontechnical users.
When working on data systems you quickly realize that often how the question was answered (how the metric is defined, what data was taken into account and so on) is just as important as the answer.
At Amplitude we built Moda which is super similar to this.
Our chief engineer Wade gave an awesome demo to Claire Vo some months back here: https://www.youtube.com/watch?v=9Q9Yrj2RTkg
I use this basically every day asking all sorts of questions
data problems are not tech problems but rather org problems
So true. In my career (anecdotally), I’ve never encountered a data problem where the answer was ‘you didn’t choose this tech/language/product over another.’ It always comes down to decisions of governance and ownership. It’s Conway’s Law all the way down.
I'm more interested in Kimi's In-House Data Agent
