Can ChatGPT or Claude Analyse Your Company's Data? Where General Models Stop
You can upload a spreadsheet to a general AI model and get charts and conclusions within a minute. For a quick look, that is useful. For a decision that moves budget or people, it is not enough. This post explains why, and what a serious analysis does differently.
What general models are good at
General-purpose models are remarkable tools. Hand one a file and it will summarise it, draw a chart, write the code for a regression and explain the result in plain language. For exploring a dataset or getting a first impression, that is hard to beat. We use these models ourselves, every day.
So the question is fair: if a model can do all that, why would a company pay anyone for data analysis?
An analysis is not the right analysis
Because there is a difference between an analysis and the right analysis for your data.
A general model is built to give a reasonable answer to anything. Ask it whether more sales visits lead to more sales, and it will usually run one regression over the whole table and report the slope. The method is plausible. The answer may be the opposite of the truth.
Here is how. Suppose three regions each show the same pattern: accounts that get more visits buy more. But the region with the fewest visits per account happens to have the largest accounts. Pool everything together and the trend reverses: more visits appear to go with lower sales. Statisticians call this Simpson's paradox, and it is not rare in business data. A model that takes the file at face value will report the pooled result, with a tidy chart, and move on.
Nothing in the output tells you which reading is correct. Knowing that you had to split by region requires knowing how the data came to be.
Four things that get skipped
Assumptions. Every method assumes something: that observations are independent, that there is no seasonality, that missing values are missing at random. When the assumption fails, the result is wrong in ways that do not show on the chart. Checking takes work, and a general model rarely does it unless asked.
Uncertainty. "Demand in March will be 1,320 units" sounds precise. A real forecast is a range, and how wide that range is should change your decision. Stock exactly 1,320 and you will run short about half the time.
Validation. The honest test of a model is whether it would have been right in the past. That means holding back data, forecasting it, and comparing. Without that step, you have a story, not evidence.
Context. Data is produced by a business, not by nature. Reps visit struggling accounts more often. Discounts go to customers who were about to leave. A price changed in the middle of the period. None of this is in the file, and all of it changes what the numbers mean.
Confident is not the same as correct
The uncomfortable part is that a wrong analysis and a right one look the same. Both come with fluent explanations, clean charts and a confident tone. A general model does not hedge more when it is on thin ice.
For a casual question this does not matter. For a decision about where next year's budget goes, the cost of a confident wrong answer shows up months later, when it is expensive to undo and hard to trace back.
What we do differently
Our background is in mathematics, statistics and machine learning research. We do not pass your data to a chatbot and forward what it says. The work looks like this:
- Start from the decision. What are you trying to decide, and what would change your mind?
- Audit the data. What can it answer, what can it not, and how was it produced?
- Compare methods. Several approaches, tested against each other, rather than the first plausible one.
- Test against reality. Every model is checked on data it has not seen.
- Report the uncertainty. A range and how far it can be relied on, not a single number.
- Deliver something usable. A system your team looks at and acts on, not a slide deck.
We use general models along the way, where they help: writing code faster, summarising, drafting. The judgment about what is true in your data stays with us.
When a general model is enough
Often. If the question is exploratory, the stakes are low and the decision is easy to reverse, a quick answer from a general model is the sensible choice.
It stops being enough when the answer will move money, stock or people: how to split a budget, how much to order, which customers to prioritise, which accounts are at risk. Those are the questions where being roughly right for the wrong reasons is the most expensive outcome.
Final thoughts
General models have made analysis look easy, and for simple questions it now is. The hard part was never producing a chart. It was knowing which chart is true.
If you have a decision that depends on your data, see how we work or tell us what you are trying to decide. We will tell you honestly what your data can and cannot answer.
See it on your own data
Tell us about your team and we'll show you what this would look like for you.
Book a Demo

