Flashpoint.AIFlashpoint.AIblog
Back

All Models Are Wrong — Part 3

Every question is worth asking now

The backlog, c. 1922. The U.S. Dead Letter Office, Washington, D.C., where undeliverable mail piled up.
The backlog, c. 1922. The U.S. Dead Letter Office, Washington, D.C.

“Essentially, all models are wrong, but some are useful.” — George E. P. Box

Every insights team has a backlog that never gets touched. Not because the questions aren't worth answering — because the process of answering them isn't worth the effort. By the time you've scoped the project, negotiated the vendor contract, cleared procurement, and waited for fieldwork, the moment has passed. So the question gets dropped, and the team moves on to whatever is urgent enough to justify the overhead.

That is the problem synthetic research actually solves. Not “faster answers to the questions you’re already asking.” More answers, period — including the ones that currently die before anyone picks them up.

It is worth saying what synthetic research is not, because the way some vendors position it — as a straight replacement for human fieldwork — sets researchers up for a credibility problem they won’t see coming. A synthetic respondent is excellent at returning what its training data already holds. Where it breaks down is on the novel: a new concept, an unfamiliar category, a behavior that hasn’t been measured before. In those cases the model generates a plausible-sounding answer with no indication that it’s guessing. As we argued earlier in this series, no model release fixes this, because the limit is mathematical rather than technical. For exploratory work in well-covered territory, that’s manageable. For the questions clients are actually paying to answer, it’s a real risk.

Synthetic research is a starting point, not a shortcut. The teams that get the most out of it treat it as a workflow tool rather than a wholesale replacement, and know exactly when to hand off to real humans.

In practice, the workflow looks like this: an AI-assisted brief generates a first-draft survey in minutes, a researcher reviews and refines it, and a synthetic panel grounded in census data returns an instant directional read. If the signal is strong enough, that's the answer. If validation is needed, the same survey is sent to real respondents through Dynata, Cint, or Prolific — no rebuild, no new procurement cycle. The path from question to early signal to validated answer compresses from weeks to days.

The keyword there is “if.” Knowing when synthetic results are reliable — and when they aren’t — is what makes that workflow defensible. That judgment is what Flashpoint.AI scores. Every run returns a Panel Calibration Score, which measures how well the panel reflects the target audience, and a Response Fit Score, which tells you how much to trust the results for the specific questions being asked. Those scores both flag risk and tell you where to aim real fieldwork budget. The synthetic read sharpens your brief and focuses your fieldwork; the human respondents go exactly where they’re needed.

The best teams use AI to handle the execution layer — drafting instruments, fielding quick reads, flagging where to invest — while researchers keep judgment, interpretation, and client relationships firmly in hand. AI-assisted, human-kept.

For agencies, that combination changes two things at once. It changes what “fast” means: not speed for its own sake, but the ability to give clients a methodologically sound directional read almost immediately, and then spend human respondent budget with precision rather than assumption. And it changes how much ground a team can cover. The questions that used to get cut — too small to scope, too fast to field, too early in the process to justify a full project — now have an answer. The more ground an agency can cover with data and insight, the harder it is to replace.

The vendors selling synthetic as an all-in replacement are solving for impressive demos. The agencies that win will be the ones who figured out that synth and real aren't in competition — and built their workflow around that.

Next in the series: how the machine actually works, end to end.