I have spent years watching organizations waste thousands of hours on vendor security assessments that produce nothing but noise.
They send out hundred-question questionnaires. They demand SOC 2 reports. They ask for ISO certifications. They request NIST maturity scores. And at the end of all that paperwork, they still have no idea if the vendor can actually handle a real security incident.
Let me tell you something directly: if you are not assessing your vendor’s incident management process with serious scrutiny, you are not assessing security at all. You are just collecting paper.
The Paper Thing!
Here is the uncomfortable truth that nobody in the compliance industry wants to admit: a SOC 2 report tells you that a vendor paid an auditor to check some boxes on a specific day. It does not tell you what happens when something actually goes wrong.
I have seen vendors with perfect compliance scores completely fall apart during a real incident. Their policies looked great. Their certifications were current. But when the moment came to actually respond, they had no process, no coordination, and no clue.
The Time Crunch
Here is the thing: most of us do not have the luxury of spending weeks on vendor assessments. You have business pressure. You have deadlines. You have a project that needs to move forward. You cannot always wait for a full security audit.
So what do you do?
You focus on the one thing that actually matters. You ignore all the noise about risk scores and vulnerability counts and change management processes. You stop pretending that you can audit everything.
You ask one question and you demand evidence…
The Million Dollar Question
If I had only one question to ask a vendor, it would be this:
“Show me the evidence from your last tabletop exercise. Not the policy. Not the PowerPoint. Show me the actual artifacts from a real exercise…the scenario, the timeline, the decisions made, the gaps identified, and the follow-up actions.”
That is it. That is the question.
Do not ask vague things like “How do you secure my data?” That tells you nothing. Any vendor can recite a marketing answer to that. Every single one of them will tell you they take security seriously. They all have “robust security measures.” They all “prioritize protection.”
But when you ask for tabletop exercise artifacts, the response is very different.
Why This Question Works
Think of it this way: your vendor’s incident management process is like the emergency evacuation plan for a building.
You can have a beautifully written plan posted on every wall. You can have certificates saying you passed inspections. You can have meetings where you talk about safety. But none of that matters if, when the fire actually happens, nobody knows which door to use and the alarms don’t work.
A tabletop exercise is the fire drill. It is where the plan meets reality.
When you review the artifacts from a tabletop exercise, you see:
- How the vendor actually makes decisions under pressure
- Whether roles and responsibilities are clear or confused
- How communication flows (or doesn’t flow) between teams
- Whether they have identified gaps and fixed them
- Whether the exercise was a checkbox exercise or a genuine stress test
You will see more about their real security posture in those artifacts than you will in a hundred pages of compliance documentation.
What You Should Expect
When you ask for tabletop exercise evidence, there are three possible responses:
The Good Response: They send you artifacts within a day or two. The artifacts are detailed. They show realistic scenarios. They include decision logs, communication records, gap analyses, and action items with owners and deadlines. You can see a clear improvement from one exercise to the next.
The Yellow Flag Response: They hesitate. They ask for more time. They say they need to check with legal. They eventually send you something generic that looks more like a marketing document than an actual exercise. The artifacts are vague. There is no real detail. This tells you their incident management process is likely weak or nonexistent.
The Red Flag Response: They cannot produce anything. They make excuses. They claim the exercises are confidential. They promise to provide something “in a few weeks.” This is not a vendor you should trust. If they cannot demonstrate how they handle an incident in a simulated environment, how will they handle your actual data when something real happens?
“A good mechanic does not just read the diagnostic. He listens to the engine.”
Your vendor’s response to this question is the sound of their engine. Listen carefully!
The One-Week Rule
I will be direct with you: I would not give a vendor more than one business week to deliver these artifacts.
One week is generous. If you have a real incident management process that you actually practice, the evidence is ready. You know where it is. You can pull it together quickly. It is not buried in a filing cabinet. It is not something you need to fabricate.
If a vendor needs more than a week, they are either fabricating something from scratch or they have just realized they do not have a process for collecting evidence. Either way, that tells you everything you need to know.
The best vendors will deliver within 24 to 48 hours. They will understand why you are asking. They will be proud to show you how they handle incidents.
The Deeper Truth
Here is what the tabletop exercise evidence reveals that nothing else can: it reveals how the vendor actually thinks about security:
Incident management is where all your other controls come together. It is the front line. It is where vulnerability management, access control, monitoring, and everything else converge when something goes wrong.
If a vendor has a solid incident management process, they probably have decent controls across the board. Because you cannot have a functional incident response capability without understanding your assets, your risks, and your dependencies.
But if a vendor has a weak incident management process, or even worse, no process at all, they are really dangerous. They will not know when they have been breached. They will not respond effectively. They will not tell you in time.
“Buying a self-driving car doesn’t mean you can close your eyes on the highway.”
Buying a vendor’s product does not mean you can ignore what happens when they get compromised.
Legal Agreements Are Not Enough
Now, I know some of you are thinking: “But we have a contract. We have legal protections. We have insurance.”
Let me stop you right there. Legal agreements are important. They are necessary. They help you after the fact. But they do not prevent incidents. They do not protect your data in real time. They do not help you when your operations are disrupted because your vendor got hacked and cannot restore service.
A contract is not a security control. It is a remedy, and only after you have already suffered damage.
Your legal team handles the agreements. You handle the security assessment. Those are different jobs. Do not confuse th
The Follow-Up Questions
This one question is not the end of the assessment. It is the beginning.
Once you have the tabletop exercise artifacts, you can ask intelligent follow-up questions:
- “What were the specific gaps identified in this exercise?”
- “How have you addressed them?”
- “Show me the evidence of those fixes being implemented.”
- “How did you communicate this incident to your customers in the scenario?”
- “What would you do differently in a real incident versus this exercise?”
The quality of the vendor’s answers to these follow-ups will tell you whether they take security seriously or are just going through motions.
One Last Warning
Do not make the mistake of thinking that because a vendor is large or famous, they are secure. Size does not equal security! Reputation does not equal incident readiness!
I have seen large, established vendors with terrible incident management. And I have seen smaller vendors with exceptional processes who genuinely cared about getting it right.
The evidence is what matters. Not the brand name. Not the marketing. Not the compliance certificates.
Ask the question. Demand the evidence. Trust what they show you. Your organization’s safety may depend on it.