Skip to content

Free tool · no signup · about 5 minutes

SaaS Security Assessment

Score one SaaS vendor, or all the SaaS you run, on 25 controls in 10 areas. You get a risk rating, the gaps and the questions to send the vendor.

Assessing
Answered 0/25Score so far 0/100

Single sign-on

SAML or OIDC through your identity provider.

  • The app supports SSO with SAML 2.0 or OpenID Connect on the plan we buy (or will buy).

    SSO puts sign-in, MFA and leaver handling under your identity provider instead of one more password store.

  • Password sign-in can be turned off for SSO users, with a small, controlled set of break-glass accounts.

    If local passwords keep working, SSO and its MFA can simply be bypassed.

MFA enforcement

Required, not optional.

  • MFA can be enforced for every user by an admin setting (directly or through SSO), not left to each user.

    Optional MFA leaves the accounts least likely to turn it on unprotected.

  • Phishing-resistant MFA (security keys or passkeys) is supported, at least for admins.

    One-time codes and push approvals can be phished or fatigued; admin accounts are the prize.

Provisioning (SCIM)

Leavers lose access automatically.

  • Users are provisioned and deprovisioned automatically with SCIM from our identity provider.

    Without automatic deprovisioning, leavers keep access until someone remembers to remove them.

  • Deprovisioning ends active sessions and revokes the user’s API tokens and personal access tokens.

    A disabled account with a live session or token still has access.

Admin roles and audit logs

Least privilege and a usable trail.

  • There are granular admin roles, so we can give people only the admin rights they need.

    If the only choice is full admin, every helper becomes a full admin.

  • Audit logs record sign-ins, admin changes, permission changes and data exports.

    In an investigation, the first questions are who signed in, what they changed and what they took.

  • Audit log retention is stated and long enough for our investigations and obligations.

    Incidents are often found weeks or months later; short retention erases the evidence.

  • Audit logs can be exported or streamed by API to our SIEM or log store.

    Logs you can only see in the vendor’s console are hard to alert on or keep.

Data residency and encryption

Where data lives and who holds the keys.

  • We can choose, or at least know, the region where our data is stored and processed.

    Residency affects your legal obligations and your customers’ contracts.

  • Data is encrypted in transit and at rest, and the vendor documents how.

    Encryption is a baseline expectation; documented detail shows it is real.

  • Customer-managed keys (BYOK) are available if we need them.

    Holding the keys lets you cut off access to sensitive data yourself; it matters most for regulated data.

Backup and export

Getting your data back, and out.

  • We can export all of our data in a usable, documented format, at any time.

    An export is your exit plan and your last backup if the vendor fails.

  • The vendor states how it backs up our data and how we can restore deleted or overwritten data.

    Ransomware, a bad sync or a malicious admin can delete data inside the app itself.

  • The contract says how and when our data is deleted when we leave.

    Data that outlives the contract is still your risk.

Integrations and OAuth grants

Third-party apps with access to your data.

  • Admins can control which third-party apps and OAuth grants users can approve.

    Connected apps with broad scopes can read your data through the front door, without a password.

  • We can list every connected app, OAuth grant and API token with its scopes, and revoke them.

    You cannot remove access you cannot see.

Incident notification terms

What the contract promises when things go wrong.

  • The contract commits the vendor to notify us of a security incident affecting our data within a stated time.

    Your own notification deadlines start when you know; a vague clause can leave you late.

  • The vendor has a named security contact and a published way to report vulnerabilities.

    You need someone to call during an incident, and a vendor that takes reports seriously.

Certifications and assurance

Independent evidence, with the right scope.

  • A current SOC 2 Type II report covering this service is available to us, and someone has read it.

    A Type II report tests whether controls operated over a period, not just whether they existed on one day; exceptions and complementary user entity controls are in the detail.

  • The vendor holds ISO/IEC 27001 certification whose scope includes this service.

    A certificate that covers a different product or office tells you little about this one.

  • The vendor publishes its subprocessors and tells us before adding new ones.

    Your data goes wherever the vendor’s suppliers go.

Shared responsibility

Who owns which control.

  • The vendor documents a shared-responsibility model: which controls it runs and which are ours.

    Most SaaS incidents involve controls on the customer side, such as access, configuration and integrations.

  • Each customer-side control (users, MFA, settings, integrations, logs, exports) has a named owner on our side.

    A control everybody assumes someone else configured is a control nobody configured.

25 unanswered will count as No.

How it fits with your other vendor reviews

This is a quick, control-by-control check of what you will depend on day to day. For the full questionnaire a cloud provider answers, use the CSA CAIQ Answer Builder, built on the Cloud Security Alliance’s Consensus Assessments Initiative Questionnaire. For a broader third-party review covering data access, breach history, continuity and right to audit, use the Vendor Risk Scorecard. Both are Pro tools with 3 free runs on a free account.

How the rating works

  • Yes = 1, Partly = 0.5, No or not answered = 0. Each control has a weight from 1 to 3.
  • Score = weighted average out of 100: 80+ Low risk, 60 to 79 Medium, 40 to 59 High, below 40 Critical.
  • Critical controls (weight 3: SSO, MFA enforcement, SCIM, audit log coverage, full export, integration control, incident notification): one No makes the rating at least High; three make it Critical.
  • Gaps are ranked by weight × what is missing, each with an action and, where it applies, a question for the vendor.

The weights and bands are PlayCISO’s own rubric, not a standard. Check the vendor’s answers against its documents and contract, not only its sales team.

Questions

How do you assess the security of a SaaS vendor?

Check the controls you will rely on every day: single sign-on, enforced MFA, SCIM deprovisioning, admin roles and audit logs you can export, where data is stored and how it is encrypted, backup and full data export, control over third-party integrations and OAuth grants, incident notification terms in the contract, and independent assurance such as a SOC 2 Type II report or ISO/IEC 27001 certification. Then agree who owns each customer-side control.

What questions should I ask a SaaS vendor about security?

Ask whether SSO and SCIM are on your plan, whether MFA can be enforced, which events the audit logs record and how long they are kept, whether logs can be streamed to your SIEM, how you export all your data, how integrations can be restricted, what the contract commits to on incident notification, and for their latest SOC 2 Type II report and ISO/IEC 27001 scope. This tool lists the questions for every gap you record.

What is the difference between SOC 2 Type I and Type II?

A Type I report looks at whether controls are designed properly at a point in time. A Type II report also tests whether they operated effectively over a period. For a vendor you will rely on, ask for the Type II report and read its scope and exceptions.

What is SCIM and why does it matter for SaaS security?

SCIM (System for Cross-domain Identity Management) lets your identity provider create, update and remove user accounts in a SaaS app automatically. Its main security value is deprovisioning: when someone leaves, their access is removed without anyone having to remember.

What is the SaaS shared responsibility model?

The vendor runs the platform and its infrastructure; you remain responsible for how you use it: who has access, MFA, configuration, connected apps and what data you put in. Many SaaS incidents start on the customer side, so each of those settings needs an owner.

How is the risk rating worked out?

Each of the 25 questions has a weight from 1 to 3; Yes scores 1, Partly 0.5, No or unanswered 0. A score of 80 or more is Low risk, 60 to 79 Medium, 40 to 59 High and below 40 Critical. A No on any weight-3 control raises the rating to at least High, and three or more make it Critical. It is PlayCISO’s own rubric, not an industry standard.

Keep going

The tools people use most on PlayCISO.

All tools
🖤 Black Friday: new customers get 3 months of Pro for $5, then $5/month. Ends 30 Nov. See the deal