A poorly designed survey is one of the most common sources of weak social value evidence. It asks too much, captures too little that is useful, and leaves practitioners trying to build a credible SROI case from data that does not hold up to scrutiny.
The problem is rarely a lack of effort. It is usually a lack of focus. This article sets out how to design surveys that are proportionate, honest, and genuinely useful for tracking progress on the outcomes that matter.
Start with your outcomes, not your questions
Before writing a single question, go back to your theory of change or outcomes framework. Which outcomes are central to your SROI case? How will you know whether you have achieved the change you set out to make? Your survey should follow directly from that logic, not run in parallel to it.
A useful discipline is to write down the three to five outcomes you most need to evidence, and then ask: what is the minimum data I need to demonstrate change against each one? That constraint is productive. It forces you to prioritise.
If you have not yet mapped your outcomes in a structured way, our guidance on building a theory of change is a useful starting point before designing your data collection tools.
Keep it short enough to get answered honestly
Survey fatigue is real and it affects data quality. Respondents who are partway through a long survey become less considered in their answers. Later questions tend to attract lower completion rates and less reliable responses.
As a general guide, a beneficiary survey for a single programme should rarely exceed twelve to fifteen questions. If you find yourself writing more, consider whether you are trying to do too much in one instrument.
It is often better to use multiple lighter touchpoints across a programme’s duration than to front-load a single long survey. A short baseline check, a mid-point prompt, and a follow-up at the end will typically give you richer data than one comprehensive form that nobody completes carefully.
For commissioners and procurement leads, the same logic applies to supplier reporting. Our article on proportionate social value reporting covers how to calibrate evidence requirements to contract size and complexity.
Make space for negative impact
Most surveys are designed with an implicit assumption that the programme has been beneficial. The questions are positive, the framing is affirming, and there is rarely an easy way for a respondent to say that things have got worse, or that the intervention has had unintended consequences.
This is a significant gap. Negative impacts are a formal part of a robust SROI analysis, and ignoring them does not make them go away. It simply means your evidence base is incomplete.
Some practical approaches:
- Use neutral framing. Instead of ‘How much has this service improved your confidence?’, ask ‘How has your confidence changed since taking part?’
- Include at least one open-response question that invites people to describe anything that has not worked well for them.
- Where possible, give respondents the option to respond anonymously, particularly for more sensitive outcomes such as mental health or financial wellbeing.
- Consider whether peer researchers or independent interviewers might surface concerns that a provider-administered survey would not.
Include the deflation questions that honest measurement requires
One of the most important differences between a credible SROI analysis and a simple output report is the treatment of deflation factors. These are the adjustments that account for the fact that not all of the change you observe can be attributed to your work, and not all of the change persists over time. SVE’s approach to SROI methodology applies these rigorously, and your survey design should support that.
The five factors to consider are:
- Deadweight: how much of the change would have happened anyway, even without the programme? A question such as ‘How likely do you think it is that you would have achieved this without the support you received?’ is direct and serviceable.
- Attribution: where multiple organisations have contributed to an outcome, how much of the change is genuinely attributable to your work? Questions that ask respondents to name other support they have received, or to estimate the relative contribution of different services, can help calibrate this.
- Displacement: has the benefit to one group come at the cost of another? This is particularly relevant in employment programmes, where placing one person in a role may mean another candidate does not get it.
- Drop-off: does the impact last? Asking the same question at different points in time, or including a follow-up survey some months after programme completion, gives you evidence on whether outcomes are sustained.
- Leakage: do the benefits accrue outside the intended population or geography? This is more common in place-based or community programmes, where knock-on effects are harder to track but can be significant.
A well-evidenced SROI that accounts honestly for deflation is considerably more credible to commissioners and funders than one that presents unmodified change as direct impact.
From survey to credible SROI
Survey data is only valuable if it connects clearly to your valuation. Before finalising your questions, check that each one maps to a specific outcome and, where applicable, to the financial proxies you are using to calculate your social return. If a question does not connect to either, it is probably not earning its place.
Share your draft survey with a small group of intended respondents before deploying it. A brief cognitive test, asking participants to talk through what each question means to them and how they are deciding on an answer, will surface ambiguities that you will not catch by reading it yourself.
Good survey design is the foundation on which a defensible, meaningful SROI case is built.
Frequently asked questions
How many questions should a beneficiary survey include for social value measurement?
As a general guide, a beneficiary survey for a single programme should not exceed twelve to fifteen questions. Beyond that, response quality tends to drop. It is usually better to run shorter surveys at multiple points during a programme than to attempt one comprehensive instrument at the end.
What is deadweight in social value measurement, and how do you ask about it in a survey?
Deadweight refers to the proportion of an outcome that would have occurred anyway, without the programme or intervention. A simple survey question to surface this is: ‘How likely is it that you would have achieved this without the support you received?’ Response options on a five-point scale from ‘very unlikely’ to ‘very likely’ allow you to calculate a deadweight percentage for use in your SROI model.
What is the difference between attribution and deadweight in SROI?
Deadweight asks how much change would have happened without any intervention. Attribution asks how much of the change that did occur is down to your specific organisation, as opposed to other services or support the person also received. Both are deflation factors in a Social Return on Investment calculation, but they address different questions. Attribution is particularly important when a beneficiary has accessed multiple programmes simultaneously.
What is drop-off in social value, and how do you measure it?
Drop-off, sometimes called duration, refers to the extent to which an outcome diminishes over time. A confidence gain immediately after completing a programme may not persist at the same level a year later. The most reliable way to measure drop-off is through follow-up surveys administered some months after programme completion, asking the same outcome questions as the original baseline and post-programme assessments.
Should social value surveys include questions about negative impacts?
Ideally, yes. Negative and unintended impacts are a formal component of a robust SROI analysis. Surveys that only ask positively framed questions will systematically underreport harm or neutral outcomes. Using neutral question framing, open-response options, and where possible anonymous submission can make it easier for respondents to report honestly when things have not gone well.
What is displacement in SROI and when does it apply?
Displacement occurs when a benefit to one person or group comes at the expense of another. It is most commonly encountered in employment programmes: if your programme helps one candidate secure a role, but another candidate who would otherwise have been appointed does not get it, some of the benefit has been displaced rather than created. Displacement is not always material, but it should be considered and, where relevant, reflected in your SROI calculations.
What is leakage in social value measurement?
Leakage refers to benefits that accrue outside the intended population or geography. In a programme targeting a specific community, some positive effects may benefit people who were not the intended recipients. Leakage can be a positive finding, but it needs to be understood and reported accurately rather than counted as direct programme impact.
