Surveys give people an opportunity to be heard and provide government with vital information about the impact of its services. BETA partnered with Department of Social Services (DSS) to encourage more people to complete the Data Exchange survey.
Data Exchange is new report platform developed by DSS to enable the collection of meaningful data about service delivery outcomes from organisations receiving community grants. As part of this platform, a client survey enables clients to self-report on why they sought assistance and the outcomes of accessing the service.
We incorporated small behaviourally-informed progress updates into the survey. Updates were presented in a form of a gradually completing pie-chart, accompanied by short messages with references to social norms and highlighting how completing the survey would benefit the individual and their community.
We found our changes had a significant impact, increasing survey completion by 6.8 percentage points, from 84.7 per cent to 91.5 per cent. These results show how simple low-cost changes to survey design may increase survey completion rates.
Pre-registration information
Data Exchange (DEX) is the Department of Social Services’ (DSS) approach to collecting and storing program performance data. The intention of DEX is to shift the focus of performance measurement from outputs to more meaningful information about service delivery outcomes. A DEX survey is offered to clients both at the beginning and end of various services they receive from DSS.
BETA and DSS will design behavioural insights within the DEX survey to: (1) increase initial survey completion rate, using pseudo-set framing, and (2) increase second survey participation rate, using a pre-commitment device in the first survey.
Theory suggests that arbitrarily grouping items or tasks together as part of an apparent “set” (or “pie”) motivates people to reach perceived completion points: to finish a pseudo-set / complete their pie. Pre-commitment has been shown to be effective at encouraging people toward a range of behaviours, such as saving more for retirement and getting vaccinated.
The aim of the study is to examine whether using these behavioural insights techniques can improve the data collection rate for the DEX surveys, which in turn will assist DSS with evaluating service delivery outcomes. It is important to collect complete feedback from clients to inform continual program and service delivery improvement, as there are a diverse range of programs covered by the DEX survey that are accessed by many Australians.
ADDITIONAL TRIAL INFORMATION
BETA ethics pre-registration number: BETA ETH 2017 - 009
Experimental design: 3-arm Randomised Controlled Trial.
Participants are expected to be enrolled in the trial during a 3-month period. Participants will be randomised into one of 3 conditions in their first client survey: (1) pseudo-set framing (completing a “pie”); (2) final page commitment; or (3) control condition (survey as usual). If the sample size is less than anticipated, participants assigned to condition 2 may be combined into the control condition.
Intervention(s):
- Pseudo-set framing of questions (completing a “pie”), intended to motivate people to reach perceived completion points
- Final page commitment device, intended to pre-commit clients to participating in the second survey
Control condition: Treatment as usual
Outcome(s):
- Proportion of clients who complete “Survey 1”
- Average percentage of "Survey 1" questions completed by clients
- Proportion of clients who commence “Survey 2”
Expected sample size: Approximately 423
Other:
Hiscox, Michael. 2017. Have your pie and complete it too: encouraging survey participation to evaluate social service program outcomes. AEA RCT Registry. May 03.
