Abstract

Push-to-web survey design has been used to increase response rates to web surveys when email addresses are unavailable. However, getting consumers to complete online surveys is challenging. Previous research has found that embedding a small cash incentive with a survey invitation letter had a significant impact on response rates (Messer and Dillman, 2011). But is it worth the cost when email addresses are available? Regrettably, there is little research on whether invitation letters are useful when a reliable list of emails is available.

To understand the impact of a survey invitation letter on response to an email survey, we embedded an experiment into a consumer experience survey of current and former customers of the Massachusetts Health Connector Authority. Our hypothesis was that invitation letters and token incentives would increase response rates and improve representativeness.

The sample was divided into two groups; a study group sent an invitation letter with incentive prior to email invitations and a control sent only emails. This invitation letter more than doubled response rates, from 6.5% among those in the control to 13.6% among those receiving the invitation and incentive prior to emails. The impact of the letter on response varied by respondent demographics.

However, this increase in response was achieved at a significant per survey cost. The increase in response did not improve representativeness of respondent sample characteristics and few differences were apparent in responses to survey questions between study and control groups.

Our conclusion is that a survey invitation letter with token incentive does significantly increase response but also increases data collection costs. Further, the results suggest the letter did little to improve the quality of the data. In this work we explore the trade-offs between per-survey costs and data quality in detail.