[activecampaign form=2]At only 11 people per square mile, North Dakota is a rural state with the fourth lowest population density in the nation. This translates into a low population of those from the age of 21 to 64 with a disability—34,000 people. Yet 15,200 people with disabilities in that age group are employed. That’s a remarkable 44.5%, almost ten percentage points higher than the average of 35.7% for the US.*

The Division of Vocational Rehabilitation helps North Dakotans with disabilities to improve their employment opportunities and achieve competitive employment and increased independence. The Division has a more than $10 million budget (2016) and a staff of 89, with one state office in Bismarck and eight regional offices.

The Division is very proud of achieving such a high level of employment, but in spite of this success, very few customers cooperated by returning satisfaction surveys that were mailed to them. The Division administered the surveys in-house, using paper surveys that were mailed on a quarterly basis. The level of cooperation was sometimes as low as 10%.

According to Robyn R. Throlson, LSW, CRC, the Division’s Planning and Evaluation Administrator, “The surveys hadn’t been changed for a long time, and the Division had no staff resources for following up on them.” The very low level of cooperation raised concerns about whether the results fairly represented all customers.

• The survey didn’t capture the respondents’ names, so the Division couldn’t follow up with individuals.

• The Division couldn’t address issues that were identified because the cases were already closed.

• Respondents were often asked to recall instances from years ago.

• The responses skewed toward extreme experiences, both positive and negative.

• The surveys provided a general measurement of satisfaction only. There was no information for quality improvement or satisfaction with different aspects of service.

In addition, the State Rehabilitation Council was eager to improve the survey. They wanted to be sure a sufficient number of customers participated so they could be certain the data was accurate and they wanted more useful information.

Beginning to Make Changes

The Division knew changes were needed, but was unsure how to proceed. “At the Summit on Vocational Rehabilitation Performance Management, Market Decisions Research gave a presentation and we learned what might be possible for our surveys,” said Throlson.

Once hired, Market Decisions Research collaborated with the Division to conduct an assessment and then design and implement a new consumer experience measurement system that goes beyond simply measuring satisfaction. The new approach surveys 200 customers per quarter by phone. Both open and closed cases are surveyed. To help with the development of the survey, Market Decisions Research provided the Division with field-tested questions from the extensive library of survey questions that the firm maintains.

Improving Data Quality and Gathering More Timely, Useful Data

In the new process, customers are contacted by telephone and asked to participate. They are given the option of participating at the time of the call or using a toll free number to call when it is convenient for them. Some customers still receive mailed surveys to ensure accessibility for people with hearing-related disabilities or without access to a telephone. These actions and the use of the phone survey encourage participation. Unlike other surveyed groups, vocational rehabilitation customers like to give feedback by phone and appreciate being asked to do so. In research with the Division in North Dakota and nationwide, Market Decisions Research has found that vocational rehabilitation customers often enjoy having someone express interest and listen to their experiences.

Market Decisions Research improved the quality and type of data being gathered to provide more than just basic information about customer satisfaction. The Division now receives more in-depth information about the consumer’s experience with services, which is used for continuous improvement.

When customers are called they are asked specific questions based on their cases, as well as probing questions to gather more precise information and more accurate explanations about issues. The customers’ comments provide specific and actionable suggestions.

Market Decisions Research modified the survey from seven to 37 closed-ended questions to measure and provide continuous improvement information for specific aspects of service and support. A number of other survey questions allow customers to talk about their experiences in their own words. Thirty or more questions can be asked in just 10 to 12 minutes and the phone is the best vehicle for collecting complete verbatim comments by customers.

In addition to closed cases, customers with open cases at various milestones are randomly selected and surveyed. More current data is captured and changes can be made to improve the services to customers while cases are still open.

Enhancing Reporting and Expanding Use of the Data

Market Decisions Research also enhanced the methods for reporting the data. The Division is provided with all of the raw data, as well as data analysis, a data compendium, and a detailed annual report. The Division is now able to drill down into detail for information that was previously unobtainable. Reporting also identifies trends and provides comparative results for pinpointing statewide versus regional issues.

The data has helped the Division detect unmet needs and gaps in services. In fact, the data is so robust the Division expanded its use to identify training needs and utilize it as part of the statewide comprehensive needs assessment.

The Division now enjoys vastly improved participation in the survey. Of those customers reached by phone, 82% (2017) cooperated and completed the interview. The Planning and Evaluation Committee, a sub-committee of the State Rehabilitation Council, is very pleased with the information. Even for a state like North Dakota, which has a lower vocational rehabilitation budget and population, an improved consumer experience measurement system is cost-effective.

In the future, the Division may drill down and collect data at the counselor level rather than the regional level. Also, the Division has begun thinking about survey questions regarding collaborating with Workforce Innovation and Opportunity Act (WIOA) core partners for evaluation of common elements. Satisfaction data can drive improvement in outcomes to help meet the common measures for WIOA.

About Market Decisions Research

Since 1977, Market Decisions Research has provided survey research and program evaluation for government agencies, nonprofits, and businesses in more than twenty-eight states.

We collect data in-house using a comprehensive set of tools—high-speed scanning of paper and mail surveys, a telephone interviewing center, and sophisticated software for online surveys. Adept at qualitative research, we offer a team of trained focus-group moderators and interviewers for executive and key informant interviews.
Our research team gets the most out of data with skillful question development, advanced analytics, and clear reporting. Our evaluation team works collaboratively with clients and stakeholders to assess needs and ensure that all are prepared to act on the information developed.

Market Decisions Research is a national leader in research and evaluation services for vocational rehabilitation agencies. We have conducted research with vocational rehabilitation customers, stakeholders, employers, and vocational rehabilitation staff for clients throughout the country. Our work improves vocational rehabilitation programs and outcomes.

Market Decisions Research Is Here to Help

For more information, contact Market Decisions Research at 800-293-1538 or
research@marketdecisions.com.