Challenge
The Predictive Index is a talent optimization company that businesses use to hire, inspire, and retain talent. They provide instructor-led training classes as a service to introduce the concepts behind their workplace assessment tools to customers.
After conducting a 9-month pilot of their virtual classes, the learning team noticed that CSAT scores of the new offering were lower than it’s in person format. The company partnered with a group of master’s students from Boise State University to investigate the cause for the low satisfaction and identify potential areas of improvement.
Approach
As project lead, I led three other masters-level students through research and analysis of the program. The team used a mixed-methods approach that included learner surveys, semi-structured interviews, ad-hoc data analysis, literature review, and observations.
Key Research Questions and Areas
The analysis focused on four areas including: program design and delivery, learner application, completion rate, and sales/marketing with the fundamental questions being:
– Is the program meeting its intended outcomes?
– What improvements could be made to the virtual training programs?
Distributing surveys understand application of skills
The project team distributed a web-based survey to learners two months after their completion of attending the virtual training.
Stakeholder interviews to gain an understanding of the learning program
While collecting survey responses, I led stakeholder interviews to better understand the current state of the program as well as the intended business goals. Based on feedback from the interviews, I developed a graphic to show the inputs and results of the program.
Data review to understand learning engagement
To get a better picture of the training program, the project team reviewed reaction data from learners who attended the virtual training, in addition to reviewing course enrollment and completion data.
Literature review to understand virtual training best practices
We conducted reviews of 20 academic articles to understand best practices used in the delivery and design of virtual training programs. The findings were used to create an observation checklist that was used to evaluate the current state of the program.
Observations of the virtual training sessions
Using the observation checklist, the project team observed four virtual training sessions and rated various elements of the program design and delivery.
Data analysis to identify insights
Following data collection, the project team created rubrics, scored data, and created a dimensional triangulation rubric. The team drew evidence-based conclusions and recommendations using a synthesis rubric.
Opportunities
The overall quality of the program was high, the program was headed in the right direction but there were some gaps that should be addresses. We provided The Predictive Index with a list of recommendations related to the different program dimensions.
Program design and delivery
Determine the overall strategy of the virtual training program. For example, is it just an online version of the existing instructor-led training, or does it have another mission and purpose?
Learner-application
Develop a level 3 (application) survey that is delivered to learners after a specified period (3 months) after completing the program to determine whether they have applied what they learned when they return to the workplace.
Completion rate
Develop a customer-facing website that makes self-registration for learners an easy one-stop-shop. The site should contain a calendar of training events where customers can see availability and register for events.
Sales and marketing strategy
Adapt the marketing strategy to indicate the learning outcomes learners can expect to receive based on their attendance.
Outcomes
As of 2019, the recommendations that the research team made, helped to inform The Predictive Index’s “digital first” learning products. The Predictive Index has seen a YoY increase in revenue and customer engagement in their learning experiences.