The Challenge
Pluralsight is an online learning platform for technology professionals. After releasing a beta version of the assessment product, Role IQ, the team had a chance to investigate how to increase learner engagement with the experience.
Approach
As Product Manager, I facilitated all discovery and hypothesis testing for this experiment, alongside a team of Engineers, Data Scientists, and UX Designers.
The Role IQ team chose to run an A/B test experiment to test a predictive skill inference model in order to determine whether it would increase engagement with the Role IQ assessment product.
Predictive skill inference is a method of predicting a learner’s Skill IQ score for assessment(s) they haven’t taken yet.
The findings from this study were used to determine whether resources should be dedicated to building out a machine learning model in the Pluralsight product.
Identify key research questions
- Does inferring a learner’s score on an assessment they haven’t taken yet increase their likelihood of taking the assessment?
- How can Role IQ utilize the predictive skill inference model in its experience?
- Should resources be dedicated to building out a predictive skill inference model?
Craft study design
Hypothesis: Prompting a learner with their predictive skill level for a given skill in their role will increase the number of assessments that they take within a role.
Experiment: We ran an A/B test around notification messaging for learners who started a Role IQ but only completed one assessment. The control group received the traditional reminder notification, while the test group received messaging about their inferred skill level for the role. The experiment window capped at three weeks; overall, 2,749 learners saw the notification, 1,412 in the control group, and 1,337 in the test group.
Target Metrics:
- Average # of assessments completed per user (who have completed at least one assessment) during the experimental timeframe
- % of users who took an assessment after viewing the experimental notification
Covariate metrics:
- # of views per notification category
- # of clicks per notification category
Outcome
The team found no significant difference in the average number of assessments completed by the control vs. test group users after viewing the experimental notification. However, we found a significant difference in the percent of control vs. treatment users who took an assessment after seeing the experimental notification.
Insight
Learners were more likely to take an assessment after viewing a notification containing a predicted Skill IQ then seeing a traditional role engagement notification.
The findings from this study were used to get buy-in for resources to build the skill inference machine learning model in the Pluralsight platform. As of June 2019, the model is in production and being used to power several experiences in the platform.