Challenge
Pluralsight is an online learning platform for technology professionals. During Fall 2019, the company received an influx of customer feedback round learning content being outdated when compared to competitors. A team was put in place to identify how the company could improve learners’ perception around content.
APPROACH
I led a cross-team research team that included a UX Designer, Engineer, Data Scientist, and Product Marketer through research planning and analysis. We used a mixed-methods approach including a data review, customer success and sales surveys, prototype testing, and semi-structured interviews with internal stakeholders and learners.
Key Research Questions and Areas
– What is the scope of the content freshness issue for customers and prospective clients?
– What are the behaviors, preferences, and barriers related to content freshness in the Acme Enterprises experience?
– What needs and opportunities could be used to help inform the product strategy around content freshness?
Refining the problem
The group performed an initial data review and distributed an internal survey to key stakeholders from customer success and sales. This initial analysis focused on the following questions:
- What is the scope of the content freshness issue for customers and prospective clients?
- Who were the impacted user groups for this issue?
- What additional research should be done to improve this problem?
Initial research confirmed that this was an opportunity area impacting learners, tech leaders inside of organizations, and Pluralsight’s internal customer success, sales, and content production teams. The research team prioritized a list of additional research areas that enabled us to get buy-in for additional discovery.
Prototype testing with learners
Leadership zeroed in on one of the recommendations the research group made because of its potential to make the largest impact on the learner experience. Because of this, the team carefully designed concepts to allow learners to verbalize their expectations around content freshness affordances in the software. We tested 3 concepts with 12 learners and intentionally left out explanations as to what verified-up-to-date meant.
Learner feedback included things like:
“I assume everything you have has been reviewed. If it’s not adding anything, it’s in the way.”
“Verified up to date means that the recordings represent the most recent version on the platform”
Insight
This let us know that learners were making incorrect assumptions about Pluralsight’s content that could lead to long term mistrust of the platform.
Conducting interviews with sales and customer success
Next, we connected with members of the customer success and sales team to learn more about the perception of content among Pluralsight learners. We learned that these teams often prepare speaking points to try and deflect customer conversations around issues.
Insight
This led us to believe that a solution to the content issue must live in the Pluralsight platform itself.
Conducting interviews with Pluralsight Learners
We led semi-structured interviews focused on the question:
- What are learners behaviors, preferences, and barriers related to content freshness in the Pluralsight experience?
Insight
We learned that learners perception of how up to date content was related to 3 primary factors: technology version, content updates, and content coverage.
Opportunities
The research team identified that content freshness is a multi-dimensional problem spanning 3 facets: freshness perception, content coverage, and discoverability.
Freshness Perception
- Identify and make technology versions available for learners and tech leaders – this is consistently asked forIdentify and make existing material updates as metadata that are relevance signals for learners and tech leaders
- Improve perception of freshness by showing that content is “coming” or soon to be published when a user searches for a certain topic
- Improve perception of freshness for potential customers and tech leaders by providing content roadmaps
- Retire old courses to remove noise in the system
- Conduct experiments to improve perception of relevance (trending courses, etc)
Content Coverage
- Continue to improve coverage for breadth and depth, shelf life, and levels across topics in the content library
- Improve perception of coverage by indexing additional context types in the search experience
Discoverability
- Improve the users ability to filter and sort by newest
- Improve perception of coverage by better phrase matching, query enhancement, and granular results in the search experience
Outcomes
There were a number of business outcomes that occurred due to this project.
- The organization avoided a potentially harmful solution that could have led to customer mistrust
- The research group conducted an aligned and strategic approach that led to a prioritized list of opportunity areas across the entire business
As of Q1 2020, the company restructured their curriculum business units to support the use of business intelligence to help scale content creation moving forward.