My Portfolio / User Experience Research / NPS
Work | My Portfolio | User Experience Research | NPS
Content analysis of NPS comments to track trends
Aim: Conduct a qualitative content analysis of NPS comments so that colleagues can identify common UX themes and track them over time by reapplying the same constructs each quarter.
Problem
Product Operations (ProdOps) owned and managed themes for reporting on our NPS data, which involved manual triage and tagging of all NPS responses so that the company could report on NPS trends every quarter. The company was very responsive to customer feedback provided in NPS comments, but struggled to stay on top of theme tagging, which was limited to two themes per comment, and could vary depending on who tagged a comment.
I was tasked with helping the ProdOps team adjust the themes for NPS comments and to categorise them into product and non-product results. I adopted a top-down approach to analysing NPS comments to create themes that could be reapplied each quarter, allowing the company to track trends over time.
Process
1
Definition of terms
Key concepts were defined and a coding framework was created based on the initial bottom-up content analysis conducted on a smaller set of customer comments.
2
Open coding
Using the coding framework, NPS comments were open-coded into product (usability) and non-product (content and support) categories of responses.
3
Content analysis
Themes were counted and mapped against NPS scores. Usability issues, sources of value, and pain points were pulled from these themes.
1. Definition of terms
To take a top-down approach to content analysis of the NPS data, I needed to disambiguate some terms and make decisions about how to code ambiguous phrasing. For example:
-
The difference and overlap in usage of the terms success, support, and service, which are often used interchangeably or generally.
-
Sentiment versus usefulness, e.g., the word "great" in"Support is great" could refer to a positive opinion of someone or the team (sentiment) and/or the quality of the support a customer received (usefulness/helpfulness).
-
Usability versus ease-of-use, which are often used interchangeably by the UX community, but are separately defined in the ISO 9241-11 standard for usability and the System Usability Scale (SUS), with ease-of-use being a sub-construct of usability.
-
Usability versus learnability. The latter typically refers to ease of first time use, understandability, and intuitiveness.
From here, an initial coding framework was developed. This framework was primarily based on:
-
Prior knowledge of sub-constructs of usability, as defined by the ISO and studied during my PhD.
-
Insights from a bottom-up qualitative content analysis of a smaller set customer comments related to a particular product in the product line and its documentation.
The framework was validated and modified as appropriate over the subsequent phases as more data was coded and analysed. The main modification made to the framework was to pull out flexibility as its own theme.
2. Open coding
Customer comments were open coded into categories of responses as they related to:
-
The product's usability.
-
The non-product experience with content and support.
A comment could be coded against multiple categories, but a category could only be coded once per comment. For example, if a comment mentioned a need for flexibility twice, or gave two different examples of a need for more flexibility, the response would be coded under Flexibility only once.
While coding, example phrases and comments were added to the coding framework developed as part of the definition phase.
3. Content analysis
Qualitative content analysis involves systematic classification of data to identify patterns. Comments were coded into themes and sub-themes, and the frequency of codes were noted, giving a sense of the more pressing issues.
The validity of the analysis is supported by the following observations:
-
Positive usability comments were associated with higher NPS scores and negative usability comments were associated with lower NPS scores. This indicates high validity of the coding since research shows that the NPS correlates positively with the SUS.
-
Almost no Promoters were coded as being dissatisfied with Pendo, indicating that the comments were coded with high validity, since a major sub-construct of the SUS is satisfaction.
498
Comments coded for product usability
82
Comments coded for content and support
2
Quarters of NPS data analysed in < 2 weeks
Results
The following results were presented to leadership and shared with the company as a whole, separated by product-related (usability) comments and non-product (content and support) comments:
-
NPS scores compared against the NPS score for all responses.
-
The distribution of Detractors, Passives, and Promoters.
-
The distribution of comments across NPS scores.
-
A summary of the top themes, both in general and related to specific products.
-
Graphs showing the differences in opinion between Detractors, Passives, and Promoters.
-
A list of the top five sources of value for customers.
-
A list of the top four pain points expressed by customers
-
A list of competitors mentioned by customers grouped by the types of services they provide.
Outcomes
The following outcomes emerged from this work:
-
The coding framework and content analysis results were used by the Product team to make decisions about the data that the company routinely uses for prioritisation and for tracking trends over time.
-
Due to usability being one of the main indicators of detraction, the insights from the first two quarters of the year were used by Product Design to reduce negative perceptions of usability for the second two quarters.
-
The Machine Learning (ML) team used the coding framework to upgrade the company's theme-tagging process with the development of the NPS Insights tool, which automatically identifies themes from open-text NPS responses.
Challenges and learnings
The request to conduct this research emerged as a result of a much smaller scale content analysis I did of NPS and Pendo Feedback comments that related to the product documentation I was managing. I conducted this volume of coding and research in a very short period of time (less than two weeks) on top of my day-to-day work, which would have been impossible to maintain. As a result, I was unable to continue coding comments and doing my full-time job after two quarters. However, it was immensely satisfying to know how appreciated and useful the insights and output were.