Program Evaluation
Background
ABC Insurance (a pseudonym) provides business, bond and specialty, and personal insurance for a range of coverages. The company primarily operates in the United States, with a small footprint in Canada and several European countries. Over 30,000 employees work for ABC Insurance across dozens of offices. This evaluation took place within the business insurance unit to determine how well the WalkMe digital adoption platform (DAP) was being utilized in the unit’s Salesforce adoption campaign.
Starting in June 2021, approximately 600 workers within business insurance implemented Salesforce for its customer relationship management (CRM) functionality as part of the sales enablement project. WalkMe, which provides performance support within Salesforce, was implemented for these employees at this time, as well. The evaluation client, the Director of Learning and Development, was interested in how well ABC Insurance is using WalkMe and to understand which improvements can be made in the content and strategic governance of the tool. This evaluation intended to provide answers to these questions.
Scope of program evaluation.
Evaluation Methodology
Dr. Chyung's 10-step evaluation model was the primary tool used to guide the evaluation process.
10-step evaluation model.
Evaluation Type: formative evaluations are designed to improve an ongoing process while summative evaluations are meant to reach conclusions once a process has ended. Stakeholders communicated to the evaluation team their intended use of the evaluation findings would be used to improve how WalkMe is used for Salesforce adoption. Since WalkMe is part of an ongoing project, a formative evaluation was most appropriate.
Evaluation Approach: after examining stakeholder needs, the evaluation team determined that comparing data about the quality of the program to the company goals for the program (a goal-based evaluation) would best address client needs. The evaluation team also added a goal-free dimension to assess any unintended consequences resulting from WalkMe implementation.
Evaluation Dimensions: the evaluation was designed to explore two dimensions to answer how well WalkMe is used at ABC Insurance:
Dimension 1 - Content and Learning Experience: how well placed is the content? How helpful is it? How good is the learning experience? What should be changed?
Dimension 2 - Strategy, Support, and Governance: how good is the learning design strategy? How well does the organization support the learning design strategy around WalkMe? What could be different?
Data Collection: the evaluation team collected data from surveys, interviews, and user performance data. All 600 learners were surveyed (~20% completed the survey) and results were anonymous, unless the respondent opted to participate in an interview. Likewise, performance data for all 600 learners was analyzed. The interviews were conducted with both WalkMe users and key stakeholders who were in charge of implementing the WalkMe and Salesforce programs. Five learners were interviewed for Dimension 1, while four stakeholders were interviewed for Dimension 2. Each interview lasted approximately one hour.
Results
Overall, the evaluation found WalkMe to be a beneficial tool for improving workplace performance when using Salesforce at ABC Insurance. When measuring the content and learning experience (dimension one), the evaluation team found that WalkMe provided useful content, was well placed, and improved the Salesforce learning experience. The final result for dimension 1 is “excellent.”
When measuring the learning strategy, support mechanisms, and governance of WalkMe (dimension 2), the evaluation revealed several gaps and areas for improvement. The final result for dimension 2 is “poor.”
The table below describes the evaluation results for each of the two dimensions.
Conclusions and Recommendations
Overall, the survey and interview data indicated the end-user experience with WalkMe is excellent. Extant data also indicated that WalkMe has a positive correlation with user performance. However, the evaluation team identified a few areas of improvement for the program:
Implement a formal WalkMe strategy: the strategy should define the lifecycle of WalkMe content from identifying needs to publishing content and should define roles and responsibilities.
Increase the use of WalkMe: provide more WalkMe content and support since it is well-liked and positively impacts performance.
Continuously gather feedback: while the voting buttons did not yield meaningful data for the evaluation, other feedback mechanisms and strategies could be considered:
Regular user surveys.
Partnering with other teams (i.e., the user experience team) to collect user data.
User education on the importance of using voting buttons as a way to gather feedback for improvements.
Review of employee job performance data connected to Salesforce activities.
Limitations and Lessons Learned
Despite an evidence-based design approach to this program evaluation, the evaluation team experienced and identified several limitations that should be considered in the whole of the evaluation’s results. The context in which the results should be interpreted is presented here:
Lack of response to voting buttons: the team implemented voting buttons within WalkMe content to collect data on whether users found the content helpful. The low response rate from the voting buttons prevented the team from using this data source to influence results and recommendations.
Change in survey messaging strategy: the team’s initial survey messaging strategy was to provide a popup message via WalkMe to users when they log into Salesforce inviting them to participate in the survey. This yielded an unexpectedly low response rate. To increase the response rate, the team successfully shifted to an email campaign.
Variation in survey visibility: the team designed the survey as primarily anonymous with the option for participants to opt into being interviewed. If a participant opted into the interview process, their survey data was no longer anonymous. The team mitigated this by having our team lead (and ABC employee) scrub personal identifiers before presenting the data to the rest of the evaluation team.
Embedded nature of WalkMe: WalkMe looks and feels like it's native to Salesforce. This is the intended use for WalkMe, but as a result, many users do not know they are using WalkMe and assume they are only using a function of Salesforce. This posed a challenge for the team in how survey questions were designed, how interviews were conducted, and the messaging to staff about the evaluation project.
Survey Question Scope Expansion: stakeholders requested the addition of several survey questions that were outside of the evaluation’s scope. While this addition did not interfere with the team’s intended data collection, the additional questions could have influenced participant responses differently than if the questions were all in scope.
OPWL Program Learning Goals
This project was part of my master's program at Boise State University. Click here to view how I aligned the project work to several of the program learning goals.
View Entire Report
You can view the report in its entirety by expanding the document below:
WalkMe Evaluation Final Report v3.1 sms.pdf