Introduction

This three-part series summarizes the key insights from our Learning Evaluation subject matter expert, Anthony Randolph, PhD, through his dissertation, Human Resources Department: Professionals’ Experience Utilizing Data Analytics in the Training Evaluation Process. View part one here, where we introduce Anthony and the work of incorporating data analytics into training evaluation.

A Note About Limitations

For this research, Anthony notes two limitations with respect to generalizing the findings of this study to other Learning and Development departments who implement analytical tools in their training evaluation. These limitations are first, that organizations vary in how they decide to use technology. And second, the Metrics that Matter (MTM) tool that we discuss in this blog series is one of many analytic solutions on the market and does not represent the only solution in the evaluation process.

Applying Learning Evaluation Analytics: Two Research Questions

Anthony’s research questions looked to uncover practices and procedures related to the use of data analytics in learning evaluation. Specifically, Anthony’s study looked to answer the following:

1. What factors influenced HR professionals to use HR analytics in the training evaluation process?
2. How do HR professionals use HR analytics in this process?

To answer these questions, Anthony designed a pilot training initiative conducted with the Organization Learning & Development Department (OL&D) within a Midwest regional hospital. The organization was not satisfied with their learning evaluation plan based on the mid-20th century Kirkpatrick Evaluation Model, which did not yield results for judging learning effectiveness. To fully investigate the Kirkpatrick model and several other frameworks, Anthony completed a thorough literature review that described the major models in Learning measurement dating back to the 1950s.

Anthony’s pilot initiative aligned the OL&D with a new analytical approach using the Metrics that Matter (MTM) program and framework. MTM evaluates training by providing benchmarks, surveys, reports, and predictive forecasting. MTM aids HR professionals with the ability to generate and track an organization’s key performance indicators (KPIs), generate employee surveys, and pull all the organization’s internal information from the LMS (Learning Management System) and HRIS (Human Resources Information System) into data for making informed business decisions.

Recall, evaluation is judging importance; analytics are converting data to information. When evaluating learning effectiveness, MTM is an essential tool to collect, report, and analyze data to decide what is most important to inform leaders about the effectiveness of their learning programs or strategy (literally, the “metrics that matter”; hence the name of the solution!) But we haven’t responded to the two questions: First, what influences HR professionals to use analytics; and second, how do these professionals use analytics? For this, Anthony introduces the theoretical basis to respond to these research questions.

Intermission: Discussing the Theoretical Basis of the Study

Ongoing, HR teams recognize an increasing promotion of analytics in HR settings, but HR professionals are provided with little instruction on practical application. Anthony set out to offer a useful reference for organizations and HRD professionals looking to apply data analytics in their learning evaluation process.

The Unified Theory of Acceptance and Use of Technology (UTAUT) and Sociomateriality theories served as Anthony’s theoretical framework for understanding how HRD professionals use data analytics in the training evaluation process. Let’s pause to define these theories, as their framework is necessary to understand Anthony’s findings.

UTAUT is a theory of technology adoption that defines four components of technology acceptance. This theory is especially concerned with the behavioral intensions of technology users. The four components include:

1. Performance expectancy: a user’s perceived sense of how effective a technology solution is to reach a performance goal.
2. Effort expectancy: a user’s perceived ease of use for a technology solution.
3. Social influence: a user’s sense of importance they perceive others hold toward using a new technology solution.
4. Enabling, sometimes called facilitating, conditions: a user’s perceived feeling that organizational support is available to them.

Anthony found three components were especially salient to his study: performance expectancy, social influence, and enabling conditions. To round out his study, Anthony wanted a more socially informed view of HR professionals in practice, and for this, he applied Sociomateriality theory as a complement to UTAUT.

The Sociomateriality theory shifts the research orientation away from technology and toward the user, focusing attention on what people do with a particular technology in their daily work. Like the social influence component of UTAUT that acknowledges perceived peer importance for technology adoption, Sociomateriality also reveals social influence, but from the perspective of personal, social, and structural “entanglement” that reveals how users are actually using a technology in their daily work.

Putting it All Together

Let’s compare and contrast these theories to see how they are different, yet complementary in Anthony’s research. UTAUT is like a roadmap that helps us understand why someone might choose to use a new tech gadget or app. It considers factors like how useful they think it will be, how easy it is to use, what their colleagues and co-workers think about it, and whether they have the right support to use it.

On the other hand, Sociomateriality is more about what happens after someone starts using that tech. It’s like a lens that helps us see how people’s daily work and the technology they use are closely intertwined and influence each other. It’s not just about the technology itself, but about how people and technology shape each other in the real world. So, while UTAUT helps us understand why we might use a technology, Sociomateriality helps us understand how we use it.

While absorbing these theories requires a little intellectual heft, having a loose grasp of UTAUT and Sociomateriality is important to understand Anthony’s findings, but for that, you will have to wait for the next post in the series!

What factors do you think influence HRD Professionals to use analytics in the learning evaluation process? Let us know in the comments!

Need to catch up? Head back to part one, or advance to the final post in the series, “Applying Analytics to Learning Evaluation, What We have Learned.”