Training Evaluation Misses the Critical Thinking Dimension

For typical training courses the following four or five evaluation steps are the norm:

Level 1: Reaction and Perceived Values

Measures reaction to, and satisfaction with, the medium, content, and value of the project or program.

Level 2: Learning and Confidence

Measures what participants understand or learned from the project or program (information, knowledge, skills, and contacts).

Level 3: Application and Implementation

Measures what participants understand or learned from the project or program (information, knowledge, skills, and contacts).

Level 4: Impact and Consequences

Measures progress after the program implemented (the use of information, knowledge, skills, and contacts).

Level 5: ROI

Monetary Benefits.

(Phillips, & Phillips, 2007)

Evaluation is further divided into formative and summative evaluation, with formative evaluation relating to evaluating the training program and summative evaluation relating to the long-term effects of the training program (see Evaluation). Level 2, Learning and Confidence, is more of a formative evaluation measure and identifies whether trainees learned the material presented in the training. Level 3, application and consequences, is categorized as summative evaluation in which it looks at whether the trainees on-the-job behavior represents their learning from the training program.  

For these two specific levels of evaluation Phillips and Phillips (2007) highlighted information, knowledge, skills, and contacts. Providing employees (trainees) with the information that they need to conduct their job functions is critical. Additionally, providing employees with the knowledge to utilize this information in a productive manner is key to success. Having the skills to perform one's job is self-explanatory, but, as experience has shown us, people often lack the proper skills required to perform their main job function. Having the right contacts as well as knowing who has the information when needed is equally important. 

Using the following variables to evaluate training programs (information, knowledge, skills, and contacts) have proven to be effective for years. However, expanding on these variables to improve the evaluation process follows the continuous improvement process. As an effort to expand on the accuracy of training I would pose adding critical thinking to the mix.

Training employees to think critically helps to eliminate issues such as functional fixedness and mental sets. Ollinger, Jones, and Knoblich (2008) termed mental set as: " the repeated application of a successful method makes blind any alternative approach, because of the mechanization of the particular solution method" (p. 270). Alternatively, Duncker (1945) identified functional fixedness as: "the tendency to fixate on the typical use of an object or one of its parts" (as cited in McCaffrey, 2012, p. 216).

Adding the dimension of critical thinking to training endeavors will help transform learners (trainees, employees) to effective learners. Brindley, Walti, and Blaschke (2009) identified effective learners as those who are capable of coping with "complexity, contradictions, and large quantities of information, who seek out various sources of knowledge" (p. 3). By seeking out new sources of knowledge employees will better avoid the aforementioned traps of mental sets and functional fixedness. 

Including critical thinking skills as part of the training program, as well as incorporating evaluation of employees critical thinking skills on-the-job, could prove to produce better training results and on-the-job performance results. Additionally, including critical thinking in both the instructional and evaluation phases of the training program could improve both the formative and summative evaluations of the overall program. 

References:

Brindley, Walti, & Blaschke (2009). Creating effective collaborative learning groups in an online environment. International Review of Research in Open and Distance Learning, 10(3), 1-18. 

McCaffrey, T. (2012). Innovation relies on the obscure: A key to overcoming the classic problem of functional fixedness. Psychological Science, 23(3), 215-218. dpi: 10.1177/0956797611429580

Ollinger, Jones, & Knoblich (2008). Investigating the effect of mental set in insight problem solving. Experimental Psychology, 55(4), 269-282. dpi: 10.1027/1618-3169.55.4.269

Phillips, & Phillips (2007). Show me the money: The use of ROI in performance improvement, part 1. Performance Improvement 46(9), 8-22. dpi: 10.1002/pfi.160

Evaluation

Evaluation is often at the end of systematic performance models.  The Instructional Systems Design (ISD) model, sometimes referred to the ADDIE model, includes the following stages: analysis, design, development, implementation, and evaluation.  Other disciplines practice models similar to the ADDIE model.  Six Sigma practices the DMAIC model: define, measure, analyze, improve, and control. In the DMAIC model measure would be comparable to evaluation.  Human Resource Development (HRD) practices analyze, propose, create, Implement, and assess.  In the HRD model, assess refers to assessment which is the same as evaluation in the ADDIE model. 

These systematic performance models are often viewed as linear, stet-by-step, models.  By viewing these models this way they become ineffective at improving performance for the long-term.  Each model is presented to be cyclical and interactive.  This means that each model is designed as a continuous improvement cycle with dynamic interactions between each stage.  In the case of evaluation, this stage affects each of the other four stages in the process.  Evaluation begins during the initial analysis phase and continues through each stage, then re-cycles again, as improvements to the new improved cycle are incorporated.  Wang and Wilcox (2006) support this view indicating: “the larger view of evaluation may not be treated as a separate phase during the process…. It is indeed an ongoing effort throughout all phases of the ADDIE process and culminating at the last phase” (p. 528).

Shrock and Geis identified evaluation as a “process of collecting information and feeding it back to those who need the information so that the system can succeed” (as cited in Stolovitch & Keeps, 1999, p. 185).  Evaluation should be designed to provide feedback during each stage in the process so that improvements can be made to the process.

Evaluation comes in two forms: formative evaluation and summative evaluation.

Scriven (1991) identified formative evaluation to be used “to provide information on improving program design and development” (as cited in Wang & Wilcox, 2009, p. 529).  Wang and Wilcox identified that the purpose of formative evaluation was “to identify weakness in instructional material, methods, or learning objectives” (p. 529). Formative evaluation can be used to evaluate the instructional methods during a training program. 

Following the training program summative evaluation will be used to determine the long-term effectiveness of the program and its instructional methods, including learning transfer.  Brown and Gerhardt (2002) described summative evaluation as those “efforts that assess the effectiveness of completed interventions in order to provide suggestions about their use” (p. 952).  A training program can be evaluated by its impact on the organization and its long-term effectiveness through summative evaluation.

A successful evaluation is one that utilizes both formative evaluation and summative evaluation.  Evaluation needs to be viewed as an iterative process that affects each of the stages in the training process that it is measuring.  Each systematic performance improvement endeavor needs to be addressed as a continuous improvement cycle with a strong emphasis on evaluation.  Evaluation is the key component that makes the systematic performance improvement process a continuous effort, allowing improvements to be made to the process during each stage.

REFERENCES

Brown, K. G. & Gerhardt, M. W. (2002).  Formative evaluation: An integrative practice model and case study.  Personnel Psychology, Vol. 55, pp. 951-983.

Stolovitch, H. D. & Keeps, E. J. (1999).  Handbook of human performance technology: Improving individual and organizational performance worldwide (2^nd ed.).  San Francisco, CA: Jossey-Bass Pfeiffer.

Wang, G. G. & Wilcox, D. (2006).  Training evaluation: Knowing more than is practiced.  Advances in Developing Human Resources, Vol. 8, No. 4, pp. 528-539.

HPT Standards & Systemic Issues

HPT has developed ten standards when dealing with systemic issues.  The first four standards are the core HPT standards, while the remaining six standards deal more specifically to systemic issues.  The first four standards must be met before any of the other six standards can be accomplished.  Without completing the first four standards first, you will be unable to accomplish a fully systemic resolution (Brethower, 2006). 


HPT's ten standards for dealing with systemic issues:
1) Focus on results and help clients focus on results.
2) Look at situations systemically, taking into consideration the larger context, including competing pressures, resource constraints, and anticipated change.
3) Add value in how you do the work and through the work itself.
4) Utilize partnerships or collaborate with clients and other experts as required (Brethower, 2006, p. 112).


Be systematic in all aspects of the process, including the:
5) assessment of the need or opportunity.
6) analysis of the work and workplace to identify the cause or factors that limit performance.
7) design of the solution or specification of the requirements of the solution.
8) development of all or some of the solution and its elements.
9) implementation of the solution.
10) evaluation of the process and the results (Brethower, 2006, p. 113).


Look closely at the standards for steps 5 through 10.  Do they look familiar? These steps include what is typically known as a needs assessment followed by the ADDIE process.  The ADDIE process is an acronym for Analysis, Design, Develop, Implementation, and Evaluation.  One additional item that could be added would be feedback.  Standard #11 could include a feedback loop to each process, thus making the process a continuous improvement process.


Reference:
Brethower, D. M. (2006). Systemic issues.  In Pershing, J. A. (Ed.), Handbook of human performance technology: Principles, practices, potential (pp. 111-137). San Francisco, CA: Pfeiffer.

Training as an Intervention

Training is a critical component in the field of Human Performance Technology (HPT). Pershing (2006) defined HPT as “the study and ethical practice of improving productivity in organizations by designing and developing effective interventions that are results-oriented, comprehensive, and systemic” (p. 6). Unfortunately, training is often looked at as the end-all solution to correcting/fixing performance problems. This fallacy of ‘training can fix it’ has lead to a large number of training interventions, along with their expenditures, that have fallen short of promised expectations.

Training programs are often selected as a matter of convenience rather than as a planned systematic intervention with a specified goal or outcome.  At times it feels as though training programs are selected primarily, as Landis and Bhagat (1996) stated, "because they are well advertised, not because they are well designed" (as cited in Weber, S., 2008, p. 51).  Often times, trainers and those in charge of training, resort to already established training programs.  Rossett (2009) highlighted this problem by sounding the following alarm: “It is time for human resources and training professionals to turn from their habitually favored interventions, like training, to solutions that match the customer and situation” (p. 19).  Additionally, training is often considered to be a quick-fix to a misunderstood performance problem. Rothwell, Stavros, and Sullivan (2010) identified the problem where quick-fix solutions are inappropriate in resolving the source of the problem as “employee training is often inappropriately perceived to be” (OD defined).

HPT takes a counter-intuitive approach when dealing with training. Training is often considered the last step in the problem resolution intervention. Thus, training should only be applied in those instances where no other cheaper and less timely intervention will work. Robinson and Robison (1995) highlight: “solutions to performance problems should be based upon a thorough analysis of causes of the problem” (p. 4). The point here is not that training is ineffective, but that training needs to address the right performance objective for it to be effective.

Training needs analyses, or performance analyses, should be conducted before any training intervention is designed.  These analyses are conducted to determine what is termed the performance gap.  This gap is the difference between what should be to what is.  Rummler and Brache (1995) view process maps that compare processes in their current state (the ‘as-is’ state) compared to how these should be (the ‘should-be’ state). The differences between the should-be and the as-is represents the performance or process gap - this is where the resolution needs to be directed.    Robinson and Robinson (1995) use a performance relationship map to determine the needs for four key performance drivers: business needs, performance needs, training needs, and work environment needs. Their performance relationship map distinguishes between the type of performance that should be demonstrated with those that is being demonstrated. The difference is the performance gap that needs to be attended to. 

When the identified performance gap identifies a deficiency in employees knowledge and skills in which new knowledge could resolve the gap then training could be a final resolving intervention.  Robinson and Robinson (1995) identified training needs as those “areas where performers lack skill or knowledge to perform satisfactorily” (p. 26). If the performance gap identifies a deficiency in either of the other three drivers (business needs, performance needs, work environment needs) then training is probably not the best intervention to resolve the performance gap.  

Selecting training as an intervention when the performance deficiency relates to employees knowledge and skills will most certainly guide you on the path to resolving the performance gap.  Selecting training as an intervention when the source of the problem is not related to knowledge or skills, or when the problem has not been identified through need analysis, will most likely lead to waisted expenditures and waisted effort from those involved.  Additionally, improperly selecting training could potentially decrease performance while lowering the motivation of the employees in the long-term. 

References:

Pershing, J. A. (2006). Human performance technology fundamentals. In Pershing, J. A. (Ed.), Handbook of human performance technology: Principles, practices, potential. San Francisco, CA: Pfeiffer.

Robinson, D. G., & Robinson, J. C. (1995). Performance consulting: Moving beyond training. San Francisco, CA: Berrett-Koehler Publishers.

Rothwell, W. J., Stavros, J. M., & Sullivan, A. (2009). Organization development and change. In Rothwell, W. J., Stavros, J.M., & Sullivan, A. (Eds.). Practicing organization development: A guide for leading change (3^rd ed.) (Chapter 1). San Francisco, CA: Pfeiffer.

Rossett, A. (2009). First things fast: A handbook of performance analysis (2^nd ed.). San Francisco, CA: Pfeiffer.

Rummler, G. A., Brache, A. P. (1995). Improving performance: How to manage the white space on the organization chart (2^nd ed.). San Francisco, CA: John Wiley & Sons.

Weber, S. (2008).  Intercultural learning in business and human resource education. In Nijhof, W. J., & Nieuwenhuis, L. F.M. (Eds.), The learning potential of the workplace (pp. 47 - 69). Rotterdam, The Netherlands: Sense Publishers.

Human Performance Technology (HPT) Definitions & Assumptions

In 2004 a Presidential Task Force of ISPI identified Human Performance Technology (HPT) as "an integrated systems approach to improving human performance" (ISPI, 2004, p. 6).  As the editor of the third edition of The Handbook of Human Performance Technology, Pershing (2006) provided the following definition of HPT in his chapter titled "Human Performance Technology Fundamentals": "Human performance technology is the study and ethical practice of improving productivity in organizations by designing and developing effective interventions that are results-oriented, comprehensive, and systemic" (p. 6).

This improvement in performance can be either in the improvement in human performance of individual workers, the improvement of human performance through the design/operation of processes, and/or the improvement in human performance through organizational initiatives and environmental / societal endeavors.  In each case HPT provides a systematic approach to provide a comprehensive performance improvement initiative, not short-term, with a results-oriented evaluative framework.

The Presidential Task Force (2004) identified the following criteria to judge whether an initiative is an HPT initiative:

1. Is focused on valuable, measured results;
2. Considers the larger system context of people's performance;
3. Provides valid and reliable measures of the effectiveness of those applications;
4. Clearly describes applications grounded in prior research or empirical evidence (or are not discouraged by either one) so that they may be replicated under the conditions and by the means for which they were recommended (p. 6).

HPT operates on the following assumptions, as outlined in the Presidential Task Force (2004):

1. A technology is a set of empirical and scientific principles and their application
2. Human performance technology is the technology concerned with all variables which impact human performance
3. All organizational processes and practices impact the production of valued results, whether positively or negatively and whether those results go measured or unmeasured, acknowledged or not.  (Everything that an organization does affects what it accomplishes, whether or not the results are acknowledged or desirable.)
4. The purpose of all organizations is the same: to create value for their stakeholders; this is accomplished by aligning all processes, practices, and resources to maximize the production of that value.
5. We collaborate with and value the expertise of other disciplines; human performance technology becomes the integrator and multiplier. (p. 6)

The presidential Task Force report provides great information regarding the HPT Framework and is worth downloading and adding to your files - ISPI Task Force Report.  I would like to thank Guy Wallace for forwarding this link to me via Linkedin.  Guy is the President at EPPIC, Inc. and you can view his web page at http://eppic.biz/ for resources relating to instructional design and performance improvement.

References:

ISPI Presidential Initiative Task Force - Stage 1 (2004).  Retrieved from http://humanperformancetechnology.files.wordpress.com/2010/07/task-force-final-report-march-31-2004.pdf

Pershng, J. (2006).  Human Performance Technology Fundamentals.  In Pershing, J. (Ed.), Handbook of Human Performance Technology: Principles, Practices, Potential, (pp. 5-34).  San Francisco, CA: Pfeiffer.