Training Evaluation Misses the Critical Thinking Dimension

For typical training courses the following four or five evaluation steps are the norm:

Level 1: Reaction and Perceived Values

Measures reaction to, and satisfaction with, the medium, content, and value of the project or program.

Level 2: Learning and Confidence

Measures what participants understand or learned from the project or program (information, knowledge, skills, and contacts).

Level 3: Application and Implementation

Measures what participants understand or learned from the project or program (information, knowledge, skills, and contacts).

Level 4: Impact and Consequences

Measures progress after the program implemented (the use of information, knowledge, skills, and contacts).

Level 5: ROI

Monetary Benefits.

(Phillips, & Phillips, 2007)

Evaluation is further divided into formative and summative evaluation, with formative evaluation relating to evaluating the training program and summative evaluation relating to the long-term effects of the training program (see Evaluation). Level 2, Learning and Confidence, is more of a formative evaluation measure and identifies whether trainees learned the material presented in the training. Level 3, application and consequences, is categorized as summative evaluation in which it looks at whether the trainees on-the-job behavior represents their learning from the training program.  

For these two specific levels of evaluation Phillips and Phillips (2007) highlighted information, knowledge, skills, and contacts. Providing employees (trainees) with the information that they need to conduct their job functions is critical. Additionally, providing employees with the knowledge to utilize this information in a productive manner is key to success. Having the skills to perform one's job is self-explanatory, but, as experience has shown us, people often lack the proper skills required to perform their main job function. Having the right contacts as well as knowing who has the information when needed is equally important. 

Using the following variables to evaluate training programs (information, knowledge, skills, and contacts) have proven to be effective for years. However, expanding on these variables to improve the evaluation process follows the continuous improvement process. As an effort to expand on the accuracy of training I would pose adding critical thinking to the mix.

Training employees to think critically helps to eliminate issues such as functional fixedness and mental sets. Ollinger, Jones, and Knoblich (2008) termed mental set as: " the repeated application of a successful method makes blind any alternative approach, because of the mechanization of the particular solution method" (p. 270). Alternatively, Duncker (1945) identified functional fixedness as: "the tendency to fixate on the typical use of an object or one of its parts" (as cited in McCaffrey, 2012, p. 216).

Adding the dimension of critical thinking to training endeavors will help transform learners (trainees, employees) to effective learners. Brindley, Walti, and Blaschke (2009) identified effective learners as those who are capable of coping with "complexity, contradictions, and large quantities of information, who seek out various sources of knowledge" (p. 3). By seeking out new sources of knowledge employees will better avoid the aforementioned traps of mental sets and functional fixedness. 

Including critical thinking skills as part of the training program, as well as incorporating evaluation of employees critical thinking skills on-the-job, could prove to produce better training results and on-the-job performance results. Additionally, including critical thinking in both the instructional and evaluation phases of the training program could improve both the formative and summative evaluations of the overall program. 

References:

Brindley, Walti, & Blaschke (2009). Creating effective collaborative learning groups in an online environment. International Review of Research in Open and Distance Learning, 10(3), 1-18. 

McCaffrey, T. (2012). Innovation relies on the obscure: A key to overcoming the classic problem of functional fixedness. Psychological Science, 23(3), 215-218. dpi: 10.1177/0956797611429580

Ollinger, Jones, & Knoblich (2008). Investigating the effect of mental set in insight problem solving. Experimental Psychology, 55(4), 269-282. dpi: 10.1027/1618-3169.55.4.269

Phillips, & Phillips (2007). Show me the money: The use of ROI in performance improvement, part 1. Performance Improvement 46(9), 8-22. dpi: 10.1002/pfi.160

Evaluation

Evaluation is often at the end of systematic performance models.  The Instructional Systems Design (ISD) model, sometimes referred to the ADDIE model, includes the following stages: analysis, design, development, implementation, and evaluation.  Other disciplines practice models similar to the ADDIE model.  Six Sigma practices the DMAIC model: define, measure, analyze, improve, and control. In the DMAIC model measure would be comparable to evaluation.  Human Resource Development (HRD) practices analyze, propose, create, Implement, and assess.  In the HRD model, assess refers to assessment which is the same as evaluation in the ADDIE model. 

These systematic performance models are often viewed as linear, stet-by-step, models.  By viewing these models this way they become ineffective at improving performance for the long-term.  Each model is presented to be cyclical and interactive.  This means that each model is designed as a continuous improvement cycle with dynamic interactions between each stage.  In the case of evaluation, this stage affects each of the other four stages in the process.  Evaluation begins during the initial analysis phase and continues through each stage, then re-cycles again, as improvements to the new improved cycle are incorporated.  Wang and Wilcox (2006) support this view indicating: “the larger view of evaluation may not be treated as a separate phase during the process…. It is indeed an ongoing effort throughout all phases of the ADDIE process and culminating at the last phase” (p. 528).

Shrock and Geis identified evaluation as a “process of collecting information and feeding it back to those who need the information so that the system can succeed” (as cited in Stolovitch & Keeps, 1999, p. 185).  Evaluation should be designed to provide feedback during each stage in the process so that improvements can be made to the process.

Evaluation comes in two forms: formative evaluation and summative evaluation.

Scriven (1991) identified formative evaluation to be used “to provide information on improving program design and development” (as cited in Wang & Wilcox, 2009, p. 529).  Wang and Wilcox identified that the purpose of formative evaluation was “to identify weakness in instructional material, methods, or learning objectives” (p. 529). Formative evaluation can be used to evaluate the instructional methods during a training program. 

Following the training program summative evaluation will be used to determine the long-term effectiveness of the program and its instructional methods, including learning transfer.  Brown and Gerhardt (2002) described summative evaluation as those “efforts that assess the effectiveness of completed interventions in order to provide suggestions about their use” (p. 952).  A training program can be evaluated by its impact on the organization and its long-term effectiveness through summative evaluation.

A successful evaluation is one that utilizes both formative evaluation and summative evaluation.  Evaluation needs to be viewed as an iterative process that affects each of the stages in the training process that it is measuring.  Each systematic performance improvement endeavor needs to be addressed as a continuous improvement cycle with a strong emphasis on evaluation.  Evaluation is the key component that makes the systematic performance improvement process a continuous effort, allowing improvements to be made to the process during each stage.

REFERENCES

Brown, K. G. & Gerhardt, M. W. (2002).  Formative evaluation: An integrative practice model and case study.  Personnel Psychology, Vol. 55, pp. 951-983.

Stolovitch, H. D. & Keeps, E. J. (1999).  Handbook of human performance technology: Improving individual and organizational performance worldwide (2^nd ed.).  San Francisco, CA: Jossey-Bass Pfeiffer.

Wang, G. G. & Wilcox, D. (2006).  Training evaluation: Knowing more than is practiced.  Advances in Developing Human Resources, Vol. 8, No. 4, pp. 528-539.