Evaluation

Evaluation is often at the end of systematic performance models.  The Instructional Systems Design (ISD) model, sometimes referred to the ADDIE model, includes the following stages: analysis, design, development, implementation, and evaluation.  Other disciplines practice models similar to the ADDIE model.  Six Sigma practices the DMAIC model: define, measure, analyze, improve, and control. In the DMAIC model measure would be comparable to evaluation.  Human Resource Development (HRD) practices analyze, propose, create, Implement, and assess.  In the HRD model, assess refers to assessment which is the same as evaluation in the ADDIE model. 

These systematic performance models are often viewed as linear, stet-by-step, models.  By viewing these models this way they become ineffective at improving performance for the long-term.  Each model is presented to be cyclical and interactive.  This means that each model is designed as a continuous improvement cycle with dynamic interactions between each stage.  In the case of evaluation, this stage affects each of the other four stages in the process.  Evaluation begins during the initial analysis phase and continues through each stage, then re-cycles again, as improvements to the new improved cycle are incorporated.  Wang and Wilcox (2006) support this view indicating: “the larger view of evaluation may not be treated as a separate phase during the process…. It is indeed an ongoing effort throughout all phases of the ADDIE process and culminating at the last phase” (p. 528).

Shrock and Geis identified evaluation as a “process of collecting information and feeding it back to those who need the information so that the system can succeed” (as cited in Stolovitch & Keeps, 1999, p. 185).  Evaluation should be designed to provide feedback during each stage in the process so that improvements can be made to the process.

Evaluation comes in two forms: formative evaluation and summative evaluation.

Scriven (1991) identified formative evaluation to be used “to provide information on improving program design and development” (as cited in Wang & Wilcox, 2009, p. 529).  Wang and Wilcox identified that the purpose of formative evaluation was “to identify weakness in instructional material, methods, or learning objectives” (p. 529). Formative evaluation can be used to evaluate the instructional methods during a training program. 

Following the training program summative evaluation will be used to determine the long-term effectiveness of the program and its instructional methods, including learning transfer.  Brown and Gerhardt (2002) described summative evaluation as those “efforts that assess the effectiveness of completed interventions in order to provide suggestions about their use” (p. 952).  A training program can be evaluated by its impact on the organization and its long-term effectiveness through summative evaluation.

A successful evaluation is one that utilizes both formative evaluation and summative evaluation.  Evaluation needs to be viewed as an iterative process that affects each of the stages in the training process that it is measuring.  Each systematic performance improvement endeavor needs to be addressed as a continuous improvement cycle with a strong emphasis on evaluation.  Evaluation is the key component that makes the systematic performance improvement process a continuous effort, allowing improvements to be made to the process during each stage.

REFERENCES

Brown, K. G. & Gerhardt, M. W. (2002).  Formative evaluation: An integrative practice model and case study.  Personnel Psychology, Vol. 55, pp. 951-983.

Stolovitch, H. D. & Keeps, E. J. (1999).  Handbook of human performance technology: Improving individual and organizational performance worldwide (2^nd ed.).  San Francisco, CA: Jossey-Bass Pfeiffer.

Wang, G. G. & Wilcox, D. (2006).  Training evaluation: Knowing more than is practiced.  Advances in Developing Human Resources, Vol. 8, No. 4, pp. 528-539.

Training as an Intervention

Training is a critical component in the field of Human Performance Technology (HPT). Pershing (2006) defined HPT as “the study and ethical practice of improving productivity in organizations by designing and developing effective interventions that are results-oriented, comprehensive, and systemic” (p. 6). Unfortunately, training is often looked at as the end-all solution to correcting/fixing performance problems. This fallacy of ‘training can fix it’ has lead to a large number of training interventions, along with their expenditures, that have fallen short of promised expectations.

Training programs are often selected as a matter of convenience rather than as a planned systematic intervention with a specified goal or outcome.  At times it feels as though training programs are selected primarily, as Landis and Bhagat (1996) stated, "because they are well advertised, not because they are well designed" (as cited in Weber, S., 2008, p. 51).  Often times, trainers and those in charge of training, resort to already established training programs.  Rossett (2009) highlighted this problem by sounding the following alarm: “It is time for human resources and training professionals to turn from their habitually favored interventions, like training, to solutions that match the customer and situation” (p. 19).  Additionally, training is often considered to be a quick-fix to a misunderstood performance problem. Rothwell, Stavros, and Sullivan (2010) identified the problem where quick-fix solutions are inappropriate in resolving the source of the problem as “employee training is often inappropriately perceived to be” (OD defined).

HPT takes a counter-intuitive approach when dealing with training. Training is often considered the last step in the problem resolution intervention. Thus, training should only be applied in those instances where no other cheaper and less timely intervention will work. Robinson and Robison (1995) highlight: “solutions to performance problems should be based upon a thorough analysis of causes of the problem” (p. 4). The point here is not that training is ineffective, but that training needs to address the right performance objective for it to be effective.

Training needs analyses, or performance analyses, should be conducted before any training intervention is designed.  These analyses are conducted to determine what is termed the performance gap.  This gap is the difference between what should be to what is.  Rummler and Brache (1995) view process maps that compare processes in their current state (the ‘as-is’ state) compared to how these should be (the ‘should-be’ state). The differences between the should-be and the as-is represents the performance or process gap - this is where the resolution needs to be directed.    Robinson and Robinson (1995) use a performance relationship map to determine the needs for four key performance drivers: business needs, performance needs, training needs, and work environment needs. Their performance relationship map distinguishes between the type of performance that should be demonstrated with those that is being demonstrated. The difference is the performance gap that needs to be attended to. 

When the identified performance gap identifies a deficiency in employees knowledge and skills in which new knowledge could resolve the gap then training could be a final resolving intervention.  Robinson and Robinson (1995) identified training needs as those “areas where performers lack skill or knowledge to perform satisfactorily” (p. 26). If the performance gap identifies a deficiency in either of the other three drivers (business needs, performance needs, work environment needs) then training is probably not the best intervention to resolve the performance gap.  

Selecting training as an intervention when the performance deficiency relates to employees knowledge and skills will most certainly guide you on the path to resolving the performance gap.  Selecting training as an intervention when the source of the problem is not related to knowledge or skills, or when the problem has not been identified through need analysis, will most likely lead to waisted expenditures and waisted effort from those involved.  Additionally, improperly selecting training could potentially decrease performance while lowering the motivation of the employees in the long-term. 

References:

Pershing, J. A. (2006). Human performance technology fundamentals. In Pershing, J. A. (Ed.), Handbook of human performance technology: Principles, practices, potential. San Francisco, CA: Pfeiffer.

Robinson, D. G., & Robinson, J. C. (1995). Performance consulting: Moving beyond training. San Francisco, CA: Berrett-Koehler Publishers.

Rothwell, W. J., Stavros, J. M., & Sullivan, A. (2009). Organization development and change. In Rothwell, W. J., Stavros, J.M., & Sullivan, A. (Eds.). Practicing organization development: A guide for leading change (3^rd ed.) (Chapter 1). San Francisco, CA: Pfeiffer.

Rossett, A. (2009). First things fast: A handbook of performance analysis (2^nd ed.). San Francisco, CA: Pfeiffer.

Rummler, G. A., Brache, A. P. (1995). Improving performance: How to manage the white space on the organization chart (2^nd ed.). San Francisco, CA: John Wiley & Sons.

Weber, S. (2008).  Intercultural learning in business and human resource education. In Nijhof, W. J., & Nieuwenhuis, L. F.M. (Eds.), The learning potential of the workplace (pp. 47 - 69). Rotterdam, The Netherlands: Sense Publishers.

Human Performance Technology Principles (HPT)

Listed below are a set of 10 Principles of Human Performance Technology (HPT) as presented by Jacobs in 1987 (Gilley, Dean, Bierema, 2001, pp. 80-91). These principles provide a guide for those who are interested in providing systematic performance related solutions for their organization. Practitioners, managers, and today's corporate leaders can all benefit by keeping these principles in the back of their minds.

1. Human performance and human behavior are different, and knowledge of their differences is important for achieving goals.
2. Any statement about human performance is about organizational performance as well.
3. Costs of improving performance should be regarded as investments in human capital, yielding returns in the form of increased performance potential.
4. Organizational and individual goals must be considered to define worthy performance.
5. Knowing how to engineer human performance and the conditions that affect it is as important as explaining why the behavior occurred.
6. Diagnosing problems required analysis of the present system and examination of differences between it and an ideal system. Avoiding anticipated problems requires analyzing the planned system and modifying it to approximate the ideal.
7. Exemplary performance provides the most logical reference for establishing job performance standards.
8. Human performance problems have differing root causes that originate either from the person, from something in the environment, or from both.
9. The performance of one subsystem affects the performance of other subsystems in somewhat predictable ways, requiring that root causes be analyzed at more than one level of the organization.
10. Many different solutions may be used to improve human performance. Selection of any one solution is dependent on the cause and nature of the performance problem, and the criteria used to evaluate a solution must include its potential to make a measurable difference in the performance system.

(Gilley, Dean, Bierema, 2001, pp. 80-91)

Reference

Gilley, J. W. (2001). Philosophy of organizational performance. In Gilley, J. W., Dean, P., & Bierema, L. (Eds.), Philosophy and practice of organizational learning, performance, and change: New perspectives in organizational learning, performance, and change, pp. 67-92. Cambridge, MA: Perseus.