Evaluation

Evaluation is often at the end of systematic performance models.  The Instructional Systems Design (ISD) model, sometimes referred to the ADDIE model, includes the following stages: analysis, design, development, implementation, and evaluation.  Other disciplines practice models similar to the ADDIE model.  Six Sigma practices the DMAIC model: define, measure, analyze, improve, and control. In the DMAIC model measure would be comparable to evaluation.  Human Resource Development (HRD) practices analyze, propose, create, Implement, and assess.  In the HRD model, assess refers to assessment which is the same as evaluation in the ADDIE model.

These systematic performance models are often viewed as linear, stet-by-step, models.  By viewing these models this way they become ineffective at improving performance for the long-term.  Each model is presented to be cyclical and interactive.  This means that each model is designed as a continuous improvement cycle with dynamic interactions between each stage.  In the case of evaluation, this stage affects each of the other four stages in the process.  Evaluation begins during the initial analysis phase and continues through each stage, then re-cycles again, as improvements to the new improved cycle are incorporated.  Wang and Wilcox (2006) support this view indicating: “the larger view of evaluation may not be treated as a separate phase during the process…. It is indeed an ongoing effort throughout all phases of the ADDIE process and culminating at the last phase” (p. 528).

Shrock and Geis identified evaluation as a “process of collecting information and feeding it back to those who need the information so that the system can succeed” (as cited in Stolovitch & Keeps, 1999, p. 185).  Evaluation should be designed to provide feedback during each stage in the process so that improvements can be made to the process.

Evaluation comes in two forms: formative evaluation and summative evaluation.

Scriven (1991) identified formative evaluation to be used “to provide information on improving program design and development” (as cited in Wang & Wilcox, 2009, p. 529).  Wang and Wilcox identified that the purpose of formative evaluation was “to identify weakness in instructional material, methods, or learning objectives” (p. 529). Formative evaluation can be used to evaluate the instructional methods during a training program.

Following the training program summative evaluation will be used to determine the long-term effectiveness of the program and its instructional methods, including learning transfer.  Brown and Gerhardt (2002) described summative evaluation as those “efforts that assess the effectiveness of completed interventions in order to provide suggestions about their use” (p. 952).  A training program can be evaluated by its impact on the organization and its long-term effectiveness through summative evaluation.

A successful evaluation is one that utilizes both formative evaluation and summative evaluation.  Evaluation needs to be viewed as an iterative process that affects each of the stages in the training process that it is measuring.  Each systematic performance improvement endeavor needs to be addressed as a continuous improvement cycle with a strong emphasis on evaluation.  Evaluation is the key component that makes the systematic performance improvement process a continuous effort, allowing improvements to be made to the process during each stage.

REFERENCES
Brown, K. G. & Gerhardt, M. W. (2002).  Formative evaluation: An integrative practice model and case study.  Personnel Psychology, Vol. 55, pp. 951-983.

Stolovitch, H. D. & Keeps, E. J. (1999).  Handbook of human performance technology: Improving individual and organizational performance worldwide (2nd ed.).  San Francisco, CA: Jossey-Bass Pfeiffer.

Wang, G. G. & Wilcox, D. (2006).  Training evaluation: Knowing more than is practiced.  Advances in Developing Human Resources, Vol. 8, No. 4, pp. 528-539.

General Systems Theory (GST)

General Systems Theory (GST), commonly referred to Systems Theory, takes the view from the Gestalt Psychologists who "emphasizes that the whole is greater than the sum of its parts" (Matlin, 2005, p. 6).  Adopting GST provides a systematic framework for the field of Human Performance Technology (HPT).  Rosenberg, Coscarelli, and Hutchison described the importance of incorporating GST into HPT: "The use of systems, or the systems approach, is essential to HPT.  Without a systemic framework, it would be extremely difficult to achieve improved performance" (as cited in Stolovitch & Keeps, 1999, p. 25). 
GST
Swanson (2007) explained that GST provides a means of viewing the world as wholes rather than piece-by-piece.  Turner (2010) described GST as providing "the assumption of non-summativity where the whole is greater than the sum of its parts" (p. 16).  The advantage of incorporating systems theory to organizations, as in the approach that HPT takes, is "because of its potential usefulness in understanding the complexities of 'live' organizations" (Kast & Rosenzweig, 1972, p. 449).  Deming's system thinking emphasizes between external environments (customers) an internal processes (the inter-correlation of elements) (Wang, 2004). Deming’s systems thinking is designed to provide a proper fit "within the external environment, the dynamic equilibrium between internal processes and their environment should be maintained" (Wang, 2004, p. 395).
Components of GST
"GST provides a method of structuring organizations into different levels of performance" (Turner, 2010, p 16).  This structuring of organizations into different levels of performance have been adopted by a number of disciplines including HPT, Human Resource Development (HRD), and Organization Development (OD), to name a few.
Kast and Rosenzweig (1972) suggested that organization's need to address three levels of analysis: the environment, the social organization, and the level of the subsystems.  Swanson (2007) identified four levels of performance: organizational, process, work team, and job levels.  Rummler and Brache (1995) identifies three levels of performance: organization, process, and job performer levels.  These levels of analysis provides an organization to be broken down into its parts for further study.


References
Kast, F. E., & Rosenzweig, J. E. (1972).  General systems theory: Applications for organization and management.  The Academy of Management Journal, 15(4) ,447-465.
Matlin, M. W. (2005).  Cognition (6th ed.).  Hoboken, NJ: John Wiley & Sons.
Stolovitch, H. D. & Keeps, E. J. (Ed.), (1999).  Handbook of human performance technology: Improving individual and organizational performance worldwide (2nd ed.).  San Francisco, CA: Jossey-Bass Pfeiffer.
Swanson, R. A. (2007).  Analysis for improving performance: Tools for diagnosing organizations and documenting workplace expertise.  San Francisco, CA: Berett-Koehler.
Turner, J. R. (2010).  Using feed process as a means of performance improvement in a dynamic environment. Performance Improvement, Vol. 49, No. 8, pp. 15-20.  DOI: 10.1002/pfi

Visit slideshare: http://www.slideshare.net/JohnTurner5/metacogn-pres-2

Cognition & Metacognition related to HPT

In today's business environment jobs, tasks, and problem-solving activities are multi-faceted in the sense that they require activation of all the cognitive sets of knowledge (declarative knowledge, conceptual knowledge, procedural knowledge, conditional knowledge). Equally, they require activation of all the associated metacognitive functions. Ignoring one of the cognitive or metacognitive domains - during training, while assigning a task to a group, or transferring information across processes, can lead to a deficit in performance. Addressing each of the four knowledge domains and their associated metacognitive domains during these activities will assure that all required information is exchanged and that the desired outcome will be successful.

Research has indicated that performance improvement occurs when the metacognitive domains are addressed. This research also includes performance improvement at the individual, team, group, and organizational levels. Addressing the cognitive and metacognitive domains during performance improvement efforts could assist the field of Human Performance Technology (HPT).

Cognition

Cognition is a relatively new field in psychology that "describes the acquisition, storage, transformation, and use of knowledge" (Matlin, 2005, p. 2). Cognitive Psychology studies phenomena in the memory system such as: memory acquisition, memory storage and retrieval, short-term memory, working memory, long-term memory, and neuronal activity.

Matlin, M. W. (2005). Cognition (6th ed.). Danvers, MA: John Wiley & Sons.

Metacognition

Metacognition can be thought of as a developing skill, which helps one to become aware of her cognitive activities and to reflect on the outcomes from these cognitive activities. This awareness and reflection about one's cognitive activities help develop new theories about her metacognitive abilities.

Metacognition relates to one's understanding of her knowledge and the ability to use that knowledge. Kuhn (2000) defined metacognition as "cognition that reflects on, monitors, or regulates first-order cognition" (p. 178).

Kuhn, D. (2000). Metacognitive development. Current directions in psychological science, Vol. 9, No. 5, pp. 178-181.

Pintrich (2002) referred to metacognitive knowledge as "knowledge about cognition in general, as well as awareness of and knowledge about one's own cognition" (p. 219).

Pintrich, P. R. (2002). The role of metacognitive knowledge in learning, teaching, and assessing. Theory into practice, Vol. 41, No. 4, pp. 219-225.

Metacognition is a cyclical activity where the cognitive functions are reflected upon and monitored by the metacognitive functions. These same metacognitive functions also change based on the outcome of the cognitive functions.

Understanding one's metacognitive abilities provides for better learning and improved task or goal achievement. Having a clear understanding of one's metacognitive abilities allows a person to adapt better to new circumstances, thus providing for improved performance on the job, as well as in personal settings.

How do you use metacognitive strategies?  What metacognitive strategies have worked for you?
Below are a few examples of metacognitive strategies that could be used:
• Problem-solving strategies.
• Mental models.
• Knowledge about self – recalling existing knowledge.
• Ability to incorporate self-assessment techniques.
• Feedback and feedforward processes.
• Concept-mapping techniques.
• Questioning the accuracy of understanding: either at the individual level or at the team/group level.
• Having a clear understanding of the task/goal among all parties involved.
• Having a clear understanding of the structure you are operating in.
• The process of how learning is shared among members, either collective learning or shared team learning.
• Information distribution available to all members during the project or task.
• Having a shared interpretation of required information to complete a task among all members.

Visit Slideshare

Performance Analysis Defined

Performance analysis is a methodical procedure designed to best identify causes to performance deficits. Performance analysis procedures can be applied to the individual, process, or organizational levels of performance. Performance analysis is the first step in determining what course of action needs to take place to address a specific problem or performance deficit. This first step is often called front-end analysis and is a pre-curser to the more in-depth needs-assessment and training development stages. The performance analysis helps to identify if a needs assessment or a training initiative is necessary, since these are exhausting and expensive interventions to perform. Additionally, the performance analysis will identify if another intervention could resolve the performance deficit or problem being addressed, other than a training intervention, which could reduce time and costs in many instances.

Performance Analysis Defined

Pershing described that performance analysis "focuses on factors that drive individual, group, and organizational performance. The factors may be causes of problems, road maps to improve quality, or ways to exploit opportunities. They are the why questions" (as cited in Pershing, 2006, p. 18).

Pershing, J. A., (ED, 2006). Handbook of performance technology: Principles, practices, potential (3rd ed.). San Francisco, CA: Pfeiffer.

Rossett reminds us that analysis provides the foundation for HPT, "a profession and a perspective that demands study before recommendations, data before decisions and involvement before actions" (p. 139).  Analysis can be defined as having three goals: 1) to find out what is going on and to disseminate and verify those perceptions, 2) to involve key figures and data in the effort, and 3) to model and employ a systematic process for improving human performance (Rossett, pp. 143-149, as cited in Pershing, 2006).

In her latest book titled First Things Fast, Rossett (2009) defines performance analysis in the following manner: "Performance analysis (PA) is partnering with clients and customers to help them define and achieve their goals. PA involves reaching out for several perspectives on a problem or opportunity; determining any and all drivers toward or barriers to successful performance; and proposing a solution system based on what is learned, not on what is typically done" (p. 20).

Rossett, A. (2009). First things fast: A handbook for performance analysis (2nd ed.). San Francisco, CA: John Wiley & Sons.

Organization Development (OD) looks at performance analysis as the first step in identifying the need for a change initiative. OD requires "an initial analysis be done of the performance requirements of the organization that can be improved through the documenting and development of planned, systemic change and the development of human expertise required to implement, maintain, and sustain workplace change and performance" (Swanson & Holton, 2001, p. 278).

Swanson, R. A., Holton III, E. F. (2001). Foundations of human resource development. San Francisco, CA: Berrett-Koehler.

Human Performance Technology (HPT)

HPT is a multi-disciplinary practice which has included as its' core behavioral psychology and systems theory. HPT has also been influenced, and remain in support of, a number of other cognate disciplines such as economics, education, organization development (OD), management sciences, human resource development (HRD), training and development (T&D), cognitive psychology, etc...

Irlbeck (2002) analyzed the historical definitions of HPT and identified three recurring themes; 1) systems approach, 2) references to performance, and 3) a focus on individuals, groups, and organizations. HPT and the International Society of Performance Improvement (ISPI), the publication for HPT, "maintain reference to the human side of the equation by alluding to individuals, groups, and/or organizations" (p. 93).

Irlbeck, S. A. (2002). Human performance technology: An examination of definitions through dependent and independent variables. Performance Improvement Quarterly, Vol. 15, No. 2, pp. 84-95.

Human Performance Technology (HPT) is best defined as: "a systematic approach to improving productivity and competence, uses a set of methods and procedures - and a strategy for solving problems - for realizing opportunities related to the performance of people" (ispi.org, retrieved 10/23/2010).

Technorati:USY63R93XKUE