Evaluation

Evaluation is often at the end of systematic performance models.  The Instructional Systems Design (ISD) model, sometimes referred to the ADDIE model, includes the following stages: analysis, design, development, implementation, and evaluation.  Other disciplines practice models similar to the ADDIE model.  Six Sigma practices the DMAIC model: define, measure, analyze, improve, and control. In the DMAIC model measure would be comparable to evaluation.  Human Resource Development (HRD) practices analyze, propose, create, Implement, and assess.  In the HRD model, assess refers to assessment which is the same as evaluation in the ADDIE model. 

These systematic performance models are often viewed as linear, stet-by-step, models.  By viewing these models this way they become ineffective at improving performance for the long-term.  Each model is presented to be cyclical and interactive.  This means that each model is designed as a continuous improvement cycle with dynamic interactions between each stage.  In the case of evaluation, this stage affects each of the other four stages in the process.  Evaluation begins during the initial analysis phase and continues through each stage, then re-cycles again, as improvements to the new improved cycle are incorporated.  Wang and Wilcox (2006) support this view indicating: “the larger view of evaluation may not be treated as a separate phase during the process…. It is indeed an ongoing effort throughout all phases of the ADDIE process and culminating at the last phase” (p. 528).

Shrock and Geis identified evaluation as a “process of collecting information and feeding it back to those who need the information so that the system can succeed” (as cited in Stolovitch & Keeps, 1999, p. 185).  Evaluation should be designed to provide feedback during each stage in the process so that improvements can be made to the process.

Evaluation comes in two forms: formative evaluation and summative evaluation.

Scriven (1991) identified formative evaluation to be used “to provide information on improving program design and development” (as cited in Wang & Wilcox, 2009, p. 529).  Wang and Wilcox identified that the purpose of formative evaluation was “to identify weakness in instructional material, methods, or learning objectives” (p. 529). Formative evaluation can be used to evaluate the instructional methods during a training program. 

Following the training program summative evaluation will be used to determine the long-term effectiveness of the program and its instructional methods, including learning transfer.  Brown and Gerhardt (2002) described summative evaluation as those “efforts that assess the effectiveness of completed interventions in order to provide suggestions about their use” (p. 952).  A training program can be evaluated by its impact on the organization and its long-term effectiveness through summative evaluation.

A successful evaluation is one that utilizes both formative evaluation and summative evaluation.  Evaluation needs to be viewed as an iterative process that affects each of the stages in the training process that it is measuring.  Each systematic performance improvement endeavor needs to be addressed as a continuous improvement cycle with a strong emphasis on evaluation.  Evaluation is the key component that makes the systematic performance improvement process a continuous effort, allowing improvements to be made to the process during each stage.

REFERENCES

Brown, K. G. & Gerhardt, M. W. (2002).  Formative evaluation: An integrative practice model and case study.  Personnel Psychology, Vol. 55, pp. 951-983.

Stolovitch, H. D. & Keeps, E. J. (1999).  Handbook of human performance technology: Improving individual and organizational performance worldwide (2^nd ed.).  San Francisco, CA: Jossey-Bass Pfeiffer.

Wang, G. G. & Wilcox, D. (2006).  Training evaluation: Knowing more than is practiced.  Advances in Developing Human Resources, Vol. 8, No. 4, pp. 528-539.

HPT Standards & Systemic Issues

HPT has developed ten standards when dealing with systemic issues.  The first four standards are the core HPT standards, while the remaining six standards deal more specifically to systemic issues.  The first four standards must be met before any of the other six standards can be accomplished.  Without completing the first four standards first, you will be unable to accomplish a fully systemic resolution (Brethower, 2006). 


HPT's ten standards for dealing with systemic issues:
1) Focus on results and help clients focus on results.
2) Look at situations systemically, taking into consideration the larger context, including competing pressures, resource constraints, and anticipated change.
3) Add value in how you do the work and through the work itself.
4) Utilize partnerships or collaborate with clients and other experts as required (Brethower, 2006, p. 112).


Be systematic in all aspects of the process, including the:
5) assessment of the need or opportunity.
6) analysis of the work and workplace to identify the cause or factors that limit performance.
7) design of the solution or specification of the requirements of the solution.
8) development of all or some of the solution and its elements.
9) implementation of the solution.
10) evaluation of the process and the results (Brethower, 2006, p. 113).


Look closely at the standards for steps 5 through 10.  Do they look familiar? These steps include what is typically known as a needs assessment followed by the ADDIE process.  The ADDIE process is an acronym for Analysis, Design, Develop, Implementation, and Evaluation.  One additional item that could be added would be feedback.  Standard #11 could include a feedback loop to each process, thus making the process a continuous improvement process.


Reference:
Brethower, D. M. (2006). Systemic issues.  In Pershing, J. A. (Ed.), Handbook of human performance technology: Principles, practices, potential (pp. 111-137). San Francisco, CA: Pfeiffer.

SlideShare uses and Resources for HPT

SlideShare is “the world’s largest community for sharing presentation.  With 55 million monthly visitors and 120 million pageviews, it is amongst the most visited 200 websites in the world” (http://www.slideshare.net/about).  SlideShare is a great way to save all your presentation slides in one location where you can access them with a common URL anytime you need access to them.  Additionally, each presentation can be either linked to or incorporated into a blog, a post, a web page, a report, etc…  Hart (2011) indicated that presentations could be:

• ·      Viewed at the site – where you can see status on usage
• ·      Individually embedded into a web or blog page
• ·      Embedded as a SlideShare Playlist (p. 154).

Bozarth (2010) highlights that SlideShare is useful for trainers who want to distribute either a powerpoint slide presentation or a pdf file via Facebook, Twitter, or any other social networking application.

In my blog post on August 26, 2011, Metacognitive Model for HPT.  I included a link to the pdf slide presentation that I used to report my journal article.  There were two versions of this pdf slide presentation.  The long version, which can be found at http://www.slideshare.net/JohnTurner5/metacognition-pres and the short version, which can be found at http://www.slideshare.net/JohnTurner5/metacogn-pres-2

Additionally, in my blog post on May 25, 2011, Complexity Leadership Theory, I also included a link to the powerpoint slide presenation for this post, http://www.slideshare.net/JohnTurner5/complex-ldrtheory-8097228  Other presentations can be found at my SlideShare URL.

One effective use of adding SlideShare presentations to a web page, or a blog page, can be found at Harold Jarche’s web page, Life in Perpetual Beta.  Jarche has a separate tab for all of his SlideShare presentations incorporated into his web page.  These presentations can be found at http://www.slideshare.net/jarche  Another use of incorporating SlideShare presentations in a blog can be found at Clive Shepherds’ blog page, Clive on Learning.  Clive add a link to his SlideShare presentations which can be found at http://www.slideshare.net/clives.

Another effective use of incorporating SlideShare into your web page or blog is to include the actual presentation into the post.  One example of this can be found at Jane Hart’s blog, From “Command & Control” to “Encourage & Engage”: the presentation.

When it comes to developing slides there is no better source than Garr Reynolds’ blog on issues related to professional presentation design.  Garr provides examples and tips on how to develop productive and attractive slides for a memorable presentation.  Additional tips and examples can be found on Garr’s blog, Garr’s posterous. 

References:

Bozarth, J. (2010). Social media for trainers: Techniques for enhancing and extending learning.  San Francisco, CA: John Wiley & Sons.

Hart, J. (2011). Social learning handbook. Center for Learning & Performance Technologies, C4LPT.

Blogs and Wikis relating to HPT

Many Web2.0 platforms available today provide opportunities for businesses to obtain and retain technical information.  The transition from Web1.0 to Web2.0 includes the implementation of interactive applications.  Hart (2011) distinguished Web1.0 as read-only web and Web2.0 as read-write-web.  Some of these applications include Facebook, Twitter, Linkedin, Myspace, blogs, wikis, and YouTube, to name only a few.  I have concentrated on two of these Web2.0 platforms, blogs and wikis, as they relate to the Human Performance Technology (HPT) field.

HPT related blogs

Numerous blogs are content driven, meaning their written for the sole purpose of publishing content related to a specific discipline.  A blog, as described by Bozarth (2010), is “an online space for posting chronologically ordered comments or ideas that can include text, photo, video, audio, and links to other sites, blogs, or documents” (p. 83).  McAfee (2009) highlighted blogs as being easy to update, easy to view or reference as a separate web page, and are permanent records.   

I searched for five blogs relating to the HPT discipline.  One such blog was Donald Clark’s Big Dog, Little Dog blog, designed to post content related to instructional design and performance (http://bdld.blogspot.com/).  A second blog is Jane Bozarth’s blog titled bozarthzone, which covers a variety of topics relating to e-learning and social media (http://bozarthzone.blogspot.com/).  A third informative blog is Donald H. Taylor’s blog dedicated to performance, learning, human capital, and talent management (http://donaldhtaylor.wordpress.com/).  A fourth blog is Harold Jarche’s blog with the mantra of ‘Life in Perpetual Beta’.  This blog is dedicated to informal learning principles in the workplace (http://www.jarche.com/).  The final blog researched was Mitch Joel’s blog titled Six Pixels of Separation – The Blog which is dedicated to marketing but covers a vast amount of information relating to social networking (http://www.twistimage.com/blog/).

One interesting and effective twist to blogs is a blog carnival.  Blog carnivals are where bloggers interested in a common content area pool together and post monthly, or quarterly, posts on a specific topic (check out my blogpost on blog carnivals at http://johnrturnerhptresource.blogspot.com/2011/07/blog-carnivals.html).  The host for a blog carnival varies from month to month.  This is a great way to find other bloggers interested in the same content and it adds a new way to get your content out to different viewers (http://blogcarnival.com/bc/).

HPT related wikis

Wikis provide a different type of interaction than blogs.  A wiki is a web page where everyone with access can change the content (Bozarth, 2010).  McAfee (2009) made the point that wikis have the ability to capture and spread knowledge as well as being an effective tool supporting strongly tied coworkers.  In the area for training and development, Bozarth (2010) identified 4 ways to employ a wiki:

1.     include learner-built development of a permanent, takeaway record of the particular course session;

2.     a record of the course over various iteration or offerings across time;

3.     a compilation of FAQs or good practices for those coming into the role that the training targets;

4.     or a single project aimed at improving overall company or work-unit operations (p. 109).

One of the most popular wikis is Wikipedia, an online encyclopedia (http://en.wikipedia.org/).  The content on Wikipedia can be changed by anyone by selecting the ‘edit’ button next to the written content on the page you wish to change.  The ‘discussion’ tab gives you a history of what changes were made to the content, when they were made, and by whom.  A second wiki is Psychology Wiki, which provides almost anything related to psychology: learning, cognition, personality, statistics, education, and industrial (http://psychology.wikia.com/).  A third wiki is called Human Science and provides knowledge from a human perspective.  Topics covered on this wiki include development, personality, management, and spirituality, to name a few (http://humanscience.wikia.com/wiki).  A fourth wiki is an index of wikis called WikiIndex (http://wikiindex.org/).  Here you can find other wikis, wiki communities, wiki ideas, and wiki people.  Another new twist to the original format of a wiki can be found at WikiAnswers.  WikiAnswers allows you to type in a question and it will give you a list of answers that have been provided by others (http://wiki.answers.com/).

Precautions

Security is always a concern with regards to corporate knowledge.  Prior to deciding which platform you want to utilize you will need to determine whether you want an open access platform or a restricted platform.  One thing to remember when deciding who you want to access these Web2.0 applications is that the more restricted you make a platform the less it takes on the features of a Web2.0 application.  The advantage of a Web2.0 application is the ability to have outsiders view the content so they can comment and add to the conversation.  If you restrict access this too much, you are defeating the purpose of using an open access platform.

Within a corporate setting specific policies and guidelines should be made available so that everyone understands what is an acceptable contribution and what is not an acceptable contribution.  It should be made clear that participants should respect the opinions of others and that all references to ‘other’ content should always be referenced or a link should be provided to the original content.  These policies and guidelines can be a working document, which can be added to as the Web2.0 platform becomes populated.  In most cases, these policies and guidelines will become self-regulated among the users of the Web2.0 platform.

  

References

Bozarth, J. (2010). Social media for trainers: Techniques for enhancing and extending learning.  San Francisco, CA: Pfeiffer.

Hart, J. (2011). Social learning handbook: A practical guide to using social media to work and learn smarter. Centre for Learning & Performance Technologies.

McAfee, A. (2009).  Enterprise 2.0: New collaborative tools for your organization’s toughest challenges.  Boston, MA: Harvard Business Press.

Human Performance Technology Principles (HPT)

Listed below are a set of 10 Principles of Human Performance Technology (HPT) as presented by Jacobs in 1987 (Gilley, Dean, Bierema, 2001, pp. 80-91). These principles provide a guide for those who are interested in providing systematic performance related solutions for their organization. Practitioners, managers, and today's corporate leaders can all benefit by keeping these principles in the back of their minds.

1. Human performance and human behavior are different, and knowledge of their differences is important for achieving goals.
2. Any statement about human performance is about organizational performance as well.
3. Costs of improving performance should be regarded as investments in human capital, yielding returns in the form of increased performance potential.
4. Organizational and individual goals must be considered to define worthy performance.
5. Knowing how to engineer human performance and the conditions that affect it is as important as explaining why the behavior occurred.
6. Diagnosing problems required analysis of the present system and examination of differences between it and an ideal system. Avoiding anticipated problems requires analyzing the planned system and modifying it to approximate the ideal.
7. Exemplary performance provides the most logical reference for establishing job performance standards.
8. Human performance problems have differing root causes that originate either from the person, from something in the environment, or from both.
9. The performance of one subsystem affects the performance of other subsystems in somewhat predictable ways, requiring that root causes be analyzed at more than one level of the organization.
10. Many different solutions may be used to improve human performance. Selection of any one solution is dependent on the cause and nature of the performance problem, and the criteria used to evaluate a solution must include its potential to make a measurable difference in the performance system.

(Gilley, Dean, Bierema, 2001, pp. 80-91)

Reference

Gilley, J. W. (2001). Philosophy of organizational performance. In Gilley, J. W., Dean, P., & Bierema, L. (Eds.), Philosophy and practice of organizational learning, performance, and change: New perspectives in organizational learning, performance, and change, pp. 67-92. Cambridge, MA: Perseus.

Is Your Workplace Designed for Learning?

As the 21^st century transitions to the knowledge worker (from the industrial / manufacturing worker of the 20^th century & from the agricultural worker from the 19^th century) companies are faced with the responsibility of training their workers to acquire the required knowledge, skills, and abilities (KSA’s) for their jobs. Nijhof and Nieuwenhuis (2008) identified that work is shifting toward knowledge-intensive work, with a focus on knowledge production. It is not uncommon today to view a journal article or a newsletter that talks about the knowledge-economy. The primary difference here is that knowledge-economy refers more to a macro perspective, where knowledge production looks more at a micro perspective.

As the requirements for the knowledge worker continually changes to keep up with the global economy and new technologies, the educational system often falls behind these new technologies, thus furthering the burden for the employer to train employees. This new emphasis on training, by the employer, was highlighted by Nijhof and Nieuwenhuis (2008) by stating: “Efficient and effective learning impose requirements on workplaces that are different from those for efficient and effective working” (pp. 6-7). This new requirement on employers causes a shift in the common practices of the organization.

Most views of the workplace take the stand that there is a place to work (the workplace), there is a place to play (at home), and there is a place to learn (at home or at school – both away from the workplace). However, learning new tasks or skills are often required to perform one’s job. Alternatively, in order to successfully perform one’s job, one must be able to learn about the job duties and processes that are involved with that particular job. Nijhof & Nieuwenhius (2008) identified that learning is an intermediate function of work, “the learning potential of the workplace therefore lies in its conditions to support or stimulate learning” (Nijhof & Nieuwenhuis, 2008, p. 5). Thus, the more learning is supported in the workplace, employees have a tendency to be more motivated and productive.

A fully supported learning environment in the workplace could be replicated using the following definition provided by Nijhof and Nieuwenhuis (2008): “The learning potential of the workplace may… be defined as the power of a work setting to integrate learning at work with the result of behavioural changes and the generation of new knowledge” (p. 7).

Is your organization set up to integrate learning in the workplace? I will follow up in the next couple of blogs with a few models that could be implemented to begin a new learning workplace for organizations.

References

Nijhof, W. J. & Nieuwenhuis, L. F. M. (2008). The learning potential of the workforce. In Nijhof, W. J. & Nieuwenhuis, L. F. M. (Eds.), The learning potential of the workplace. Rotterdam, The Netherlands: Sense.

Human Performance Technology (HPT) Definitions & Assumptions

In 2004 a Presidential Task Force of ISPI identified Human Performance Technology (HPT) as "an integrated systems approach to improving human performance" (ISPI, 2004, p. 6).  As the editor of the third edition of The Handbook of Human Performance Technology, Pershing (2006) provided the following definition of HPT in his chapter titled "Human Performance Technology Fundamentals": "Human performance technology is the study and ethical practice of improving productivity in organizations by designing and developing effective interventions that are results-oriented, comprehensive, and systemic" (p. 6).

This improvement in performance can be either in the improvement in human performance of individual workers, the improvement of human performance through the design/operation of processes, and/or the improvement in human performance through organizational initiatives and environmental / societal endeavors.  In each case HPT provides a systematic approach to provide a comprehensive performance improvement initiative, not short-term, with a results-oriented evaluative framework.

The Presidential Task Force (2004) identified the following criteria to judge whether an initiative is an HPT initiative:

1. Is focused on valuable, measured results;
2. Considers the larger system context of people's performance;
3. Provides valid and reliable measures of the effectiveness of those applications;
4. Clearly describes applications grounded in prior research or empirical evidence (or are not discouraged by either one) so that they may be replicated under the conditions and by the means for which they were recommended (p. 6).

HPT operates on the following assumptions, as outlined in the Presidential Task Force (2004):

1. A technology is a set of empirical and scientific principles and their application
2. Human performance technology is the technology concerned with all variables which impact human performance
3. All organizational processes and practices impact the production of valued results, whether positively or negatively and whether those results go measured or unmeasured, acknowledged or not.  (Everything that an organization does affects what it accomplishes, whether or not the results are acknowledged or desirable.)
4. The purpose of all organizations is the same: to create value for their stakeholders; this is accomplished by aligning all processes, practices, and resources to maximize the production of that value.
5. We collaborate with and value the expertise of other disciplines; human performance technology becomes the integrator and multiplier. (p. 6)

The presidential Task Force report provides great information regarding the HPT Framework and is worth downloading and adding to your files - ISPI Task Force Report.  I would like to thank Guy Wallace for forwarding this link to me via Linkedin.  Guy is the President at EPPIC, Inc. and you can view his web page at http://eppic.biz/ for resources relating to instructional design and performance improvement.

References:

ISPI Presidential Initiative Task Force - Stage 1 (2004).  Retrieved from http://humanperformancetechnology.files.wordpress.com/2010/07/task-force-final-report-march-31-2004.pdf

Pershng, J. (2006).  Human Performance Technology Fundamentals.  In Pershing, J. (Ed.), Handbook of Human Performance Technology: Principles, Practices, Potential, (pp. 5-34).  San Francisco, CA: Pfeiffer.