Saturday, December 22, 2012

Training Evaluation Misses the Critical Thinking Dimension

For typical training courses the following four or five evaluation steps are the norm:

Level 1: Reaction and Perceived Values
Measures reaction to, and satisfaction with, the medium, content, and value of the project or program.

Level 2: Learning and Confidence
Measures what participants understand or learned from the project or program (information, knowledge, skills, and contacts).

Level 3: Application and Implementation
Measures what participants understand or learned from the project or program (information, knowledge, skills, and contacts).

Level 4: Impact and Consequences
Measures progress after the program implemented (the use of information, knowledge, skills, and contacts).

Level 5: ROI
Monetary Benefits.
(Phillips, & Phillips, 2007)

Evaluation is further divided into formative and summative evaluation, with formative evaluation relating to evaluating the training program and summative evaluation relating to the long-term effects of the training program (see Evaluation). Level 2, Learning and Confidence, is more of a formative evaluation measure and identifies whether trainees learned the material presented in the training. Level 3, application and consequences, is categorized as summative evaluation in which it looks at whether the trainees on-the-job behavior represents their learning from the training program.  

For these two specific levels of evaluation Phillips and Phillips (2007) highlighted information, knowledge, skills, and contacts. Providing employees (trainees) with the information that they need to conduct their job functions is critical. Additionally, providing employees with the knowledge to utilize this information in a productive manner is key to success. Having the skills to perform one's job is self-explanatory, but, as experience has shown us, people often lack the proper skills required to perform their main job function. Having the right contacts as well as knowing who has the information when needed is equally important. 

Using the following variables to evaluate training programs (information, knowledge, skills, and contacts) have proven to be effective for years. However, expanding on these variables to improve the evaluation process follows the continuous improvement process. As an effort to expand on the accuracy of training I would pose adding critical thinking to the mix.

Training employees to think critically helps to eliminate issues such as functional fixedness and mental sets. Ollinger, Jones, and Knoblich (2008) termed mental set as: " the repeated application of a successful method makes blind any alternative approach, because of the mechanization of the particular solution method" (p. 270). Alternatively, Duncker (1945) identified functional fixedness as: "the tendency to fixate on the typical use of an object or one of its parts" (as cited in McCaffrey, 2012, p. 216).

Adding the dimension of critical thinking to training endeavors will help transform learners (trainees, employees) to effective learners. Brindley, Walti, and Blaschke (2009) identified effective learners as those who are capable of coping with "complexity, contradictions, and large quantities of information, who seek out various sources of knowledge" (p. 3). By seeking out new sources of knowledge employees will better avoid the aforementioned traps of mental sets and functional fixedness. 

Including critical thinking skills as part of the training program, as well as incorporating evaluation of employees critical thinking skills on-the-job, could prove to produce better training results and on-the-job performance results. Additionally, including critical thinking in both the instructional and evaluation phases of the training program could improve both the formative and summative evaluations of the overall program. 

Brindley, Walti, & Blaschke (2009). Creating effective collaborative learning groups in an online environment. International Review of Research in Open and Distance Learning, 10(3), 1-18. 

McCaffrey, T. (2012). Innovation relies on the obscure: A key to overcoming the classic problem of functional fixedness. Psychological Science, 23(3), 215-218. dpi: 10.1177/0956797611429580

Ollinger, Jones, & Knoblich (2008). Investigating the effect of mental set in insight problem solving. Experimental Psychology, 55(4), 269-282. dpi: 10.1027/1618-3169.55.4.269

Phillips, & Phillips (2007). Show me the money: The use of ROI in performance improvement, part 1. Performance Improvement 46(9), 8-22. dpi: 10.1002/pfi.160

Tuesday, December 18, 2012

Representativeness (Intuition) versus Probability (Statistical accuracy)

"Experts are led astray not by what they believe, but by how they think" 
(Kahneman, 2011, pp. 219-220).

Every day decisions are made that affect the organization as well as the workplace and the workers. For example, decisions are made during an interview process to determine who the best candidate for the job will be. Additionally, decisions are made to identify who should be promoted from within the organization. Numerous other similar types of decisions are made weekly, sometimes daily, within organizations. These decisions are made using the information at hand (resume, work related performance records, referrals, manager's evaluation, etc…) as a predictor of future performance.

Basing a decision on intuition alone has been shown to be ineffective. Utilizing the information at hand can prove to produce slightly better predictions, depending on the validity of the information. The best that one can do when faced with having to make such a decision in a short time frame is to separate your subjectivity and base your decision on the data. Kahneman (2011) supported this position: "prediction  by representativeness is not statistically optimal" (pp. 150-151).  Here Kahneman refers to 'representativeness' as the decision-makers subjectivity, the decision-makers intuitive judgements about a particular candidate. Decisions based on representativeness have been shown to be no more accurate than in random assignment. Accuracy in decision making comes when statistical evidence (empirical data) guides the decision process. Kahneman made the following recommendations when making decisions:

  • Anchor your judgement of the probability of an outcome on a plausible base rate.
  • Question the diagnosticity of your evidence (p. 154).

Additional guide-lines, or rules, for making predictions are provided by  Kahneman (2011):

  • Errors of prediction are inevitable because the world is unpredictable.
  • High subjective confidence is not to be trusted as an indicator of accuracy (p. 220).

Basing decisions on 'plausible base rates' can lead to more accurate predictions in the long-term. The flexibility here is the term plausible, utilizing the best information made available in conjunction with your current knowledge of the field will assist in making more accuracy predictions. Avoiding subjective, or representative decisions, will improve the accuracy of predictions for the short-term as well as the long-term.

Kahneman, D. (2010). Thinking, Fast and Slow. New York, NY: Farrar, Straus, and Giroux.

Monday, December 10, 2012

Book Review

I have recently had my book-review for Edmondson's teaming (2012) published in the Learning and Performance Quarterly online journal. LPQ is a student-led, blind peer-review journal. LPQ is an open-access publication designed to make research available to the public and to support a greater exchange of global knowledge with articles supporting innovative learning and performance across disciplines (LPQ). As a reviewer for LPQ we are looking for additional reviewers and potential editors. If anyone is interested or needs the experience of being a reviewer for their resume go the the web page and submit your name. By being an open-source publication there is no membership required,  all articles are available for reading at any time. I hope you enjoy LPQ. You can also follow LPQ on Facebook.

The book-review is listed below:

Learning and Performance Quarterly, 1(3), 2012 31 
 Book Review 
Teaming: How Organizations Learn, Innovate, and Compete in the Knowledge Economy 
Jossey-Bass; 2012; 334 Pages; ISBN: 978-0-7879-7093-2 
New emerging constructs found in today's literature include those referring to collaboration, complexity, and globalization. One concept that builds on each of these aforementioned constructs is teaming. Edmondson has been involved with noteworthy research in the areas of small group and team research, including that of team psychological safety. Team psychological safety represents a climate in which group and/or team members feel comfortable questioning other's ideas, feel comfortable being challenged to defend one's own point of view, and are more open to constructive criticism. In her research, Edmondson identified psychological safety as a team construct in which a psychological safety measurement was developed showing that psychological safety affects learning behavior and also team performance. In her recent book, teaming, team psychological safety is but one construct identifying the learning construct teaming. 
Edmondson stated that “teaming is teamwork on the fly” (p. 13). Teams are traditionally portrayed in the literature as being a noun, consisting of fixed groups in pursuit of a common goal. Edmondson changes the discussion by placing teams as a verb, representing a dynamic activity, determined by the mindset and practices of teamwork. In Edmondson’s foundation for learning model, teaming is the first foundation, representing the structural support required for all other activities to take place. Edmondson identified four behaviors to accompany the foundation of teaming: speaking up, experimentation, collaboration, and reflection, 
Organizing to Learn 
Teaming is further differentiated from recent literature in that teaming is an organizational learning model. Edmondson’s model puts teaming as the driver for successful organizational learning functions. While teaming provides an environment for learning, Organizing to Learn, the second foundation for Edmondson’s learning model, promotes collective learning. Collective learning includes the following individual learning behaviors: a) asking questions, b) sharing information, c) seeking help, d) experimenting with unproven actions, e) talking about mistakes, and f) seeking feedback (p. 27). Edmondson identified the following four steps for the foundation organizing to learn: framing for learning (mental maps), creating psychological safety, learning from failure, and reaching across boundaries. 
The third and final foundation in Edmondson’s foundation for learning model was Execution-as-Learning, paralleling the same idea as that of action learning. Action learning follows four general principles, a) learning is acquired by doing, b) participants address organizational problems as well as personal development, c) participants work in Learning and Performance Quarterly, 1(3), 2012 32 
teams with peers, and d) participants follow an attitude of learning-to-learn. Execution-as-Learning was best described by Edmondson as the place in which "action and reflection go hand in hand” (p. 222). Four steps were included in Edmondson’s foundation for learning model to represent the foundation Execution-as-Learning: diagnose, design, act, and reflect. 
Together, these three foundations structure a learning environment (teaming) functioning on collective learning principles (Organizing to Learn) in which problems are addressed through systematic action learning steps (Execution-as-Learning). In Edmondson’s model a heavy emphasis was placed on leadership, which is required to get the process rolling. This heavy emphasis on leadership could be viewed as a weakness if an organization does not have a supportive leader. Edmondson indicated that leadership is what makes the process work, tying the foundations together. Successful teaming and learning thrive when leadership is able to focus on the foundations for learning, thus creating the by-product of a learning culture. Teaming is about more than just teams and their internal interactions. Teaming becomes the building block for a learning organization, which is the strength of Edmondson’s book. 
John R. Turner is currently a doctoral student in the Applied Technology & Performance Improvement (ATPI) program at the University of North Texas. His background is in engineering, with a second bachelor’s degree in Psychology from the University of Arkansas at Little Rock and a Master’s degree in Human Resource Development (HRD) from the University of Texas at Tyler. His research interests include performance improvement, team performance, team cognition, cognition/metacognition, outcomes-based evaluation, and meta-analysis techniques, and he has published in Performance Improvement and Journal of Knowledge Management. 

9th International Technology, Knowledge, & Society Conference

Coming up at the beginning of 2013 I will be presenting my paper Multiagent Systems as a Team Member at the 9th International Technology, Knowledge and Society conference on January 13-14, 2013, in Vancouver, Canada. This conference is being presented by Common Ground Publishing, USA. There is a great line-up of speakers from all disciplines in which you are sure to find something interesting. The website for the conference is:

My paper is being considered for publication by The International Journal of Technology, Knowledge and Society journal. Listed below is the abstract for the paper that I will be presenting. Once the presentation slides are put together, edited, and finalized I will have them posted on slideshare. I will post the link to the presentation slides once I have them completed. I am currently scheduled to present on on Jan. 14 under the Business Management and Organizational Technologies section around 1:30 pm.


With the increasing complex business environment that organizations have to operate in today, teams are being utilized to complete complex tasks.  Teams are capable of completing complex tasks that no one individual can achieve.  Effective team decision-making requires team members to discuss new information (unshared knowledge) and to consider this new information along with existing information (shared knowledge).  Research has shown that shared knowledge is favored over unshared knowledge during team discussions.  One method of transferring unshared knowledge to shared knowledge is to take advantage of new multiagent systems that are designed to support teams.  Multiagent systems are capable of filtering information without the bias toward shared information over unshared information.  This conceptual manuscript presents a model that incorporates individual intelligent agents and multiagent systems that monitor and actively interact with team members as an effort to address the unshared knowledge barrier, resulting in better team decision-making and problem solving outcomes. 

If you are planning on attending the conference drop a line and we can try to meet up at the conference.
Related Posts Plugin for WordPress, Blogger...