DAVID J. PUCEL
Industrial Education, University of Minnesota
JOHN C. CERRITO
Department of /twitters Administration and MIS, Augsburg College
DESIGNING POST-TRAINING TESTS TO
MEASURE THE IMPACT
OF MANAGEMENT LEADERSHIP TRAINING PROGRAMS
ON ACTUAL MANAGEMENT JOB PERFORMANCE
SUMMARY
'The dynamics of
change in an increasingly competitive environment will require that
organizations develop management-leadership training programs that actually
impact on performance. A very critical step in the development of a well
conceived management-leadership training program is the measurement of learning
and the ability of the learner to transfer knowledge back to the job.
The focus of
this report is to describe a three year (1992-1995) management
training-evaluation project was conducted at a large Fortune 500 company headquartered
in St. Paul, Minnesota. The project involved the development of a
post-training test administered to supervisors and managers at the conclusion
of an intensive 3 week company "Commitment to Excellence" training
program.
Me purpose of
developing a post-training test for the training program was to measure learner
comprehension and transferability of knowledge gained in the training program
back to the workplace. Consequently, performance feedback indices
were developed to measure transfer of training back to the actual job site. It was
expected that correlating scores between the post-training lest and performance
feedback indices would establish a cud-off score for the post-test and
would also demonstrate predictive validity.
Thus far,
transfer of knowledge from the training program back to the work-place has yet
to be established. As of the time of this report the company was unable to
produce 30 cases of completed "Commitment to Excellence"
post-training tests and 30 training-participant performance feedback indices
from the work-place. However, the project has resulted in the development of a
comprehensive 58-item post-training test. A content validity study yielded high
results for most items on the post-training test. Consequently, the
post-training test developed for the "Commitment to Excellence"
training program is a good measure of material learned as a result of the
training program.
INTRODUCTION
The company is an international
leader in the field of industrial paints and abrasives. The company conducts
business on a world-wide scale and operates manufacturing plants in the United
States, South America and Europe.
Requests for
reprints should be sent to John C. Cerrito, Augsburg College, 2211 Riverside
Avenue, Minneapolis MN 55454, USA. Tel. (612) 330-1768.
JOURNAI. FOR MENTAI. CHANGES, VOI.. II,
NO 2, 1996, p. 19-24.
20 DAVID J. PUCEI., JOIIN C. CERRITO
In strategic effort to up-grade and enhance the overall
performance of its supervisory and management corps, the company developed a
comprehensive training program. The company "Commitment to
Excellence" training program consisted of three weeks of core training for
all supervisors and managers.
The first week of the "Commitment to Excellence" training program
focuses on managing employees to improve individual performance. Specifically,
the first week of the program included content such as dynamics of leadership,
principles of supervision, changing organizational culture, motivation,
coaching, legal is-sues and alcohol and drug abuse in the workplace. The second
week of the pro-gram focused on working within groups. Which included such
topics as team building, dealing with diversity in groups, communicating,
fostering group creativity, group problem solving techniques and dealing with
change.
The third week of the program focuses on the general aspects of the
company business. Including such topics as understanding the company vision,
ethical responsibilities, financial statement analysis, strategic planning and
marketing analysis.
Company managers and supervisors progressed through the 3 week program
in the order presented above. Each managers and supervisors participation in
each of week of the training program was separated by approximately 6 months. Training
participants would normally complete the program in about one and one-half
years.
The company, was very interested in assessing transfer of knowledge
gained in the management training program back to the workplace. Specifically,
the company was interested in studying the impact that the training program
might have on improving individual management/supervisor performance behaviors.
Consequently, the company was interested in conducting a project/study
to develop evaluation tools such as post-training tests and performance
feedback in-dices that would provide information on the ability of the program
participants to transfer training back to the job (Kirkpatrick, 1977).
Also, the company was interested in considering information generated by
the project study in making other human resource management decisions,
regarding promotion and retention of supervisory or management personnel.
SPECIFIC PROJECT GOALS
The project was designed to develop
evaluation tools that measure the knowledge individual supervisors or managers
have retained after completing the training program and the extent to
which they can generalize and apply the con-tent to typical on-the-job
situations at the company. Because the work-force within the company is
composed of people in 3 job families (production, technical and sales),
DESIGNING POST-TRAINING TESTS 21
the evaluation
tools needed to be built around realistic job situations relative to each of
the job families.
Listed below are specific project
steps:
1. To identify the
expected outcomes from the completion of each week of the training program.
2. To identify job
samples representative of activities which supervisors and managers perform
within the three job families that require the application of the program
content.
3. To develop
post-training tests that measure the knowledge individual supervisors or
managers have retained after completing the program, and the extent to which they
can generalize and apply the content to typical situations within the company.
4. To validate the
tests in accordance with the requirements of EEOC guide-lines.
STUDY METHODS
The study methodology used for the
project proceeded under the premise that in order to produce valid test and
performance feedback tools it is important to establish content validity.
Consequently, in order for the these tools to be most useful within the
company's environment, they must be developed with the sup-port and involvement
of all concerned members of the organization (Jackson & Culp 1979).
The project methodology was also founded on the assumption that the company's
Human Resources staff would gather and provide information from within the
company, needed to conduct the project. This included gathering important
information such as; lists of duties for jobs within each job family, sample
situations in which supervisors and managers would apply the content, and
complying the results of tests scores obtained from people who have completed
the training (Pucel, Cerrito & Noe 1989).
The following is a list of the major
methodological components of the project
1. Identify the
expected outcomes of each week of training.
2. Identify
examples of the realistic job applications of those outcomes (work samples).
3. Create test
items that measure the outcomes within the context of realistic job applications.
4. Conduct a
content validity study of the test.
This would
include having company Human Resources staff that developed the training
program, examine the list of outcomes for each week of the training
22 DAVID J. PUCEL„ JOI IN C. CERRITO
program to determine if, they adequately represent the expected training
out-comes.
Content validity studies also require
input from a representative group of company supervisory and management
employees who would be participants in the training program and the in
post-training evaluation study. The training pro-gram development staff and a
representative group of training participants re-viewed the test items to judge
the extent to which the test items actually measured competence relative to
the desired outcomes.
5. The test would
be administered to participant groups that completed each week of training. The
results would be used to conduct an item analysis of each item and to determine
test reliability.
6. After the
training participants complete the training program and return to their jobs,
each person will be evaluated by their direct manager. Each training
participant will be evaluated on key outcomes that were taught in the training
program.
7. Each training
participants score on the post-training test will be correlated with their
score from their manager's performance indices when they returned to the
workplace. This would help establish a post-training test cut-off score and
allow for evaluating transfer of training back to the job site.
8. A project report
will be presented to the company which documents the test development process,
the validity study data supporting the test, the reliability study data, the
correlation between the training post-test and the manager's performance
evaluation which was the procedure recommended for establishing the cut-off
score.
OUTCOMES
The results of the content validity
were very encouraging. A group of subject matter experts (company Human
Resources personnel and a representative group of supervisor/management
trainees) were asked to rate each item on the test on how well the item measures
application of the content learned. The ratters used a lour point scale, with 4=provides
excellent assessment of important information covered in the training program,
3=provides adequate assessment of important in-formation covered in
the program, 2=provides inadequate assessment and 1=does not assess important
information covered in the training program.
A content
validity index (CVl) was calculated using Lawshe's (1975) formula CV1=(Ne-N/2)/(N/2)
where Ne equals the number of experts indicating an item is excellent
(4) or adequate (3) and N equals the total number of rafters. The CVI ranges
from +1.00 indicating complete agreement that the test item provides an
excellent or adequate assessment of important covered in the training program
to
DESIGNING POST-TRAINING TESTS 23
1.00 indicating that the test item provides an inadequate
assessment or does not asses important information covered in the training
program.
The CVIs calculated based on the
subject matter experts judgments ranged from 0.900 to 0.230. While 45 of the 58
items on the test were ranged between 0.900 to 0.400 indicating that the
subject matter experts judges these items provided excellent or adequate
assessment of important information covered. The remaining items ranged in CVIs
from 0.2300 to 0.3700 and some were re-drafted.
The results of the validity study
showed that the training post-test was content valid. Consequently, after
completing the content validity study, the post-test was administered to
supervisors and managers completing the company's Commitment to Excellence
training program.
To establish a post-training test
cut-off score and to measure transfer of training it was agreed with company
management that the scores from the post-training test be correlated with
performance indices once the training participant returns to the work place.
Performance feedback indices were developed and some were completed by the
training participants direct-manager 6 months after the trainee completed
training and returned to the work place. Consequently, some correlations
between the training post-test and performance feedback indices were
completed.
However, in order to establish
predictive validity 30 to 50 cases (correlations between the post-tests and
performance feedback indices) were needed. Specifically, the company was
unable to provide 30 to 50 completed training post-tests and performance
feedback indices, as a result predictive validity could not be established.
Thus far, the results of the study has
been the development of a post-training test for the Commitment to Excellence
training program that was content valid and consequently is a good measure of
material learned as a result of the training program.
CONCLUSIONS
The growing competitive environment
will require that organizations place more emphasis on management training and
development. Management training programs should be developed to enhance
management performance. However, organizations should also expect specific
performance outcomes as a result of their management training programs.
Management
training programs should be developed with a strong evaluation component. As a
result of the training program, the organization should expect that the manager
will transfer what they have learned in training back to the workplace.
24 DAVID J. PUCEL.,
JOHN C. CERRITO
Measuring
knowledge gained as a result of completing a management training program and
evaluating the ability of the manager to transfer what they learned
in training back to the workplace will have a positive impact on the
performance and the profitability of the organization.
REFERENCES
Jackson, S. and Kulp, M.J.,
"Designing Guidelines for Evaluating the Outcomes of Management Training",
Determining the Payoff of Management Training, ASTD, Washington, D.C., 1979, pp
7-8.
Kirkpatrick, D.I.,
"Evaluating Training Programs: Evidence vs Proof", Training and
Development Journal
(November 1977), pp. 9-12.
lastra, B and Nichols,
M.B., Define and Broadcast Your Training Philosophy", The Best of
Training I IRD Interviews and Profiles, Lakewood Publications, Minneapolis, 1981,
pp 120-122.
Iawshe, C. 11.(1975).
"A Qualitative Approach to Content Validity". Personnel Psychology,
28,563-575.
Pucel, David J.,
Cerrito, John C. and Noe, Raymond, "Integrating Selection, Training and
Performance Evaluation", Performance
Improvement Quarterly, National Society for Performance and Instruction,
1989.
Warr, P., Bird,
M., and Rackham, N., "Evaluation of Management Training", Gowar Press,
London, 1970.