Chapter
5
Discussion
This study was designed to
address questions about the two major components of the TAS program, the
distance education course and the mentoring workshop. Results regarding the
distance education course will be discussed at Kirkpatrick's first two levels
of evaluation, reaction and learning. Results regarding the mentoring workshop
will be discussed at Kirkpatrick's first three levels of evaluation, reaction,
learning, and behavior. Correlation between reactions and learning, and
reactions and behavior will be discussed along with Kirkpatrick's fourth level,
results, in light of the program's costs and benefits. The chapter concludes
with a summary of findings and implications for future studies.
The questions this study was
designed to address regarding the distance learning component were:
·
How
successful was the distance education program based on student participation,
satisfaction and grades?
·
(b) Is
distance education the most effective way of reaching these students? and
·
Was
it beneficial to choose the Internet over other methods of education delivery?
The questions this study was
designed to address regarding the mentoring component were:
·
How
successful was the mentoring workshop based on student satisfaction levels and
grades?
·
Was
it beneficial to include mentoring as an integral component of the program?
·
Does
TAS help gifted students make informed college and career choices?
Questions regarding correlation were
·
Is
there a correlation between levels of satisfaction and post-program interest in
engineering?
·
Is
there a correlation between levels of satisfaction and post program choice of
college major?
Questions regarding Kirkpatrick's fourth level of
evaluation, results, were
·
Are
there other less costly ways of meeting the educational need?
·
How
does this study benefit stakeholders, future students, other organizations and
instructional designers?
The discussion of the
results will address each of the major research questions in light of the
review of literature. This section is divided into five sections: distance
education, mentoring, correlation, results, and conclusions.
Distance Education
The discussion regarding the
distance education component of the program is addressed in two sections based
on Kirkpatrick's first two levels of evaluation, reaction and learning.
Level I: Reaction
How successful was the distance education program based on student
participation and satisfaction? Based on the results at level one, a majority of
the original 196 students participated in the program through its conclusion
(89%), and a majority of those students were satisfied with the distance
learning program, rating it good or excellent (93%). According to Kirkpatrick
(1994) if participants are not satisfied, they will not be as motivated to
learn or continue participating in a program. Satisfaction is therefore a
variable that influences participation levels. These results are an indication
that students who continued with the program were influenced to do so by their
high level of satisfaction with the course.
Students rated the distance
education units between good and above average. Students rated the units that
had an original design component the highest. The units that had original
design components in them required students to take a real world problem such
as building a spacecraft or space colony and use their creativity to design the
component based on the information provided to them in the lesson. The review
of literature indicates that creativity and real-world problem solving are
important characteristics of distance education and science interventions
(Gomez, 2000; Rodriguez, 1997; UNITE, 2001). The higher ratings of these units
over the others indicate that students valued creative real-world problem
solving activities in the course, thus supporting the research findings reported
in the review of literature.
The course was described by
some students as challenging but not too hard, indicating a balance between
levels of involvement and the ability for students to complete the assignments
while handling their school workload. The Web-based format of the course
allowed students to complete the course at their own pace. This was an
important assumption made by program designers to accommodate students who are
trying to balance school with extra-curricular activities. Gifted students were chosen
to participate in TAS, and an effort was made to recognize multiple workloads
on these students. The review of literature notes that distance courses may be
too time consuming for regular students but may be well suited to gifted scholars
(Casey & Shore, 2000). Students' ability to complete the course while
handling regular school workloads supports the findings reported in the review
of literature regarding gifted students and distance learning.
The Web-site components
including chat sessions and e-mail were rated by students between good and
excellent. The review of literature regarding distance learning indicates the
importance of interactivity in on-line science activities (Amazing Space 2001;
Center for Educational Technology, 2001; Explore Science, 2001; GLOBE, 2001;
NASA Quest, 2001). High satisfaction levels of students regarding the
interactive portion of the program uphold the research findings reported in the
review of literature.
In their comments, students
requested even higher levels of interactivity in the course. Interaction with
experts and peer-to-peer mentoring was found in the review of literature to be
an important component in programs of this type (Coppula, 1997; Marable, 1999;
Panitz, 1996). In the second year of the program more chat sessions and an
interactive bulletin board was added to enhance course interactivity between
mentors and students and between students themselves.
In the second year of the
program students were also given access to their own-on-line portfolios, and
educators were assigned to critique each assignment. The review of literature
supports these changes, as feedback is considered an essential factor in both
student satisfaction (reaction) and in student learning (Dick, 1996; Gagne,
Briggs, & Wagner, 1992; Miodoser, Nachmias, Lahav, & Oren, 2000).
So, how successful was the
distance education program based on student participation and satisfaction? Based
on student participation and satisfaction levels in the course, the distance
education can be considered a success at Kirkpatrick's first level of
evaluation, reaction, at the close of its first year. Kirkpatrick's second
level of evaluation, learning, was also measured because no change in behavior
(Kirkpatrick's third level) can be attributed to a program unless one or more
of the learning objectives were accomplished.
Level II: Learning
How successful was the distance education program based on student grades? In the distance learning course, the average quiz scores ranged between 87 and 96. 99% of the students who completed all six quizzes passed with a grade of 75 or higher. The high level of passing grades indicates that students mastered the learning objectives in each unit. In the post-program evaluations students were rated above average to excellent by their mentors in terms of academic skill and effort indicating students had mastered the learning objectives and indicating a measure of reliability of the two evaluation instruments. The student grades indicate that the distance education program was successful in its first year.
Is distance education an effective way of reaching these students? Was it beneficial to choose the Internet over other methods of education delivery? Distance education was the choice of the program designers since students were distributed all over the state of Texas and other methods were impractical and costly. Distance education using the Internet allows for a program to be accessed by a large number of students at a reduced rate of cost (Tuttle, 1998). According to research, distance education seems to be as effective as traditional education with regards to learning outcomes (Hanson & Maushak, 1996).
Students rated the web-site between good and excellent, with no students indicating any difficulties with the level of technology employed in the web-site or required of the students. That the TAS students had an initial expertise in the use of technology was an assumption made by the course designers. The review of literature indicates that gifted students in particular have an interest in and an ability to use the Internet (Casey & Shore, 2000). While Web-based learning may not be the answer for all students, the review of literature supports the use of the Web for gifted students who are self-motivated and have initial expertise in the use of technology (Tuttle, 1998).
One of the major limitations of using the Internet is that not all students may have a computer (Tuttle, 1998). While not all TAS families had computers in their home, those students without computers noted that they used school and library facilities to complete the course. Grants to provide a computer to students without easy access to the Internet could be considered in the future to help support these students.
Another limitation with Web-based courses is slow modem connection speeds. Some students noted in their evaluations that download time was an issue for those with slower modem connections. The decision was made at the close of the second year to put the curriculum on CD-ROM to facilitate access for students since a CD-ROM would allow students faster access to the course content (although a browser connection would still be needed to explore external links).
Students rated the units
with a design component higher than the other lessons and also rated the
interactive portions of the course between good and excellent. The TAS
curriculum was designed with constructivist learning principles and used
interactivity as a major component in student learning. Incorporating
interactive activities in the delivery of the on-line course is recommended by
the review of literature regarding distance learning, and is considered to be
important in sustaining student interest and learning (Hammonds, 1998; Web66,
2001; Yahoo, 2001). Interactive simulations, interactions with experts and
peers, and creative real-world problem solving were found to be common in NASA
and NASA-related education web-sites (Amazing Space 2001; Center for
Educational Technology, 2001; Explore Science, 2001; GLOBE, 2001; NASA Quest,
2001). A majority of other non-NASA education sites do not currently use
constructivist learning principles or promote interactivity, both of which are
considered by researchers as important factors in distance learning (Miodoser,
Nachmias, Lahav, & Oren, 2000).
So, was the distance
education program effective, and was it beneficial to use the Internet as a
method of delivery? Students' high levels of satisfaction and learning in the
distance education course indicate that distance education using the Internet
was an effective way to deliver the instruction and prepare students for the
mentoring workshop. Even though there were a number of limitations, the results
of this study suggest that the use of the Internet was the right choice. In
addition, students' high levels of satisfaction and learning in the distance
education course may have had an impact on their mentoring workshop experience.
Mentoring Workshop
The discussion of the
results regarding the mentoring workshop component of the program will be
addressed in the following three sections based on Kirkpatrick's first three
levels of evaluation, reaction, learning, and behavior.
Level I: Reaction
How successful was the mentoring workshop based on student satisfaction
levels? The
average ratings by students in post-program evaluations of the mentoring
workshop were between above average and excellent. The majority of all students
rated the mentors and co-ops between above average and excellent. In the
post-program survey, 83% of the responding students rated the workshop
excellent, and 89% rated the mentors good or excellent indicating a measure of
reliability of the post-program evaluation and survey instruments. However,
workshop logistics (e.g. hotel, food, and transportation) were only rated as
good. Improvements in logistics should be considered when making future
improvements to the program.
The research on intervention
programs indicate that site visits, interaction with professionals in the
field, peer-mentoring, real world problem solving and hands-on activities are
important factors contributing to a program's success (Hamilton, 1997; Johns
Hopkins University, 2001; Rodriguez, 1997). The results of this study support
such claims. The expert briefings, tours, and hands-on activities were noted by
students as being the highlight of their stay at NASA.
A very high degree of
satisfaction by the students validates the decision to include the workshop
components in the program and indicate that it was a significant contributor to
the program's success. Student's reactions, however, are not sufficient to
determine the overall effectiveness of the workshop. Kirkpatrick's second
level, learning, measures whether students mastered the program objectives.
Level II: Learning
How successful was the mentoring workshop based on student grades and
changed attitudes? In the post-program evaluations students were rated above average to
excellent by their mentors in terms of academic skill and effort indicating
that students successfully mastered the project skills required for the summer
workshop (completion of a Mars mission design).
Quantitative measures of
learning outcome indicators in the mentoring component of the program were
based on the review of literature (National Mentoring Center, 1999; U.S. Dept.
of Ed., 1998). The learning outcome indicators in this study include an
increase in interest in engineering and a choice of future career in
engineering and related disciplines. A majority of students in this study indicated
a high level of interest in engineering after the program and a majority of
students indicated that the course had a moderate or high impact on their
future career choices.
In the post-program
evaluation, 75% of the students rated their attitudes towards engineering were
improved as a result of the program (25% indicated they were about the same).
In the post-program survey 92% of the students indicated the program had some
or high impact on their future career choices. In the post-program survey, 86%
of the students rated their interest in engineering as moderate or high. These
high levels of improved attitudes, program impact and interest levels indicate
that the mentoring program had a significant measure of impact on the students'
interest in engineering and on their future choice of career.
In post-program surveys, 77%
of students indicated a moderate (45%) or high (32%) degree of interest in
engineering prior to the program. 93%
of the students indicated a moderate (23%) or high (70%) degree of interest in
engineering after the program. These
statistics indicate an increase (38%) in the percentage of students who had a
high interest in the topic after completing the program. The review of literature indicated that students
experiencing an intervention program are significantly impacted in their choice
of college major (American Association of University Women, 1992; Molkenthin,
2001; Panitz, 1996)
Qualitative measures of
learning outcomes of mentoring programs were identified by the review of
literature (Flaxman, 1993, NCLAM, 2001). The learning outcomes used in this
study were an increased level of knowledge, reduced feelings of isolation, and
an enhanced sense of the ability to achieve. Comments from students indicated
that students gained a great deal of knowledge from the program, and
experienced reduced feelings of isolation as a result of being part of a group
of students with their same interests, and by working with mentors. Students also indicated that they had an
enhanced sense of the ability to achieve as a result of the mentoring workshop.
An unanticipated outcome
indictor that resulted from the evaluation was a significant increase in degree
intentions after the program. A large percentage of students indicated that
their post-program degree intentions were higher post-program than pre-program.
Students, who had initially been content with attaining a Bachelors degree, had
higher aspirations to Masters or Doctorate degrees after the program. Research
has shown that mentoring not only can increase student’s self-confidence, but
also their aspirations (McIntosh & Greenlaw, 1990). This is supported by
the significant increase in degree attainment intentions by TAS students after
the program.
So, how successful was the
mentoring workshop based on student grades and changed attitudes? The results
of this study indicate that students mastered the program's objectives and experienced
a distinct change in attitudes as a result of the intervention.
Was it beneficial to include mentoring as an integral component of the
program? Students
rated the mentors and co-ops in the program between above average and
excellent, in addition they rated the components of the mentoring workshop
between above average and excellent. The high ratings of the mentors and the
mentoring workshop indicate that the decision to include mentoring was the
right choice for the TAS students. Many types of mentoring were designed to be
included in the TAS program. Adult and peer mentoring (with engineers,
scientists and co-ops), the use of technology (on-line chat and e-mail), the
workshop experience (face-to-face mentoring), academic tutoring (student
project assessment by mentors) and long-term mentoring (continued guidance by
mentors for up to four years) were all incorporated into the program. According
to the review of literature each of these components are considered key factors
in successful mentoring programs (Marable, 1999; Molkenthin, 2001; Panitz,
1996).
Scholars reported that the
mentors provided role models and helped them define options and career paths
indicating that mentoring was an important component in the TAS program.
Research indicates that mentors can give gifted students intellectual
stimulation, communicate an excitement about the field of science, and help
student self-confidence and aspirations grow while feelings of isolation are
lessened (Bennett, 1997; Berger, 1990; Casey & Shore, 2000; Kerr, 1985).
Gifted students particularly have a desire for adult role models, and have many
career options open to them (because of ability) (Kerr, 1991). High levels of
post-program interest in engineering support the decision to use mentoring to
inspire and guide these students.
The TAS program designers
chose to have mentors work with students to design a mission to Mars for
presentation at the end of the week. Students rated mentor support of the team
projects highly, between above average and excellent. Involving students and mentors
in activities and projects that encourage them to work together is considered
one of the best ways to facilitate successful mentoring relationships (JETS,
2001; Molkenthin, 2001).
Students rated mentor e-mail,
chat sessions and review of project assignments between good and excellent
indicating that these methods of interaction with mentors were successful. The
chat sessions were rated by students between good and above average, and
anecdotally as a good way to meet mentors beforehand. According to the review
of literature, the use of the Web to facilitate mentoring relationships is a
choice considered appropriate for self-motivated students who seek out and
benefit from adult relationships (Bennett, 1997; Electronic Emissary Project,
2001; International Telementor Program, 2001). Additional chat sessions were
requested by students, and are recommended by these findings.
Mentors are encouraged to
stay in touch with their team members for up to four years. It is recommended
by these results that future evaluations address the extent and quality of
those continued relationships and their impact on the program's ultimate goal,
a choice of career.
Results from the evaluation
regarding the mentoring workshop indicate high levels of satisfaction
(reaction) and learning, Kirkpatrick's first two levels of evaluation.
Student's behavior, Kirkpatrick's third level of evaluation, can indicate
whether the program has met its initial goal of impacting students' choice of
major in college and future careers.
Level
III: Behavior
Does TAS help gifted students make informed college and career choices? The distance education
course was designed to give students information about different careers in
science, mathematics, engineering and technology in preparation for the
mentoring workshop. The mentoring component was included to give students
access to role models in various careers, and allow students to visit a
facility where professionals were working in those fields. The tours, briefings
and activities were intended to provide students with opportunities to talk to
professionals other than their mentors and to give them hands on experiences in
the field of space science.
The results from the
post-program surveys indicate a high number of students actively chose a
college major in science, mathematics, engineering or technology at the end of
their senior year. Some students noted that while they were initially
interested in these majors it wasn't until after the conclusion of the program
that they actually decided on a specific choice of major. In the first
post-program survey (in the beginning of their senior year), 86% of responding
students noted they planned to major in science or engineering. In survey two
(after students had applied to colleges), 90% of responding students said they
planned to major in science or engineering indicating a measure of reliability
of the two survey instruments. Fewer students responded to the second survey
however, and thus the increase in numbers may be a result of the more
interested students responding to the second survey.
Anecdotally, a number of
students indicated that the program gave them specific options regarding their
future career plans. The review of literature indicates that students who
experience some kind of intervention during their high school years are
positively impacted in their decision to choose a major in those fields
(American Association of University Women, 1992). In addition, national
statistics regarding number of freshmen who initially choose science,
mathematics, engineering or technology as a major, note a decline in interest
during their time at college (National Science Foundation, 1993). Continuing
involvement with mentors in the TAS program may be an effective way for those
with an initial interest to sustain interest over time. Future surveys and
active solicitation of more participants in the follow-up program are needed to
ascertain whether students continue on the same career track.
Correlation
Is there a correlation between levels of satisfaction and post-program
interest in engineering? Students rating of the program components were correlated with their
interest in engineering at the conclusion of the program to measure whether
their attitudes (learning) were related to their levels of satisfaction
(reaction). A correlation factor (r) of .14 was calculated between student's
satisfaction ratings of the program components and their level of interest in
engineering after the program's conclusion. The resulting correlation indicates
a relationship between the first and second level of Kirkpatrick's evaluation.
To support the correlation
statistics, the percentage of students who rated the program components as good
or excellent (94%) can be compared to the percentage (90%) that indicated they would
be majoring in engineering or science in college. These percentages
descriptively suggest that there is some relationship between student's
reaction and behavior, Kirkpatrick's first and third level of evaluation.
While the correlation
statistics are not high, it should be taken into consideration that many of the
students were already interested in these fields prior to beginning the
program, in fact the students were initially chosen on that basis. Students'
change in plans regarding their level of degree attainment is perhaps a more
significant behavior to consider as a measure of program success. Higher levels of aspiration after the
program can be considered to be due in part to high levels of satisfaction (reaction)
and learning as a result of the program.
While there appears to be an
association between reaction and learning, and between reaction and behavior,
future surveys are needed to determine ultimate program impact (Kirkpatrick's
fourth level, results) by student's final choice of college major and career.
Level IV: Results
Kirkpatrick's fourth level, results, addresses the tangible results that can be measured at the close of a program. While student's final choice of career is still several years away, we can look at two outcomes, costs and benefits, of the first year of the program. Costs and benefits of the program will be addressed separately in the following two sections.
Costs of virtual schools
match approximately the cost of the TAS program (Hammonds, 1998; Web66, 2001;
Yahoo, 2001). Summer camp costs for a six-day program (such as Space Camp) are
comparable as well. Organizations like NASA interested in long-term results
need to match their ability to provide such programs with their investment in
the intended outcomes (a larger workforce). Since long-term results are not a
focus of this particular study, one can only say that considerations should be
made on a case by case basis by the organizations developing similar programs.
Future studies to determine how many of the TAS students enter the science,
mathematics, engineering and technology workforce can then be used to determine
whether the cost of the program has was worth the return on investment.
Benefits
How does this study benefit stakeholders, future students, other
organizations and instructional designers? The results of this study can benefit key
stakeholders (entities with a vested interest in the increase of engineers and
scientists in the workforce) by providing positive statistics regarding the
levels of student reaction, learning and behavior. The first three levels are
however merely an indication of the final program outcome, results, (students'
choice of careers).
It is believed that continued
funding of the program is warranted as a result of this initial program
evaluation at Kirkpatrick's first three levels of evaluation, with suggested
improvements regarding levels of interactivity in the distance education
course, and by choice of hotel, food and transportation.
Other industries considering
replicating the program (to meet the need for future high levels of scientists
and engineers entering the workforce) can use the results of this first
evaluation to support their plans. Educational organizations (who are looking
for model Web and mentoring programs) and instructional designers (who are
looking for Web-based models that utilize mentoring) can use the TAS model as a
blueprint should they encounter a similar need for an interactive Web-based
educational and mentoring program.
The evaluation results will
be used by TAS program managers to improve the quality of the Web-site, the
on-line course and the summer workshop for future students. Several changes and
improvements were made after informal analysis of the results at the end of the
first year. The evaluation indicates additional improvements in levels of
interactivity and workshop logistics for future years.
Conclusions
Currently most existing science,
mathematics, engineering and technology intervention programs are aimed at high
school students, some are aimed at gifted students, and a few utilize
mentoring. TAS uniquely targets gifted high school students using Web-based
distance learning technology and real-time mentoring.
Based on the results at
level one, reaction, a majority of the original 196 students participated in
the program through its conclusion (89%) and a majority of those students rated
the program good or excellent (94%) indicating a proportional correlation.
Level two results of
learning, indicate a high knowledge acquisition (a 99% passing rate), and high
impact on attitudes towards engineering (92%) as a result of the program. Student
testimonials noted creativity and interactivity as important factors in the
distance learning component validating initial program assumptions, and
requested even more of both, factors noted in the review of literature on
distance education as sustaining student interest and fostering student
learning.
The results of the two
follow-up surveys indicate that the behavior of a significant number of
students was impacted in their intended choice of science, mathematics,
engineering and technology majors (86% in survey one, and 90% in survey two).
The level one, two and three
results obtained in this study indicate that the program was successful at all
three levels of Kirkpatrick's evaluation model. However some may argue that the success is due to student's prior
interest in the topic. While it is true that students may have already
considered choosing a science or engineering major prior to participation in
the TAS program; some measures of this tendency can still be attributed to the
program itself. Research has shown that gifted students have multi-potentiality
(Berger, 1990; Kerr, 1991) and often a more difficult time choosing a path
without experiencing an intervention of some kind. Interventions can include distance learning, workshops and
mentoring. After the TAS intervention, a combination of all three elements, student's
final choices of college majors (behavior) can if not entirely, at least in
part, be attributed to high levels of satisfaction (reaction) and learning.
According to the review of
literature, one of the best ways to improve the validity of an evaluation study
is to replicate it (Wolf, 1979). One way is to carry out the evaluation study
with successive groups of learners over time. The results of the two studies
can be compared and judgments and decisions about the program's fate made at a
future time. While inconsistencies can arise if the program is significantly
altered from one year to the next, the results of more than one study are
generally considered to be superior as a basis for making important decisions
about a program. It is recommended that future studies continue to use the
Kirkpatrick model as a structure for evaluation, moving up to the fourth level,
results, scholar's choice of career, as an indicator of the program's ultimate
success.