This study was designed to address questions about the two major components of the TAS program, the distance education course and the mentoring workshop. Results regarding the distance education course will be discussed at Kirkpatrick's first two levels of evaluation, reaction and learning. Results regarding the mentoring workshop will be discussed at Kirkpatrick's first three levels of evaluation, reaction, learning, and behavior. Correlation between reactions and learning, and reactions and behavior will be discussed along with Kirkpatrick's fourth level, results, in light of the program's costs and benefits. The chapter concludes with a summary of findings and implications for future studies.
The questions this study was designed to address regarding the distance learning component were:
∑ How successful was the distance education program based on student participation, satisfaction and grades?
distance education the most effective way of reaching these students? and
∑ Was it beneficial to choose the Internet over other methods of education delivery?
The questions this study was designed to address regarding the mentoring component were:
∑ How successful was the mentoring workshop based on student satisfaction levels and grades?
∑ Was it beneficial to include mentoring as an integral component of the program?
∑ Does TAS help gifted students make informed college and career choices?
Questions regarding correlation were
∑ Is there a correlation between levels of satisfaction and post-program interest in engineering?
∑ Is there a correlation between levels of satisfaction and post program choice of college major?
Questions regarding Kirkpatrick's fourth level of evaluation, results, were
∑ Are there other less costly ways of meeting the educational need?
∑ How does this study benefit stakeholders, future students, other organizations and instructional designers?
The discussion of the results will address each of the major research questions in light of the review of literature. This section is divided into five sections: distance education, mentoring, correlation, results, and conclusions.
The discussion regarding the distance education component of the program is addressed in two sections based on Kirkpatrick's first two levels of evaluation, reaction and learning.
Level I: Reaction
How successful was the distance education program based on student participation and satisfaction? Based on the results at level one, a majority of the original 196 students participated in the program through its conclusion (89%), and a majority of those students were satisfied with the distance learning program, rating it good or excellent (93%). According to Kirkpatrick (1994) if participants are not satisfied, they will not be as motivated to learn or continue participating in a program. Satisfaction is therefore a variable that influences participation levels. These results are an indication that students who continued with the program were influenced to do so by their high level of satisfaction with the course.
Students rated the distance education units between good and above average. Students rated the units that had an original design component the highest. The units that had original design components in them required students to take a real world problem such as building a spacecraft or space colony and use their creativity to design the component based on the information provided to them in the lesson. The review of literature indicates that creativity and real-world problem solving are important characteristics of distance education and science interventions (Gomez, 2000; Rodriguez, 1997; UNITE, 2001). The higher ratings of these units over the others indicate that students valued creative real-world problem solving activities in the course, thus supporting the research findings reported in the review of literature.
The course was described by some students as challenging but not too hard, indicating a balance between levels of involvement and the ability for students to complete the assignments while handling their school workload. The Web-based format of the course allowed students to complete the course at their own pace. This was an important assumption made by program designers to accommodate students who are trying to balance school with extra-curricular activities. Gifted students were chosen to participate in TAS, and an effort was made to recognize multiple workloads on these students. The review of literature notes that distance courses may be too time consuming for regular students but may be well suited to gifted scholars (Casey & Shore, 2000). Students' ability to complete the course while handling regular school workloads supports the findings reported in the review of literature regarding gifted students and distance learning.
The Web-site components including chat sessions and e-mail were rated by students between good and excellent. The review of literature regarding distance learning indicates the importance of interactivity in on-line science activities (Amazing Space 2001; Center for Educational Technology, 2001; Explore Science, 2001; GLOBE, 2001; NASA Quest, 2001). High satisfaction levels of students regarding the interactive portion of the program uphold the research findings reported in the review of literature.
In their comments, students requested even higher levels of interactivity in the course. Interaction with experts and peer-to-peer mentoring was found in the review of literature to be an important component in programs of this type (Coppula, 1997; Marable, 1999; Panitz, 1996). In the second year of the program more chat sessions and an interactive bulletin board was added to enhance course interactivity between mentors and students and between students themselves.
In the second year of the program students were also given access to their own-on-line portfolios, and educators were assigned to critique each assignment. The review of literature supports these changes, as feedback is considered an essential factor in both student satisfaction (reaction) and in student learning (Dick, 1996; Gagne, Briggs, & Wagner, 1992; Miodoser, Nachmias, Lahav, & Oren, 2000).
So, how successful was the distance education program based on student participation and satisfaction? Based on student participation and satisfaction levels in the course, the distance education can be considered a success at Kirkpatrick's first level of evaluation, reaction, at the close of its first year. Kirkpatrick's second level of evaluation, learning, was also measured because no change in behavior (Kirkpatrick's third level) can be attributed to a program unless one or more of the learning objectives were accomplished.
Level II: Learning
How successful was the distance education program based on student grades? In the distance learning course, the average quiz scores ranged between 87 and 96. 99% of the students who completed all six quizzes passed with a grade of 75 or higher. The high level of passing grades indicates that students mastered the learning objectives in each unit. In the post-program evaluations students were rated above average to excellent by their mentors in terms of academic skill and effort indicating students had mastered the learning objectives and indicating a measure of reliability of the two evaluation instruments. The student grades indicate that the distance education program was successful in its first year.
Is distance education an effective way of reaching these students? Was it beneficial to choose the Internet over other methods of education delivery? Distance education was the choice of the program designers since students were distributed all over the state of Texas and other methods were impractical and costly. Distance education using the Internet allows for a program to be accessed by a large number of students at a reduced rate of cost (Tuttle, 1998). According to research, distance education seems to be as effective as traditional education with regards to learning outcomes (Hanson & Maushak, 1996).
Students rated the web-site between good and excellent, with no students indicating any difficulties with the level of technology employed in the web-site or required of the students. That the TAS students had an initial expertise in the use of technology was an assumption made by the course designers. The review of literature indicates that gifted students in particular have an interest in and an ability to use the Internet (Casey & Shore, 2000). While Web-based learning may not be the answer for all students, the review of literature supports the use of the Web for gifted students who are self-motivated and have initial expertise in the use of technology (Tuttle, 1998).
One of the major limitations of using the Internet is that not all students may have a computer (Tuttle, 1998). While not all TAS families had computers in their home, those students without computers noted that they used school and library facilities to complete the course. Grants to provide a computer to students without easy access to the Internet could be considered in the future to help support these students.
Another limitation with Web-based courses is slow modem connection speeds. Some students noted in their evaluations that download time was an issue for those with slower modem connections. The decision was made at the close of the second year to put the curriculum on CD-ROM to facilitate access for students since a CD-ROM would allow students faster access to the course content (although a browser connection would still be needed to explore external links).
Students rated the units with a design component higher than the other lessons and also rated the interactive portions of the course between good and excellent. The TAS curriculum was designed with constructivist learning principles and used interactivity as a major component in student learning. Incorporating interactive activities in the delivery of the on-line course is recommended by the review of literature regarding distance learning, and is considered to be important in sustaining student interest and learning (Hammonds, 1998; Web66, 2001; Yahoo, 2001). Interactive simulations, interactions with experts and peers, and creative real-world problem solving were found to be common in NASA and NASA-related education web-sites (Amazing Space 2001; Center for Educational Technology, 2001; Explore Science, 2001; GLOBE, 2001; NASA Quest, 2001). A majority of other non-NASA education sites do not currently use constructivist learning principles or promote interactivity, both of which are considered by researchers as important factors in distance learning (Miodoser, Nachmias, Lahav, & Oren, 2000).
So, was the distance education program effective, and was it beneficial to use the Internet as a method of delivery? Students' high levels of satisfaction and learning in the distance education course indicate that distance education using the Internet was an effective way to deliver the instruction and prepare students for the mentoring workshop. Even though there were a number of limitations, the results of this study suggest that the use of the Internet was the right choice. In addition, students' high levels of satisfaction and learning in the distance education course may have had an impact on their mentoring workshop experience.
The discussion of the results regarding the mentoring workshop component of the program will be addressed in the following three sections based on Kirkpatrick's first three levels of evaluation, reaction, learning, and behavior.
Level I: Reaction
How successful was the mentoring workshop based on student satisfaction levels? The average ratings by students in post-program evaluations of the mentoring workshop were between above average and excellent. The majority of all students rated the mentors and co-ops between above average and excellent. In the post-program survey, 83% of the responding students rated the workshop excellent, and 89% rated the mentors good or excellent indicating a measure of reliability of the post-program evaluation and survey instruments. However, workshop logistics (e.g. hotel, food, and transportation) were only rated as good. Improvements in logistics should be considered when making future improvements to the program.
The research on intervention programs indicate that site visits, interaction with professionals in the field, peer-mentoring, real world problem solving and hands-on activities are important factors contributing to a program's success (Hamilton, 1997; Johns Hopkins University, 2001; Rodriguez, 1997). The results of this study support such claims. The expert briefings, tours, and hands-on activities were noted by students as being the highlight of their stay at NASA.
A very high degree of satisfaction by the students validates the decision to include the workshop components in the program and indicate that it was a significant contributor to the program's success. Student's reactions, however, are not sufficient to determine the overall effectiveness of the workshop. Kirkpatrick's second level, learning, measures whether students mastered the program objectives.
Level II: Learning
How successful was the mentoring workshop based on student grades and changed attitudes? In the post-program evaluations students were rated above average to excellent by their mentors in terms of academic skill and effort indicating that students successfully mastered the project skills required for the summer workshop (completion of a Mars mission design).†
Quantitative measures of learning outcome indicators in the mentoring component of the program were based on the review of literature (National Mentoring Center, 1999; U.S. Dept. of Ed., 1998). The learning outcome indicators in this study include an increase in interest in engineering and a choice of future career in engineering and related disciplines. A majority of students in this study indicated a high level of interest in engineering after the program and a majority of students indicated that the course had a moderate or high impact on their future career choices.
In the post-program evaluation, 75% of the students rated their attitudes towards engineering were improved as a result of the program (25% indicated they were about the same). In the post-program survey 92% of the students indicated the program had some or high impact on their future career choices. In the post-program survey, 86% of the students rated their interest in engineering as moderate or high. These high levels of improved attitudes, program impact and interest levels indicate that the mentoring program had a significant measure of impact on the students' interest in engineering and on their future choice of career.
In post-program surveys, 77% of students indicated a moderate (45%) or high (32%) degree of interest in engineering prior to the program. 93% of the students indicated a moderate (23%) or high (70%) degree of interest in engineering after the program. These statistics indicate an increase (38%) in the percentage of students who had a high interest in the topic after completing the program.† The review of literature indicated that students experiencing an intervention program are significantly impacted in their choice of college major (American Association of University Women, 1992; Molkenthin, 2001; Panitz, 1996)
Qualitative measures of learning outcomes of mentoring programs were identified by the review of literature (Flaxman, 1993, NCLAM, 2001). The learning outcomes used in this study were an increased level of knowledge, reduced feelings of isolation, and an enhanced sense of the ability to achieve. Comments from students indicated that students gained a great deal of knowledge from the program, and experienced reduced feelings of isolation as a result of being part of a group of students with their same interests, and by working with mentors.† Students also indicated that they had an enhanced sense of the ability to achieve as a result of the mentoring workshop.
An unanticipated outcome indictor that resulted from the evaluation was a significant increase in degree intentions after the program. A large percentage of students indicated that their post-program degree intentions were higher post-program than pre-program. Students, who had initially been content with attaining a Bachelors degree, had higher aspirations to Masters or Doctorate degrees after the program. Research has shown that mentoring not only can increase studentís self-confidence, but also their aspirations (McIntosh & Greenlaw, 1990). This is supported by the significant increase in degree attainment intentions by TAS students after the program.
So, how successful was the mentoring workshop based on student grades and changed attitudes? The results of this study indicate that students mastered the program's objectives and experienced a distinct change in attitudes as a result of the intervention.
Was it beneficial to include mentoring as an integral component of the program? Students rated the mentors and co-ops in the program between above average and excellent, in addition they rated the components of the mentoring workshop between above average and excellent. The high ratings of the mentors and the mentoring workshop indicate that the decision to include mentoring was the right choice for the TAS students. Many types of mentoring were designed to be included in the TAS program. Adult and peer mentoring (with engineers, scientists and co-ops), the use of technology (on-line chat and e-mail), the workshop experience (face-to-face mentoring), academic tutoring (student project assessment by mentors) and long-term mentoring (continued guidance by mentors for up to four years) were all incorporated into the program. According to the review of literature each of these components are considered key factors in successful mentoring programs (Marable, 1999; Molkenthin, 2001; Panitz, 1996).†
Scholars reported that the mentors provided role models and helped them define options and career paths indicating that mentoring was an important component in the TAS program. Research indicates that mentors can give gifted students intellectual stimulation, communicate an excitement about the field of science, and help student self-confidence and aspirations grow while feelings of isolation are lessened (Bennett, 1997; Berger, 1990; Casey & Shore, 2000; Kerr, 1985). Gifted students particularly have a desire for adult role models, and have many career options open to them (because of ability) (Kerr, 1991). High levels of post-program interest in engineering support the decision to use mentoring to inspire and guide these students.
The TAS program designers chose to have mentors work with students to design a mission to Mars for presentation at the end of the week. Students rated mentor support of the team projects highly, between above average and excellent. Involving students and mentors in activities and projects that encourage them to work together is considered one of the best ways to facilitate successful mentoring relationships (JETS, 2001; Molkenthin, 2001).
Students rated mentor e-mail, chat sessions and review of project assignments between good and excellent indicating that these methods of interaction with mentors were successful. The chat sessions were rated by students between good and above average, and anecdotally as a good way to meet mentors beforehand. According to the review of literature, the use of the Web to facilitate mentoring relationships is a choice considered appropriate for self-motivated students who seek out and benefit from adult relationships (Bennett, 1997; Electronic Emissary Project, 2001; International Telementor Program, 2001). Additional chat sessions were requested by students, and are recommended by these findings.
Mentors are encouraged to stay in touch with their team members for up to four years. It is recommended by these results that future evaluations address the extent and quality of those continued relationships and their impact on the program's ultimate goal, a choice of career.
Results from the evaluation regarding the mentoring workshop indicate high levels of satisfaction (reaction) and learning, Kirkpatrick's first two levels of evaluation. Student's behavior, Kirkpatrick's third level of evaluation, can indicate whether the program has met its initial goal of impacting students' choice of major in college and future careers.
Level III: Behavior
Does TAS help gifted students make informed college and career choices? The distance education course was designed to give students information about different careers in science, mathematics, engineering and technology in preparation for the mentoring workshop. The mentoring component was included to give students access to role models in various careers, and allow students to visit a facility where professionals were working in those fields. The tours, briefings and activities were intended to provide students with opportunities to talk to professionals other than their mentors and to give them hands on experiences in the field of space science.
The results from the post-program surveys indicate a high number of students actively chose a college major in science, mathematics, engineering or technology at the end of their senior year. Some students noted that while they were initially interested in these majors it wasn't until after the conclusion of the program that they actually decided on a specific choice of major. In the first post-program survey (in the beginning of their senior year), 86% of responding students noted they planned to major in science or engineering. In survey two (after students had applied to colleges), 90% of responding students said they planned to major in science or engineering indicating a measure of reliability of the two survey instruments. Fewer students responded to the second survey however, and thus the increase in numbers may be a result of the more interested students responding to the second survey.
Anecdotally, a number of students indicated that the program gave them specific options regarding their future career plans. The review of literature indicates that students who experience some kind of intervention during their high school years are positively impacted in their decision to choose a major in those fields (American Association of University Women, 1992). In addition, national statistics regarding number of freshmen who initially choose science, mathematics, engineering or technology as a major, note a decline in interest during their time at college (National Science Foundation, 1993). Continuing involvement with mentors in the TAS program may be an effective way for those with an initial interest to sustain interest over time. Future surveys and active solicitation of more participants in the follow-up program are needed to ascertain whether students continue on the same career track.
Is there a correlation between levels of satisfaction and post-program interest in engineering? Students rating of the program components were correlated with their interest in engineering at the conclusion of the program to measure whether their attitudes (learning) were related to their levels of satisfaction (reaction). A correlation factor (r) of .14 was calculated between student's satisfaction ratings of the program components and their level of interest in engineering after the program's conclusion. The resulting correlation indicates a relationship between the first and second level of Kirkpatrick's evaluation.
To support the correlation statistics, the percentage of students who rated the program components as good or excellent (94%) can be compared to the percentage (90%) that indicated they would be majoring in engineering or science in college. These percentages descriptively suggest that there is some relationship between student's reaction and behavior, Kirkpatrick's first and third level of evaluation.
While the correlation statistics are not high, it should be taken into consideration that many of the students were already interested in these fields prior to beginning the program, in fact the students were initially chosen on that basis. Students' change in plans regarding their level of degree attainment is perhaps a more significant behavior to consider as a measure of program success.† Higher levels of aspiration after the program can be considered to be due in part to high levels of satisfaction (reaction) and learning as a result of the program.
While there appears to be an association between reaction and learning, and between reaction and behavior, future surveys are needed to determine ultimate program impact (Kirkpatrick's fourth level, results) by student's final choice of college major and career.
Level IV: Results
Kirkpatrick's fourth level, results, addresses the tangible results that can be measured at the close of a program.† While student's final choice of career is still several years away, we can look at two outcomes, costs and benefits, of the first year of the program. Costs and benefits of the program will be addressed separately in the following two sections.
Costs of virtual schools match approximately the cost of the TAS program (Hammonds, 1998; Web66, 2001; Yahoo, 2001). Summer camp costs for a six-day program (such as Space Camp) are comparable as well. Organizations like NASA interested in long-term results need to match their ability to provide such programs with their investment in the intended outcomes (a larger workforce). Since long-term results are not a focus of this particular study, one can only say that considerations should be made on a case by case basis by the organizations developing similar programs. Future studies to determine how many of the TAS students enter the science, mathematics, engineering and technology workforce can then be used to determine whether the cost of the program has was worth the return on investment.
How does this study benefit stakeholders, future students, other organizations and instructional designers? The results of this study can benefit key stakeholders (entities with a vested interest in the increase of engineers and scientists in the workforce) by providing positive statistics regarding the levels of student reaction, learning and behavior. The first three levels are however merely an indication of the final program outcome, results, (students' choice of careers).
It is believed that continued funding of the program is warranted as a result of this initial program evaluation at Kirkpatrick's first three levels of evaluation, with suggested improvements regarding levels of interactivity in the distance education course, and by choice of hotel, food and transportation.
Other industries considering replicating the program (to meet the need for future high levels of scientists and engineers entering the workforce) can use the results of this first evaluation to support their plans. Educational organizations (who are looking for model Web and mentoring programs) and instructional designers (who are looking for Web-based models that utilize mentoring) can use the TAS model as a blueprint should they encounter a similar need for an interactive Web-based educational and mentoring program.
The evaluation results will be used by TAS program managers to improve the quality of the Web-site, the on-line course and the summer workshop for future students. Several changes and improvements were made after informal analysis of the results at the end of the first year. The evaluation indicates additional improvements in levels of interactivity and workshop logistics for future years.
Currently most existing science, mathematics, engineering and technology intervention programs are aimed at high school students, some are aimed at gifted students, and a few utilize mentoring. TAS uniquely targets gifted high school students using Web-based distance learning technology and real-time mentoring.
Based on the results at level one, reaction, a majority of the original 196 students participated in the program through its conclusion (89%) and a majority of those students rated the program good or excellent (94%) indicating a proportional correlation.
Level two results of learning, indicate a high knowledge acquisition (a 99% passing rate), and high impact on attitudes towards engineering (92%) as a result of the program. Student testimonials noted creativity and interactivity as important factors in the distance learning component validating initial program assumptions, and requested even more of both, factors noted in the review of literature on distance education as sustaining student interest and fostering student learning.
The results of the two follow-up surveys indicate that the behavior of a significant number of students was impacted in their intended choice of science, mathematics, engineering and technology majors (86% in survey one, and 90% in survey two).
The level one, two and three results obtained in this study indicate that the program was successful at all three levels of Kirkpatrick's evaluation model. †However some may argue that the success is due to student's prior interest in the topic. While it is true that students may have already considered choosing a science or engineering major prior to participation in the TAS program; some measures of this tendency can still be attributed to the program itself. Research has shown that gifted students have multi-potentiality (Berger, 1990; Kerr, 1991) and often a more difficult time choosing a path without experiencing an intervention of some kind.† Interventions can include distance learning, workshops and mentoring. After the TAS intervention, a combination of all three elements, student's final choices of college majors (behavior) can if not entirely, at least in part, be attributed to high levels of satisfaction (reaction) and learning.
According to the review of literature, one of the best ways to improve the validity of an evaluation study is to replicate it (Wolf, 1979). One way is to carry out the evaluation study with successive groups of learners over time. The results of the two studies can be compared and judgments and decisions about the program's fate made at a future time. While inconsistencies can arise if the program is significantly altered from one year to the next, the results of more than one study are generally considered to be superior as a basis for making important decisions about a program. It is recommended that future studies continue to use the Kirkpatrick model as a structure for evaluation, moving up to the fourth level, results, scholar's choice of career, as an indicator of the program's ultimate success.