Chapter 4

Results

The TAS program has two major components: a distance education course and a mentoring workshop. Reactions, learning and behavior have been assessed quantitatively for both of the components using unit tests, post-program evaluations, and follow-up surveys, and qualitatively using mentor roundtable discussions, unsolicited testimonials, and video interviews. A correlation between program satisfaction and decision to study engineering was also completed. The next four sections cover respectively the first three levels of the Kirkpatrick model of evaluation: reactions, learning and behavior, and the correlation.

Level one of Kirkpatrick's model is reaction. Reaction is simply defined as customer satisfaction. If participants are not happy, they will not be as motivated to learn. Satisfaction is also a major variable in participation levels.

Scholar satisfaction levels with the distance education course and mentoring workshop were gathered twice, from post-program scholar evaluations and a post-program survey. The next two sections cover the results from the data gathered from scholars regarding the distance education and mentoring workshops.

The distance learning course was rated by 131 (75%) of the scholars in the post-program evaluation and by 109 (62%) eight months later in post-program survey two. Post-program evaluation questions addressed satisfaction levels for each of the six course units, and Web-site organization, content, navigation, and downloads. The post-program survey asked scholars how they would rate the entire on-line course.

Units. Each of the four unit components, Liftoff (reading), Mission (assignment), Extended Mission (extensions), and Quiz were rated by scholars from 1-5, with 5 being the highest rating (1 – Poor; 2 – Fair; 3 – Good; 4 – Above Average; 5 – Excellent).

The satisfaction (reaction) scores for each of the six units are described by the mean, median and standard deviation of the raw scores, and illustrated with a frequency polygon of those scores below. Table v and Chart x illustrates the mean scholar ratings for each unit section.

Chart v. Mean scholar ratings of satisfaction with each unit section (n=131).

The Unit 3, 5 and 6 missions were rated highest by the scholars. All three of these missions had an original design component in them while the other three lessons did not. Unit 6 (designing a Mars colony) received the highest single rating, and the average rating of all units was 3.89, between good and above average.

Anecdotal comments about the units from the scholar evaluations are given below.

• I think it was a great idea for the students to get a fresh head start on what the program was going to be about. It was a great unit.
• I particularly like this section for the fact that it gave useful information on Mars itself.

• I found through this section that I really love geology!
• This was a really interesting topic and I really enjoyed it.

• I liked this design phase the most. The only problem I had was worrying about drawing my design, but after I got it done it showed me that when I stick to something, I can do it.
• I liked the design your own this or that; these kinds of activities are a huge benefit because they make you think more.
• This unit was helpful for me because I was able to take a nice peek into the robots that NASA was using. It also gave a great opportunity for all of us to apply our creativity in building plans for a rover.

• Even though I'm not interested in the medical sector of space exploration, this unit was still fun to read. I liked learning about the present astronauts and cosmonauts along with what they took with them, and what they have to say about long-duration space voyages.
• Mission four was great to know about the human factors. I also corrected myself on some stuff that I had wrong.

• The lessons explained things concisely and informatively. They got the point across without becoming boring. I learned a great deal about the space program!
• This unit was so much fun. I actually got to use all of my knowledge and apply it to other aspects of the mission. Creativity was also involved, so that was fun.
• I loved the assignment. I went all the way to the agricultural extension agent's office to ask questions. I wished that more was done on hydroponics/space agriculture.

• This one was the best. It had awesome information and even better, an AWESOME assignment. I had so much fun doing this one.
• I think I liked this one most of all because none of it was really told to us. We got to go explore and think about things all on our own. The theories were there, but we got to put our own spin on things, and that was exciting. The units kept getting better.
• I thought that the material in the correspondence course was a little simplistic for high-school aged students with an interest in science and engineering. However, if the course work had been very rigorous, I'm not sure that I could have balanced it with my academic load.

Web-site. Web-site components and interactive features were rated by 131 (75%) of the scholars in a post-program evaluation. Scholars rated the four major components of the Web-site: organization, content, downloadable forms and navigation, from 1-5, with 5 being the highest rating: 1 – Poor; 2 – Fair; 3 – Good; 4 – Above Average; 5 – Excellent.

The questions regarding the Web-site were: How well was the Web-site organized; How would you rate the content of the Web-site; How well were you able to download important forms; and Was the Web-site easy to navigate?

The evaluation scores for each component of the Web-site are described by the mean, median and standard deviation of the raw scores, and illustrated with a frequency polygon of those scores.

Table n. Scholar satisfaction rating of Web-site components.

Chart n. Frequency polygon of scholar satisfaction ratings of web-site components.

Post-program Surveys. The distance education course component was rated by 109 (62%) of the scholars, eight months after the program ended, in the second post-program survey. The course was rated by scholars from 1-4, with 4 being the highest rating: 1 – Poor; 2 – Fair; 3 – Good; 4 – Excellent. Table x and Charts x illustrate the rating of the distance education course from the follow-up survey.

Table x. Scholar satisfaction ratings of the on-line course (n=109).

Chart 19. Scholar satisfaction ratings of the on-line course (n=109).

Anecdotal information about the Web-site from the evaluation comments are given below.

• I was able to get everything I needed very quick and easily.
• I didn't have any problem navigating or finding information because I constantly wanted to know more. I did hear though that many people didn't read the dress code or rules or schedule.
• Need automatic message board that pops up when you open student page to tell us we have forms to fill out, because it was by chance I saw that link and clicked it - the day before the deadline.
• I would like to be able to access individual pages quick than having to flip through several pages.
• Generally, everything was really good, but some of the forms and links wouldn't work for me and made some of the assignments a little harder.
• Navigating through the Web-site was extremely easy. The information in the Web-site was very useful. I experienced a quick downloading time. However, the server was down sometimes.

Scholars evaluated the on-site summer mentoring workshop twice, once in the post-program evaluation and again, eight months later in the second post-program survey. Post-program evaluation questions addressed specific program components including briefings, tours, trips, special events, logistics and e-mail. Scholars were also asked to rate the mentor's e-mail, final project review and chat session. The post-program survey asked scholars to rate the workshop and the mentoring experience separately.

The following sections present post-program evaluation workshop and mentor ratings, and the survey ratings of both components. Anecdotal comments follow each section.

Workshop ratings. Workshop components were rated by scholars in the post-program evaluation. Ratings were from 1-5, with 5 being the highest rating: 1 – Poor; 2 – Fair; 3 – Good; 4 – Above Average; 5 – Excellent. Each component of the mentoring workshop are described by the mean, median and standard deviation of the raw scores, and illustrated with a frequency polygon of those scores. Table v and Chart x illustrates the average scores for all components.

Table x. Scholar satisfaction ratings of workshop components (n=131).

Anecdotal comments about the summer workshop from scholar evaluations and testimonials are given below.

• I thought the e-mails were all-around very helpful to the preparation and planning of the week's activities.
• I haven't seen this much organization, communication, and a structured learning environment like this since boarding school in seventh grade.
• I did enjoy Astronaut Gus L.'s presentation and how he took time out of his busy schedule to have lunch (pizza) with us. I also had a blast talking to several informative NASA JSC engineers.
• I really enjoyed the astronaut presentation. The other presentations were also helpful, especially getting the opportunity to ask questions about the different engineering professions.
• Especially enjoyed the ones who had been with NASA for 25+ years and really knew their stuff and had a passion for their work.
• These tours were one of my most favorite parts of the whole program. Getting to actually see firsthand what research was being done and how astronauts trained.
• You have to do the mission control center every year, it was amazing. I cried when Randy S. was telling us about Challenger. It was awesome to know that it all happened right where I was standing.
• While I enjoyed all the field trips the most memorable was the Challenger Center [simulation]. That experience was probably my favorite of the entire week.
• I loved shooting off the rockets! It was a real challenge trying to get them close to the launch pad because it was really windy that day!
• The hotel was a great place to stay. The meals were okay except having to eat pizza and sandwiches every other day for lunch.
• The banquet was set up lovely and a great way to conclude the week.

Mentor ratings. Scholars rated the mentoring experience by ranking the mentor’s e-mails, final project reviews, and chat sessions. In addition, the support given by the mentors, staff and co-ops for scholars' team projects were ranked. Components were rated by scholars from 1-5, with 5 being the highest rating: 1 – Poor; 2 – Fair; 3 – Good; 4 – Above Average; 5 – Excellent. Tables 31 and 32 and Charts 17 and 18 illustrate the statistics compiled from the evaluations using mean, median, standard deviation and a frequency polygon of scores.

 Anecdotal comments about the mentoring experience from the evaluations are given below.

• I wish we would have several chat sessions throughout the year.
• Dan B. was excellent! He really helped us out, but he made us think.
• My mentor Amy R. was very prompt with e-mail messages and interested in with keeping e-mail relations. The chat session is an excellent concept for team members to get acquainted and ask the mentor and co-op about the project and trip.
• Having a mentor and co-op was very helpful and guidance and on referencing for background information while working on the team project.
• We had the very coolest guy as our mentor, and our co-op was wonderful. I don't think I've ever heard someone use so many acronyms in one sentence.

Post-Program Surveys. The mentoring workshop and mentoring relationship were rated by 109 of the 175 scholars who participated in the summer workshop (62%) eight months after the program ended in the second post-program survey.

Components were rated by scholars from 1-4, with 4 being the highest rating: 1 – Poor; 2 – Fair; 3 – Good; 4 – Excellent. Tables y and z and Charts y and z illustrate the scholar satisfaction ratings of the mentoring workshop and the mentoring relationship.

Anecdotal comments from the survey regarding the workshop and mentoring relationship are given below.

• I think this is an excellent program for the youth to participate in. The amount of knowledge that I have gained is so great that I have to convince many others to go, too.
• I think that it was a lot of fun and it really encouraged me to pursue a career in engineering.
• The best part was meeting other kids with the same interests, passions, and aspirations as myself.
• I would like more time to actually work on our project, we only had a few hours a day.
• You might want to be more strict with the application process so that people who don't finish their work can't come.
• Overall, the opportunity to tour the facilities, meet the people actually working there, get involved, and spend a week on site was a wonderful chance to focus on the future. It has certainly influenced my plans, and I hope that many more students will be able to experience Johnson Space Center in the upcoming years.

Level two of Kirkpatrick's model learning is a measure of the extent to which participants improve knowledge and skills, and change attitudes as a result of a program.

Scholar's acquisition of knowledge and skills were assessed for the distance education course and the mentoring workshop. Quiz scores were used to measure the level of learning in the distance education course. Scholar grades for the summer workshop were used to measure the level of skill and effort.

Attitudes regarding engineering were gathered from post-program evaluations, two surveys and scholar testimonials. The following three sections cover scholar achievement levels in the distance education course, the mentoring workshop, and attitudinal changes as a result of the program.

Quizzes. The quizzes reflected the distance learning course objectives listed at the beginning of each unit. A 75% or better grade was deemed a passing level. 100% of the scholars completed at least one quiz. 129 (66%) of the scholars completed all six quizzes, 99% of those scholars passed each course unit with a score of 75% or better. Table x and Chart x illustrate the mean quiz scores for each individual quiz described by mean, median and standard deviation, and a frequency polygon of raw scores.

Chart x. Frequency polygon of all scholar quiz scores.

Mentors evaluated 41 scholars (23% of the total workshop participants) in their evaluations. 17 of the 48 mentors completed the on-line evaluation, or 35.4%. Each scholar was assessed by their mentor on academic skills and interest level from 1-5, with 5 being the highest rating: 1 – Poor; 2 – Fair; 3 – Good; 4 – Above Average; 5 – Excellent. Scholar skill and effort in the program intervention are described in Table x.

Table x. Academic skill and effort level of scholars as rated by mentors.

Scholar attitudes regarding the program's impact on future career choices and their interest in engineering were gathered from post-program evaluations, post-program surveys and testimonials.

Post-program evaluations. Post-program evaluation questions addressed scholars' change in attitudes towards engineering. Changes in attitude were rated as improved, about the same, or less than before, and are described in Table x.

Table x. Scholars attitudes towards engineering (n=131).

Post-program surveys. Post-program attitudes towards engineering were gathered from two post-program surveys. 130 (74%) scholars completed the first post-program survey. 109 (62%) scholars participated in the second post-program survey.

Program impact on future career, interest in engineering, pre- and post-program degree intentions were gathered in survey one. Data from this survey are illustrated in Table x, y, z, and a and Charts x, y, and a.

Table x. Survey one: Scholar ratings of impact on future career choices (n=130).

Chart x. Survey one: Scholar ratings of impact on future career choices (n=130).

Chart y. Survey one: Scholar ratings of interest in engineering.

Table z. Survey one: Scholar ratings of degree intentions pre-program (n=130).

Table z. Survey one: Scholar ratings of degree intentions post-program (n=130).

Pre-and post program attitudes towards in engineering were gathered in survey two and are shown in Tables x and y, and Chart y.

Testimonials. Video testimonial questions included an articulation of attitudinal changes as a result of the program. Unsolicited testimonials were received from scholars, mentors, teachers and parents over the course of several months following the completion of the program (and continue to the present). Anecdotal comments from the testimonials are given below.

• I never had considered working for NASA as a reality, but now I can’t wait to seize the opportunity.
• I've decided I'll be majoring in Mechanical Engineering, probably with a concentration in propulsion, and a double minor in math and chemistry.
• Before I met the engineers and saw how and where they work, I was pretty sure that I wanted to be an engineer. Now, though, I am certain that being an engineer is what I truly want to do.
• I would just like to say that being in your program has opened so many doors for me and I am eternally thankful and in your debt for the opportunity that you gave me. I just hope that someday I will be able to contribute to the NASA program as much as it has to me.
• My plans for the future have changed after this week.  Before the week I was very unsure about what I wanted for my life.  Now I have aspirations to work for NASA as a mechanical engineer. 
• Honestly the week at NASA changed my life. I has always wanted to be an astronaut but never had any idea of how to go about doing so.
• Thank you for helping me to realize what I want to be a part of… You don’t know how much it means to someone who has worried what he might do. This program has changed my life.
• The week was great and it has really impacted my career goals and college plans. I would probably still be lost as to what I wanted to study in college if I had never gone to TAS.
• Because of this program I have decided to continue taking science even though it is not mandatory.
• I would recommend the program to anyone. I learned a lot during the week and it opened my eyes to the aerospace industry as a career.

Kirkpatrick's third level, behavior, describes a participant's action after an intervention. For a program to be considered a success at level three, a participant must actively choose a specific behavior as a result of the intervention.

Scholars' choice of college major was elicited three times from scholars. Initially from the post-program evaluation, and again in two follow-up surveys. The first survey occurred two months after the program was completed, and the second survey, eight months after the program was completed. Scholars were in the final months of their high school senior year when they completed the second survey and had been accepted into colleges.

Post-program evaluation. 131 (62%) scholars rated their choice of major in the post-program evaluation. 110 (84%) indicated a major in science, mathematics, engineering or technology. Table 1 indicates scholars' choice of majors.

Table x. Post-program evaluation: scholars' choice of major (n=131).

Post-program surveys. Post-program surveys questions polled scholar's choice of majors. 130 (74%) scholars participated in the first post-program survey, 109 (62%) scholars participated in the second post-program survey. Scholars choice of majors indicated in survey one and survey two are given in the Tables x and y. Chart y indicates scholars choice of major in both surveys by discipline.

Table x. Survey one: scholars choice of major (n=130).

Table x. Survey two: scholars choice of major (n=109).

Chart x. Surveys one and two: scholars intended choice of major by discipline.

Correlation Statistics

A correlation between scholar satisfaction levels with their post-program interest in engineering was calculated from the second survey data (most recent). Scholars rated the distance education, mentoring workshop component and mentoring relationship from 1-4 (with 4 being the highest rating). Scholars rated their interest in engineering from 1-4 (with 4 being the highest rating). Mean satisfaction levels of the program components were correlated with each scholar's rating of their interest in engineering using the Pearson product-moment correlation. A correlation factor (r) of .14 was the result, indicating some level of correlation (Appendix H).

Of the 109 scholars who completed survey two, 93% rated the distance education component as good or excellent, 100% rated the mentoring workshop at good or excellent, and 89% rated the mentoring relationship as good or excellent. Of these scholars, 90% indicated they will be majoring in engineering or science in college, descriptively supporting a moderate correlation between satisfaction levels and choice of major.

Appendix H

Correlation Statistics

n	Attitudes	Satisfaction
1	3.00	3.33
2	4.00	3.66
3	3.00	3.33
4	4.00	3.66
5	1.00	3.33
6	4.00	3.33
7	3.00	3.00
8	4.00	3.66
9	2.00	3.66
10	3.00	3.00
11	4.00	3.33
12	4.00	4.00
13	4.00	3.66
14	4.00	2.66
15	1.00	3.00
16	4.00	4.00
17	3.00	3.00
18	4.00	3.66
19	3.00	4.00
20	4.00	4.00
21	4.00	3.33
22	4.00	3.66
23	4.00	3.66
24	4.00	3.66
25	4.00	2.66
26	4.00	3.66
27	4.00	3.66
28	3.00	3.66
29	4.00	3.66
30	4.00	3.66
31	3.00	4.00
32	2.00	3.66
33	3.00	3.33
34	3.00	3.00
35	4.00	3.66
36	4.00	3.66
37	4.00	4.00
38	3.00	3.66
39	3.00	4.00
40	3.00	4.00
41	4.00	3.66
42	2.00	3.66
43	4.00	3.66
44	4.00	3.33
45	4.00	3.66
46	3.00	3.66
47	4.00	4.00
48	4.00	3.33
49	4.00	4.00
50	4.00	4.00
51	3.00	2.66
52	4.00	4.00
53	4.00	2.66
54	3.00	4.00
55	4.00	3.66
56	4.00	4.00
57	4.00	3.66
58	3.00	3.33
59	4.00	3.66
60	4.00	3.00
61	4.00	3.33
62	4.00	3.66
63	3.00	2.66
64	4.00	3.66
65	4.00	4.00
66	3.00	3.33
67	4.00	3.66
68	4.00	2.66
69	4.00	3.66
70	3.00	2.66
71	4.00	3.66
72	4.00	3.66
73	4.00	3.66
74	4.00	3.66
75	4.00	3.66
76	3.00	3.66
77	4.00	3.00
78	4.00	3.33
79	4.00	3.00
80	4.00	3.66
81	4.00	3.00
82	4.00	3.00
83	4.00	4.00
84	3.00	3.66
85	4.00	4.00
86	4.00	3.33
87	4.00	3.66
88	3.00	4.00
89	1.00	3.00
90	4.00	3.66
91	4.00	3.33
92	4.00	3.66
93	3.00	3.00
94	4.00	4.00
95	4.00	4.00
96	4.00	3.66
97	2.00	3.66
98	4.00	3.66
99	4.00	4.00
100	4.00	3.66
101	4.00	3.66
102	4.00	2.66
103	4.00	2.66
104	3.00	4.00
105	4.00	4.00
106	4.00	3.66
107	3.00	4.00
108	4.00	3.00
109	4.00	3.00
Mean	3.61	3.52
Standard Deviation	0.69	0.40
Correlation	0.14