Chapter 4

Results

 

Introduction

 

The TAS program has two major components: a distance education course and a mentoring workshop. Reactions, learning and behavior have been assessed quantitatively for both of the components using unit tests, post-program evaluations, and follow-up surveys, and qualitatively using mentor roundtable discussions, unsolicited testimonials, and video interviews. A correlation between program satisfaction and decision to study engineering was also completed. The next four sections cover respectively the first three levels of the Kirkpatrick model of evaluation: reactions, learning and behavior, and the correlation.

Reaction

 

Level one of Kirkpatrick's model is reaction. Reaction is simply defined as customer satisfaction. If participants are not happy, they will not be as motivated to learn. Satisfaction is also a major variable in participation levels.

Scholar satisfaction levels with the distance education course and mentoring workshop were gathered twice, from post-program scholar evaluations and a post-program survey. The next two sections cover the results from the data gathered from scholars regarding the distance education and mentoring workshops.

 

Distance Education

 

The distance learning course was rated by 131 (75%) of the scholars in the post-program evaluation and by 109 (62%) eight months later in post-program survey two. Post-program evaluation questions addressed satisfaction levels for each of the six course units, and Web-site organization, content, navigation, and downloads. The post-program survey asked scholars how they would rate the entire on-line course.

Units. Each of the four unit components, Liftoff (reading), Mission (assignment), Extended Mission (extensions), and Quiz were rated by scholars from 1-5, with 5 being the highest rating (1 – Poor; 2 – Fair; 3 – Good; 4 – Above Average; 5 – Excellent).

The satisfaction (reaction) scores for each of the six units are described by the mean, median and standard deviation of the raw scores, and illustrated with a frequency polygon of those scores below. Table v and Chart x illustrates the mean scholar ratings for each unit section.

Table x. Mean scholar satisfaction ratings for all units, each unit, and each section (n=131).

Liftoff

Mission

Extension

Quiz

Mean

Unit 1

3.9

3.6

3.9

3.7

3.78

Unit 2

3.9

3.7

3.8

3.8

3.80

Unit 3

4

4

3.9

3.8

3.93

Unit 4

4

3.8

3.8

3.8

3.85

Unit 5

4.1

4

3.9

3.8

3.95

Unit 6

4.1

4.2

4

3.8

4.03

Mean

3.89

 

Chart v. Mean scholar ratings of satisfaction with each unit section (n=131).

 

 

The Unit 3, 5 and 6 missions were rated highest by the scholars. All three of these missions had an original design component in them while the other three lessons did not. Unit 6 (designing a Mars colony) received the highest single rating, and the average rating of all units was 3.89, between good and above average.

Anecdotal comments about the units from the scholar evaluations are given below.

Unit 1 - Mars of the Mind (History).

Unit 2 - Mars Rocks (Geology).

Unit 3 - Robotic Missions to Mars.

Unit 4 - The Human Factor.

Unit 5 - Mission: Possible - Engineering designs for human missions to Mars.

 

Unit 6 - Design a Mars Colony.

Web-site. Web-site components and interactive features were rated by 131 (75%) of the scholars in a post-program evaluation. Scholars rated the four major components of the Web-site: organization, content, downloadable forms and navigation, from 1-5, with 5 being the highest rating: 1 – Poor; 2 – Fair; 3 – Good; 4 – Above Average; 5 – Excellent.

The questions regarding the Web-site were: How well was the Web-site organized; How would you rate the content of the Web-site; How well were you able to download important forms; and Was the Web-site easy to navigate?

The evaluation scores for each component of the Web-site are described by the mean, median and standard deviation of the raw scores, and illustrated with a frequency polygon of those scores.

 

Table n. Scholar satisfaction rating of Web-site components.

Organization

Content

Forms

Navigation

Mean

4.2

4.3

4.0

3.8

Median

4

4

4

4

Standard Deviation

0.92

0.87

1.01

1.07

 

Chart n. Frequency polygon of scholar satisfaction ratings of web-site components.

 

Post-program Surveys. The distance education course component was rated by 109 (62%) of the scholars, eight months after the program ended, in the second post-program survey. The course was rated by scholars from 1-4, with 4 being the highest rating: 1 – Poor; 2 – Fair; 3 – Good; 4 – Excellent. Table x and Charts x illustrate the rating of the distance education course from the follow-up survey.

Table x. Scholar satisfaction ratings of the on-line course (n=109).

Scholars

Percentage

Excellent

35

32%

Good

66

61%

Fair

8

7%

Poor

0

0%

 

Chart 19. Scholar satisfaction ratings of the on-line course (n=109).

Anecdotal information about the Web-site from the evaluation comments are given below.

Mentoring Workshop

 

Scholars evaluated the on-site summer mentoring workshop twice, once in the post-program evaluation and again, eight months later in the second post-program survey. Post-program evaluation questions addressed specific program components including briefings, tours, trips, special events, logistics and e-mail. Scholars were also asked to rate the mentor's e-mail, final project review and chat session. The post-program survey asked scholars to rate the workshop and the mentoring experience separately.

The following sections present post-program evaluation workshop and mentor ratings, and the survey ratings of both components. Anecdotal comments follow each section.

Workshop ratings. Workshop components were rated by scholars in the post-program evaluation. Ratings were from 1-5, with 5 being the highest rating: 1 – Poor; 2 – Fair; 3 – Good; 4 – Above Average; 5 – Excellent. Each component of the mentoring workshop are described by the mean, median and standard deviation of the raw scores, and illustrated with a frequency polygon of those scores. Table v and Chart x illustrates the average scores for all components.

Table x. Scholar satisfaction ratings of workshop components (n=131).

Briefings

Tours

Trips

Events

Logistics

E-mail

Mean

4.1

4.2

4.3

4.2

3.8

4.3

Median

4.1

4.2

4.3

4.4

4

4.2

Standard Deviation

0.34

0.34

0.15

0.38

0.47

0.21

Chart x. Scholar satisfaction ratings of workshop components (n=131).

Anecdotal comments about the summer workshop from scholar evaluations and testimonials are given below.

Mentor ratings. Scholars rated the mentoring experience by ranking the mentor’s e-mails, final project reviews, and chat sessions. In addition, the support given by the mentors, staff and co-ops for scholars' team projects were ranked. Components were rated by scholars from 1-5, with 5 being the highest rating: 1 – Poor; 2 – Fair; 3 – Good; 4 – Above Average; 5 – Excellent. Tables 31 and 32 and Charts 17 and 18 illustrate the statistics compiled from the evaluations using mean, median, standard deviation and a frequency polygon of scores.

Table 31. Scholar satisfaction ratings of mentor e-mail, project reviews and chat

sessions.

Mentor E-mail

Mentor Project

Mentor Chat

n

120

107

96

Mean

4.29

4.23

3.82

Median

5

5

4

Standard Deviation

1.02

0.98

1.20

Chart 17. Frequency polygon of scholar satisfaction ratings of mentor e-mail,

project reviews and chat sessions.

 

 

 

Table 32. Scholar satisfaction ratings of support by mentors, staff and co-ops on

team projects.

Mentors

Staff

Co-ops

n

129

84

129

Mean

4.58

4.44

4.66

Median

5

5

5

Standard Deviation

0.87

0.80

0.73

 

Chart 18. Frequency polygon of scholar satisfaction ratings of support by mentors and co-ops on team projects.

 

 Anecdotal comments about the mentoring experience from the evaluations are given below.

Post-Program Surveys. The mentoring workshop and mentoring relationship were rated by 109 of the 175 scholars who participated in the summer workshop (62%) eight months after the program ended in the second post-program survey.

Components were rated by scholars from 1-4, with 4 being the highest rating: 1 – Poor; 2 – Fair; 3 – Good; 4 – Excellent. Tables y and z and Charts y and z illustrate the scholar satisfaction ratings of the mentoring workshop and the mentoring relationship.

 

Table x. Survey two: scholar satisfaction ratings of the mentoring workshop (n=109).

Scholars

Percentage

Excellent

91

83%

Good

18

17%

Fair

0

0%

Poor

0

0%

 

Chart 20. Survey two: scholar satisfaction ratings of the mentoring workshop

(n=109).

Table x. Survey two: scholar satisfaction ratings of the mentoring relationship.

Scholars

Percentage

Excellent

66

60%

Good

32

29%

Fair

10

10%

Poor

1

1%

Total

109

100%

 

Chart 21. Survey two: scholar satisfaction ratings of the mentoring relationship.

 

Anecdotal comments from the survey regarding the workshop and mentoring relationship are given below.

Learning

 

Level two of Kirkpatrick's model learning is a measure of the extent to which participants improve knowledge and skills, and change attitudes as a result of a program.

Scholar's acquisition of knowledge and skills were assessed for the distance education course and the mentoring workshop. Quiz scores were used to measure the level of learning in the distance education course. Scholar grades for the summer workshop were used to measure the level of skill and effort.

Attitudes regarding engineering were gathered from post-program evaluations, two surveys and scholar testimonials. The following three sections cover scholar achievement levels in the distance education course, the mentoring workshop, and attitudinal changes as a result of the program.

Quizzes. The quizzes reflected the distance learning course objectives listed at the beginning of each unit. A 75% or better grade was deemed a passing level. 100% of the scholars completed at least one quiz. 129 (66%) of the scholars completed all six quizzes, 99% of those scholars passed each course unit with a score of 75% or better. Table x and Chart x illustrate the mean quiz scores for each individual quiz described by mean, median and standard deviation, and a frequency polygon of raw scores.

Table x. Average scholar grades on quizzes 1-6.

Quiz 1

Quiz 2

Quiz 3

Quiz 4

Quiz 5

Quiz 6

n

196

182

169

166

150

143

Mean

91.75

93.66

87.27

95.85

88.65

92.24

Median

100

100

87.5

100

87.5

100

Standard Deviation

10.93

9.94

16.19

9.17

12.84

11.90

Chart x. Frequency polygon of all scholar quiz scores.

Scholar grades

Mentors evaluated 41 scholars (23% of the total workshop participants) in their evaluations. 17 of the 48 mentors completed the on-line evaluation, or 35.4%. Each scholar was assessed by their mentor on academic skills and interest level from 1-5, with 5 being the highest rating: 1 – Poor; 2 – Fair; 3 – Good; 4 – Above Average; 5 – Excellent. Scholar skill and effort in the program intervention are described in Table x.

 

 

 

 

Table x. Academic skill and effort level of scholars as rated by mentors.

Academic Skill

Effort

n

41

41

Mean

4.07

4.17

Median

4

5

Standard Deviation

0.75

1.06

 

Attitudes

Scholar attitudes regarding the program's impact on future career choices and their interest in engineering were gathered from post-program evaluations, post-program surveys and testimonials.

Post-program evaluations. Post-program evaluation questions addressed scholars' change in attitudes towards engineering. Changes in attitude were rated as improved, about the same, or less than before, and are described in Table x.

Table x. Scholars attitudes towards engineering (n=131).

Attitudes towards engineering

Scholars

Percentage

Improved

99

75%

About the same

33

25%

Less than before

0

0%

 

Post-program surveys. Post-program attitudes towards engineering were gathered from two post-program surveys. 130 (74%) scholars completed the first post-program survey. 109 (62%) scholars participated in the second post-program survey.

Program impact on future career, interest in engineering, pre- and post-program degree intentions were gathered in survey one. Data from this survey are illustrated in Table x, y, z, and a and Charts x, y, and a.

Table x. Survey one: Scholar ratings of impact on future career choices (n=130).

Impact

Scholars

Percentage

None

2

1%

Very little

9

7%

Some

74

57%

High

45

35%

Chart x. Survey one: Scholar ratings of impact on future career choices (n=130).

 

Table y. Survey one: Scholar ratings of interest in engineering.

Interest level

Scholars

Percentage

Not interested

5

4%

Neutral

13

10%

Moderately Interested

25

19%

Highly Interested

87

67%

Total

130

100%

Chart y. Survey one: Scholar ratings of interest in engineering.

 

 

Table z. Survey one: Scholar ratings of degree intentions pre-program (n=130).

Intended degree

Scholars

Percentage

Doctorate

42

32%

Masters

44

34%

Bachelors

34

26%

No college

0

0%

Undecided

9

7%

None

1

1%

 

Table z. Survey one: Scholar ratings of degree intentions post-program (n=130).

Intended degree

Scholars

Percentage

Doctorate

52

40%

Masters

50

38%

Bachelors

20

15%

No college

0

0%

Undecided

7

5%

None

1

1%

 

Chart z. Survey one: Scholar ratings of degree intentions pre- and post-program (n=130).

Pre-and post program attitudes towards in engineering were gathered in survey two and are shown in Tables x and y, and Chart y.

Table z. Survey two: Scholar ratings of interest in engineering pre-program (n=109).

Interest level

Scholars

Percentage

Strongly against

0

0%

Not interested

8

7%

Neutral

17

16%

Moderately Interested

49

45%

Highly Interested

35

32%

 

Table z. Survey two: Scholar ratings of interest in engineering post-program (n=109).

Interest level

Scholars

Percentage

Not interested

3

3%

Neutral

4

4%

Moderately Interested

26

23%

Highly Interested

76

70%

 

Chart z. Survey two: Scholar ratings of interest in engineering pre- and post-program (n=109).

Testimonials. Video testimonial questions included an articulation of attitudinal changes as a result of the program. Unsolicited testimonials were received from scholars, mentors, teachers and parents over the course of several months following the completion of the program (and continue to the present). Anecdotal comments from the testimonials are given below.

Behavior

Kirkpatrick's third level, behavior, describes a participant's action after an intervention. For a program to be considered a success at level three, a participant must actively choose a specific behavior as a result of the intervention.

Scholars' choice of college major was elicited three times from scholars. Initially from the post-program evaluation, and again in two follow-up surveys. The first survey occurred two months after the program was completed, and the second survey, eight months after the program was completed. Scholars were in the final months of their high school senior year when they completed the second survey and had been accepted into colleges.

Post-program evaluation. 131 (62%) scholars rated their choice of major in the post-program evaluation. 110 (84%) indicated a major in science, mathematics, engineering or technology. Table 1 indicates scholars' choice of majors.

  

Table x. Post-program evaluation: scholars' choice of major (n=131).

Intended Career

Scholars

Percentage

Engineering, Science, Mathematics or Technology

110

84%

Other

21

16%

 

Post-program surveys. Post-program surveys questions polled scholar's choice of majors. 130 (74%) scholars participated in the first post-program survey, 109 (62%) scholars participated in the second post-program survey. Scholars choice of majors indicated in survey one and survey two are given in the Tables x and y. Chart y indicates scholars choice of major in both surveys by discipline.

Table x. Survey one: scholars choice of major (n=130).

Intended major

Scholars

Percentage

Engineering, Science, Mathematics or Technology

112

86%

Other

18

14%

 

Table x. Survey two: scholars choice of major (n=109).

Intended major

Scholars

Percentage

Engineering, Science, Mathematics or Technology

98

90%

Other

11

10%

Chart x. Surveys one and two: scholars intended choice of major by discipline.

Correlation Statistics

A correlation between scholar satisfaction levels with their post-program interest in engineering was calculated from the second survey data (most recent). Scholars rated the distance education, mentoring workshop component and mentoring relationship from 1-4 (with 4 being the highest rating). Scholars rated their interest in engineering from 1-4 (with 4 being the highest rating). Mean satisfaction levels of the program components were correlated with each scholar's rating of their interest in engineering using the Pearson product-moment correlation. A correlation factor (r) of .14 was the result, indicating some level of correlation (Appendix H).

Of the 109 scholars who completed survey two, 93% rated the distance education component as good or excellent, 100% rated the mentoring workshop at good or excellent, and 89% rated the mentoring relationship as good or excellent. Of these scholars, 90% indicated they will be majoring in engineering or science in college, descriptively supporting a moderate correlation between satisfaction levels and choice of major.

 

 

Appendix H

Correlation Statistics

n

Attitudes

Satisfaction

1

3.00

3.33

2

4.00

3.66

3

3.00

3.33

4

4.00

3.66

5

1.00

3.33

6

4.00

3.33

7

3.00

3.00

8

4.00

3.66

9

2.00

3.66

10

3.00

3.00

11

4.00

3.33

12

4.00

4.00

13

4.00

3.66

14

4.00

2.66

15

1.00

3.00

16

4.00

4.00

17

3.00

3.00

18

4.00

3.66

19

3.00

4.00

20

4.00

4.00

21

4.00

3.33

22

4.00

3.66

23

4.00

3.66

24

4.00

3.66

25

4.00

2.66

26

4.00

3.66

27

4.00

3.66

28

3.00

3.66

29

4.00

3.66

30

4.00

3.66

31

3.00

4.00

32

2.00

3.66

33

3.00

3.33

34

3.00

3.00

35

4.00

3.66

36

4.00

3.66

37

4.00

4.00

38

3.00

3.66

39

3.00

4.00

40

3.00

4.00

41

4.00

3.66

42

2.00

3.66

43

4.00

3.66

44

4.00

3.33

45

4.00

3.66

46

3.00

3.66

47

4.00

4.00

48

4.00

3.33

49

4.00

4.00

50

4.00

4.00

51

3.00

2.66

52

4.00

4.00

53

4.00

2.66

54

3.00

4.00

55

4.00

3.66

56

4.00

4.00

57

4.00

3.66

58

3.00

3.33

59

4.00

3.66

60

4.00

3.00

61

4.00

3.33

62

4.00

3.66

63

3.00

2.66

64

4.00

3.66

65

4.00

4.00

66

3.00

3.33

67

4.00

3.66

68

4.00

2.66

69

4.00

3.66

70

3.00

2.66

71

4.00

3.66

72

4.00

3.66

73

4.00

3.66

74

4.00

3.66

75

4.00

3.66

76

3.00

3.66

77

4.00

3.00

78

4.00

3.33

79

4.00

3.00

80

4.00

3.66

81

4.00

3.00

82

4.00

3.00

83

4.00

4.00

84

3.00

3.66

85

4.00

4.00

86

4.00

3.33

87

4.00

3.66

88

3.00

4.00

89

1.00

3.00

90

4.00

3.66

91

4.00

3.33

92

4.00

3.66

93

3.00

3.00

94

4.00

4.00

95

4.00

4.00

96

4.00

3.66

97

2.00

3.66

98

4.00

3.66

99

4.00

4.00

100

4.00

3.66

101

4.00

3.66

102

4.00

2.66

103

4.00

2.66

104

3.00

4.00

105

4.00

4.00

106

4.00

3.66

107

3.00

4.00

108

4.00

3.00

109

4.00

3.00

Mean

3.61

3.52

Standard Deviation

0.69

0.40

Correlation

0.14