Rita Karl, Education and Curriculum Manager
Universities Space Research Association
NASA Johnson Space Center

Introduction

The proficiency of technology skills is a topic that relates not only to on-line distance education courses like Texas Aerospace Scholars, but to any course that requires students to complete a variety of assignments that require the use of the Internet, PowerPoint, Adobe Acrobat Reader, word-processing and graphics applications.  A proficiency test that assesses students’ expertise in the above disciplines can be used to design instruction appropriate for that particular group of students.

Texas Aerospace Scholars (TAS)

TAS is an on-line distance education course that introduces over 300 eleventh-grade gifted scholars to engineering and the field of aerospace.  Scholars complete a seven-month course that covers various topics in aerospace including the space shuttle, the International Space Station, robotic and human exploration of the Moon and Mars.  Students are selected by their state senator or representative to participate in TAS and having completed the on-line curriculum are eligible for a one-week summer workshop at the NASA Johnson Space Center. http://aerospacescholars.jsc.nasa.gov

Test Structure

A multiple-choice test of 35 questions was administered to participating scholars that addressed proficiency skills in use of the Internet, PowerPoint, Adobe Acrobat Reader, word-processing and graphics applications.  The results of the test analysis will be used to make decisions about the levels of technology that will be required of future participants. Questions included 15 True/False questions, 10 matching items, and 10 multiple choice  questions.  Question items were based on course goals and objectives.   A three question  survey at the end of the test asked scholars about their access and proficiency with specific hardware and software to provide a more accurate profile of scholars' access and abilities. Link to Test Blueprint.

Link toTest Items.

Course Goals and Objectives

The goal of the Texas Aerospace Scholars aerospace engineering curriculum is to encourage scholars to consider the fields of math, science, engineering and technology as future careers.  Course objectives that relate to technology specifically include:

• To give scholars hands-on experience in using e-mail, Internet research skills, Bulletin Board posting sites, interactive chat sessions, PowerPoint, Adobe Acrobat Reader, word-processing and graphics applications.
• To encourage students to explore various multimedia links available on the Internet.
• To help guide students in the creation of original engineering designs using available graphics applications.
• To teach scholars how to make and send word-processing documents and to attach graphics files to e-mails and downloads.
• To give scholars experience in interactive Internet communications including real-time (chat) sessions with NASA engineers and scholar-to-scholar and scholar-to-staff interactions using e-mail and the TAS bulletin board.
• To have scholars experiment with downloading and opening word-processing, video, audio, Adobe Acrobat Reeader and Powerpoint files.

The results of the test analysis will be used to form a general profile of Texas Aerospace Scholars and will be used to make decisions about the levels of technology that will be required of future participants.

Written Report

The written report provides a

• Description of the instrument
• Rationale for item development
• The population, and sample
• The sampling procedure

• Table of specifications
• Raw and standard scores for each individual
• Frequency polygon of raw scores
• Mean, median, and standard deviation
• Item analysis (including difficulty and discrimination)
• Discrimination Index
• Reliability and validity (using the appropriate formulas)
• Test Blueprint
• Test
• Answers

Thorndike, R. M. (1997). Measurement and Evaluation in Psychology and Education. (6th ed.) Columbus, Ohio: Prentice-Hall.

Related links:

Multiple Choice Tests:

Basic Item Analysis for Multiple Choice Tests

Scoring and Analysis of Multiple Choice Tests

Designing and Managing Multiple Choice Tests