Assessment Details

Results as of March 2010

The Assessment Team conducted a literature review on various aspects of and problems in engineering education. The quantitative literature review focused on engineering students’ retention and attitude toward engineering. The qualitative literature review examined the cultural and social aspects of engineering programs in higher education institutes and students’ sense of belongingness and access to support systems across different subgroups of students.

In Fall 2009 and Spring 2010, the Assessment Team collected both quantitative and qualitative data to create a baseline data-base. The following types of data collection activities were completed up to date.

A. Quantitative Data Collection Activities:

• Evaluating Academic Success Effectively (EASE) survey results conducted by Student Affairs are being reviewed to examine the overall academic and social engagement of new students.

• The Pittsburgh Freshman Engineering Attitude Survey (PFEAS) data were integrated with academic and demographic data from Banner. The merged data are currently being analyzed.

• Strategic Planning & Assessment Resource Team (SPART) survey data are being collected in the spring of 2010 to obtain information on the academic experience of engineering students as well as retention and graduation rate. The SPART data will be merged with the PFEAS data for analysis.

B. Qualitative Data Collection Activities:

• In order to create a qualitative baseline data-base, three types of qualitative data, document, observation, and interviews, were collected from all sections of ENGR 1201 (Introduction to Engineering) classes between October 10, 2009 to February 28, 2010. The document data included basic demographic information about all ENGR 1201 course instructors, Teaching Assistants, and students, course syllabus, instructional materials, lesson plans, major assignment guidelines, and materials used for training of the Teaching Assistants.

• In Fall, 2009, seven in-depth individual interviews were conducted with three instructors and four Teaching Assistants. Each interview was based on a semi-structured interview protocol listing several key questions about what they view as major strengths and/weaknesses of their program in general and ENGR 1201 class in particular. The instructor interviews took 60 to 90 minutes and Teaching Assistant interviews took about 45 minutes. Except for one Teaching Assistant interview, all instructor and Teaching Assistant interviews were conducted by the qualitative evaluator.

• In Fall 2009, eight selected ENGR 1201 lecture and lab sections were observed by the qualitative evaluator (Jae Hoon Lim) and her Research Assistants. Each observation lasted throughout the entire class period and therefore took about 1 hour 45 minutes. The classroom observation was based on an observation protocol created to document several key social and cultural aspects of the classes. In later observations, a modified observation protocol was also used to document the general patterns of interactions between the instructor and students in large lecture sessions.

• In late Fall 2009 and early Spring 2010, the Assessment Team interviewed 46 freshmen engineering students, 38 males and 8 females, who were enrolled in ENGR 1201 classes in Fall 2009. The student sample consisted of 32 Caucasians, 8 Asians (including 4 middle eastern), 2 African-Americans, 3 Hispanics, and 1 Native-American. Forty interviews were conducted as individual interviews while 3 interviews were conducted as pair interviews that included two students in one interview process. Both types of interview took about 30 to 45 minutes and were also based on a semi-structured interview protocol listing several key questions about their campus and classroom learning experiences. All interviews were audio-recorded for verbatim transcription later. All interviews were conducted by the qualitative evaluator or her Research Assistants who had been trained by the evaluator.

• High School Visit Log was created to collect data for each high school visit made by a designated representative of the UNC Charlotte Engineering Program. The log includes sections documenting basic demographic information about the school, the number of students and school faculty/staff who interacted with the university representative during the visit, and overall responses from the high school faculty and students.

2. The major findings resulting from these activities.

A. Results from Quantitative Assessment

The Pittsburgh Freshman Engineering Attitude Survey

• The Pittsburgh Freshman Engineering Attitude Survey (PFEAS) was administered to 341 undergraduate students enrolled in Introduction to Engineering (ENGR 1201) in the fall of 2009. The survey consisted of 50 statements designed to measure student attitudes towards the field of engineering. Survey items related to 13 different categories (i.e. jobs, abilities, family) and were measured on a five point Likert scale or ordinal-based self-assessed confidence scales. Survey was administered at the beginning of the fall semester and again prior to the end of the semester to all participants. Participation was voluntary and participants were not compensated for survey completion. Participants who did not complete both the pre- and post-test were not included in the final analysis. The final analysis included data from 256 students. Data collected in the fall 2009 will serve as baseline data for future year comparisons.

• All participants were undergraduate students, with the majority reporting freshman year status. The sample was predominately both male at 92% (n=235); and Caucasian at 75.8% (n=194). Hispanic and Asian/Pacific Islander each comprised approximately 8% of the sample, while the ethnicity of African-American made less than 3%. Underrepresentation of minority participants resulted in unequal comparison groups and caution is used in report all results and interpretations.

• Secondary institution information was collected and examined for trends. Participants had a mean high school GPA of M=3.77 (n=223, SD=.46). The SATM mean was M=607(n=230, SD=62.12) and the mean score for SATV was M=535 (n=230, SD=70.64). The participants that took the ACT (n=39) had a mean score of M=22.77 (SD=3.41). ENGR 1201 student who took the math placement test scored M=19.8 (n=229, SD=3.2). The cumulative college GPA was 2.812 (n=255, SD=.82).The majority of the participants were classified as mechanical, computer, or general engineering majors. The majority of students were North Carolina residents at 83.6% (n=214) and three quarters of the participants applied for financial aid.

• The vast majority of the items had the mean score of 3.0 or higher, indicating that participants’ attitude was positive. Pre- and post-survey changes were examined for statistical significance using paired sample t-test. Statistically significant response change was demonstrated in the categories of “basic”, “career”, “communication”, “compatible”, “exact”, and “perception” (p<.001). In these categories students showed an increase score, equating to a greater response in the affirmative. These data were further analyzed with one-way repeated measures ANOVAs to examine a relationship between student attitudinal responses and factors such as gender, ethnicity, and major. Analysis was also conducted on secondary information, residency, and course sections. No significant differences were found between groups in regard to pre- and post-survey attitudinal differences.

• A validation study of the PFEAS is currently underway. The internal structure of the survey is being examined using factor analytic methods. Further analysis will be conducted to evaluate the validity and the psychometric properties of the survey. The PFEAS may be modeified for use in ENGR 1201 based on the validiation study results.

• The PFEAS will be slightly modified by adding more items relating to ENGR1202 and administered to evaluate the impact of ENGR 1202 at the end of April, 2010.

B. Findings from Qualitative Assessment

A preliminary analysis was conducted with document and class observation data as well as student interview data that have been transcribed up to date. Major findings from the data-base available at this point include:

• As a whole, freshmen students are satisfied with their engineering program and highly motivated to remain in the program.

• Freshmen students perceive that their program faculty, particularly ENGR 1201 instructors, provide excellent support and personal connection.

• Three major challenges are identified regarding the structure and instructional practices of ENGR 1201 classes. They are; difficulty in providing an effective training for Teaching Assistants; difficulty in maintaining consistency in instruction and grading practices across multiple sections of ENGR 1201, and difficulty in ensuring personal relationship and connection with a growing number of students in the program.

• Freshmen students who are currently in the Engineering Learning Community are highly satisfied with the residential arrangement and instructional support provided through the community. These students also developed a solid social and instructional support network among them and benefit from the network.

• Male students tend to build a solid social network quickly with other male students in their class or program. To the contrary, female students exhibit a more independent work disposition and are less active in a social networking activity with other students. This tendency is also observed among minority and international students. Yet, minority and international students tend to have their own small social network beyond the disciplinary boundary.

Publications and Products

1. What other specific products (databases, physical collections, educational aids, software, instruments, or the like) have you developed?

A. Products from Quantitative Data Collection and Analysis

• The PFEAS may be further modified for use in ENGR 1201 based on the validation study results.

• The PFEAS will be slightly modified by adding more items relating to ENGR1202.

• A data file has been created that incorporates all available data (PFEAS and academic and demographic data from the Banner). The file is continually being updated with the new data.

B. Products from Qualitative Data Collection and Analysis

• Three different interview protocols were developed for instructors, Teaching Assistants, and ENGR 1201 students. Two classroom observation protocols were developed, one focusing on several key social and cultural aspects of the ENGR 1201 classes and the other documenting the general patterns of interactions between the instructor and students in a large lecture sesssion of the classes.

• For the second round of baseline data collection scheduled in later Spring 2010, three new interview protocols were developed for ENGR 1202 course instructors, Teaching Assistants, and students. The two classroom observation protocols originally developed for ENGR 1201 are modified for ENGR 1202 class observations expected in Spring 2010.

• All qualitative interview data transcribed up to date are stored in a secure external drive that requires a password. The Assessment Team will conduct a systematic and thorough analysis of the entire baseline qualitative data between May 1, 2010 and June 15, 2010.