Electronic Portfolio Learning at Northern Illinois University

2006 WPA Assessment Institute
Electronic Portfolios for First Year Composition
(http://www.engl.niu.edu/mday/niueportf.html)

Thursday, July 13, 2006

Michael Day
Northern Illinois University

Why Portfolios?

They help us come to terms with growing pressure for authentic assessment from administration and accrediting organizations.

They encourage students to "collect, select, and reflect" (Yancey 15, "Introduction" in Electronic Portfolios) upon pieces that demonstrate their learning. Thus, the students become more actively involved in assessing their own learning.

They help students connect their learning in a class to other classes and life experiences. They look backward (reviewing), forward (projecting), and around them (connecting).

They allow students to use writing to demonstrate and reflect upon learning. Reflecting in writing "makes thinking visible" (Yancey 17,"Introduction" in Electronic Portfolios). "Writing objectifies thought" (Ong), making it possible for students to see and manipulate the words that represent their learning process.

They broaden the spectrum of assessment, which, in some cases, is limited to multiple-choice tests.

They have the potential to change the climate of learning on a campus. Reflecting on how learning takes place is the key to dialogue: student to student, student to teacher, students and teachers to administrators, and so on.

Why Electronic Portfolios?

They take up infinitely less space than paper-based portfolios.

They can be reproduced, shared, or sent at almost no cost. (CD, email, the Web)

They can be updated easily, yet previous copies can be archived.

They can be repurposed for class and program assessment, graduation requirements, or the job search.

They can include a variety of media that the computer is capable of displaying.

They are interactive; students can link from document to document, or to outside resources to show how the learning is embedded in a social and intellectual context.

They can be set up with customized access features, ranging from completely private, to student/instructor, to the entire class, to prospective employers, or to the whole world.

A Humble Beginning: Electronic Portfolios in First-Year Composition at NIU
1. History and Overview
A. Electronic portfolio initiatives
Faculty development workshop on electronic portfolios, 2001
Society for Technical Communication workshop on professional electronic portfolios, 2001
Undergraduate Research Apprenticeship Program on electronic portfolios, 2002
Professional electronic teaching portfolios for pre-certification teachers, 2002

B. Assessment initiatives in First-Year Composition

We were aware of

The demand for authentic assessment, locally and nationally

The demand for consistency across many sections of FYComp, balanced with

The need for instructors and TAs to be able to teach using their individual strengths, not try to fit into a preset curriculum

Where should assessment start? Program Outcomes?
We looked to the national organization for writing program administrators, the Council of Writing Program Administrators.
WPA Outcomes Statement

It defines what students should be able to DO at the end of a FYComp class or program.

It doesn't really describe our program, so it had to be adapted.

It uses language we did not completely understand.

We modified it to create our own outcomes statement.

Through this process of discussing and debating just about every word, we learned a great deal about our strengths and weaknesses as a program.

I recommend this process to anyone contemplating program assessment initiatives.

Our FYComp committee approved draft versions of these outcomes in 2003.

They can be found under "Goals and Guidelines" on our FYComp web page.

Northern Illinois University's First Year Composition Program Outcomes Statements

2. Pulling the assessment and electronic portfolio strands together: the pilot program
Spring 2003:

Two local grants gave us support to go ahead with an electronic portfolio pilot project to:

Test our outcomes

See how our students measured up to our outcomes

Implement systematic program assessment

Close the feedback loop: what can we learn from the electronic portfolios that we can use to improve student learning?

Summer 2003:

Two of us attended an Office of Assessment Portfolio Workshop.

A FYComp assessment subcommittee developed benchmarks to show teachers how to judge outcomes from student portfolios.

One of us began investigating software solutions, such as the Open Source Portfolio Initiative.

Fall 2003:

We decided not to reinvent the wheel. Using existing WebBoard software and our well-calibrated Holistic Scoring team saves money and time. No group -- teachers, students, or scorers -- needs to be specially trained.

We introduced the new TAs in our seminar on teaching college writing to the electronic portfolio process. Their English 103 classes were used for the first semester pilot.

We derived a rubric for scoring the portfolios from our outcomes statement and benchmarks.

We collected electronic portfolios at the end of fall 2003

Students submitted their papers and reflections to a private WebBoard Conference

Instructors could grade them online or print them out.

Our tech specialist set up a PHP script to "harvest" the papers from two randomly selected students per class and generate a scoring sheet.

In fall, all papers were printed out for scoring, since we were in "amphibious" territory. This seemed like a waste of paper!

Our holistic scoring team evaluated the printed out eportfolios using our rubric.

We applied for membership in the first cohort of the National Coalition for Electronic Portfolio Research (NCEPR), and we were successful.

Spring 2004:

Our tech specialist fed the data into an Excel spreadsheet, so we could compare scores in different areas.

The results have allowed us to declare that our data collection is both valid and reliable.

We have collected portfolios from spring semester classes, but since the assignment sequence is different, we had to develop a new rubric for scoring.

The holistic scoring team evaluated the new set of portfolios, for the first time on the computer. The process went more smoothly than we had expected.

Fall 2004:

After consulting with Kathleen Yancey at the NCEPR meeting about the importance of student agency and reflection, we made a drastic revision of the electronic portfolio project, as well as the FYComp curriculum.
We began asking the new TAs in our program (a captive audience, since they must take our pedagogy seminar) to create and revise their own electronic teaching portfolios over the course of their first year of teaching. This strategy has several benefits:

They must work alongside students to create an eportfolio.
They focus their energies on reflecting on and improving their own teaching.
They have a portfolio that they can easily share with future employers.

Based on comments from the Holistic Scoring Team, we revised the scoring sheet to achieve a balance between what our outcomes require and what we can reasonably be expected to assess in student eportfolios.

Spring 2005:

We saw the fantastic work that the TAs were doing with their electronic teaching portfolios, and this reinforced our belief that this is a better final project than a term paper, because it meets professional development needs.

Fall 2005:

Our electronic portfolio RA collected qualitative and quantitative data from the previous year's new TAs about their perceptions of the strengths and weaknesses of their own portfolios, their students portfolios, and their teaching of the electronic portfolio. View some of the results here.

Spring 2006:

We were asked to report on our three years of data, only to realize that we have no statistics expert on our staff who could analyze the data and help us report results. We have asked for help in this area.

Recent Developments:

The Provost and Assesment offices have asked us to consider ways to expand the eportfolio project across campus, and perhaps to make it longitudinal. They are interested in using it to assess writing and technology skills, as well as attitudes toward learning, across a larger cross section of NIU students.

We now focus much more on reflection in both the portfolio and the syllabus.

We're learning that for many students, the reflection in a webbed environment makes students much more aware of audience.

We also found that a lot of students react negatively to the word "reflection," but not as negatively to the same assignment when it is called a "reaction" or "progress report."

3. General results of the pilot project so far

Developing outcomes and assessment has sparked discussion: more than ever, FYComp faculty are talking to each other about why and how we teach what we teach, and what students take away from our classes.

We hold electronic portfolio and technology workshops several times a year.

Electronic portfolios have become an integral feature of the articulation/calibration and program retreat sessions we require all staff to attend.

To a modest degree, the pilot assessment encouraged the teachers in the program to buy in to the idea that all students should be able to demonstrate a common set of skills at the end of our classes. So, rather than lockstep the program with a single syllabus, we can insure consistency and teach to our strengths.

We are receiving recognition on campus and nationally for following best practices in assessment. With authentic assessment measures in place, we are more and more confident that outside accreditation bodies will be satisfied, if not impressed, with our efforts to use assessment and the feedback loop to improve student learning.

NIU has remained an integral member of The National Coalition for Electronic Portfolio Research.

The university has continued to support the project, as we have begun to demonstrate that our eportfolio assessment gives us information to improve the program for students.

4. Implications: Where we go from here

Now that we know our assessment process has validity and reliability, and have collected three years of atomistic, holisting, qualitative and quantitative data, we might correlate electronic portfolio scores with some of the following factors to see how well we are serving different groups of our students:

Course grades

Gender, race, ethnicity

Non-native speakers

Parental education level

The scores on various features of the writing process may help us identify areas in which we could improve our instruction, through faculty development workshops, for example.

5. Your experiences and questions? General Discussion

Created by Michael Day
July 9, 2006
Email mday