David R. Russell's Homepage
Portfolios Across the Curriculum: Whole School Assessment in Kentucky
By Lizabeth Moore and David R. Russell
When the Kentucky Supreme Court declared the public education system unconstitutional in 1989 and the legislature passed the Kentucky Education Reform Act (KERA) to revamp the existing system, we Kentucky English teachers became involved in the broadest reform ever attempted by any state in the nation. As part of the reform, a yearly state-wide performance-based assessment of each school was instituted in 1991. Along with other components, the assessment included a writing portfolio, holistically-graded by teachers in each school, that would count 14% in the total assessment.
What, we asked, would this writing portfolio be? Collecting a portfolio of student writing and grading student writing holistically were not new ideas, and many of us English teachers had been doing similar things for years. However, a standard portfolio and state-wide grading rubrics were a bit much to comprehend all at one time, along with the rest of the complex reform.
Moreover, we heard that the portfolio was designed to improve the quality of student writing in general by moving pedagogy across the curriculum toward more student-centered focus and "real-world" writing. We heard that the portfolio was an assessment of the writing instruction of every Kentucky school, not only English classrooms. But the reform was so massive and the time for implementing it so limited that little professional development was available for teachers. In the panic to design, collect, revise, and compile the portfolio in those first years, the purpose of the portfolio was lost. Only some years later did we discover the possibilities it held for our teaching and the students' learning.
In my high school, Paul Laurence Dunbar High in Lexington, Kentucky, we first suspiciously viewed the portfolio as just another "new" idea that would go by the way of many other innovative educational projects that are introduced every few years by our state educational department. Only several years later did we learn the long-range benefits that portfolios across the curriculum could hold for our school through collaborative portfolio assessment from all teachers. In 1991 we just knew that suddenly we English teachers were handed additional work to do.
According to the state guidelines, each senior in the state (as well as students in grades 4 and 8later changed to grade 7) would complete a portfolio of their writing across the curriculum. The 12th grade portfolio would consist of six pieces: a Table of Contents, a Letter to the Reviewer, a personal experience piece, a literary (creative writing) piece, andthis was new to ustwo "transactive" pieces, writing to communicate with a real-world audience. And the biggest innovation of allat least two pieces had to be from a content area other than English, to encourage writing across the curriculum. Even more surprisingly, instead of sending the portfolios off to be scored by some testing company, we teachers would assess them!
We would assign each portfolio one of four performance levels: Novice, Apprentice, Proficient, and Distinguished. We would use a standard Holistic Scoring Guide developed by a state-wide committee of teachers. Six criteria would guide us: purpose/audience, idea development/support, organization, sentences, language (word choice and usage), and correctness. No portfolio would receive a lower score solely on the basis of correctnessto keep the emphasis on communication, not merely correctness. To achieve consistency across schools, the portfolio scoring from each school could be audited by a state panel of teachers who had proved to be consistently accurate scorers. Eventually every school would be audited
Finally, we learned that the stakes were high for schools, but low for students. If the students in our school, on average, showed improvement on the portfolio and the other assessment components over a two year period, our school could get financial rewards, as the goal of the reform was to improve all the schools, not simply reward those that were already good. If significant improvement was not demonstrated over a two-year period, a school could receive sanctions. The students were required to submit a complete portfolio to graduatebut there was no minimum score required. The goal was to assess the whole schools writing program over time, not individual studentsor even the English department alone, as we found out.
Sharing Responsibility for Writing
The writing portfolio was the one area of the statewide assessment where we English teachers thought we could easily improve students' scores and demonstrate our expertise. We simply assumed that the responsibility for the portfolio would fall to the English department. English meant writing, didn't it? Therefore, we accepted as fact that we would have our seniors compile the required portfolio in a few weeks, and then we could go about our business of teaching literature!
This seemed to make sense. In our school most teachers in other disciplines taught their content without using writing much as a learning tool. Most of them used writing to test learning: traditional research papers and perhaps traditional essays, often as part of an objective exam. There was very little writing with a purpose beyond demonstrating learning or an audience beyond the teacher as examiner. There was little transactive writing, as we learned to call writing for communicationthe kind of writing that was necessary for our students to do well on the portfolios. Even in our English classes, writing assignments often were the traditional literary essays: "Compare the play Romeo and Juliet to the musical West Side Story in character, setting, and plot" or "How does Beowulf embody the characteristics of the Anglo-Saxon hero?" We, the overworked English department, grudgingly accepted the job of having our seniors compile portfolios, without thinking much about our colleagues in other departments.
In the spring, a team of us English teachers went through the state-mandated scoring training and scored our students' portfolios. The scored portfolios were then sent to the team of outside scorers for auditing. Imagine our surprise when the scores that first year were mostly Novice (the lowest ranking) with a scattering of Apprentice, Proficient (the KERA goal) and Distinguished. Surprise turned into dismay over the next three years, as our scores did not demonstrate a significant improvement. We could not figure this out. We were a large suburban high school with a very capable staff of highly motivated teachers, a knowledgeable and supportive administration, a strong professional environment, and a strong academic student body. Many of us English teachers had written district-wide writing curriculum, taught workshops on the writing process, spent many hours with our students on writing. In short, we were working hard! We were meeting students' needs. Then why were our scores on this type of assessment close to the bottom?
In retrospect, the cause of the problem seems obvious. Our emphasis in those early days was on compiling the portfolio, helping students revise existing pieces (especially those from other disciplines), record-keeping, checking to make sure portfolios were complete"just getting the damn thing done" so we could go about the business of teaching English. For the content area teachers it was a matter of assigning "that one portfolio writing piece" the principal had asked them to assign, and getting it to the English department (in whatever condition), so they could get back to the business of "teaching the content." We did not understand that the goal of portfolio assessment, in actuality, was improving writing instructionand learningin all Kentucky schools by all teachers in all classes. We had nothing in place that would address this issue..
In the summer of 1994 I attended a voluntary three-day summer
workshop where teachers from across the state analyzed student portfolios, conducted by Sharon Hatton, the Department of Education Writing consultant for our region. In that setting, with the luxury of large blocks of time devoted to discussing student writing with other teachers, I realized for the first time that scoring portfolio pieces could be a way to rethink teaching and learning, to define weaknesses and establish an instructional plan for a department or school-wide effort. It became crystal clear to me that our English entries were definitely lacking in authentic purpose and audiencethe first criteria.
The subject area teachers did not understand what transactive writing was or how to implement writing in their classrooms as a tool for learning, rather than for assessment only. Subject area teachers (and many of us in English) hadn't imagined the vastly varied forms of writing students could use to help them learn a subject and communicate to real-world audiences. Teachers didn't share a common set of terms for talking about writing, a language of writing. Many subject area teachers hadn't even viewed one of the compiled portfolios. We in English just assumed that because they were teachers, they should know how to teach writing.
It was a rather large epiphany. I came to the conclusion that English departments could no longer be the sole responsible body that assigns and collects pieces and manages the entire portfolio while the whole school can be rewarded (or sanctioned) on its merit. If this portfolio was designed to improve the writing instruction in our whole school, all departments should be investing their time, effort, and pride in the process. But, how to do this? How to inform all teachers of the process? How to get teachers to invest in a philosophy so different from the existing one? I concluded that every teacher must learn about student writing in order to improve it.
Whole-School Portfolio Assessment
In the spring of 1995, I proposed that the entire high school facultynot just a team of English teachersscore the writing portfolios. The goal was that all teachers would take ownership for the writing assigned for the portfolio and take responsibility for improving student writing in their own courses and disciplines. If teachers understood the holistic scoring system (purpose, audience, etc.) and saw student writing from other courses, they might discover how their own classroom writing tasks could improve, and how they could get their students to enter into a deeper level of critical engagement with the course material through writing. And that, after all, was the goal of the whole reform.
With our students' scores still low, it did not take much convincing for the administration and departmental chairs to agree that all of us should score the portfolios. Every administrator (including the principal), counselor, media specialist, and resource teacher would receive training and score portfolios. (The math department was dealing with its own portfolio and was excluded from the scoring.) It would be a whole school portfolio assessment.
That summer, the whole faculty, administration, and staff spent twelve hours in professional development training, reading and discussing samples of student writing in terms of the writing concepts on the scoring guide. (Having a real-world purpose and audience for school writing was a radical idea to many!). We discussed student writing in a wide range of content areas and formsfrom poetry to laboratory reports. We especially looked at the "exemplar portfolios" used for scoring training, which illustrate sustained performance at each of the four performance levels. We spent a great deal of time using the scoring materials to actually score sample portfolios.
It was also a time for building camaraderie among the departments. We laughed, joked, and ate good food during the training. Our school has always believed that offering food during trials and tribulations makes the work easier. "Just a spoonful of sugar . . ." We offered continental breakfasts, sit-down lunch (complete with white tablecloths and center arrangements) and afternoon snacks. We enjoyed the days in spite of it being required training.
At the end of the training sessions, my colleagues had generally favorable comments: "So, this is what you wanted us to do!" "This is not so difficult; we can do this," and the one I especially liked, "My assignments have been all wrong. Now I know what you want."
It was obvious throughout that next school year that many (though by no means all) teachers' awareness was raised about writing. A number of teachers revised or created new assignments to include writing. Several took a further step to make their assignments grow out of their curriculum and further their course goals, not just added-on writing assignments. In general, the staff began to talk the language of the scoring guide when they discussed student writingpurpose, audience, idea development/support, and voice, as well as spelling, grammar, and punctuation. The social studies teachers even posted the scoring guide in their classrooms to refer to during discussions with students about writing.
When scoring time came in April, we all gathered two afternoons after school and divided up into thirteen teams of six teachers, with an English teacher or experienced scorer functioning as a table leader, to discuss borderline portfolios, handle paperwork, and so on. Thus, we English teachers, who had guided the students in compiling the portfolios, did not actually score them unless there was a difference among scorers or a portfolio posed special difficulty. We divided the 380 senior portfolios into thirteen stacks and conducted three rounds of scoring. This meant that each portfolio was scored a minimum of three times by three different scorers, over the two afternoons of scoring.
After that first session the comments were varied. Some content-area teachers were touched by the personal writing of the students, that incidents from their personal lives were shared in such depth. Some were impressed with the creativity demonstrated by so many. However, some were intimidated by the poetry and not sure how to score it. Many were dismayed when they saw that papers written for their own classroom assignments were included in a portfolio and scored as weak. Overall, subject-area teachers were surprised at how much they learned about their students by reading their writing across the curriculum.
As the scoring continued, it became clear that it was certainly the single most important professional development we had as a faculty that year, or perhaps any year. We were moving toward the goal: for all teachers to use writing as a learning tool in their classrooms. Though a few days of training and one year of experience do not make one a master writing teacher, at least all teachers had some basic tools nowmore than we'd ever given them before.
The results of collaborative portfolio assessment by teachers across the curriculum were many. Some were predictable; some were startling. One startling outcome was how accurately subject-area teachers can score student writing, after some training. The Kentucky Department of Education periodically has an "audit" team of teachers who have proven reliability as scorers take a sample of portfolios and rescore them, to check for consistency across schools. The statewide rate is 75% exact agreement. We at Dunbar had an 86% exact agreement that first year of whole-school assessment. Teachers across the curriculum can be very reliable readers of student writing when they get adequate training and collegial support.
As we English teachers worked with our colleagues, we could feel them developing a sense of confidence about evaluating writing. One science teacher who was worried and resistant when she began said, "Last year after we actually . . . sat down and graded the portfolios, I was actually surprised at how well I was able to do on it. I liked the idea of having the English teacher sitting there that I could hand [a difficult portfolio] to if I just went, "Holy cow, what I do with this?" I could hand it to her and she could, you know, be my back up. But after grading last year I thought, 'Well, you know, I think maybe I know what Im doing.'"
Professional Development through Whole School Assessment
But scoring accuracy isn't the point, of course; improving teaching and learning through writing is the point. And that means changing teaching for the long run. During the second year of whole-school assessment, David Russell, a researcher from Iowa State University, interviewed 26 teachers from across the curriculum, primarily from science and technology, and did some follow-up interviews over the next year.
He found a wide range of responses, from teachers who did not assign writing and resented having to assess portfolios, to teachers who felt that writing had revolutionized their whole classroom approach. Overall, teachers were very positive about the changes. There quickly developed a critical mass of teachers in each department committed to making it work, and as time goes on, others (though never all!) are finding useful things for their teaching as a result of whole-school portfolio assessment. To illustrate those changes, some anonymous comments from teachers are included, especially those from science and technologytraditionally the fields we think of as "furthest" from English.
Reading and discussing student writing together increased the level of knowledge and comfort about teaching with writing. One technology teacher said, "I think it helps me as a teacher. If I were never involved in this I might not be able to relate to the student as well. . . . You know it gives me some things to use in my classroom that help them be better writers." Another technology teacher said, "I think its made me aware of how to be a better writer, and how to teach students to be better writers. I mean, obviously Im not an English major, so I didnt spend a lot of time in composition and stuff. I think its helped me."
A science teacher found she knew "more of what to look for. I think probably the thing that helped me the most was actually sitting down last year and grading portfolios. Because then I got to read what was in the portfolios . . . I saw some really good ones. I saw some that werent so good, so that gave me a better idea of what I needed to look for. The [science] content wasnt a problem. I knew what to do for that, but as far as the way these kids put the papers together, it really helped me reading over the ones from last year. . . . I feel a whole lot more comfortable with assigning writing assignments in my class because I know more what to look for, not just the subject content, but the whole writing process. I know what to do."
More teachers began to work with students to improve the writing, rather than assuming the English teachers would "fix" existing content area pieces for the portfolio. After teachers became familiar with the requirements of our writing portfolio (and realized that the KERA reforms were not "going to go away"), many subject area teachers became the biggest advocates for the writing process, designing assignments that led students through writing in stages.
Some science teachers redesigned assignments to take students' writing and learning processes into account. As one said, "I think a lot of times before you do an activity, you can get students to write down what they think is going to happen and to make a proposal, and then a lab can be part of a pre-write. Gathering your data, etc. And then, doing something with it. Whether it's real life writing or whether it's more creative writing." Another science teacher found, "The way that were doing this now, I feel like I can at least talk somewhat intelligently about this writing process and maybe just take a class period to discuss subjects [for writing]what I expect of them and what they need to do and what they need to remember to do from, you know, English. And thats no big deal. I dont mind that."
As the comments of the this science teacher suggest, whole-school portfolio assessment using a holistic scoring guide gave teachers a common language for talking about writing, among themselves and with students. There is strength when an art teacher can address the lack of authentic purpose and audience in a piece of writing. There is strength when a science teacher can discuss idea development and a physical education teacher can discuss lapses in organization. As a science teacher put it, "The kids are starting to see that . . . writing is not something that you just do in English. Writing is something that youre going to do in all different areas."
I don't live with rose-colored glasses. I know that some teachers are more excited, more knowledgeable, more invested than others. As a science teacher put it, "There is definitely more writing. There is definitely a better attitude about it now than when we first started. But there are still quite a few teachers that are very uncomfortable with the whole process." It takes timeyearsto accomplish fundamental changes like making writing an integral part of learning across the curriculum. However, we have a greater awareness of writing as being important in every discipline, not just English.
Curriculum Development and Whole School Assessment
Through reading and discussing student portfolios, many teachers began to see how writing can be used as a tool for learning--not only demonstrating learning, for assessment, but also for making a connection between curriculum and the worlds of writing students will enter after secondary school. In other words, the portfolio assessment was broad enough that it made room for new things in the curriculums of various disciplines, instead of crowding things out of their curriculums, as most external assessments do. And because teachers assessed the student work, instead of an external testing agency, they discussed the assessment in terms of their curriculum, what goes on in class. And that dialogue can spur curricular change.
Because it was clear that overall the weakest selections in the portfolios were the required subject-area pieces, more subject-area teachers began to take responsibility for improving their students' writing by assigning writing that expanded the students involvement with the subject matter. They began to design writing tasks that were transactive, with a real purpose and audience beyond the teacher-as-examiner. We had far fewer "cookie cutter" writing tasks and far more real-world writing with student choice. Twelve of the eighteen non-English teachers interviewed reported changing their assignments. The comments of a science teacher were typical. "I do quite a bit that is different. The kids are more likely to write up their labs rather than just kind of do a fill-in form that we used to do all the time."
Because transactive writing is included as one of the four required types of writing (and the form must be a real-world form), it takes some imagination as a teacher. More subject-area teachers are excited about creating writing tasks that indeed become connections with what James Moffett calls the "universe of discourse" beyond the school. It is exciting to hear them discuss the writing that is occurring in their classrooms. They are learning to adapt the traditional research papers into proposals, abstracts, oral presentations, multi-media presentations, instruction manuals, and position papers. They are learning to adapt traditional methods of testing to develop students' critical thinking skills and engagement with the content; they are learning to assign book reviews, feature stories, editorials, pamphlets, brochures, memos, business letters, recommendation letters, and application essays as writing within their curriculum.
Some content-area teachers were (and are) concerned that writing would compromise their content, and these tend to view writing as an add-on (often unwelcome). But others are finding that using writing is "just approaching something differently," as a science teacher put it. "I don't feel that it's taken away from my content. I don't feel like I've changed the content of what I teach." Another was adamant that writing is a way of teaching content. "I pride myself--those kids that come back to me and they say, 'You took me through that first three weeks of college chemistry.' And Im still going to do that. Now I may use writing to accomplish that goal, and I may lecture less, but you know, Im not going to compromise [the content]. The writing will just be incorporated into that and Ive got to figure out ways to do that."
Gradually, this dialogue about writing is beginning to affect discussions of curriculum within individual departments. The impact of the portfolio in science, according to one teacher, will be "on the curriculum and the different things that we do. So I think that were all growing and its not one of those processes that will ever quit and will never reach the outside boundary." Another science teacher emphasized the importance for curriculum discussions in working with "teachers that I otherwise wouldnt" in whole-school portfolio assessment. "When we do this whole scoring thing, were working with a diverse section of teachers that we usually don't spend a lot of time with. So, thats good. Now we talk more within the department too about, well, what kind of writing assignment did you do, hoping to pick up triggers or ideas or something that we can do."
Teachers Learning TogetherCollegiality for a Change
Whole-school assessment for us is the one tool that brought all of us, administrators and teachers, together in a common effort. As we addressed the needs of the writing portfolio, we were discussing learning needs of all kinds: the need to critically engage students with the content as well as present it; the need to communicate for a variety of purposes and audiences as well as for the teacher-as-examiner. A dialogue about learning started that almost certainly would never have occurred without whole-school portfolio assessment. Our discussions gave us a very practical sense that all of us are responsible for the kind of learning that goes on in our school.
Finally, whole-school involvement in the writing portfolio tore down some of the barriers between US vs. THEMthe English department versus the other departments, teachers versus administrators. It provided a common ground for all of us as educators and writers, where we could learn together. A science teacher expressed it this way: "You're seeing some of the work that is being done. You also get to know the teachers better. This is a big school and you get the little pods. And we don't . . . think science is any better than others, but in our daily contact, these are the people we contact more than anyone else."
Another science teacher added, "I think its wonderful. I think that it was a real injustice to the English teachers the years that we did not help them [assess portfolios]. Because its an all-type project. I mean, we should all participate in it and Ive learned a great deal and feel fairly comfortable about my assignments. . . . It was a very good experience." Several teachers echoed this science teacher. As one science teacher said, "I think it's very good. Before English teachers were doing all of it, and it put the burden on them. They were feeling overloaded and then we were feeling kind of guilty. So it's just . . . you feel more it's a team effort. And so I think that's good. I don't like the extra work, nobody likes the extra work, but it's not as bad as it sounds. Two afternoons after school. But I think it's been very good for the school. I think it's brought us together."
Indeed, we English teachers are also learning from our colleagues in other areas to value kinds of writing that are important to students in many fields but have traditionally not been well understood or valued in English. A technology teacher was glad that English teachers were coming to appreciate as valuable portfolio writing the "types of pieces my students were writing [e.g., brochures, instructions]. Theyre moving more towards the technical pieces and I think . . . from my perspective, thats good. It might not be from a [traditional] English teachers perspective." As an English teacher said, "I see now that literary criticism is really a kind of technical writing for English . . . Scoring the portfolios with the other teachers has given me a broader kind of thinking about writing."
Conclusion: Change for the Long Run
We are still seeing the long-term effects of whole-school assessment, through this new teacher awareness of writing. The writing portfolio is no longer looked upon as the sole responsibility of the English department. Many teachers work seriously at assigning writing that is purposeful and meaningful in terms of teachingnot just assessingtheir content. And many have attended professional development workshops on writing geared to their needs, such as a summer workshop on technical writing. In several departments over the past year, a group of teachers within each department has worked together to construct writing assignments that help students engage with and answer the essential content questions of a unit, so that writing grows out of the content of the unit.
Of course we hoped that there would be moreand betterstudent writing as a result of the discussions and changes whole-school assessment created. There was. The scores did not soar out the top of the scale, but the difference in the writing ability of our students from one year to another was gratifying. The biggest change was that the bulk of our students' writing had moved from Novice to Apprentice, and the Novice writing was definitely better, closer to being Apprentice. The number of Proficient and Distinguished writers increased as well. We were convinced this happened because teachers were assigning more meaningful tasks and possessed some shared terminology and experience of writing across the curriculum to guide their students.
The teachers have also perceived a difference in student writing. All agree that there is more writing and a greater variety of writing. 24 of the 26 teachers interviewed believed that student writing has improved, and as one technology teacher put it, "It's just like anything else, they've just put more emphasis on it and they just kept doing it. And the more they do it the better they get." Teachers sharing expectations, as one science teacher said, has made student writing "more organized. Theyre able to put things together betterthe transitions are better. Their vocabulary I think is improved. Dont think their spelling is improved. But theyre just more aware of what they need to put in paper." As another science teacher explained it, "I mean, if you go back to the whole basis of having portfolios in the first place, it's to show that youre a writer, and writing doesnt take place in one particular situation."
After five years, we can say it is not a perfect system yet, but a viable work in progress. The students are writing more, in a wider range of forms, and we teachers are learning to produce more meaningful writing tasks, both for teaching the content of our courses and helping students critically engage with the "universe of discourse" beyond the school. The system has worked because it gave teachers a way to talk in a practical way about teaching as they evaluate student writing together.
Our work grew out of the state-mandated teacher assessment of a flexible-content portfolio assessed by a common rubric. But in any school, teachers from one department or across the curriculum can grow by getting together to discuss student writing (as Spandel and Stigall have delightfully shown in their book, Creating Writers: Linking Writing Assessment and Instruction). The questions are: How can we teachers learn from each other how to value student writing? And how can we improve teaching and learning through our dialogue with each other about student writing? Teacher-assessed portfolios across the curriculum have given us a valuable start in answering these questions.
For Further Reading
Kentucky Writing Portfolio. Kentucky Department of Education. http://www.kde.state.ky.us/oapd/curric/portfolios/
Moffett, James. Teaching the Universe of Discourse. Boston: Houghton Mifflin,1968.
Raines, Peggy A. "Writing Portfolios: Turning the House into a Home." English Journal 85 (January 1996): 41-45.
Russell, David R., Starr Lewis, and Anella Riggs. "Growing Together: Curricular and Professional Development Through Collaborative Portfolio Assessment." English Leadership Quarterly 18 (1996): 13-18.
Russell, David R. "Collaborative Portfolio Assessment in the English Secondary School System." Clearing House 68 (March/April 1995): 244-47.
Spandel, Vicki, and Richard J. Stiggins. Creating Writers : Linking Writing Assessment and Instruction. New York: Addison-Wesley, 1997.