« July 2007 | Main | October 2007 »
August 17, 2007
Data in the classroom?
After almost two months at Akshara I can finally claim to have interacted with at least a few people at almost all levels of the system; the core of KLP, teachers, the most recent addition to my list of people I’ve pestered with questions. I had the opportunity to test out my survey with teachers for the first time. Aside from that, I also had the chance to start working with quantitative data.
One thing I noticed, as I worked on a random sample for auditing the math programme and calculated predicted numbers of children that would enter the KLP statewide programme in September, was how immediately useful quantitative data can be in planning. Suddenly, Stata was a boon, and simple enrollment information could reveal a lot about the system. For example, as I looked through the Karnataka data, I found a stark decrease in enrollment from Class 1 to Class 8; in some blocks, this drop in enrollment was particularly pronounced for girls.
Yet the usefulness of quantitative data did not change the fact that there were strange disconnects between quantitative and qualitative information. For example, as I interviewed teachers, I realized how difficult it can be to gain complete qualitative information as an outside observer. The teachers in the schools I visited thanked me profusely for the programme’s existence, though I had nothing to do with it; they balked at my question about what could be eliminated from the math kit, perhaps because they would rather not lose any resources, even if they don’t use all of them. The teachers seemed committed, of course, and children seemed happy – in some cases, to such an extent that I wondered if it was hopeless to train teachers, if really only changing their home environments would make a long-run impact on their performance.
However, though my position as an outside observer, connected to Akshara and perceived as a person with power, made it difficult to tell how authentic the information I was receiving was, there were still something jarring about the contrast between the qualitative and the quantitative. For me, this observation reinforced the importance of triangulation – looking at the same issue from multiple viewpoints. One teacher told me, for example, that the final results in her school from the reading programme came out badly because that day, inexplicably, all the children who had been attending the programme and a new set of children happened to come to school. Though this might have been an exaggeration, I couldn’t help but wonder how well our assesssments really reflect the benefits of our programmes.
Out of this teacher’s comment, I also noticed her deep desire to impress me with the performance of her children. While I observed her class, she asked me over and over whether they were doing well. To her, assessment was not something she could use -- it was only something that could either validate or invalidate her work. Early this week, I presented my theory of change model to the KLP coordinators. Many of the resulting comments were about data use: how could data be made into something that people could use at all levels? Where was data feeding back into the system? In this teacher’s case, data was a yes-or-no question: did they do well? Did they not? If one of the aims of KLP is to shift the system into thinking about data as a useful tool, it may prove extremely difficult to ignite that shift among teachers who are used to being constantly evaluated on summative terms.
Posted by gowriv at 12:11 PM
August 04, 2007
About data and how to use it
It’s amazing that after over a month at Akshara I continue to learn more about how the organization works. In the past two weeks, I’ve met more cluster volunteers, two BEOs, some BRPs and CRPs, P, the academic who designed the math curriculum, a few teachers, a group of third-standard students, and two representatives from H, who provided library content and training.
Two large questions have become a recurring theme, both related to assessment. The first is this: What does assessment actually assess? And the second is: What should assessment be used for? In fact, the questions are actually almost the same.
When I visited the school in Jeevanbhima Nagar with research students from Columbia, the first thing I noticed was how nice it was to see kids again – it seemed that over the years, the more I learned about development and community work, I spent less and less time with kids and more and more time burrowing in my laptop, discussing kids with people who were not kids, and reading what other people had written about kids.
I also started to think more about the question about what assessment should assess. Assessment is a powerful tool in curriculum, and not just in its role at the end of the process: it shapes the focus of the curriculum and makes a final statement about what was important and what was not. An assessment is almost a philosophy of education, in that it identifies the key elements that educators seek, ultimately, to impart to their students in a lasting way. Are you being marked on your writing skills? Your ability to question? The quickness of your memory? The strength of your foundations?
So what was the pearl of wisdom that the Columbia economics team had to offer on what matters in education? Ostensibly, the mantra that permeates through all of Akshara’s activities: basic skills so that children have something to build upon. As I was watching the children take the pilot test, though, I couldn’t help thinking that it actually stifled the things I thought mattered in education. As proctors of the pilot test, we were responsible for telling the children not to copy, not to talk. As the CV explained to the children what they needed to do, the burst out with a chorus of questions, directed at her, at us, and at each other. There was lots of collaborating, lots of copying. It seemed like these children thought of the test as a group exercise rather than a chance to demonstrate their individual progress.
Watching the children interact with the teacher and each other made me question the values testing imparts. After all our talk of interactive learning, of bringing the teacher and the child closer together, the endpoint is the message that what’s really important is for you to sit at your desk, not look to your left or right, not help anyone, not ask for help, not receive support from the teacher, and get the right answer. It seems like a mismatch. Wouldn’t it be interesting to test students and reward them for their ability to work together to solve a problem, instead of their ability to struggle with things in silence alone? What is the boundary between individual success and group success?
I think, in the end, this is an ideological question, but it seems, ironically, like the culture of testing has had enough power to offer support to things like cooperative learning. Slavin’s research, for example, argues for cooperative learning because it improves test scores – a cooperative process to achieve an individualistic outcome. Interestingly, though, Slavin also concludes that cooperative learning works best when students know that their individual assessments depend on those of others in the group: their successes are interdependent. So cooperative learning is both an end in itself – because, obviously, it teaches cooperation -- and a means by which to improve students’ individual performance. Testing students individually also has the advantage of being much more apppealing to institutional culture than allowing students to “cheat” – it’s able to show the results that can be used for good advocacy for good causes.
The H meeting I attended danced around a similar question – what is important to assess? – though this time, it related to the relationship between a child and a book. The main conflict was between the importance of processing a book as a whole and the importance of being able to read words on a page. H, leaning on the premise that books with less text on each page are easier for children to understand, cares about whether a child can access and understand, generally, a book. Akshara cares about whether the words on the page are made up of letters that children can read. Complicating the whole discussion was the reality of Kannada publishing – there simply aren’t many books for low-level readers with simple words. Then there’s the question of development: is an older child who can’t read simple words interested in the same book as a younger child? Probably not.
Here too, there was the concern that what you assess defines what you value. For H, what is important is that kids love books, even if they aren’t literate enough to read all the words in them – thus, assessment hinges on the book layout and accessibility. For KLP, what is important is that kids can read, and assessment focuses on letters much more than on complexity of meaning. I wonder how these two axes of reading skill could be incorporated into an assessment system. For example, what if books were assessed by word complexity, with an additional rating for book complexity? Then we’d assume that a young, poor reader would read a book with low complexity and low word complexity, but an older poor reader would read a book with low word complexity and high book complexity – a more mature story, fewer graphics, and perhaps a more challenging vocabulary.
At the end of all this assessment is the question that I began working through with A this week – what should assessment be used for? In an a large meeting, P nsisted that there was a difference between diagnostic testing and assessment. Diagnostics are used to analyze problems and design solutions for them; assessments are used to make a final judgment on something’s success or failure. In program evaluation, this is the distinction between formative and summative assessment.
It seems to me that KLP’s foundation, at least in theory, is the idea of diagnostics – remedial programmes should be setting off a long-term cycle of improvement, with each test offering the information that can inspire the next round of improvement. This was my work with A on policy models for KLP. Yet my evaluation professor’s distinction between formative and summative evaluation is key here: the final deciding factor in whether an evaluation is formative or summative is the way it is used. In the case of the math programme, the tests are not diagnostic: teachers likely will not use them to shape their teaching. Even the reading programme’s tests were not diagnostic – the data mostly served to affirm success rather than to identify problems and solutions.
This is the question that a BRP from the South 4 district raised when we went to visit the BEO office there. Sitting in the back of the room, she began to contribute her thoughts on KLP. She had a lot to say, but the very first thing she said was that KLP made teachers feel “down” – by assessing students so regularly, the programme only managed to affirm that teachers were doing a bad job and someone else could do a better job than they could. The worst part, to her, was that the teachers were actually doing all the work in both cases.
I think that underlying her problems with KLP was the same distinction between diagnostics and assessment that I’ve been talking about. To her, the reading programme had been assessed, not diagnosed: KLP came in, imposed a programme, declared success, and left. This had left teachers, according to this BRP, feeling attacked rather than supported. The last part of the cycle was missing: the feedback and planning for the next strategy for improvement. The BEO told us that she hadn’t seen much impact of the reading programme in her block – perhaps an indication that one turn of the wheel is not enough to make a lasting change. What defines KLP’s testing as diagnostic will be the way it is used, and so far it has not been. This space is where I am hoping to insert myself over the course of the year.
Posted by gowriv at 12:09 PM