What is impact evaluation and why ?
When we start an intervention either as an individual or an organization, we start the things with a set of objectives to meet at the certain point of time. When we started our library we started with a program theory that libraries will improve the reading skills and efficiency of the whole school. At certain period of time of the intervention a quest appears- what has the intervention made changes to the beneficiaries, what would happen if the intervention has not taken place, is the intervention going in the track we intended or what are things we need to correct or do better so to make the target audience get benefit in a meaningful way? The attempt to answer these questions falls into impact evaluation or assessment. These questions definitely apply to any library intervention whether financially funded or not funded by an external agency. Therefore, the need of impact evaluation is nowhere limited to serve the purpose of accountability and transparency rather also to make sure in serving the value of the intervention. It is rather a basic nature of enquiry for a reflective professional including librarian.
Hence, impact evaluation is a process identifying and evaluating any change happen as an effect of a service or an initiative on an individual or group. From the words of Virginia Walter, in his book titled Getting It Right, he extensively mentioned about the significance of evaluation of library service- “Systematic evaluation of our work will improve library services for children. We need document our productivity in order to be accountable for the resources given to us to do the job. We need to know if we are meeting our own objectives. Children librarians do good work. Evaluation allows us to prove it”. For a country like India where library has not parallelly been a part of school education organisations are advocating for school libraries in the form of a movement towards making libraries as an integral part of school system. There is strong link between impact evaluation and advocacy. In order to frame a strong advocacy case we need strong impact evidence, thus a policy is framed.
Challenges of impact evaluation of school libraries
Traditionally, one of the primary objectives of school library program is to enhance reading competency. Therefore, reading assessment is major part of the impact evaluation of such programs, which is partly true for us too. It is easy to identify who is reader and who is not, who has got better reading skills and who has got less. Our assessment tool is weak enough that we cannot measure the children who are at the level of visual level who are at pre-reading. Library is a space for them too, or they could be dominant audience among kindergarten children. One common method of impact evaluation of such library program is standardized large scale testing of reading. A dominant weakness of this method is that it is not inclusive and hard to identify attribute of reading growth of children. Children come from different background and have different style and pace of learning who do not fit well into the standardized testing. Children who are from well off families tend to have better environment that boost their non-cognitive aspect and consequently perform better in the reading testing also.
A classic example for our case is that India’s Right to Education Act 2009 allows children to enroll in government school in any grade appropriate to their age without any fundamental knowledge to read in the respective grade. This has created an epidemic of learning disparity in classroom across public schools, which is to be addressed with a bridge course as suggested by the same Act but practically happening with very few well performing schools. Children enrolled in public schools are mostly from families of lower economic status whose parents might not have interest in their children’s learning or other factors that hinder mental and physical well-being of the child could show up as weaker performer in reading test. Thus, standardized large scale testing doesn’t cover these non-cognitive aspects, yet creates a huge difference in children’s performance. Therefore, the prevalent impact evaluation method is weak to support to claim the intervention as an attribute for the reading growth of the children being tested.
Library programs that enhancing reading skill is not a primary objective tend to end up doing the input assessment as an impact evaluation of the library program. The program theory of such library program is based on input-output theory that the more and better the input yeilds better output or outcome. This is an assumptive projection of outcome by assessing but without assessing actual outcome, which may not be valid. This method of evaluation does not answer the basic questions that a reflective librarian would ask, which is discussed at first para.
Since the emergence of sophisticated technologies and 21st Century Learning Standards a massive access to infinite information is leveraged in the libraries. And, now learning is no more limited to a predefined set of content but an explorative or constructivist way of learning in which what is learnt is more important than how and from where it is learnt where libraries plays a crucial role. Therefore, impact of a library cannot be understood by assessing how much input is arranged but an assessment of learning outcome. Despite this being a current trend of learning and requirement for assessment, conducting such as assessment still really be a challenging for a librarian since learning outcome would be very diverse and could be subject specific that might not be the subject the librarian has knowledge of.
Ways ahead
The two objectives of standardized testing are to understand how many learners are at the standard level of knowledge the respective grade should have and another is to see the growth of learning happen between two points of time of an intervention. I don’t have any question with using standardized testing for the first objective but with the second one.
One way ahead for evaluating the growth of learning attributed to an intervention, library program in our case, is to de-standardized and design assessment tools for multiple level of learning by categorizing children based on their learning nature. Designing assessment tool for every child is practically not feasible at all. Additionally, interpreting the assessment results for different learning level groups.
In addition, a quick solution that we seem is continuous evaluation because it is mandatorily a tracking of every child’s learning. Here, an understanding of the difference between evaluation and impact evaluation is crucial, that the previous is usually of formative while the latter is summative. A continuous evaluation fits into formative evaluation. Impact evaluation is credible and sensible when done by an external agency. Evaluation by third party is a norm in development impact evaluation and hence continuous evaluation is not usually done by external agencies. Therefore, traditional impact evaluation methods are comparison of base data and end data of the intervention, or comparison between the target group and none target group of the intervention.
Ideally, according to Global Libraries of Milinda and Gates Foundation, an impact evaluation mean to evaluate changes in knowledge and skills, changes in perceptions and confidence, changes in specific behaviour, changes in quality of life and changes in society and economy in a sequential order. This is a shift in impact evaluation of libraries which is more relevant than Walter’s (Output Measures for Public Library Service to Children, 1992) measure of output of libraries. Walter still majorly measures the quantity of input and quality of process but ignored the outcome side. Impact is the effect of the outcomes on the environment and is usually long-term, which is not in Walter’s tools.
For our context, I would think of only three stages- changes in knowledge and skills, changes in perceptions and confidence and changes in specific behaviour. With our school library program we usually want to see the changes in these above mentioned three areas. So, our impact evaluation would be around changes dominantly in reading skill, changes in perception about school library among school teachers, student and parents and changes in behaviour, change in culture of library and reading such as students and teachers’ frequency and multiplicity of using library resource and teachers and school administration’s attention in operating and integrating a school library in the whole school curriculum.
There is no dearth of evaluation techniques for measuring such behavioural intangible aspects of impact of libraries. There is already answer for “can we quantity library’s influence” with a standard of ISO 16439. The ISO standard deals with potential methods for identifying, verifying and quantifying the influence of libraries on individuals and society which is applicable in every kind of library including school libraries.
A way ahead towards measuring learning outcome in 21st Century Learning Standard in context of library is to make a strong collaboration with subject teachers thereby making library an integral part of the subject taught in the classroom, which technically is in 21st Century Learning Environment.
Conclusion
Therefore, impact evaluation is an inevitable aspect of a reflective library professional. Time has gone to believe that the important services libraries are providing are intangible and unquantifiable. Despite the challenges of standardize testing, it is a reality now to measure the immeasurable effect of library in a manner that the measuring tools covers all kind of learner or library user. Librarians need to constantly learn and adapt to new things, such as to the changing trend of learning and hence the new methods of looking at the outcomes of the libraries.