Table of Contents

Role of Information in Performance Indicators, Impact Assessment and Program Evaluation

Shilpa Kedar

Mandel Center for Nonprofit Organizations

Case Western Reserve University, Ohio, USA

Shilpa.kedar@case.edu

The critical task before an evaluator of a developmental project is to decide which parameters should be measured and how to best reflect performance and impact. Organizations may find themselves capturing a significant amount of data, which is not sufficiently linked to the need to monitor performance, or the need to convincingly demonstrate the impact.

My experience in India this summer working with SEWA (Self- Employed Women’s Association) focused my attention on the value of judiciously captured and employed data for designing performance metrics and an impact assessment. A student of nonprofit management at CASE, my role in SEWA was that of an intern, charged with two tasks. The first was to develop performance metrics for all the activities undertaken in the drought and earthquake affected districts of Gujarat, India. The second was to design a study for assessing the impact of the land and water activities in the districts. Following my project with SEWA, I spent two weeks in visiting two Non Governmental Organizations (NGOs) in India that are partially funded by a U.S. based international development agency. My task was to file a report on my observations at the site.  I hesitate to use the term ‘evaluation’ for describing the scope of my role, and do so only for lack of a convenient substitute.

What follows, are reflections of my experience in developing performance indicators and a design for impact assessment, and in ‘evaluating’ programs. The underlying theme, as the title suggests, is the capture and use of information for measuring performance, assessing impact and evaluating programs. The article also draws attention to the role and scope of the external consultant or evaluator, or intern as in my case, in accessing, collating and presenting data in a manner most useful to the NGOs themselves, and to its external stakeholders.

Background

SEWA is a member-based organization of poor, self-employed women workers, striving towards full employment and self-reliance. It has been working in both urban and rural areas for over thirty years, has a strong membership base and a rich understanding of the grassroots. Since 2002, SEWA is implementing a government-aided program called Jeevika. The program has been designed in response to the devastating earthquake that struck Gujarat on 26 January, 2001, and is administered through a network of grassroots members spearhead teams, district associations and the different rural institutions established by SEWA. In tune with the integrated approach followed by SEWA, Jeevika has thirteen activities being implemented in the target districts – Land and Water activities, Agriculture, Livestock and Fodder, Nursery and Plantations, Adult Education, Micro-finance, Health, Gum Collection, Salt Farming, Craft, Housing, Food distribution and Childcare.

The two tasks assigned to me were driven by SEWA’s desire to have a scientific basis to measure and reward performance, and to collate information about the outputs and outcomes of Jeevika’s activities. Besides demonstrating results, the impact assessment design when implemented, will help analyze and share data and lessons across Jeevika and SEWA’s programs and district offices. Importantly, it will help identify critical success factors and understand challenges to successful implementation. As Jeevika extends its activities in more villages and blocks in the earthquake-affected districts in the next few years; this input is very critical in shaping strategy, program design, and replicability of the program.

Aiding my projects significantly was the fact that Jeevika has invited an independent market research agency to collect data on numerous aspects of livelihood, financial security, health and other development indicators from the villages in which the program is to be operational. Data is to be collected and shared with the Jeevika program staff over the seven-year program period. The first report with baseline data for some of the targeted villages where implementation had started, was available to me and helpful in understanding the nature of data that would be accessible for performance monitoring and evaluation purposes.

Nature of impact assessment                                  

The developmental approach adopted by SEWA has been formulated and tried and tested by SEWA over a number of years. I therefore took the program theory as proven, and the need for such a program as given.

With the implementation of the seven-year program having started two years ago, and the program in the stage of rapid expansion, it seemed appropriate that the assessment of the impact be preceded or accompanied by, an evaluation of the implementation processes of the program. Process evaluation is important to eliminate the possibility that a possible failure in impact is attributed to a failure in program theory instead of a failure in process implementation. The impact assessment study was therefore designed to include both process and impact evaluation. 

Studying the impact of one activity within an integrated approach

Given the brief time at my disposal, the scope of my project was limited to assessing the impact of the land and water activities. A necessary restriction, this however led to the challenge of alienating the impact of one activity from amongst the impact of the other activities in Jeevika’s integrated approach.

Programmatically titled ‘the water campaign’, the land and water activities fructify in the form of increased agricultural produce, healthier livestock, and increased plantations, amongst others. Many rounds of discussions later, I eventually decided on a design that would evaluate the direct impact of the land and water activities, and also the impact of these activities as it was manifested in agriculture, livestock and fodder. It can be argued that the impact on agriculture and livestock is not purely a result of land and water activities, but also of other activities like training and other capacity building efforts. Towards this end, the assessment design takes into account the input provided in the realm of Jeeviakas’s agriculture, livestock and fodder activities.

Moving towards a higher level of data analysis and presentation

I perceived SEWA to be very diligent in capturing data at the grassroots level where services were administered, or where community based activities took place. Yet, most of the organizational documents that I came across presented information in a very rudimentary form. The program staff was very convinced about the impact they are having, and had basic data related to growth in membership base, trainings, beneficiaries, etc., collated in the reporting systems. Nonetheless, they seemed a step away from being able to present information in a way that will effectively reflect the impact their efforts were having.

A good percentage of my time was spent in interacting with the program staff and consultants associated with SEWA, and the formulation of the performance metrics, and in turn the impact assessment, was possible only with their whole-hearted support and sharing of knowledge.  The process of developing the performance indicators required at least two to three meetings, where we discussed the nuances of the performance metrics as they evolved. It is in these iterative conversations, that it was possible to go beyond the basic figures and stories, and reach the main issues that the staff was trying to monitor, or impact.  Devising ways to measure and present these critical parameters became the focus of my efforts.

Assets that the impact assessment design could build on

As mentioned earlier, SEWA and Jeevika have well established systems to capture data at the grassroots, and a professional research agency already recruited to capture comprehensive baseline and endline survey data of households targeted by the Jeevika program. The healthy level of funding from the Government of India meant that adequate funds were available for evaluation efforts. This, and the attitude of program coordinators, made it possible to have an expensive but rigorous design for an evaluation involving control groups and a time-series evaluation.

Critical role of qualitative data

As an external consultant (intern) on time-bound project terms, it was challenging to efficiently and adequately learn about the field in which the program is to be evaluated, and research the best practices therein. With a few reports on evaluation efforts of similar programs downloaded from various sources on the Internet helping me get started, I found the subject knowledge of the program staff and consultants of SEWA to be instrumental in understanding the issues involved in implementing the numerous activities of Jeevika. As elaborated in the next section, respecting and using the knowledge base of the internal stakeholders of SEWA was contributed to the process of building ownership of the performance metrics.

There was a good amount of literature that also fed into the framing of the performance indicators and assessment design. Particularly helpful was the literature generated by visiting scholars and interns, as they succinctly captured the story of SEWA and its work. Documents generated by SEWA provided detailed accounts of its various activities, and struggles therein, often accompanied by case studies and rudimentary statistics on service provision and coverage.  Supplementing reading of books and reports with viewing video films was very useful to get a visual picture, and understand the multifaceted organization and its services.

Visiting the villages to see the various activities, and working the rounds with the field staff was beyond doubt, the most effective way to get a feel for the struggles and the successes of the program. Especially in the case of the two NGOs whose programs I ‘evaluated’ on behalf of the US based agency, there was no doubt that personally seeing the services being administered and the involvement of the community was very valuable in getting a true sense of the programs.

Iterative process of development leading to ownership

It is my belief and hope that the joint effort and iterative process adopted for interactions with the program staff and consultants to develop the indicators and the assessment design was also instrumental in building ownership of the metrics. This is critical, as the metrics may need to evolve and be fine-tuned as they are put into practical use. In the same vein, acceptance and ownership of the performance indicators is vital, as the indicators will also be used for internal program monitoring and performance evaluation of the staff.

Identification of need for data

Implementing the performance design of performance metrics and impact assessment necessitates the capture of more data than what is currently being captured, and the reports spell out the additional data required. An example would be the need to capture the average height of the water table in the target villages, something that is currently not being captured, but conversations with the consultant indicated it was a valid, effective and viable measure to indicate impact of the water campaign.  At a more macro level, it was evident that data on numerous parameters would need to be collected from the control group villages – something that till date was not required.

Identification of data needs at this stage of the program implementation is useful, as the two years of program implementation experience provides an indication of performance elements and special challenges that need to be measured. At the same time, it is not too late to initiate capture of additional data in villages where work has already begun.

Brainstorming with program staff on devising new ways to measure performance and impact meant that in some instances data capturing systems needed to be put in place for metrics that were developed with ‘future organizational structures’ in mind. These are for activities that have not yet been implemented on the ground, and for which the organizational structures are still falling into place.  In some instances, the metrics are based on monitoring and evaluating systems that are in the process of being designed. In the case of the adult education activity of Jeevika for example, a system for internal monitoring of the quality of the teachers was being designed, and the performance metrics we identified, planned on drawing data from this system.

Some of the additional data that is required, can possibly be collected by the market research agency appointed by SEWA, and some would need to be collected by, or generated by, the program staff in the process of implementing the program.

Capacity building, and the need for a mentor

During my six-week term, I was assigned to work in close conjunction with a member of SEWA’s staff. The collaborative work ensures some level of continuity to the effort done this summer, and provides a local contact point for other program staff in case of queries in implementation of the metrics or the impact assessment design. Without being presumptuous, it is my hope that the joint exercise of logically thinking through and discussing the challenges in measuring performance and impact has enhanced capacity to think critically and logically, and equipped SEWA with additional tools and skills to develop performance metrics and impact assessment designs for its other activities and programs.

My experience as a young professional undertaking complex projects of this magnitude suggests that it is tremendously helpful to be have access to experts who are committed to guiding your work. In addition to the supervisors and consultants at SEWA who guided me, I had the privilege of remote assistance from my professor of program evaluation and impact assessment at CASE. His input was invaluable whenever I found myself mired in a dilemma about the approach to adopt, or about  specific elements of the design. The text books were valuable, but customized advice in SEWA’s context was even more so. I believe a mentor is particularly important for intensive and time bound projects, as there is little luxury of time to step away from the complexities of the project and revisit the decisions being taken.

Shades of my role as a designer and evaluator

Program staff of any NGO, just as with SEWA, typically knows the impact the program is having, and has ample anecdotal evidence or individual case studies where the impact has been perceived. An evaluator (or designer of an assessment study as in my case) presents a means to quantifiably prove the impact, and therefore is a welcome intervener. I believe that my intervention nudged thinking on evaluating performance and demonstrating outcomes, and provided a good reason for the program staff to step back from the daily activities and work with me to articulate challenges and identify ways to measure the performance and impact. The effort at times pushed the activity heads to think beyond data capture and reporting, to data analysis and presentation in ways that would effectively reflect the performance and impact of the activities of the program.

A professional from outside the realm of SEWA, I possibly also contributed in terms of introducing a fresh style of conceptualization and writing, using terminology and report elements familiar to and expected by an International audience. I felt it was important to attempt to write the reports in a manner that made the document readable not only for the staff intimately associated with the program, but also for those (such as funding agencies, researchers and staff of similar programs) who may not be so familiar with SEWA. This I perceived as an important step towards enabling the sharing of practices and lessons within the development sector.

The future of the project reports

The program director of the Jeevika program will be supervising the impact assessment, with a staff person who will be charged with spearheading the effort. Since it was not possible for me to interact with the implementer of the design, the impact assessment document has been drafted to be very explanatory. It provides the rationale for taking design decisions, and sensitizes the reader to issues involved in decisions that need to be taken. For each evaluation question, a summary table clearly lays out the data that needs to be captured, the source of information, the method of data collection, and the point of time that the data is to be collected.

The performance indicators will be disseminated within the organization to guide data capture, analysis and presentation. The performance indicators spreadsheet too, is prefixed by an explanation of the intent, rationale and scope of the metrics to allow for easier dissemination and understanding within the organization.

It was heartening to see that even whilst I was still finalizing the reports, there were calls from program staff to see if the performance metrics could be of help to them as they created grant proposals and reports. The positive reactions from the program staff and consultants, and the interest and excitement as they worked with me to identify and shape the metrics were by far the most powerful motivator, and made my experience all the more valuable.

Back