By: Blair Glencorse and Anne Sophie Ranjbar. This blog post was originally published by Feedback Labs. The Accountability Lab, a member of Feedback Labs, catalyzes a new generation of active citizens and responsible leaders around the world. They train, mentor, and resource local changemakers to strengthen systems of accountability and unleash positive social and economic change. Blair Glencorse is Executive Director and Anne Sophie Ranjbar is Associate Director of the Accountability Lab. Follow the Lab on Twitter @accountlab.

Last month, we released our most recent Impact and Learning Report– a document which is the result of a year-long process of trying to understand  how well we are doing at the Accountability Lab, what challenges we are facing and how we can improve. We asked five sets of key stakeholders around the world (our “accountapreneurs”, participants in our programs, governments, peer organizations and staff/volunteers) a series of short questions that relate to our Theory of Change. In this, at every point of the logic model, we are trying understand how we can better achieve the outcomes we are looking for- namely improved lives for citizens through greater accountability.

We received nearly 200 total responses across the groups, averaged responses for each question, and calculated the total non-weighted average across all stakeholders (see the complete dataset here). The results proved interesting for us in a number of ways. They indicated, for example, progress in making our target groups more aware of accountability (a 7.0 weighted score on a scale of 0 to 10); but that there is further to go in providing the kinds of support these stakeholders need (we scored 6.6 on a scale of 0 to 10). Equally, the survey showed progress in building a collaborative movement for accountability. Our efforts to generate public dialogue, change mindsets among young people and build communities around ideas are taking root.

This useful feedback has already helped us improve in 2016 through improving collaboration, focusing on sustainability and working on our communication. But I think the most valuable lessons we can share with the broader community is what we are learning about the process of learning itself. Plenty of organizations talk about learning, but concrete lessons are often few and far between. Here are five that come to mind based on our recent experience:

1. Learning in a meaningful, adaptive way is really, really hard. As the team at Global Integrity has outlined in their excellent strategy, adaptive learning matters because challenges are complex and solutions are emergent. It means understanding not just what works on the ground but where, why, and how successful approaches can be used to inform collective action. After all, individual learning is useful but learning that helps to shape shared efforts and push the field forward as a whole can be even more impactful. Learning should be a collaborative process, which amplifies its effects. We’ve found thinking about this in relation to our work on accountability in practice is tough. But, we now understand that learning is one part of the process of improvement- in the short-term to iterate quickly; and in the long-term to shape strategy. And if we can work with others around our findings, we can improve accountability outcomes over time.

2. Comparing internal and external learning is useful. Previously we were primarily concerned with how those outside the Lab perceived our work, as is natural- they are the stakeholders we aim to support. But this year we added staff and volunteers to our survey; the comparative perspective has been fascinating. For example, we discovered that the Lab has influenced the behavior of staff and volunteers less than of our stakeholders. While our staff may have already had a different starting point in terms of understanding, this still indicates that perhaps we need to put more effort into internal training, communications and network building.

3. Adaptive learning is a process that has to grow and develop from within an organization from the bottom-up as well as top-down. It has to become part of the DNA of systems and operations. Top-down calls from management to consistently gather feedback data and learn from mistakes fall on deaf ears if the process is seen by staff to be burdensome or superfluous, or if there is no incentive to do so. When delivery of projects and programs is tied to pre-defined logframes or matrices, for example, teams can become absorbed with simple learning and agreed deliverables rather than a more in-depth process of understanding how to improve over time. We’ve been talking about this incessantly and we are lucky to have great teams on the ground who are working hard on learning, but the challenges of changing to a learning mindset are real- and for every organization it is an ongoing process.

4. Trying to learn too much can be a mistake. Given limited time and capacity, keeping the feedback process simple makes it feasible to minimize transaction costs and improve outcomes. This year’s survey included shorter, universal questions that required responses for all of them, which generated cleaner, quicker comparable data. We also favored multiple-choice questions over essay questions to prevent survey fatigue and to accommodate recipients who were not as confident in their writing abilities. As a result, we gathered more information that was more meaningful and useful.

5. While a rigorous external control trial might be out of reach for a small organization like ours, an independent impact evaluation process is entirely feasible. In our survey this year we tried to minimize “courtesy bias” through non-leading questions and a broader spectrum of answer choices. This helped generate more honest feedback, but we know there’s still more we can learn. We discovered that PhD students love learning (surprise!) and are excited to work with the political science department at Auburn University on our impact and learning process. The students will be helping us refine, deliver and interpret our survey this year in an objective way. It’s a win-win- the students get useful experience and we benefit from low-cost, independent, academic rigor.

One final piece of feedback we received through our survey is that we should document more of what we’re learning and share this with the field more systematically. In the next month we’ll be hosting events on adaptive learning in all of our focus countries (Pakistan, Nepal and Liberia) for communities working on accountability and governance- and we’ll blog about the outcomes here. Let us know if you’d like to be part of the conversation!

eedev