What stays human? AI, accountability and the choices facing civil society (Transcript)

As AI becomes part of civil society’s work, where can it help, which decisions should stay in human hands, and who answers when something goes wrong?

In What Stays Human?, Accountability Lab Co-CEO Cheri-Leigh Erasmus speaks with Danai Nhando Danai Nhando, a human rights lawyer and responsible AI governance expert, and Natalia “Nati” Carfi, Executive Director of the Open Data Charter. From grantmaking and recruitment to agricultural AI projects in Africa and Latin America, they explore whose knowledge is reflected in these tools, what meaningful consent looks like when people depend on an organization for support, and where people can turn when an AI-supported decision causes harm. They also consider how organizations can be open about their use of AI and agree on safeguards that guide everyday decisions.

Read the full conversation below, listen to or watch the episode on Spotify, Apple Podcasts, or YouTube.

Episode produced by Naseera Noor-Mahomed, Global Communications and Marketing Manager at Accountability Lab.

Episode transcript

This transcript has been lightly edited for clarity and readability.

Nati Carfi [00:00]: AI is the answer, but what was the question? Nobody asked. This is the cool kid in town, so we all want to use AI. We jump in without asking: do we actually need it, and what is the purpose?

Cheri-Leigh Erasmus [00:22]: I’m Cheri-Leigh Erasmus, and this is the AccountabiliTea Podcast. Today, we’re digging into where value really lies for NGOs in the age of artificial intelligence. I’m joined by Danai Nhando, a human rights lawyer working at the intersection of technology and digital rights, and Nati Carfi, Executive Director of the Open Data Charter. We’re talking about shadow AI use, data risks for the world’s most vulnerable populations, and what it takes to build AI principles your organization can stand behind. Let’s get into it.

I’m excited to dive into today’s topic. We always try to make governance relatable – for example, talking about corruption through its effects on people’s daily lives. Rather than describing it as a big number, we ask what it means for healthcare or education in their community.

Yet many of these things still feel a couple of levels removed from us as individuals. AI affects us both organisationally and deeply as humans. We are all making decisions about it daily, and not just at work.

Last week, I noticed that my Instacart shopping app now has an AI assistant that can pull from my past baskets and make up my grocery list. I thought: do we need this?

I’m already saving a lot of time buying groceries online and having them delivered. Did we need another layer?

I find myself having these pendulum swings: one moment thinking, “This would be great for civil society. We should use it here,” and the next, “We’re probably all going to die on this road we’re on.” I’m excited to hear what those swings look like for our guests.

Nati and Danai, we’ll start with a lighthearted question, because this can be a difficult topic to explore.

What is one task you would happily hand over to AI tomorrow if you could? Danai, let’s start with you.

Danai Nhando [02:45]: Waking up early and doing the kids’ lunches. If there were a tool that could get that out of the way for me, I’d sign up.

This morning I was going back and forth with my son: “Hurry up! We need to get out. We’re going to be late.” Every morning feels frantic. My 12-year-old said, “Mum, you’re going to be doing this until he’s 18, and he’s only seven.” That put it into perspective.

So, a lighthearted answer: if an AI tool could get the kids out the door much quicker, I’d sign up for that.

Cheri-Leigh Erasmus [03:27]: There’s probably a market for an au pair robot. It sounds like that’s what you need.

Danai Nhando [03:34]: Exactly.

Cheri-Leigh Erasmus [03:36]: Nati, what would you hand over?

Nati Carfi [03:39]: I would also choose something AI doesn’t do yet: laundry, folding and ironing. I would absolutely leave that to an AI that could solve my chores.

Cheri-Leigh Erasmus [03:59]: That might not be too far-fetched. A couple of weeks ago, I saw a somewhat dystopian clip of humanoid hands training with hotel staff in India, learning to make beds, put sheets on, and swap things out.

Nati, the laundry humanoid might be coming sooner than you think. I don’t know how you feel about that.

Nati Carfi [04:32]: No.

Cheri-Leigh Erasmus [04:34]: With the lighthearted questions out of the way, let’s explore what AI means for civil society and NGOs more broadly. Both of you have a global presence and understand that its implications differ from place to place. I’m excited to hear those perspectives.

Danai, what are the risks when NGOs adopt AI faster than they develop policies and safeguards around it?

Danai Nhando [05:12]: That’s a good starting point, and there is data to back this up. It isn’t simply what we think is happening in the nonprofit sector; it is what the data shows.

A study by the Humanitarian Leadership Academy and Data Friendly Space looked at more than 2,500 respondents within nonprofits across [country count unclear in supplied transcript]. It found that at least 93% of people working in those organizations said they had used an AI tool, 70% used it daily, and 69% were using free tools.

That’s shadow AI: use happening under the desk, without others knowing. Yet only 22% of those organizations had formal AI policies. We are seeing substantial adoption in the nonprofit and humanitarian sector without the policies to match.

Funding sits at the heart of this. After the past two years of tight budgets and major cuts, are organizations going to invest in an enterprise model of the latest large language model?

If people are using AI, you may think, “Great,” while hoping they are using it safely – or act as though you can’t see it happening.

The risks are broad because nonprofit and humanitarian organizations exist to serve people. They hold possibly some of the most sensitive data in the world. We cannot allow this sector to lag behind. If anything, it should be prioritized when it comes to policy and governance, with much more opportunity across the sector.

One immediate risk concerns information. A few months ago, I wrote about the “unconsenting beneficiary” within nonprofit work: children, refugees and people who have to give their data just to live.

When you’re a refugee, you aren’t asking, “Are you protecting this? What safeguards do you have around putting my data into AI?” You want food and shelter. That unconsenting beneficiary is at the center of this.

Earlier this year, I think around June, there was a major cyberattack involving the World Food Program. It put the personal information of about 600,000 households in Gaza [at risk], including names, phone numbers and locations. In a setting where people are already so vulnerable, you can imagine what that means.

The risks vary according to what an organization does and the information it holds. They are amplified when there are no safeguards, checks or accountability.

If there is a cyberattack, will it be reported? Will anyone know the incident happened?

I strongly feel that the implications of AI for the nonprofit and humanitarian sector need much more attention and scrutiny than they are receiving.

Cheri-Leigh Erasmus [09:07]: What I’m hearing is that, in some instances, we’ve deployed a tool without doing the work. Now we’re crossing our fingers and hoping the worst doesn’t happen. In some cases, we’re already seeing major challenges.

Nati, where are you seeing the consequences of the gap between policy guidance and implementation?

Nati Carfi [09:34]: I’ve been having this conversation with partners and colleagues around the world. We are seeing deployment without safeguards or policies in place.

In a way, it is understandable given the funding cuts civil society organizations received last year and this year. Some teams have fewer people to do the work. AI can seem like an easy way to navigate those shortages by helping with drafting, revising, and other tasks.

As Danai said, few institutions have policies in place before they start using AI. Some team members don’t acknowledge using it, which makes it difficult for leaders to understand where, when, and how it is being used.

The Open Data Charter works under a fiscal sponsor. We’ve been speaking with its leadership about developing an AI and data governance framework for all the initiatives it supports.

We can all contribute to drafting it and learn from the practical challenges our teams encounter. It’s better to acknowledge that AI is being used and understand the shortcomings of that use.

Developing policies across the initiatives gives the whole community a role in making the governance and safeguards framework work for everyone. We will encounter different challenges, but the more we collaborate on improving the policy, the better.

Another positive trend is the growth of online learning tools that help civil society organizations understand how to use AI in their work.

There are free tools to evaluate your starting point and identify where ethical AI use could add value. They help you understand the available tools and their challenges, improve your databases, and meet privacy requirements.

They can also help you ensure that private information within your data isn’t shown or shared with AI tools. These are practical steps towards using AI ethically and consciously, without being left behind or endangering the personal data you collect and use every day.

Collaboration is the way forward. I don’t think ODC’s policy would be as good as it is without the other organizations working under our fiscal sponsor.

We learn from each other. The challenges others encounter and their good uses of AI feed into our own work. Conversations between teams and leaders make both our work and our policy better.

There are still shortcomings. I wouldn’t say we have solved this, because things are changing so quickly. Staying in conversation with other organizations helps us keep pace.

Cheri-Leigh Erasmus [14:23]: Absolutely. At Accountability Lab, as a fiscal sponsor, we’ve gone through a similar process: making sure everyone either uses the policy we reworked or includes equivalent clauses in their own policies. We share data, so this is critical.

It is also something for civil society and membership networks to consider, particularly where organizations share data.

I’d like to return to accountability. There are tools to help organizations think through privacy, consent, bias and security. We hope they are using those tools to develop policies suited to the communities they engage with, their work, the sensitivity of the information they hold and the harm they could cause.

Danai, if an AI-supported decision causes harm, who should we hold accountable, and how? What happens if you’re a refugee or program participant whose livelihood partly depends on an organization that could also be causing harm?

Danai Nhando [15:54]: That’s a big, difficult question on several levels, because AI enters organizations through so many different doors.

As Nati mentioned, there is shadow AI use: a staff member uses it behind the scenes, sees its impact on their work, but doesn’t fully understand the risk. Others use free tools because that’s all they can access. They can’t keep paying $20 a month for an enterprise version of Claude.

There are also third-party tools, as in your Instacart example. You wake up and find AI embedded in a tool you’ve already been using. You didn’t specifically consent to that, but your organization already uses the tool and may have data stored in it, so you may not move away.

Then there are formally procured tools. You’re intentional: you evaluate the risks and procure a tool rather than sleepwalking into AI use. Three months later, the provider introduces an AI chatbot within it, and you have no control over that.

Nonprofits also work in ecosystems. Your implementing partners might use AI even if your organization doesn’t, while you continue sharing data. There may also be a vendor breach. There are many ways sensitive information can be put at risk.

That makes governance and accountability central to this conversation. It doesn’t begin only at the organizational level; it also begins with the leading companies that have opened the doors to AI use globally.

When you look at OpenAI or Anthropic, one question people still ask is whether the sector is regulated. The US, for example, doesn’t have one overarching AI policy. [Unclear wording about individual states developing protections.] African countries are racing to launch national AI policies, but people ask: “You’ve written it – what does implementation look like?”

To give an anecdote, South Africa released an AI policy, and people noticed that it had been written by AI and was riddled with errors.

By the time people began putting regulation, safeguards and governance in place, the train was moving so fast that the response became largely patchwork, both organisationally and nationally.

Some places, such as the EU, are trying to get further ahead. Others look at that approach and try to follow it.

Accountability after a breach is difficult. In the US, the recourse we’ve seen has often involved people taking companies to court. Someone has had to decide to pursue the legal route.

A company voluntarily saying, “We caused the harm,” or, “Our tool did it,” is rare, if it happens at all. What accountability looks like remains, for me, a grey area that isn’t clearly defined.

Cheri-Leigh Erasmus [19:54]: There are so many places where a breach can happen. As an organizational leader, my biggest fear is shadow use. We have policies for our enterprise-level tools, but I don’t sit in front of everyone’s laptop each day. I don’t know what’s happening. In some ways, we don’t even know whether we are inadvertently causing harm. That’s the position we’re in.

Moving from organizations to the broader open data movement, we know that open data has moved power closer to communities.

It enables activism and helps hold governments and other power holders accountable. Yet those same datasets can be used in many different ways.

How is this conversation developing within the open data community, and how might its beliefs need to change in the AI era?

Nati Carfi [21:03]: It’s a great question, and we’re still debating it. One positive effect of the conversation about data for AI is that it has brought the open data and data privacy communities together. Previously, we often didn’t sit at the same table; in some data governance discussions, we were even seen as antagonists.

We’re now having more mature conversations about what open data means for AI and rights. We’ve always tried to balance access to information with the right to privacy.

AI has made that conversation more important than ever. I see a community bringing expertise together to navigate those two rights. Sometimes they conflict; sometimes they can work together, depending on the case.

I don’t mean to speak for the whole open data community, but one recognition I’ve seen is that most data used to train AI comes from the Global North.

That matters for representation in the answers AI systems produce. We need to consider what data is fed into these systems, where access to information plays a role, and how to get representation right.

In one project, we explored AI use in agriculture in Africa and Latin America. Many people we interviewed were developing small AI tools for crop growth and related purposes.

They recognized that, for consistency, they were using data from the Global North, which didn’t reflect the realities of the crops they were working with. They were willing to help create datasets to improve their own work.

How do we help create those datasets so AI can add value for people working with crops in Africa and South America, while ensuring the data is created, shared and used ethically?

Finding that balance is an ongoing conversation. I don’t know whether it will have an endpoint; it will continue as AI develops.

One positive is that people working on open data and those working on privacy and personal data are now having a more open, mature conversation.

Cheri-Leigh Erasmus [24:43]: It feels as though this has opened up 20 conversations we need to have. Some are happening, but at the wrong level or without the people affected by or using the tools.

My pendulum is swinging towards pulling the cord, because I don’t know how we [unclear] all of this back.

Danai, what advice would you give NGO leaders facing these decisions? How should they approach AI use, and what structures should they put in place before adopting it for efficiency? How should they think about what they might lose in the process?

Danai Nhando [25:38]: The nonprofit sector exists to serve people and communities. We want impact; our reports are all about being impact-driven. We need to return to that as the foundation.

I strongly feel that AI adoption in any organization should not begin with, “There’s a new tool available; let’s try it.” Start with the core work: where could AI streamline workflows or improve efficiency in service of the mission and greater impact?

Map the work, understand the workflows, and then ask which tools could support the mission without compromising what you value. It becomes less about adopting whatever is available or easy to access.

In my work, I approach AI transformation as a journey rather than an event. You don’t simply announce, “We’ve subscribed to Claude at an enterprise level. Everyone has access. Here’s 30 or 40 minutes of training,” and hope for the best.

AI transformation requires intentional change management, which many organizations are missing. You’re asking people to change how they work and use tools that can directly affect data sharing and access to information.

You’re also working with people whose existing approaches have worked for them. A tool might enhance that work, but not everyone will feel equally positive about using it.

You want people to be able to adopt and use the tool responsibly, rather than simply make it available.

Before developing an AI strategy, think about the long-term transformation journey. Don’t focus only on tools because they’re trendy or exciting. Ground yourself in the work you’re doing, then consider the tools.

That makes you think twice about adopting tools for their own sake. You reconsider why you want them.

In our approach, we started with AI principles. We looked at the core mission and developed principles to guide us when assessing tools. Take image creation: people often don’t think twice about creating an image in ChatGPT.

I created an infographic about the environmental impact of an image. When you put that into perspective – for example, the comparison with running a fridge for two days—people ask, “What do you mean?” The same applies when considering the environmental impact of a 30-second video.

That changes how you look at and use AI tools. You begin approaching them more responsibly. My core advice is not to focus solely on adoption.

Build an AI transformation strategy anchored in your mission first. Map the work, then identify specific workflows, inefficiencies, capacity gaps or intensive administrative tasks where you need support.

Only then start looking at tools. Don’t start with tools and try to figure out where they fit.

Cheri-Leigh Erasmus [30:33]: I’m hearing that we need to pause, step back and be strategic, as we do with most organizational work. Yet many organizations have moved too quickly without piecing together the implications.

Nati, could you close by discussing a task you don’t think civil society should automate, even when a tool works well? What should we never take away from the human doing the work?

Nati Carfi [31:16]: Thank you for that question. Danai, it was wonderful to hear you speak about this. AI is the answer, but what was the question? Nobody asked. It’s the cool kid in town, so everyone wants to use it without asking whether we need it or what its purpose is.

You can automate some things and make time for other tasks, but first we need to focus on the questions. Then we can ask whether AI is the answer, which AI, and for what purpose.

You may both have had this conversation with funders: they’re seeing increasingly similar proposals in response to their calls.

They recognize that AI is being used to draft proposals, but struggle to understand an organization’s original thinking or strategic vision within them. All the proposals are beautifully written and edited, which makes it harder to assess the value of the team and organization. They’re also receiving more proposals than ever.

We’ve been discussing with our board where AI should or could be used.

For now, we’re keeping AI out of our strategic documents. We’re doing the work ourselves: having the discussions and asking difficult questions about ODC’s strategic vision and future.

At the same time, I’m interviewing adopters and endorsers of the Open Data Charter, and AI helps me take notes in some conversations involving several people. I can only take so many notes myself.

It’s about finding a balance. What do we do with those notes? We plan to erase them once the interviews are complete. Will that be enough?

These are ongoing questions we shouldn’t leave behind. We need to understand the problem first and ask where we can add more value than harm. It’s complex, but we must keep asking ourselves, our organization, and our team.

We’ve also had a longstanding partnership with an organization that does our graphic design. We discussed this with the team because it has budget implications. We value the relationship, the work we’ve done together, and our understanding of each other’s organizations. They know our values and principles and have designed everything with us. We decided not to replace that with AI.

We also recently put out an open call for researchers.

In one day, we received CVs – around 117. We’re a five-person team, and the call still had about three days to run.

We asked ourselves whether to use AI for a first round of evaluations. We decided not to. Even though the volume felt like too much, we chose to take more time and evaluate every candidate ourselves.

We did ask whether AI could help and save time. But is this a task we want it to help with? Not for now.

The temptation is there. You have to decide when and where to use it, what question you’re trying to answer, and whether AI is actually the answer.

Danai Nhando [36:39]: Can I build on what Nati said? People focus so much on what they choose to outsource to AI. We hear less about what happens to the capacity it frees up. Where does that capacity go? What are you doing with it? Is it tangible?

Organizations are moving quickly to choose what to outsource, but [unclear wording] what happens to the capacity opened up in an employee’s work. Where does it go, and does it add value?

The goal becomes, “It saves me time,” and there’s a full stop. We don’t go on to say, “It saves me time, and then…” We need to talk much more about what happens after we’ve outsourced work and saved time.

Cheri-Leigh Erasmus [37:37]: There are a few things worth sitting with. Danai’s framework starts with mission, then workflow, then tools. The order matters. Unfortunately, much AI adoption in the sector runs backward.

Nati gave us a useful test: AI is the answer, but what was the question? Ask that before adopting a new tool.

The real risk here isn’t the technology itself. It’s organizations holding highly sensitive data moving quickly on AI with almost no policy to guide them.

Here’s a question to take with you: when AI saves your team time, where does that newly liberated capacity go?

If you can’t answer that, you may already be sleepwalking your organization into exactly what this episode warned about.

Outro [38:31]: Everything you need to know about governance.