Why Participatory Research Methods Are the Missing Link in AI Bias Detection.
For over a decade, I have been working as a researcher and a journalist on human-centric development issues across South Asia. My work involved doing extensive studies on sex and Informal entertainment workers, framing policies and programmes to tackle violence against women, building empowerment tools for people with lived experience of modern slavery, and understanding labour relations in the supply chain.
Across all projects, primarily UK government-funded, there was one issue that always loomed large — ethics and safeguarding. To get ethics right, every project generally begins with deep scoping work, which involves speaking to multiple stakeholders, learning from their experiences, interacting with the community in question, understanding their concerns, and points of view. This approach helps in understanding the contextual causes for marginalisation.
Marginality, amongst other factors such as poverty and access, is also strongly shaped by biases that stem from factors such as perception of those in power, caste, religion, ethnicity and gender. Some are cultural, some are historical, and the majority are systematic. With the coming of AI, there are growing assumptions that AI could overcome social biases through innovation and inclusion, and that public services will become more efficient because AI as technology can leapfrog systematic gaps created by human systems.
However, move to urban India, the narrative changes dramatically: tech optimism is a popular notion because the uptake of it both by the population and the government has been revolutionary. The Indian government’s support for sophisticated digital infrastructure, such as a unique identification number and direct beneficiary transfer (DBT), has made the problem of identity and delivery smoother, with tens of millions of people receiving free rations and cash transfers directly into their accounts.
India has also made the digital payment system nearly universal and introduced a seamless AI-enabled system for contactless movement in the airports. These are, however, mostly examples of digital plumbing, and it’s not the same as developing AI systems as smart gatekeepers.
While AI innovations in India are fast-paced and with a well-intentional national policy for “AI for everyone”, it promises to build AI-based systems in areas such as agriculture, health, education and criminal justice, which can be a game changer for India’s economy and the public service is delivered. What is, however, worrying is the lack of a concrete policy on ethical implications.
Studies show that, unlike in the West, data in India is not always reliable. Sometimes communities are missing or misrepresented in databases. Also, large swathes of rural population, especially women, indigenous tribes, and elders, do not use the internet at all. The issue of the digital divide is enormous.
In my interaction with vulnerable communities, I found that internet usage amongst poor, marginalised and vulnerable populations is almost negligible. The semi-literate population use voice-enabled services but doesn’t use the internet for service-related work.
In many places, the internet has made inroads, and secondary databases based on internet-aggregated information have emerged across South Asia. Furthermore, proxies such as surname, occupation, skin colour, mobility, and geography often influence other factors that contribute to bias in the Indian context.
On a more global level, the UN Human Rights Council flagged such disparity (report A/HRC/56/68). It warned that AI-built systems can amplify and perpetuate racism and racial discrimination in the absence of regulatory mechanisms. It argues that biased datasets cannot achieve technological neutrality. It calls for inclusionary algorithm design and a lack of accountability. The report especially signals out predictive policing as deeply problematic, as it tends to profile minorities and people of colour routinely.
One of the most recent and interesting examples of AI-bias has emerged from a study titled “Inside Amsterdam’s high-stakes experiment to create fair welfare AI”, published in MIT Technology Review. The analysis assumes great significance for the general public everywhere because it makes us pause and be less optimistic about the supreme powers of AI.
Pure mathematical computation drives AI models. Its interpretation and classification of data will be superior, especially in contexts where the quality of public data has been historically poor and compromised. Unfortunately, many AI entrepreneurs are already using data without any guardrails in their rush to innovate new AI-enabled applications and services.
Amsterdam’s AI-enabled welfare system “smart check” was built to evaluate applications for potential fraud, determine if applications needed further investigations, and enhance accuracy. It was built on OECD guidelines and sound technical safeguards, expert monitoring and transparency measures. Despite attempted debiasing, the final model showed the same biases as humans, and the original model before debiasing was much more biased than human coworkers.
The authorities disclosed technical materials, including the actual machine learning model, all the source code, and details of the bias test. What then went wrong? The study found that when one bias was removed, new biases against different groups crept back because different statistical notions of fairness were found to be mathematically incompatible with one another.
One ethical question that emerged was, “What does society deem as fair?” The AI model analysed historical data of welfare recipients who had been investigated for fraud. The journalists in a separate podcast, which ed concerns .
One factor that explains this is that AI was unable to comprehend the complex socio-economic factors that led to welfare fraud. AI experts state that AI models suffer from logical coherence errors and are prone to mimicking patterns that could create reverse causality. In this case, the AI model was echoing the patterns of biases in the data it was trained on, thereby recreating the bias.
Murry Shanahan, Professor of Imperial College and a Principal scientist at Google DeepMind, offers a philosophical deep dive into AI, defining it as “folk psychology”.
“Folk psychology” is how humans interpret the world by applying concepts such as “belief”, “desire”, “intention”, and capabilities that AI cannot. In hindsight, the journalists who investigated the Amsterdam study remarked that the process should have better addressed whether it was considered “who are we including in the conversation” or “really listening to”. So it was not just technical, it was also a question of ethics, policy and values.
The Dutch story has important lessons for citizens elsewhere about how AI companies are using data for new services. To what extent are new AI surveillance measures increasingly being used by the state or private agencies to ensure they are bias-proof? Presumably, it is not.
In my own experience of using an AI Assistant with consented data of positive voice involving 17 participants. The project was training a vulnerable group on collecting data on lived experiences. I found that LLM models started hallucinating and generalising sentiments, often making up the transcripts, which sounded genuine.
The Amsterdam case study has significant lessons for India’s AI policy makers and enthusiasts. India has declared its ambition to be a global leader in Artificial Intelligence (AI) governance. India is already embedding AI into defence, military, and financial hardware, giving it a strategic defence. As the world’s largest democracy and a tech-savvy nation, AI offers an opportunity for wider social good and to use it to leapfrog the current state of the digital divide. Not sufficiently training foundational AI models in Indian languages will miss the opportunity for India to leapfrog the digital divide. Data show that currently, Indian languages make up less than one per cent of online content.
That is, however, changing: in 2023, Indian AI startup, Sarvam AI, released the first open-source Hindi language model called OpenHathi-Hi-0.1. The AI model was the first in a series of models that will contribute to the ecosystem with open models and datasets to encourage innovation in Indian language AI. Google DeepMind is also actively working on supporting Indian languages, such as Morni, Multimodal Representation for India, which involves creating models that can understand a range of Indian languages and dialects.
India stands at a critical juncture. India has declared its ambition to be a global leader in Artificial Intelligence (AI) governance. As a tech-savvy nation, it is well-positioned to champion an inclusive human-centric approach. Unlike Amsterdam, which had robust regulatory frameworks yet still failed, India is building AI systems without comprehensive ethical guidelines. The lessons are clear: technical excellence alone cannot eliminate bias.
From my research experience, India might consider the following:
· Mandatory participatory design processes where affected communities help define problems before solutions are built.
· Bias auditing by researchers who understand marginalisation patterns, not just technical metrics.
· Making algorithmic decision-making processes public.
· For entrepreneurs working on developing AI products using data, it’s important to go beyond the usual market research. First-hand understanding of users’ needs are important,
India has the opportunity to pioneer inclusive AI development for the Global South. The question is whether we’ll choose innovation that empowers or excludes — and whether those most affected will have a meaningful voice in shaping that choice.