Unsupervised Thoughts A blog on machine learning

What Are the Ethical Risks of Your AI Project?

How to prevent AI systems from causing unintended harms?

AI systems may cause unintended harms and their increasing adoption leads to additional risks and scrutiny. In this blog post, I present tools project teams can use to identify ethical concerns associated with their AI use cases. I first illustrate why this is so challenging, I then discuss various ethical checklists that were designed for this purpose, and I finally provide some recommendations on how to select such a checklist.

This article has also been published on Dataiku’s blog.

Identifying Potential Ethical Concerns Is Challenging

Let’s start with two real-life examples to show the complexity of such assessment. Imagine first that you manage an online advertising campaign on Facebook to promote STEM careers. You want to reach both men and women so you set the same “maximum bid per click” (i.e., the maximum price you’re willing to pay for one click on the ad) for these groups. But then, something surprising happens: your ad ends up being shown more to men, and up to 45% more for the 35-44 age group. How come? Does the ad-serving algorithm simply reflect actual consumer behavior and maybe the fact that women click less on your ads? Or has the algorithm been contaminated by biased data which made it assume such a behavior?

Through a field test in a similar context, two researchers concluded that neither of these explanations was correct. Women tended to click more on the ads and the disparate number of impressions was uncorrelated with the likely level of institutional bias in the countries covered by the campaign. The real cause was the “economics of ad delivery” — since women are more valuable targets for advertisers, the ad was more likely to be outbid for female consumers.

Let’s say now that you’re a machine learning engineer in charge of developing the recommendation engine of a streaming platform. How do you set the objective for your algorithm? After discussing with various internal stakeholders, it seems clear that you should maximize “engagement”, i.e. the time spent by users on your platform or the number of videos viewed. This metric is directly connected to your platform’s revenues. Thanks to the “wisdom of the crowd,” maximizing engagement also seems a way to promote the best quality content and provide a great user experience. However you soon realize the alarming consequences of this simple choice. Given the “natural pattern of borderline content getting more engagement” (in Mark Zuckerberg’s words), your recommendation engine now makes conspiracy or extremist videos more visible and soon enough, ill-intentioned users discover this trend and create purposely divisive content to generate revenues.

These examples illustrate some of the challenges of anticipating the negative side effects of your AI projects:

Ethical Checklists Have Been Specifically Designed for AI Use Cases

Fortunately, several organizations published ethical checklists to guide practitioners through the identification of the potential ethical concerns of AI use cases. For example:

Examples of ethical checklists

In a nutshell, these ethical checklists are just lists of questions to structure your assessment of the ethical aspects of your AI projects. They facilitate the communication between project team members and stakeholders, they contribute to harmonizing practices, and they help make and document decisions. Furthermore, their main benefit may simply be their cultural impact throughout your organization: by opening debates about the ethical implications of AI projects, they contribute to raise awareness on this topic and foster a questioning attitude.

So, what questions are included in these ethical checklists? They of course differ from one another but some essential questions are often or always present:

How to Choose Between These Ethical Checklists?

The ethical checklists share other important features. First, they are broad in scope and illustrate the complexity of the Responsible AI topic, as well as the need to combine various expertises and perspectives. In particular, they generally cover various categories of concerns:

The ethical checklists also tend to touch upon all stages of the AI project lifecycle: the scoping of potential AI use cases, the exploratory analysis of datasets, the training of machine learning models, their inclusion in a broader IT system and the deployment and operation of such systems. For some of the ethical checklists, the questions are specifically adapted to each of these stages. In any case, the four checklists encourage their users to periodically revisit their answers.

A final commonality between the checklists is that they do not provide an assessment or score that would tell you, for example, whether the identified ethical concerns are significant or whether your mitigation measures are sufficient. This is adequate because these complex issues require fine-grained analyses and any kind of mechanical appraisal would probably be too coarse to help.

If you’re now convinced that you should use such an ethical checklist to assess your use cases in a structured manner, the question becomes: which one to choose? There isn’t an obvious answer because none is clearly more appropriate than the others for all situations. With the various ethical checklists, the level of detail, the guidance, but also the “administrative burden” differ. For example, there are more pages in the fourth checklist above than there are questions in the first one! You then need to find the right balance between them, given the severity of the potential impacts of your use cases, the AI maturity of your internal stakeholders, and the culture of your organization, so that these ethical assessments stay meaningful and efficient. To help you with this choice, you can find a more detailed comparison of the four checklists at the end of this blog post.

In any case, don’t hesitate to customize your ethical checklist to better take into account the lessons learned from its use and reflect the needs of your organization. For example, if your organization carries out Data Protection Impact Assessments (DPIA) under article 35 of the GDPR, you should align the two approaches to avoid redundancies or inconsistencies.


Assessing potential ethical concerns for an AI use case is crucial given the significant risks of inadvertent harms. This should be done early in the project lifecycle and regularly reconsidered to take into account the progress made or context changes.

You can use one of the ethical checklists that were designed for this purpose. They’ll give you a head start but it’ll still be up to you and your organization to draw the necessary conclusions. This requires leaders to be aware of the potential ethical implications of AI, as well as thoughtful processes to incorporate technical and business expertise, engage stakeholders, and make difficult decisions. Various practices have recently emerged to enable such deliberation, for example: AI ethics committees, accountability reports, and public notice of existing and proposed uses of AI by public agencies.

More broadly, these ethical checklists should be part of a coherent Responsible AI framework, that would also include other methodological assets, such as ethics guidelines, a template for algorithmic impact assessments, an audit framework, and templates to document datasets and models. If you’re interested in all this, make sure to follow the progress of the “Responsible AI Working Group” of the “Global Partnership on AI.” In December 2020, it will report on the results of its first project which aims at analyzing existing and potential Responsible AI initiatives and “recommend new initiatives and how they could, in practice, be implemented and contribute to promote the responsible development, use and governance of human-centered AI systems.”

Bonus: Comparison of the Ethical Checklists

Title Ethics Guidelines for Trustworthy AI AI Fairness Checklist Of Oaths and Checklists Stakeholder Impact Assessment
Authors High-level expert group on AI set up by the European Commission M. Madaio et al. (Microsoft) DJ Patil, H. Mason and M. Loukides D. Leslie (Turing Institute)
Responsible AI Themes Covered Human agency and oversight Safety Security Privacy Transparency Fairness Inclusion Societal impacts Accountability Fairness Fairness Privacy Security Safety Privacy Fairness Inclusion Societal impacts Environmental sustainability
Advantages -Tested in the context of a wide public consultation -Comprehensive and well explained -Particularly adapted to the European context -Co-designed with machine learning practitioners -Different questions at the various project milestones -Very concise and simple -Yes/no questions which leave little room for ambiguity -Embedded in a comprehensive Responsible AI framework -Different questions at the various project milestones
Limitations -Quite long -Specific to the European context for certain questions -Quite long -Restricted to fairness issues Limited guidance for inexperienced practitioners General questions which may be too high level for inexperienced practitioners
comments powered by Disqus