The ethical implications of AI decision-making

You may or may not remember that in late 2021, online real estate giant Zillow made its departure from the ‘iBuying’ market, laying off around 25% of staff whilst plummeting in net worth. The company started 2021 valued at a high of $48 billion, and left 2021 at $14 billion. Significant losses and disruption that can ultimately be chalked up to artificial intelligence (AI) algorithms that made some seriously misguided decisions. In this case, an AI system was misvaluing properties to such an extent that Zillow was burning through cash at a rate unprecedented to them – and their value has not since recovered.

AI decision-making is not an entirely new concept, but it is one that has boomed in prevalence in recent years, especially with its sudden surge in popularity and use cases. With AI increasingly influencing decisions, either human-made or fully autonomous, the ethical implications of the technology call more than just mistakes or errors into question. Where should AI be used? Where do we draw the line? AI’s capacity to make decisions for us touches on the fundamental questions of human autonomy, fairness, bias, and our trust as a species in an ever-more automated world that will continue to shape how we live our lives.

AI is undoubtedly a revolutionary technology that can grant unparalleled opportunities and, if used correctly, should be used where applicable. However, the real question is, how much control are we willing to give up?

The key ethical concerns of AI decision-making

The ethics debate surrounding AI is a fire that has been raging since its inception. There is no catch-all solution nor any singular correct way to implement it. At its core, it is a technology that must be approached pragmatically and with care.

However, this does not answer the burning questions surrounding its ability to make decisions, especially decisions that can reverberate through our society: who bears responsibility when AI makes mistakes? How do we protect individual privacy while training these systems on vast datasets? The delicate balance between advancing AI capabilities and maintaining human autonomy requires careful consideration of accountability frameworks and fair implementation practices. Perhaps most of all, it requires a deep understanding of the ethical implications that come with its use. So, let's take a look at a couple of the key concerns.

The transparency paradox

At the heart of AI ethics lies a fundamental and unfortunate paradox: the more powerful and complex AI systems become, the less transparent their decision-making processes can be to human oversight. This is the fundamental concept behind what is known as the 'black box' problem. This issue is particularly prevalent with deep learning AI models that utilise complex neural networks where data is processed through many layers of interconnected nodes with inputs being assigned 'tokens' that arrange data hierarchically.

The problem here is that, in many cases, no one is able to determine how an AI comes to its conclusions. We can know the input and the output; everything in between is a mystery. The inability to see how an AI model comes to any given decision presents a myriad of ethical and technical problems. Technically, this means that if there is a problem with an AI's output, it is tough to know where things are going wrong and what to fix. Issues in standard software can be identified more easily and fixed with a patch, complex AI is a different beast. Ethically, it begs the question: how can we place complete trust in a system we do not understand, especially if it comes to decisions that can drastically affect someone's life?

Algorithmic bias and fairness

Algorithmic bias can have a significantly negative impact on the people for whom it makes decisions. There have been cases of algorithmic discrimination where people of colour have been denied healthcare or men being favoured over women for job positions. Even if unintentional, these biases perpetuate existing inequalities and, if left unchecked and trusted entirely, could have, and have had, severely adverse outcomes for people.

The biases held by AI models can come from multiple sources – historical datasets that reflect contemporary societal prejudices, underrepresentation in the development teams that create AI models, and human error through things like flawed testing protocols. With AI decision-making being used and prospectively introduced to more areas, such as housing (as examined at the beginning), loan approvals, and criminal risk assessments, it is crucial to ensure integrity and fairness.

However, whilst algorithmic bias has been an issue in the past and will undoubtedly continue to pop up here and there, it is not an issue that has gone unnoticed. The black box problem can make this a complex issue to fix. Still, with continuous monitoring, more human oversight, and bias detection tools, AI models can become more bias-aware in the future.

The data problem

Many would argue that this is ‘the big one’, and they would not necessarily be wrong. The biggest risks, especially for organisations looking to use AI, is how AI models, collect, use, and learn from data. Many big hitters like Apple, Verizon, and iHeartRadio have banned the use of models like ChatGPT over fears of data mishandling – Samsung, specifically, restricted the chatbot’s use after finding out that workers had input sensitive code to it.

The risk of unintended corporate trade secret disclosure represents a critical ethical frontier. AI systems might inadvertently reveal proprietary information and create substantial legal and competitive risks for organisations. For organisations wanting to incorporate AI elements, especially generative AI, sophisticated data governance strategies that protect both individual privacy and corporate intellectual assets are critical.

At a wider, societal level, generative AI models are trained on both input data and data scraped from the internet, meaning that they have the potential to, and often do, memorise all the data that is fed to them, regardless of sensitivity. When paired with things like biometric data, expressions, and other personally identifiable information (such as financial records and credit scores), individuals may find themselves unwittingly exposed, with intimate personal characteristics transformed into computational data points without meaningful consent. An individual's economic opportunities might be dramatically constrained by opaque computational assessments that reduce human complexity to numerical scores.

AI’s best safeguard: Human oversight

As mentioned, continuous monitoring and human oversight have become crucial factors in keeping AI in check. The key to more effective, risk-averse, and beneficial AI decision-making is to ensure that human-in-the-loop processes are in place. These frameworks maintain human judgment at critical decision points while leveraging AI's processing capabilities. Businesses stand to benefit immensely from what AI can offer, but mitigating risk requires the human touch.

Where AI is implemented, a cost/benefit ratio analysis is crucial. Full automation is more appropriate if a decision-making function has little to no wider effects. Organisations that intend to implement AI should establish a set of best practices for their employees, a set that balances AI's efficiency with a corporation's accountability. Oversight thresholds are a great example. This is where AI decisions exceeding certain risk levels automatically trigger human review. For parties affected by AI-made decisions, appeal mechanisms should be put in place. To ensure optimal functionality, regular system performance audits should be made.

A robust AI governance framework can also supplement human oversight for further safeguarding. At a governance level, a systematic approach that integrates structured accountability mechanisms, ethical design principles, continuous monitoring, and interdisciplinary oversight will help to mitigate risks from the get-go. For particularly risk-averse organisations, creating multi-tiered review processes, embedding ethical considerations directly into any and all AI-based architectures, and establishing adaptive governance models, companies can ensure that AI systems remain fundamentally subject to human judgment. The framework should mandate clear chains of responsibility and develop transparent audit trails for further assurances.
The path forward is one that warrants excitement but care. AI is an unbelievably powerful tool with near-limitless application potential. It is easy to get caught up in the excitement of this technology and run with it without doing the essential due diligence. Organisations implementing AI systems should invest in robust oversight mechanisms with clear guidelines for deployment to ensure that risks are mitigated. As said, AI is a great tool, but a tool nonetheless. A hammer is only as safe or dangerous as the wielder allows.

Additionally, laws and policies across the world have either been made or are in the process of being put in place, many to provide extra safeguards for the improper use of AI. As a society, the onus falls on all of us, from AI development to implementation, to ensure it serves the collective good before the gap between technological capability and ethical framework becomes too wide to bridge. As we tread forward to exciting possibilities, it is essential to tread with care.