The Actual Risks of AI

Moving beyond the existential doom scenarios, what are some ways in which AI adoption could actually go massively wrong?

The most prominent news in the AI world recently has been the removal of Sam Altman and his subsequent re-appointment as CEO at OpenAI. A non-profit branch of OpenAI, overseen by a board that included Altman, was established to ensure that the organization pursued the mission of creating a safe AGI for humanity. Meanwhile, the commercial branch of OpenAI seemed to be advancing rapidly without adequate attention to safety, leading the board to take action to redirect its trajectory.

Ironically, it seems the mechanism put in place to safeguard OpenAI from excessive commercialization and mission drift may have unintentionally accelerated its move toward more commercialization following the ousting of Altman. This decision faced strong resistance from business partners such as Microsoft, and almost 90% of OpenAI's employees reportedly contemplated leaving in response.

The outcome of this upheaval appears to have steered OpenAI toward even greater commercialization, raising questions about whether this trajectory was accidental or a deliberate strategic move by those within OpenAI focused on commercial interests. Regardless of how the story concludes, the prevailing sentiment in the AI research community right now is that safe AI development cannot happen with purely commercial incentives. Many argue that capitalistic incentives will inevitably prevail, leading to the development of commercial AI systems regardless of safety concerns.

I am troubled by this brand of techno-pessimism for several reasons. First and foremost, I am generally bothered by pessimism, as I firmly believe that adopting a pessimistic outlook is the easier choice. Pessimism is often regarded as a sign of maturity, while optimism can be perceived as immature and naive. Therefore, it is safer to align oneself with pessimism.

Conversely, if one is an optimist and fails to deliver on promises, they face significant backlash. However, a pessimist who warns of negative outcomes but then things improve can justify their stance by claiming that their pessimism contributed to the positive outcome! In essence, playing the role of a pessimist is a safety net, but it's also a display of cowardice. I believe it takes considerable courage to be an optimist and truly believe in the potential of technology to enhance human welfare, because genuine optimism means you cannot stop fighting the good fight.

However, it's important to note that I do not identify with other types of optimists, such as the techno-optimism commonly associated with the Silicon Valley techbros. The unjustified optimism these individuals espouse is just as dangerous as the most cowardly pessimism. This kind of optimism is often disingenuous and hypocritical, coming from individuals who know the system works in their favor and are the ultimate beneficiaries of technological wealth.

(I actually prefer to label myself a techno-pragmatist, a concept I can delve into further in a different essay. But enough of a tangent already.)

So, in this edition of the Mostly Harmless newsletter, I aim to revisit the topic of AI safety. I previously talked extensively about existential risks and strongly encourage you to review that content if you haven't already. In doing so, I aimed to dispel the notion that existential risks should be our attention's sole or even main focus. While I acknowledge the presence of long-term existential risks from AI that cannot be ignored, I believe that the short-term, non-existential, yet impactful risks stemming from the mishandling and careless development of AI today deserve equal, if not greater, attention.

---

---

AI safety encompasses several important topics. To categorize the various ways in which technology can harm society, we can examine several different categories. One such category is existential risk, which revolves around the concern that AI technology may become so powerful that it threatens civilization's continued existence. This could occur through intentional actions, where the technology develops the motivation and ability to destroy us, or inadvertently, such as attempting to address climate change and unintentionally causing harm to humanity.

But setting aside existential risk for the moment, there are numerous other ways in which technology, including AI, can negatively impact society, either accidentally or by the purposeful action of malicious individuals. This includes military use, job displacement, but also disinformation, polarization, and the exacerbation of societal biases.

Let’s review what I consider the most relevant types of AI danger. I will sort them in increasing order of immediateness, so I will start with the more far-fetched—but still realistic— scenarios and move over to more present concerns. Unfortunately, as we will see, there is little —although not zero— we can do today to mitigate most of these risks. That’s why discussing them transparently is so crucial.

Autonomous weaponry

Autonomous weaponry can seem like a long-term and slightly abstract concern related to existential risks. But while debating the possibility of AI leading to Skynet scenarios may be futile, it's essential not to overlook the potential for catastrophic military uses of AI in more realistic and short-to-medium-term scenarios.

The first issue arises from fully autonomous weapons like drones or even more traditional ones like assault rifles and self-guided missiles, which can be entirely automated using AI technology. Such fully automated weaponry is horrifying, of course, if only because all forms of technology employed to kill human beings are deplorable.

But even in conventional warfare, where humans kill other humans, there still exists some room for empathy and consideration despite armed conflict being a barbaric practice in this day and age. At least in traditional warfare, an individual remains behind the trigger of a weapon, which provides some space for respecting human rights and refraining from harming civilians or innocent people.

In a fully autonomous weaponry case, warfare can be completely dehumanized. if autonomous drones are deployed on battlefields without human operators, it could lead to the total dissolution of the distinction between innocent and active military enemies. Semi-autonomous weapons like drones operated by humans have already shown how being behind a screen resembling a video game can cause dehumanization, leading to an increase in civilian casualties. Imagine the consequences when such weapons become fully automated.

This extreme use case is closer to Terminator-like scenarios, but an easier and equally terrifying option is the use of AI for biochemical warfare.

Using artificial intelligence, we can now solve biochemical problems that were considered unsolvable five years ago through tools like AlphaFold. This new capability allows us to design proteins and chemical substances at a scale previously unimaginable. This is good, of course, as we have the potential to create powerful new medicines and treatments for some of our worst diseases.

However, this also opens the door for creating highly efficient biochemical weapons capable of targeting specific genetic markers in certain populations based on their, say, race or location. The potential for such weapons to be used for racial killing is terrifying. Further advancements may enable us to create powerful viruses that could wipe out humanity if released. It's important to note that this does not involve an AI gaining the motivation to do so and deciding to eliminate humankind. It’s a purely humans-kill-humans scenario.

The development of viruses capable of eliminating entire populations can be likened to the proliferation of nuclear weaponry among various superpowers, which has so far avoided mutual annihilation due to the high costs and challenges associated with producing such weapons. However, unlike nuclear arms, designing lethal viruses requires only a small nation or a terrorist organization to execute it successfully, as it has much lower entry costs than nuclear weaponry.

In the near future, it may be possible to download and print custom-designed proteins at home using a chemical printer. This technology could potentially allow small terrorist organizations to design deadly viruses and release them in crowded places like subways or stadiums. Unlike nuclear weapons that only superpowers can possess, bioweapons like these pose a greater threat because they don't require extensive resources or expertise.

This scenario is alarming, and there are currently no technical solutions to prevent it from happening, as anyone with a computer may have access to these capabilities. The only plausible course of action seems to be banning certain things through international treaties upheld by countries and governments. But terrorist organizations and rogue countries won't abide by agreements against producing chemical weapons or nuclear arms, making it an intractable dilemma for which I can see no short-term solution.

Massive workplace displacement

One of the ways in which technology can negatively impact society is by causing significant economic upheaval that is challenging for a large portion of society to adapt to. This issue is particularly evident in the widespread job displacement resulting from increased automation across various industries.

In the next decade, artificial intelligence is expected to reach a level where it can potentially surpass or match human capabilities in many economically viable occupations, including agriculture, manufacturing, white-collar jobs, education, science, research, and entertainment. If this scenario unfolds as projected, it is natural to experience concern about the fate of the millions of individuals whose jobs will be replaced by automation.

This widespread job disruption can potentially have catastrophic effects on individuals and the broader society. It raises critical questions about how these individuals will transition to new employment opportunities and maintain their livelihoods in the face of technological advancements.

Massive job disruption, a recurring phenomenon throughout history, has been associated with every industrial revolution. Examples of this abound. When we introduced sewing machines, there was an expectation that they would improve the working conditions for women employed in sewing. However, in reality, this did not materialize. These women simply lost their jobs to the machines, as a whole new industry displaced an existing and massive workforce.

One of the arguments often advocated by tech enthusiasts is that the advancement of technology will inevitably result in the destruction of numerous jobs, but it will also generate entirely new job opportunities, ultimately leading to a more prosperous society. They often cite examples such as the emergence of new professions like YouTubers and internet influencers or the rapidly evolving role of AI engineers in recent years. But while technology undoubtedly creates new jobs and value, it has historically also led to an imbalance in the distribution of benefits and drawbacks.

So, it is crucial to approach this issue cautiously, as it has frequently resulted in disparities between the positive and negative impacts. Technology has the potential to bring about societal profitability for the average citizen while simultaneously causing hardship for a significant proportion of the lower-income population.

For those of us concerned with social justice and equal opportunities for all individuals to earn a livelihood, the potential consequences of significant job disruption demand attention and thoughtful consideration. Despite the potential for creating new jobs and overall improvement in the long and midterm, there is no guarantee that those displaced from their jobs will possess the necessary skills or opportunities to transition to new roles. Consequently, many individuals may struggle to leave their positions and lack the marketable skills to secure alternative employment.

One suggested solution to this problem is Universal Basic Income (UBI), which proposes that the rise of digital intelligence or the fourth industrial revolution will generate sufficient value to provide a basic income for all, eliminating the necessity to work for survival.

However, many, myself included, are skeptical of the feasibility of UBI in a purely capitalistic society due to the way market incentives work. You can argue that certain well-developed countries already produce enough value to guarantee a minimum income for everyone. Yet, income inequality has persisted or worsened in many of these nations over the last few decades.

So, while Universal Basic Income is seen as a progressive and promising concept for an increasingly industrialized and automated society, I don’t think it is obviously the natural progression of our current social and economic structures. Implementing UBI would require significant social restructuring and government intervention, which many may not embrace, often due to valid concerns. We will explore this topic next.

Informational hazards

Artificial intelligence has the potential to revolutionize how we create and consume information. In a way, that is already happening with generative AI, and even before, with algorithmically curated feeds and recommendations. But the sheer scale of what may be coming makes anything currently available pale in comparison.

The first informational risk associated with AI is disinformation or fake news, created through generative AI's capacity to produce content almost indistinguishable from reality. This technology can be used for malicious purposes, such as spreading false information and convincing people of scandals related to political candidates through manipulated evidence like pictures, videos, audio recordings, and transcriptions.

One malicious use of this tech is making people think someone did something they didn't. But another way it can be used is just to make people doubt whether anything is true at all. This erodes trust in institutions, including news, government, science, and other organizations. If people stop believing everything because anyone could create convincing fake content with AI, democracy will fail, as it requires informed citizens who can tell the truth from falsehood. Disinformation can lead to chaos.

Disinformation can be intentionally spread by malevolent organizations and individuals, but there's also an unintentional effect called polarization. As AI recommends content based on your preferences, it creates separate realities for everyone because people keep consuming content aligned with their beliefs. We've witnessed such bubbles frequently. For instance, if someone watches only a few flat Earth videos on YouTube, it may lead YouTube to assume them a believer in flat Earth theories and present them with ever more conspiracy theories.

However, this isn't necessarily caused by any ill-intentioned entity; it's merely how recommendations operate. Recommendations for movies and other entertainment content work well because taste in films, music, or art generally is subjective. People enjoy different genres, and it's okay to suggest similar movies based on what they’ve liked before.

However, recommending news sources and experts should be done based on objective measures of quality or truthfulness rather than personal preference. Using the same algorithms for recommending news and entertainment doesn’t work since these domains are inherently incompatible due to their differing value propositions.

To solve this problem, we need more intelligent algorithms together with massive human curation. This can be achieved by processing recommendations differently on entertainment versus educational channels, emphasizing factual and accurate content in the latter. However, the complexity of this issue also raises questions about who is responsible for the curation process itself.

Moving on, surveillance and censorship pose another major societal challenge arising from two interconnected trends. Firstly, as you conduct virtually all activities online —including communication, creation, and consumption— your digital footprint leaves an indelible trail. Secondly, advanced technologies enable predicting future actions based on past behaviour, such as identifying individuals' contacts, preferences or even their thoughts via their online activity.

While predominantly employed for targeted ads today, this capability may also facilitate dictatorial regimes and institutions placing an inescapable surveillance net on individuals. In fact, the marketing sector now comprises the planet's largest and most pervasive surveillance network. Every sound, action, or message posted or viewed online is captured, archived and distributed among thousands of advertisers worldwide, each analysing user histories for insights regarding identity and inclination, in service of pitching products accordingly.

This machinery can be used for surveillance, such as identifying dissenters, and censorship. It allows direct censorship through platform filters or indirectly by employers and governments implicitly threatening you with dire consequences. This could lead to a dystopian society where everything done online is tracked, analyzed, scored, and punished with denied access to services, jobs, education, or even fines and imprisonment if your “score” falls below a certain level.

In the future, a technologically advanced version of George Orwell's 1984 dystopia becomes possible. With enough data and computing power, law enforcement can implement a "thought police" in its most sophisticated form without screens or microphones. No longer do they need to monitor what people say or type through screens or microphones; instead, they have access to personal devices like smartphones that capture everything individuals do, say, or write.

This goes beyond just recorded statements as it enables authorities to predict thoughts based on behavior patterns. The worst kind of dystopia thus arises, where nothing remains private anymore since all actions are public or available for analysis by government agencies. As you think, your ideas manifest in your conduct online, making it impossible not to express them. Consumption habits such as movies watched and length spent reading online reveal insights into your internal musings.

Think of this in the most creatively insidious way you can. For example, while you’re browsing a legal website, if I'm the surveyor, I can add a hidden feature that shows two images quickly for only your subconscious to see. Then, by tracking how long you look at each image, I may have enough information to predict any unusual thoughts you might have without you even realizing you’ve been scanned.

There is no escape from this reality. every government or institution that can censor will censor. The only solution is to have strong democratic principles that help citizens fight back for their right to privacy.

Exacerbating and perpetuating harmful biases

Technology in general, and AI in particular, poses significant risks when it stands to automate away the human from tasks that involve subjective judgment. One notable example is in the realm of crime, where we’ve already seen dystopic crime rating systems attempting to predict the likelihood of re-offense and the probability of bail or committing a crime again.

Another area heavily reliant on human judgment is job applications, where efforts have been made —though largely unsuccessful— to replace human recruiters with AI for hiring a range of positions, from white-collar roles to other job categories.

Similarly, credit rating, a crucial element of the developed world's financial system, has seen attempts to automate the system and related financial services, using AI to determine who qualifies for a loan or certain credit thresholds based on a complex web of historical data and predictive models.

In all of these scenarios, the central issue revolves around bias, presenting a formidable challenge in the implementation of automated systems. These models have been over and over shown to display insidious gender and racial biases.

These systems are trained using past human judgments and are inevitably influenced by the biases with which humans judge one another. Consequently, our financial, judicial, criminal, and job market records are rife with discrimination against minority groups, including racial and gender discrimination, discrimination against neurodivergent individuals, and bias against people with non-traditional backgrounds or education.

The prevalence of biases in the data heavily influences the output of the systems trained on it. Most predictive systems today are trained using a large amount of supervised or self-supervised learning from historical data, which means that they inevitably perpetuate and encode these biases. Unfortunately, we currently lack a clear understanding of how to design a system that both performs well and effectively removes biases.

There is extensive research being conducted in the area of AI fairness. Many AI labs, including my own, are actively engaged in this field. Trade-offs are being considered by various researchers in order to achieve fairness within AI systems.

When aiming for fairness, one approach involves regularizing the model to ensure that it produces consistent or similar performance across different subsets of inputs. For example, equalizing the probability of making a mistake among different subgroups. Various mathematical frameworks exist to define what constitutes a fair outcome, often involving the partitioning of the population into subgroups to ensure an equitable distribution of outcomes. Crucially, the most common definitions of fairness metrics are incompatible with each other, so there’s always a trade-off.

Considerations for fairness extend to the idea of protected attributes, such as race, gender, and educational background. The objective here is to develop systems that are independent of these variables. However, simply omitting gender or race from the input data is not sufficient, as there are numerous proxy variables that are correlated with these attributes.

However, removing variables correlated with gender and race may inadvertently eliminate crucial information, as these variables are often associated with important factors. For instance, factors such as educational background, childhood experiences, and personal preferences may influence performance and fairness at the same time. One cannot simply ignore these factors because then there’s no prediction at all to be made. This underscores the complexity involved in addressing fairness in AI systems.

This problem presents a tremendous challenge. So far, all the solutions that I am aware of involve some trade-off of performance in order to achieve fairness. This trade-off seems to be non-negotiable, as part of the performance given away is actually due to the discrimination itself. This exacerbates existing biases and discrimination.

Why will this be worse with AI than it really is? Society is already unfair. AI may improve some aspects but not others. There is a concern that AI not only captures discriminating biases but exacerbates them, making them more extreme and profound. Mathematically, it has been shown that if left unchecked, a predictive model will tend to exploit these biases to their maximum potential if the sole focus is on performance, and this effect has been empirically demonstrated in numerous papers.

The problem lies not only the technical aspect, but in the widespread adoption of theses systems as well. Making fairness a priority, and even more, a legal requirement is imperative because a system that is fairer may not perform as well as one that isn't, all other factors being equal. For example, in the context of developing systems for hiring applicants, a fairer system may yield lower performance when compared with historical data, because by definition, historical data is biased. In a purely market-driven economy, there are no inherent incentives to prioritize fairness, hence the need to inject such incentives externally, possibly through government regulation.

---

Conclusions

From a pragmatic perspective, it is essential to address the problems in artificial intelligence that warrant our focus. In particular, I am concerned about the for-profit military-industrial complex's inability and lack of incentives to solve these issues. I firmly believe that government regulation and societal oversight are necessary to implement safeguards that enable us to harness the potential of artificial intelligence for the greater good, rather than for the profit of a select few, or as a tool for the benefit of the technocratic elites.

I strongly advocate for a balanced approach to AI regulation that does not stifle innovation and development but rather complements these endeavors with safeguards to ensure responsible and ethical AI advancements. It is pivotal that we prioritize the societal implications and ethical considerations of AI, and take proactive measures to steer its development in a direction that benefits humanity as a whole.

I believe we need not fear a sensible government regulation in the case of AI adoption, by regulating the commercial applications of AI rather than the basic research. This type of regulation is in place for many consumer products, from food to electronics to pharmaceuticals. In general, new products —whether GMOs, cars, or drugs— cannot be introduced into the market without demonstrated safety and efficacy. Similarly, AI systems should not be allowed to operate commercially if they demonstratively harm some individuals.

This post is my attempt to shed light into the many ways in which AI can be misused, either accidentally or on purpose, to harm some individuals or populations. The looming question remaining is of course, what can we do, technically and otherwise, to solve these issues? If you’re interested, I can dive into the active research in mitigating AI harms in a future issue.