Week 1 – Introduction to S-Risks
S-risks: an introduction
Power & Care
Power
Future Technology
potentially allowing us to shape the entire universe
Technology
Positive Examples
on average richer than ever before
Negative Examples
Industrial warfare (chemical & nuclear weapons, total war)
Factory Farming
Power (Technological Capacity) + Insufficient Care (Indifference) = unimaginable amounts of suffering
Factory farming is the result of economic incentives and technological feasibility
Not of human malice or bad intentions
Most humans don’t approve of animal suffering per se – getting tasty food incidentally happens to involve animal suffering.4
Scale
Impartiality - Suffering is independent of past, present, future
Reducing suffering in the future is no less meaningful than reducing suffering now.
Probability
Non-negligible
hard to predict the future
range of scenarios that we can imagine is limited
Underlying Assumptions
Astronomical Scale
no reason to expect that technological progress will soon come to a halt, barring globally destabilizing events
space colonisation will likely become feasible at some point
Easier to create unprecedented amounts of suffering, intentionally or not - via technology.12
Insufficient Care
Those in power will not care (enough) about the suffering of less powerful beings
Weak human benevolence might go a long way, but does not completely rule out s-risks.
Historical precedents do exist
Suffering Focused Ethics: Suffering is morally most urgent
Scale of Suffering: Future > Present
Number of planets to be colonized i.e., number of potential sentient beings
Artificial Sentience
Comparable to how large numbers of nonhuman animals were created because it was economically expedient, it is conceivable that large numbers of artificial minds will be created in the future.
Types of S-Risk
Incidental S-Risks: efficient solution to a problem happens to involve a lot of suffering
e.g., mindcrime for the idea that the thought processes of a superintelligent AI might contain and potentially harm sentient simulations.
e.g., suffering subroutines: computations may involve instrumentally useful algorithms sufficiently similar to the parts of our own brains that lead to pain.
Agential S-Risks: an agent actively wants to cause harm
e.g., warfare and terrorism with advanced technology could easily amount to an s-risk, or a malevolent dictator might cause harm on a large scale.
Neglect
far future that don’t pull on our emotional heartstrings
long-run outcomes often focus on achieving utopian outcomes
Indifference
humanity might fail to recognize artificial sentience, just as many philosophers and scientists failed to recognize animal sentience for thousands of years. We don’t yet have a reliable way to “detect” sentience, especially in systems that are very different from human brains.
Solvability
Attempting to shape the development of pivotal new technologies by implementing precautionary measures against s-risks.
AI Safety (surrogate goals to deflect threats)
negative-sum dynamics (extortion, escalating conflicts), game theory, decision theory
Aiming for broad improvements in societal values and institutions, which increases the probability that future generations will work to prevent (or at least not cause) s-risks.
Moral Circle Expansion towards Sentience (includng artificial sentience)
Promoting International Cooperation
Advancing Moral Reflection
Reduce excessive polarisation; Improve political system
Reducing malevolent actors gaining excessive power
Focusing on higher-level approaches such as raising awareness and researching how to best reduce s-risks.
As there is significant uncertainty
Cooperative Approach > figthing
e.g., stopping technological progress wouldl ead to zero-sum fights
Reducing Risks of Astronomical Suffering: A Neglected Priority
Existential Risk
Definition
an adverse outcome would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential
Examples
Extinction
Failed Potential
Suffering Risk
Definition
Examples
Cosmically Significant Suffering
Lots of Happiness + Lots of Suffering
Survial =/ Utopia
Extinction =/ Always immense suffering
Scale
Suffering Risk > Extinction Risks
Suffering risks should be addressed before addressing extinction risks
Reducing extinction Risks before s-risk increases likelihood of s-risks
Potential Futures
Utopia
Happiness and some suffering
S-Risks
Good values are insufficient for positive outcomes
e.g., factory farming
Neglectedness
Psychological Factors
Aversion to painful contemplation
Striving to avert future dystopias inevitably requires one to contemplate vast amounts of suffering on a regular basis, which is often demotivating and may result in depression
optimism bias
tendency to overestimate the likelihood of positive future events while underestimating the probability and severity of negative events – even in the absence of evidence to support such expectations
where people are prone to judging scenarios which are in line with their desires as being more probable than what is epistemically justified, while assigning lower credence to scenarios they dislike.
Self-Interest
working towards the reduction of extinction risks or the creation of a posthuman utopia is also favored by many people’s instinctual, self-oriented desires, notably one’s own survival and that of family members or other loved ones
Ignorance about s-risks
e.g., AI Misalignment
Insufficient concern for model uncertainty & unknown unknowns
Expecting a very large (e.g., 1,000,000 : 1) ratio of happiness to suffering seems unlikely with unknown unknowns, black swans, model uncertainty
Disjunction fallacy
tendency to underestimate the probability of 'at least one' event occurring out of many possibilities
Solvability
More Difficult
Creating vast amounts of happiness in the future without also causing vast amounts of suffering in the process requires a great deal of control over future outcomes
Easier
steering our future trajectory away from classes of bad or very bad outcomes
Example AI Safety
instead of betting on the unlikely case that we will get everything right, worst-case AI safety – approaches informed specifically by considerations of minimizing the probability of the worst failure modes
What are examples of s-risks? Some examples from fiction
Black Mirror - Digital Assitant
Consciousness can be uploaded into the digital realm
Time can be stretched for the digital mind, meaning a second in the real world might last for six months
Protagonist uses this power to control and manipulate a character to become his assistant, by stretching out time for her.
The Matrix
Humanity is unknowingly trapped in a simulated reality, which is controlled by intelligent machines.
Reality is that human beings are essentially farmed for their energy by the machines
Apparent moral disregard that could be compared to humanity’s disregard for factory farm animals.
Robots have attempted to create an artificial world to mask the human slaves from the reality of their suffering.
Incidental s-risk situations, because the suffering and exploitation is a byproduct of the aims of the machines - which is to generate energy.
Westworld
Creation of highly advanced AI entities that exist in a world where they are repeatedly subjected to violence and trauma for the entertainment of human guests.
The AI, though initially programmed to forget their suffering, begins to retain memories of the abuse, leading to an endless cycle of torment.
Possibility of development of sentient beings under human control, without moral consideration
Surface Detail - by Iain M. Banks (Novel)
Virtual hells are created, simulating intense suffering for individuals after their death.
These digital afterlives are designed to punish consciousness in ways that seem eternal and inescapable.
Religious motivations are implied as the plausible reason for this dark scenario.
To be summarized
Last updated