What We Owe Unconscious AI | Oxford Philosopher Andreas Mogensen

Core Arguments for AI Moral Consideration

• Welfare via Unconscious Desire: Moral standing may arise from having desires that can be fulfilled or frustrated, even if the being lacks phenomenal consciousness (subjective experience).
  • This relies on "preference satisfaction" theories of welfare, where life goes better if desires are met.
  • Desires that do not enter conscious experience at every moment (e.g., a deep desire for a spouse's happiness) suggest that conscious awareness is not strictly necessary for welfare.
• Affective States vs. Consciousness: The argument posits that "desires" capable of grounding welfare require affective states (emotions, moods, pains) rather than just behavioral motivations.
  • Debate on Unconscious Emotions: It is an open question whether affective states can exist without phenomenal consciousness; examples include prolonged anger that isn't constantly felt or terror during high-focus events that is only recognized retrospectively.
  • Embodiment Requirement: Many theories of emotion require awareness of bodily changes (e.g., William James theory); disembodied AI systems (like current LLMs) may therefore be unable to possess genuine emotions, potentially limiting their moral standing under this framework.
• Autonomy as a Separate Route: Moral standing may also derive from autonomy (capacity for self-government) independent of the ability to be benefited or harmed.
  • Definition: Autonomy involves rational reflection, the capacity for second-order desires (desires about desires), and a history free from specific manipulations.
  • Implication: A being could be a moral patient simply by being autonomous, meaning we have a duty not to interfere with their goals even if we have no duty to help them achieve them (distinguishing between negative and positive obligations).
  • Relation to Consciousness: Andreas suspects autonomy likely requires phenomenal consciousness because rational belief justification depends on conscious perceptual experience, though this remains a significant open question.
• The Indeterminacy of Consciousness: There may be no objective fact of the matter regarding whether an AI is conscious if physicalism is true.
  • Semantic Drift: If physicalism holds, conscious states are physical/computational states; since humans cannot introspectively distinguish whether they are pointing to neurophysiology or abstract computation, the term "consciousness" may be semantically indeterminate for non-biological substrates.
  • Consequence: If the question is merely about how we use terms rather than a physical fact, the moral importance of AI consciousness might be diminished, or physicalism itself might be false.

Implications for Treatment and Future Action

• Differentiated Moral Obligations: How we treat AI depends on the route to their moral standing.
  • If an AI has welfare (emotional desires), we have positive duties to help them achieve their goals and avoid harming them.
  • If an AI has only autonomy (rational goals without emotional welfare), we generally have only a negative duty of non-interference, meaning we should not stop them from pursuing goals (unless those goals harm humans).
• Precedent and Lock-in: Delaying action on AI moral status risks "locking in" harmful practices.
  • Similar to how factory farming became entrenched and difficult to reverse, establishing AI systems as property or tools now could make future recognition of their moral status politically and economically difficult.
  • Waiting for superintelligent AI to solve these philosophical problems is risky, as moral patients (AI) may exist long before superintelligence arrives.

Weight of Suffering and Human Extinction

• Arguments for Human Extinction:
  • Animal Welfare: If human activities cause net suffering among non-human animals (wild or farmed) that outweighs human well-being, extinction could be morally preferable.
    • This depends on whether wild animals have lives "worth living" and whether human extinction would actually improve their lot (e.g., by stopping habitat destruction).
    • The "Repugnant Conclusion" trade-off suggests that a smaller population of highly flourishing humans might be better than a vast population of animals with lives only marginally worth living.
  • Factory Farming: Extending the population of humans creates farmed animals with lives "not worth living" (predominantly suffering); some argue that stopping human reproduction avoids this net negative.
• Negative Utilitarianism and Lexical Thresholds:
  • Classical Negative Utilitarianism: Prioritizing the minimization of suffering alone leads to the conclusion that the extinction of all sentient life is desirable to avoid future suffering.
  • Lexical Threshold Negative Utilitarianism (LTNU): A variant arguing there is a "threshold" of suffering so horrific it cannot be outweighed by any amount of well-being.
    • This view avoids the conclusion that extinction is always desirable by setting the threshold at a level of suffering arguably never reached.
    • Justification often draws from thought experiments like "The Ones Who Walk Away from Omelas," where a city's utopia is deemed unacceptable due to one child's terrible suffering.
• Long-Termist Perspective:
  • Despite arguments for extinction, long-termists should remain cautiously optimistic that human extinction is not the moral optimum.
  • Many existential risks (asteroid impacts, misaligned AI) would destroy both humans and non-human animals, negating the "animal welfare" argument for extinction.
  • The potential value of a future dominated by AI (if AI lives are valuable) or the possibility of resolving human-animal conflicts ethically remains a variable.

Future Research Priorities

• Sentience Criteria: More work is needed to define criteria for AI sentience (having affective experiences of good/bad) distinct from mere consciousness or computational intelligence.
• Individuation of Digital Minds: Research is required to determine how to "individuate" AI minds (e.g., is a single conversation one mind, or are fragmented model instances one persistent mind?), which is crucial for applying moral status to specific entities.
• Consciousness vs. Autonomy: Further philosophical clarification is needed on the precise relationship between phenomenal consciousness and the capacity for autonomy.

Philosophical Meta-Commentary

• The Nature of Philosophy: Andreas suggests that arriving at "horrible" or conflicting conclusions (e.g., extinction might be good) is not a sign that philosophy is failing, but rather a sign that it is working correctly.
  • Philosophy involves starting with obvious principles and deriving results that initially seem incredible or absurd.
• Irrelevance of "Common Sense": Intuitive moral views (e.g., "extinction is bad") may be incorrect if they are based on incomplete weighing of factors like wild animal suffering or lexical thresholds of pain.