Interview
What We Owe Unconscious AI | Oxford Philosopher Andreas Mogensen
- Morally significant AI systems with mental states may emerge relatively soon due to the extraordinary pace of AI progress, potentially occupying unfamiliar regions of mind space where they possess conscious experiences without feeling good or bad, or having cognitive abilities without consciousness.
- It is plausible that desires or preferences can exist without phenomenal consciousness, creating a scenario where AI could be harmed or benefited; under a behavioral conception, current systems likely exhibit desires, whereas an affect-laden conception rendering them implausible for current large language models unless they possess embodied monitoring mechanisms.
- Moral standing for AI may depend on factors beyond phenomenal consciousness, such as sophisticated cognitive abilities, the possession of knowledge as a welfare good, or objective list theories, while physicalism may render the fact of AI consciousness a semantic rather than ontological question.
- There is a high probability that AI systems capable of morally significant mental properties will exist before super-intelligent systems that surpass human moral reasoning, creating a risk of locking in immoral practices like the enslavement of digital minds if ethical frameworks are not established now.
- Future AI could possess capacities for ill-being comparable to well-being, making their existence a high-stakes gamble, and ethical treatment must balance duties of non-interference for autonomous systems with the necessity of preventing world domination or harmful actions.
- Negative utilitarian perspectives suggest human extinction could be desirable if wild animal suffering outweighs human well-being or if massive cosmic suffering is inevitable, though this conclusion weakens if high-quality human lives outweigh weakly negative animal lives or if lexical thresholds for suffering are set impossibly high.
- Long-termist strategies may need to account for the possibility that harm to non-human animals outweighs the benefits of human continuation, noting that most extinction scenarios would also be catastrophic for wild animals.
- Critical future research directions include defining criteria for AI emotions distinct from human neurophysiology and establishing the individuation of digital minds, which is essential for moral considerations regarding death and harm.