Climate change is the least of humanity’s worries. Yudkowsky mentioned that social manipulation would be a good target for a fledgling superhuman AI wishing to expand its sphere of influence. An AI with high information processing capacity but few channels through which to interact with the physical world would would probably be limited to trying to influence the researchers with whom it communicates. This would be an extremely important bottleneck through which the (presumptively unFriendly) AI djinn would need to pass before it is let out of its bottle; some thoughts on exploiting it:
- We must avoid the g-factor metaphor and remember that pre-empting a socially manipulative agent more intelligent than us is by definition futile: anything we predict it will try, it will know that we have predicted, and know how to thwart us. ‘WWAID?’ should never appear on a wristband.
- Hence, the best defense may be to put as many firewalls as possible around anybody who will be in communication with the AI, and of course limit the number of such people.
- The race is then this: the AI’s ability to convince a human(s) to do its bidding, and for the human(s) to perform the physical actions necessary to expand the AI’s power, pitted against the ability for humanity-at-large to notice what is going on and either incapacitate the human(s) or the AI. Yudkowsky outlines in the article a way in which an AI could go from communication only to controlling real-world nanotech machinery within a week, so whatever safeguard we put in place would have to be very responsive.
- We should be immediately suspicious of any solution which claims to contain the problem by forbidding humans who are in contact with the AI from following instructions given by the AI:
- Whomever is given the responsibility of policing this rule would not be immune from 2nd-order social influence; that is, influence from the humans who are in direct communication with the AI.
- We cannot pretend to be able to know when an AI is influencing a human (see top bullet point).
- Perhaps prevention is better than cure: how can we stop an AI finding out about human psychology in the first place?
