Don't know how or why this discussion got off onto this particular set of rails, but hey: this is my wheelhouse, actually.
I've done a lot of thinking recently about AI and "consciousness." The new GPT-3 AI program is formidable; what it can do led to these reflections. I'll try to be brief.
For one thing, I don't think it's killer robots we have to worry about. Rather, it's that, little by little, AI is taking over the various cognitive tasks we used to perform for ourselves. Our cars decide when to turn the headlights on, when to lock the doors, when it's wet enough for windshield wipers...they even park themselves now, and drive themselves for a while. Our refrigerators decide what food we need to buy, and will even order it for us. GPS guides us to our destinations. But as a result, no one has a sense of their own physical geography anymore, no one knows where north is or how to read a map. The human brain is the most astonishingly capable problem solving technology ever developed by evolution; why would we willingly give up exercising it? The machines will take over not by force, but because we've gradually ceded control over our own lives to "labor saving devices" that do supposedly trivial chores better.
Anyway, GPT-3 is so damn good at, for example, writing creative texts that it has made me wonder if we're posing the problem in the wrong way when it comes to the "Turing Test" and the questions about artificial "intelligence." Maybe the question is not "Can machines think?" but rather: "Do we really do anything more than manipulate language when we 'think'?" Maybe a machine can be structured to manipulate a complex sign system, a language, in "creative" ways. But...I don't believe a machine can be structured to CARE about the result. The human experience, then, is perhaps not to be located in the creative act itself, but in the response to it—in the reader instead of the writer or, better, in the interaction between them. GPT-3 can even compose music "in the style of" whoever you like. But does GPT-3 enjoy music, listen to music?