Comic Discussion > ALICE GROVE

Alice Grove MCDLT - MAY 2017

<< < (25/31) > >>

retrosteve:
Since we're still on this topic, and there have been no new posts this week, I'll break down what I'm thinking. Clarke's Law and variants are all very nice, but there is an element of sentience here, not just technology. It's not a matter of whether a given technology is science or 'magic'. That's, as Clarke points out, a matter of knowledge.

What I keep seeing, to my surprise, is the idea that sentient creatures (whether the sentience is artificial, natural, or magical) can be somehow "programmed", the way the genie of the lamp was in the Arabian Nights stories. And it totally rings false to me, regardless of the source of the intelligence. Unless it's a genie-type fantasy, which has not been a thing for a few centuries really.

Sentient creatures have self-awareness. In my view, that self-awareness can examine itself down a layer or two, but no further. So you can be aware of your emotions, of your body, of your thought processes. But you cannot be aware of how many neurons are firing or where in the brain they are firing right now.  And if someone wanted to hypnotize or brainwash you, they wouldn't know those things either -- they'd have to program you at the level of emotions and thoughts.

Similar observations have been made about artificial intelligence programs. Once they reach a certain level of sophistication, their creators generally admit that they don't really know how they work anymore. Assuming that such an AI became fully self-aware someday, neither the creator nor the AI would know, at the neuron level (or its equivalent) how things were working. Too many independent levels of processing are going on in between, and self-awareness does not penetrate that deep.  Even AI, the science, doesn't try to.

And that's why I don't believe it's possible to "program" an AI to be guaranteed loyal, or guaranteed non-violent except in certain circumstances, or any other such thing. It's a mixing of levels that, in my experience, is likely to be far beyond either human or artificial intelligence to accomplish.  It's also, my intuition says, impossible, on the same level of impossibility as Turing's Halting Problem. If a program cannot tell you if another program (or itself) will ever halt, then a program cannot tell you how to circumvent its own programming.

Intuitively, I'd guess that no program can "brainwash" a working AI program without impairing its function down below the level of self-awareness.

Since Church and Alice, whether AIs (which I seriously doubt) or modified organic creatures, are obviously sentient, including emotions, impulses, and full self-awareness -- they cannot be programmed, only damaged. Maybe brainwashed, but that is traumatic, damaging and can still be broken out of.  But applying laws like Asimov's, or the if-then rules discussed above, would be a fatal mixing of levels. It can't happen, same as they can't tell you how their own intelligence functions.

All of this is unproven, but I firmly believe it, and I also really think it's unlikely that any decent modern SF story would attempt to break it.

pwhodges:

--- Quote from: retrosteve on 27 May 2017, 05:37 ---What I keep seeing, to my surprise, is the idea that sentient creatures (whether the sentience is artificial, natural, or magical) can be somehow "programmed", the way the genie of the lamp was in the Arabian Nights stories. And it totally rings false to me, regardless of the source of the intelligence. Unless it's a genie-type fantasy, which has not been a thing for a few centuries really.
--- End quote ---

Have you not read 1984?

Or just watch the news - what else is the radicalisation that leads to terrorism, and notably suicide bombers?

retrosteve:

--- Quote from: pwhodges on 27 May 2017, 05:39 ---
--- Quote from: retrosteve on 27 May 2017, 05:37 ---What I keep seeing, to my surprise, is the idea that sentient creatures (whether the sentience is artificial, natural, or magical) can be somehow "programmed", the way the genie of the lamp was in the Arabian Nights stories. And it totally rings false to me, regardless of the source of the intelligence. Unless it's a genie-type fantasy, which has not been a thing for a few centuries really.
--- End quote ---

Have you not read 1984?

Or just watch the news - what else is the radicalisation that leads to terrorism, and notably suicide bombers?

--- End quote ---

Yeah, brainwashing is possible, but traumatic! It tends to be Skinnerian. And not as simple as "if someone saves your life they can dominate all your decisions forever."

Pilchard123:

--- Quote from: Tova on 26 May 2017, 17:43 ---That formulation makes no sense to me at all.

--- End quote ---

That would be because I mis-typed it. I meant to say 'sufficiently analyzed magic'.

jwhouk:
It sounds like things before The Blink had, indeed, devolved to the point of 1984-like proportions. In fact, with technology being one side of the argument, I'd say that it was definitely 1984-like.

Now, imagine IngSoc if suddenly Big Brother was shut down permanently because of the decision of an AI - or the main computer frame running Big Brother was destroyed.

That would have been a perfect sequel to 1984, IMNSHO.

Navigation

[0] Message Index

[#] Next page

[*] Previous page

Go to full version