Apple released its plans to automatically scan phones for child abuse material which on the face of it is good policing – and the response has been loud and angry and calls out the dangerous slippery slope of surveillance. But I think what is also being revealed is a particularly 21st century phenomenon, and that is mass social paranoia.
The slippy slope argument is not hard to see. The plan is for Apple to continuously scan photos sent in messages and stored in iCloud, testing them against a known database of child abuse images, and escalating matching photos to human review.
But now the mechanism is in place, what else could it be used for? Could the Chinese government coerce Apple to locate dissidents, by adding certain non-child-abuse images to the central database? I mean, what are Apple going to do – not sell phones in China? Or can the GDPR “right to be forgotten” be wielded to force erasure of (say) unwisely shared nudes? Hard to argue with that, and Google hides links from search results under GDPR so maybe not such a stretch. But then why not automate removal of embarrassing photos of celebrities with expensive lawyers?
The EFF response is far more articulate on the details and mission creep
potential of the new system: Apple’s Plan to “Think Different” About Encryption Opens a Backdoor to Your Private Life.
BUT:
I have a friend who has worked in positions where she can see the global traffic of child exploitation material, and I’ve spoken with her just a little bit about this in the past. It is horrific and huge. We need good policing, that’s my view, and mechanisms to achieve that, and we can debate how that happens. (We’re a long way from any answers, but my thoughts on a good approach are a whole other topic.)
So there’s a line for society to walk.
And it really doesn’t help that we have to trust a corporation to walk this line, without democratic accountability.
Yet this isn’t just a privacy debate.
It feels different because the photos are being scanned on-device. The surveillance is on our phones. And that triggers a whole other kind of response.
Cory Doctorow, way back in 2002: My Blog, My Outboard Brain (O’Reilly): Being deprived of my blog right now would be akin to suffering extensive brain-damage. Huge swaths of acquired knowledge would simply vanish.
Clive Thompson, in 2007: Your Outboard Brain Knows All (Wired): This summer, neuroscientist Ian Robertson polled 3,000 people and found that the younger ones were less able than their elders to recall standard personal info.
This feels obvious now, but it was new then:
And when he asked them their own phone number, fully one-third of the youngsters drew a blank. They had to whip out their handsets to look it up.
So smartphones become, somehow, part of the mind.
Cognitive scientist Andy Clark makes the point that the mind doesn’t stop at the skull. He lays out the extended mind hypothesis in his astoundingly prescient book Natural-Born Cyborgs (2003). Highly recommended.
He makes the argument that we don’t just use pen and paper to work out a sum, but the tool becomes part of our thinking. Information on the web isn’t just consulted on our phones, but is in a real way part of our memory.
Humans are special precisely because our brains have this ability to side-load the world into self:
In embracing our hybrid natures, we give up the idea of the mind and the self as a kind of wafer-thin inner essence, dramatically distinct from all its physical trappings. In place of this elusive essence, the human person emerges as a shifting matrix of biological and nonbiological parts. The self, the mind, and the person are no more to be extracted from that complex matrix than the smile from the Cheshire Cat.
Phones are part of us.
Scanning the photos on your phone isn’t like steaming open the mail and peeping inside the envelopes. It’s like rifling through your memory.
And when those memories may at any time be silently observed or removed… even if it never happens but there is the possibility of it…
Well.
Every culture, big and small, has a feeling that it swims in but is often slow to put its finger on, like the proverbial fish in the ocean unable to see the water. That’s my take.
I think in the 70s and 80s that feeling was the end of the world. I was pretty sure, as a little kid, that by the time I was my age, now, I would be living in a post-apocalyptic nuclear wasteland. It wasn’t a conviction, it was more like an unspoken understanding. And goodness knows what that did to us.
Ironically the end of the world is coming, in the shape of the climate crisis, and I wonder how those of us who grew up taking the Cold War for granted are coloured by that experience and how it is tainting our response. We probably feel like the climate crisis, or at least some kind of apocalypse is inevitable somehow? Or alternatively, that if we wait around for long enough then the threat will just somehow… recede? Like the way the peril lifted in the 90s. Dangerous templating for us to have; thank god for the zoomers.
What’s in the air now?
We swim in paranoia, I think.
We’re always potentially being watched.
RELATED: I ran across Zizek riffing on Donald Rumsfeld (YouTube) and specifically developing the concept of unknown knowns. Here’s Ted Hunt on Twitter with a quote/summary:
”.. the main dangers lie in the unknown knowns–the disavowed beliefs, suppositions and obscene practices we pretend not to know about, even though they form the background of our public values.” – Slavoj Zizek
So paranoia is like our culture’s current unknown known. That’s where it sits, somewhere in the social unconscious.
James Bridle’s 2014 work The Nor was an investigation into paranoia, electromagnetism, and infrastructure.
It’s a sequence of essays telling the story of a participatory, documentary act: Bridle’s walk across London, photographing every CCTV camera he passed. SPOILER: It doesn’t end well.
Here are the essays:
The sense of being watched is a classic symptom of paranoia, often a sign of deeper psychosis, or dismissed as illusory. In the mirror city, which exists at the juncture of the street and CCTV, of bodily space and the electromagnetic spectrum, one is always being watched. So who’s paranoid now?
And it was this work that really opened my eyes to the pervasive sensation of surveillance. (Which is why art is vital, right?) Especially because Bridle makes explicit the role of the network and what that does: the first essay is titled All Cameras are Police Cameras.
The camera network today is Instagram, TikTok, other people’s phones. It’s the pictures taken at parties, previously private spaces, and it’s the acquisition of the breakthrough facial recognition startup Face.com by Facebook in 2012, and everything that opened a door to across the industry.
A lot has been said about the Panopticon, Jeremy Bentham’s 1786 concept of a prison where the prisoners are controlled by the mere possibility of being observed, and of sousveillance: surveillance from the same level; we watch one-another. That’s what a networked camera in every pocket leads to.
The debate, over the last 20 years as this has been happening, has been framed around the loss of privacy and whether that matters: the younger generation has different privacy expectations to us, that’s one statement; the absolutist privacy ideals of the EFF are another part of the debate.
(And the responses to this shift are fascinating. For me, the go-to here is danah boyd’s work, and I’ve recently been diving into her work on networked privacy from the early 2010s, and the sophisticated ways that teens are finding control and agency in this world.)
But how does it feel?
It feels like paranoia. You don’t know how the image of you has spread, or your words passed on. You don’t know how it will be interpreted; you don’t know if you’re going to wake up one morning in the middle of a context collapse Twitter pile-on – or be fine as normal – or arrested by the police.
ASIDE, just to say that Covid-19 is a very 2020s disease, very paranoid.
Unlike the Blitz in London in the Second World War where the risk was external, and everyone has to pull together. (I reference this simply because it’s the event which is also mentioned here in the UK whenever there’s a new national crisis.) Everyone could pull together because everyone could be trusted. All in the same boat.
But with Covid…
Anyone you meet may be infectious. Or not. There’s a risk in every interaction that, later, you find out they have “betrayed” you. Further, there’s a risk that you, yourself, may have Covid. You may be spreading it, infecting your neighbours, your parents – you can unknowingly betray yourself.
So there’s this questioning of self and one-another, and we’ve responded with surveillance and sousveillance: we continuously monitor one-another with contact tracing apps, ourselves with self-administered tests. We’re reminded to be suspicious.
This uncertainty about self and other is so similar to social media. When you talk to people online, are they really people or are they bots? Are they stealing your data? Have you exposed yourself, given yourself away? Are you, yourself, tainted – have you fallen into a Facebook rabbit hole and been radicalised… how would you know? Is there a home-administered lateral flow test for extremism?
I am not saying that the Covid response is inappropriate.
But what I am saying is that the mutual suspicion and monitoring is (looking at it with this particular framing) a forced paranoid state, which is very in keeping with social media and networked technology.
And it would be interesting to consider how we would have tackled Covid if we had instead a different dominant social scaffolding to conceptualise “threat,” for example if we had still been in the tail end of the Cold War.
Credit to the current generation, they are responding to this paranoid milieu of the 2010s/2020s and developing new language to point at it and discuss it.
The emergence of the term gaslighting has been a joy to see: this new ability to discern when memory is being undermined for the purposes of manipulation and control – well, that’s a word we all needed and thank you.
Jumping to a definition for a second:
Gaslighting is a technique that undermines your entire perception of reality. When someone is gaslighting you, you often second-guess yourself, your memories, and your perceptions.
Let me bring this back to Apple, and why I think the initial response to their child abuse material scanning announcement has been so angry and so strong.
Our phones aren’t computers. They are our outboard brains. Our photos aren’t simply stored; they are part of our memory.
We live in a state of forced paranoia, developed over the last almost twenty years. We don’t know who’s watching or what will be done with this. But we’ve found accommodations. We’ve managed. We have new language to talk about it.
Except now somebody is proposing to look at our memories. We won’t feel anything; we won’t hear anything; probably nothing will happen. We all know from previous experiences with algorithms that misinterpretations will happen. And of course there are human monitors involved too, which means we have to consider, at some level, what they will think of us. So now we have to police ourselves, just in case we take a photo of - have a memory of - happen to think the wrong thing.
And if somebody else is now inside your memories, can you be sure that they’re not being edited? Is gaslighting occuring with these most personal of devices? Even if it never happens… that’s the lesson of the Panopticon, the mere possibility is enough to affect behaviour.
What the word for paranoia when it’s true?
Covid, phone surveillance, social media, mass paranoia – all of these are of a type and in resonance; nonlinear sympathetic consequences are kicking off all over the place.
I don’t know what should be done, what the rights and wrongs are here.
But I wanted to make the connection.
Apple released its plans to automatically scan phones for child abuse material which on the face of it is good policing – and the response has been loud and angry and calls out the dangerous slippery slope of surveillance. But I think what is also being revealed is a particularly 21st century phenomenon, and that is mass social paranoia.
The slippy slope argument is not hard to see. The plan is for Apple to continuously scan photos sent in messages and stored in iCloud, testing them against a known database of child abuse images, and escalating matching photos to human review.
But now the mechanism is in place, what else could it be used for? Could the Chinese government coerce Apple to locate dissidents, by adding certain non-child-abuse images to the central database? I mean, what are Apple going to do – not sell phones in China? Or can the GDPR “right to be forgotten” be wielded to force erasure of (say) unwisely shared nudes? Hard to argue with that, and Google hides links from search results under GDPR so maybe not such a stretch. But then why not automate removal of embarrassing photos of celebrities with expensive lawyers?
The EFF response is far more articulate on the details and potential of the new system: Apple’s Plan to “Think Different” About Encryption Opens a Backdoor to Your Private Life.
BUT:
I have a friend who has worked in positions where she can see the global traffic of child exploitation material, and I’ve spoken with her just a little bit about this in the past. It is horrific and huge. We need good policing, that’s my view, and mechanisms to achieve that, and we can debate how that happens. (We’re a long way from any answers, but my thoughts on a good approach are a whole other topic.)
So there’s a line for society to walk.
And it really doesn’t help that we have to trust a corporation to walk this line, without democratic accountability.
Yet this isn’t just a privacy debate.
It feels different because the photos are being scanned on-device. The surveillance is on our phones. And that triggers a whole other kind of response.
Cory Doctorow, way back in 2002: My Blog, My Outboard Brain (O’Reilly):
Clive Thompson, in 2007: Your Outboard Brain Knows All (Wired):
This feels obvious now, but it was new then:
So smartphones become, somehow, part of the mind.
Cognitive scientist Andy Clark makes the point that the mind doesn’t stop at the skull. He lays out the extended mind hypothesis in his astoundingly prescient book Natural-Born Cyborgs (2003). Highly recommended.
He makes the argument that we don’t just use pen and paper to work out a sum, but the tool becomes part of our thinking. Information on the web isn’t just consulted on our phones, but is in a real way part of our memory.
Humans are special precisely because our brains have this ability to side-load the world into self:
Phones are part of us.
Scanning the photos on your phone isn’t like steaming open the mail and peeping inside the envelopes. It’s like rifling through your memory.
And when those memories may at any time be silently observed or removed… even if it never happens but there is the possibility of it…
Well.
Every culture, big and small, has a feeling that it swims in but is often slow to put its finger on, like the proverbial fish in the ocean unable to see the water. That’s my take.
I think in the 70s and 80s that feeling was the end of the world. I was pretty sure, as a little kid, that by the time I was my age, now, I would be living in a post-apocalyptic nuclear wasteland. It wasn’t a conviction, it was more like an unspoken understanding. And goodness knows what that did to us.
Ironically the end of the world is coming, in the shape of the climate crisis, and I wonder how those of us who grew up taking the Cold War for granted are coloured by that experience and how it is tainting our response. We probably feel like the climate crisis, or at least some kind of apocalypse is inevitable somehow? Or alternatively, that if we wait around for long enough then the threat will just somehow… recede? Like the way the peril lifted in the 90s. Dangerous templating for us to have; thank god for the zoomers.
What’s in the air now?
We swim in paranoia, I think.
We’re always potentially being watched.
RELATED: I ran across Zizek riffing on Donald Rumsfeld (YouTube) and specifically developing the concept of unknown knowns. Here’s Ted Hunt on Twitter with a quote/summary:
So paranoia is like our culture’s current unknown known. That’s where it sits, somewhere in the social unconscious.
James Bridle’s 2014 work The Nor was
It’s a sequence of essays telling the story of a participatory, documentary act: Bridle’s walk across London, photographing every CCTV camera he passed. SPOILER: It doesn’t end well.
Here are the essays:
And it was this work that really opened my eyes to the pervasive sensation of surveillance. (Which is why art is vital, right?) Especially because Bridle makes explicit the role of the network and what that does: the first essay is titled All Cameras are Police Cameras.
The camera network today is Instagram, TikTok, other people’s phones. It’s the pictures taken at parties, previously private spaces, and it’s the acquisition of the breakthrough facial recognition startup Face.com by Facebook in 2012, and everything that opened a door to across the industry.
A lot has been said about the Panopticon, Jeremy Bentham’s 1786 concept of a prison where the prisoners are controlled by the mere possibility of being observed, and of sousveillance: surveillance from the same level; we watch one-another. That’s what a networked camera in every pocket leads to.
The debate, over the last 20 years as this has been happening, has been framed around the loss of privacy and whether that matters: the younger generation has different privacy expectations to us, that’s one statement; the absolutist privacy ideals of the EFF are another part of the debate.
(And the responses to this shift are fascinating. For me, the go-to here is danah boyd’s work, and I’ve recently been diving into her work on networked privacy from the early 2010s, and the sophisticated ways that teens are finding control and agency in this world.)
But how does it feel?
It feels like paranoia. You don’t know how the image of you has spread, or your words passed on. You don’t know how it will be interpreted; you don’t know if you’re going to wake up one morning in the middle of a context collapse Twitter pile-on – or be fine as normal – or arrested by the police.
ASIDE, just to say that Covid-19 is a very 2020s disease, very paranoid.
Unlike the Blitz in London in the Second World War where the risk was external, and everyone has to pull together. (I reference this simply because it’s the event which is also mentioned here in the UK whenever there’s a new national crisis.) Everyone could pull together because everyone could be trusted. All in the same boat.
But with Covid…
Anyone you meet may be infectious. Or not. There’s a risk in every interaction that, later, you find out they have “betrayed” you. Further, there’s a risk that you, yourself, may have Covid. You may be spreading it, infecting your neighbours, your parents – you can unknowingly betray yourself.
So there’s this questioning of self and one-another, and we’ve responded with surveillance and sousveillance: we continuously monitor one-another with contact tracing apps, ourselves with self-administered tests. We’re reminded to be suspicious.
This uncertainty about self and other is so similar to social media. When you talk to people online, are they really people or are they bots? Are they stealing your data? Have you exposed yourself, given yourself away? Are you, yourself, tainted – have you fallen into a Facebook rabbit hole and been radicalised… how would you know? Is there a home-administered lateral flow test for extremism?
I am not saying that the Covid response is inappropriate.
But what I am saying is that the mutual suspicion and monitoring is (looking at it with this particular framing) a forced paranoid state, which is very in keeping with social media and networked technology.
And it would be interesting to consider how we would have tackled Covid if we had instead a different dominant social scaffolding to conceptualise “threat,” for example if we had still been in the tail end of the Cold War.
Credit to the current generation, they are responding to this paranoid milieu of the 2010s/2020s and developing new language to point at it and discuss it.
The emergence of the term gaslighting has been a joy to see: this new ability to discern when memory is being undermined for the purposes of manipulation and control – well, that’s a word we all needed and thank you.
Jumping to a definition for a second:
Let me bring this back to Apple, and why I think the initial response to their child abuse material scanning announcement has been so angry and so strong.
Our phones aren’t computers. They are our outboard brains. Our photos aren’t simply stored; they are part of our memory.
We live in a state of forced paranoia, developed over the last almost twenty years. We don’t know who’s watching or what will be done with this. But we’ve found accommodations. We’ve managed. We have new language to talk about it.
Except now somebody is proposing to look at our memories. We won’t feel anything; we won’t hear anything; probably nothing will happen. We all know from previous experiences with algorithms that misinterpretations will happen. And of course there are human monitors involved too, which means we have to consider, at some level, what they will think of us. So now we have to police ourselves, just in case we take a photo of - have a memory of - happen to think the wrong thing.
And if somebody else is now inside your memories, can you be sure that they’re not being edited? Is gaslighting occuring with these most personal of devices? Even if it never happens… that’s the lesson of the Panopticon, the mere possibility is enough to affect behaviour.
What the word for paranoia when it’s true?
Covid, phone surveillance, social media, mass paranoia – all of these are of a type and in resonance; nonlinear sympathetic consequences are kicking off all over the place.
I don’t know what should be done, what the rights and wrongs are here.
But I wanted to make the connection.