2004-02-10

Eric Bonabeau- Evolving the Bad Guy
http://conferences.oreillynet.com/cs/et2004/view/e_sess/4847


our brains are wired not to embrace complexity but to ignore it.

so we can create things more complicated than our brains can comprehend.

the trouble with the world:
- numerous components, heterogeneity
- distributed
- interconnected, non linearly
- non stationary, dynamically changing
- chance events, surprise

inevitable: any complex system has loopholes. weaknesses, vulnerabilities,
pathalogical behaviours.

and loopholes can (and will) be discovered.

the sounds that your cat produces have been shaped by evolution to make you feel
guity.

[this is the gaming of interfaces thing. but is it actually a loophole? it's
only a loophole in a constructed system: anywhere else - in protein/receptors,
in nature - it's a feature.]

examples:
- tax code
- software
- frequent flier programs
- health care, medicare
- 1996 telecommunications act (eg, data would be taxed less than voice: hence,
companies transform voice into data. MCI was routing calls into canada and back
again.)
- NBA rules
- elections (redistricting)
- power grid, water distribution [okay, so why does this happen? we've got the
City of Bits thing about resources being deterritorialized and then re.t. on
different spaces, eg village well becomes water bottles and pipelines. so at
what point does the externalised constructed space provide abilities for
gaming/loopholes?]

awesome example:
- french critic made a movie about a serial killer killing members of tv
audience share panel
- when the movie aired it completely ruined the statistics for that day, because
a large proportion of the panel members watched the film
- called "Target Audience"

on how to find the loopholes in the system,
take hints from nature;

- parasites
- viruses
- co-evolutionary arms races

=> rapid evolution
as a method of discovering the loopholes in extremely complex systems.

but it's the red queen's race: every time you fix a loophole, you add another.

[there's really something here about the invariance in loopholes. there's a
difference between the "perfect" system (how do we quantify that?) and the
constructed, real system. what i guess I'm talking about is the shape of the
manifolds. the difference between the manifold surfaces is to do with the
resolution of how we understand dynamics processes. we've got a very low
resolution: that gap is where loopholes occur. it's the warm layer at the top of
the ocean where life exists.]


* Evolutionary computation

individuals are represented by a genetic string: 1101011000

[flattening a meshwork to a turing ribbon. i was thinking about this in one of
yesterday's notes.]

use mutation and crossover as genetic operations to evolve better creatures. in
this sense, the peaks in the adaptive landscape are the loopholes.

[there are loopholes within loopholes. or rather, the peak goes on to infinity,
but the top of the peak is altered in complicity with the affordances when we
move into it. that creates a new peak that we move towards that creates a peak
on top of the peak on top of the peak... continuing in generations. not a
garden, not a knot (or rather, the system-dynamic is a knot, but the thing as a
singularity isn't), it's an unfolding thing, like electricity, or people.]

In 2002, Layzell and Bird of Sussex evolving circuits. evolved an oscillator
circuit -- but it turned out it was picking up the EM from a local computer.
found a loophole in the lab.


three examples:

* 1. Naval research

a control engineer can only test small fraction of possible failure modes: those
that make sense to him.

[this is a really good talk -- bonabeau has a great sense of narrative and
making a complex argument followable.]

can we identify small changes that'll massively improve the system?

so he's demonstrating a way of exploring a unconstrained state space of 10^34
states, by using evolutionary computing. (it's ruptures on a warship and ones
which are dangerous.)

question: is there a combination of 3 uncorrelated rupture that'll have a
disproportionate effect on the control system? similar with putting poison in
the water supply?

[i'm thinking now of the middle ages identifying connectors in the social
network to stamp down on heresy, or that paper recently about vaccinating
connectors to have a disproportionate effect of the spread of disease.]

so surprise: the worst 3-point ruptures involved places that you wouldn't think
were important (the 13th most important rupture point for single point
ruptures). so by identifying that point, you increase the robustness of the
vessel.

another examples in submarines: oscillating roll could cause a failure
condition, but that wasn't "designed in" (of course!).

* 2. Hacker scripts

he's talking about designing a grammar to think about hacker goals, different
actions.

the evolutionary constraint is to not have any traces in the log files.

* 3. Search engine

googlebombs: can this be done systematically?

evolve googlebombs (they evolved against a model, but could have done against
the real google) by evolving link networks so that a given search text will put
a given website at the top.

[ok, so this is a really simple, *really* smart idea. why aren't loads of people
doing stuff like this? this could be hobbyist stuff. i wonder what kind of
worldview you need to see this kind of stuff immediately?]

topological solutions.

key point here: you don't need to know the scoring algorithm of google to do
this.


[he's basically showing a probe (a sense) to explore the space of interface
shape. awesome. like the knife with the sensor on the end that won't let you cut
through nerve cells. i still don't quite understand how loopholes arrive. i
mean, i do, but i need to have a picture of what the picture no-loophole
interface would be. i guess the point is that as soon as you make a system that
has leaky abstraction or can be used for unintended things, you have a loophole.
the no-loophole system is spiky: it's discontinuous. there are peaks where use
is possible and nothing inbetween. the smoothed system is more proteinish,
allows implicature. large features show basic functionality, fine features show
better functionality. hm. hm. needs thinking about.]