Horsehistory study and the automated discovery of new areas of thought

12.23, Wednesday 16 Jun 2021

This starts speculative and ends up in an interesting place: an algorithm for improving language. (My personal algorithm for new ideas appears to include: think about nonsense for longer than most others are prepared to tolerate.)

These words all start with horse-:

  • Horsefly – a fly found near horses, fair enough
  • Horseplay – totally understandable, but a metaphor that only makes sense if you have been friends with horses
  • Horsepower – sadly not like “sea power”
  • Horseradish – a root and a condiment
  • Horsetrade – now a useful way to talk about an equitable negotiation that involved a lot of back-and-forth, originally I’m sure to do with horses

Note that these are unlike: horsehair, horserider, horseshoe, etc, which are direct attributes of a horse. That is you could equally say “horse’s hair” and so on.

So it’s fun to invent other horse- words and speculate what they might mean. Some at random:

  • Horsefunctor
  • Horsehistory
  • Horseorbital
  • Horseroman
  • Horsesane

For example, #1: Rob Miller suggested on Twitter: Black Beauty is my favourite horseroman.

And that’s neat, right? “Roman” as in roman à clef, a novel not-so-secretly based on real life events, and Black Beauty is a novel about a horse. BUT ALSO maybe a horseroman could be like the famously equine senator appointed by Nero in Ancient Rome.

Let’s blend those two and throw in some semantic drift, then define horseroman as one of those thought leaders who is appointed by the people in charge, but is actually way out of their comfort zone, and answers all questions with lengthy anecdotes about their own life. Like a bad TED talk.

For example, #2: Peter Bleakley pointed out that: The “horse” in “horseradish” is nothing to do with horses. It’s cognate with “coarse” and indicates “inedible”. Same as “horse chestnut”.

So let’s take that and apply it!

Maybe horsehistory could be the socially uncomfortable parts of our history that we brush under the carpet? A useful new term!

(Thank you Tom Carden for continuing this conversation on Twitter yesterday.)


The first analysis from a horsehistorical perspective

This section is a tangent but I want to show you that the concept is useful.

I was never really taught about the British Empire or colonisation at school. Sure I knew about it, but it was always in the background, taken for granted. Sure there were some ugly aspects but aren’t there always.

Then in my 30s I visited a museum in Kolkata and unexpectedly began to learn about the atrocities committed by my country, and the feeling of sickness and shame that started on that day has never left me. I have continued to educate myself. What’s worse is that I did know some of the events, but I hadn’t stopped to consider them.

Empire is not, in the UK, ignored history. We all know it. But when you grow up with something from before you can speak, and leave it unanalysed, you accept facts that you would never accept as an adult. I imagine it’s a little like abuse: if you grow up in an abusive household, it takes work as an adult to realise: that wasn’t normal! That was not ok!

Now some of this is unconscious, but some of it is very much deliberate. We have leaders who do know history, who are able to talk about Empire (what it was, what we did, what we are still doing), but keep silent and make use of the idea of Empire (“Global Britain”, now). And because there’s utility in maintaining this myth in its unanalysed state, a kind of systemic resistance arises: anybody who does attempt to talk about Empire in an adult, clear-eyed way is aggressively shouted down. See the current culture war about “decolonising the curriculum” in universities which, to my mind, is simply about saying: let’s not take this history for granted.

My pet theory:

You can’t really talk about society as an individual, but give me this rope for a second. “British society” is aware of Empire and its atrocities, but conscious acceptance of that knowledge is repressed – for whatever reason: because it runs counter to our identity, because it is inconvenient, because then we would have to do something about it, because it was awful, take your pick.

In short, the history of the British Empire is impossible for British society to digest.

In an individual, repressed feelings find other ways to come out. So the repressed idea of colonisation came out as Brexit, which was this almost fanatical belief that we were being colonised by a larger state, EU. Or, to be blunt: that the pains that the British Empire had inflicted on others were now being inflicted on us.

I read this as a way to psychologically square the circle of the repression: to say, it’s ok to avoid looking at the history of Empire squarely in the face, because look it’s fair now, we’ve been punished.

Anyway, that’s my amateur read.

It’s also the first application of horsehistory: the study of undigestable histories and what they do to us. Agree or disagree with my analysis in this instance, the area opened up is interesting.

Maybe this study could also look at the ways that horsehistories become regular histories, and how we could take British society through that journey (several other countries with atrocities in their past have been able to look at their history with clear eyes, act accordingly, and are healthier for it).

Or we might also examine how horsehistories ossify over time (or not), or catalogue them globally, or re-analyse existing histories.

A new word becomes a new lens for understanding the world.


Words as coordinates in the space of all possible concepts

Back to the horse- prefix. What does it do to words?

It’s not a straight modifier. Horseplay is not the play of a horse (not any longer), nor horseradish a horse’s radish. It’s an unexpectedly transformative operator, in a way that I don’t yet understand.

Maybe: it’s a matrix rotation in embedding space?

To unpack that:

An “embedding” is how machine learning encodes concepts.

A good example is word2vec, an old technique (old meaning 2013) that takes words and translates them into coordinates in a multi-dimensional space of all possible concepts. The set of coordinates is called the embedding.

What’s neat is that the embeddings can be mathematically combined. That is to say:

king - man + woman = queen

If you take the coordinates for king, subtract man and add woman, you get the coordinates for queen. Approximately…

The resulting vector from “king-man+woman” doesn’t exactly equal “queen”, but “queen” is the closest word to it from the 400,000 word embeddings we have in this collection.

That example is from the EXCELLENT guide from Jay Allamar, The Illustrated Word2vec.

So thinking about the concept horsehistory using this model and attempting to decompose it, what we see is that it’s not the mathematical addition of the horse embedding and the history embedding. The horse- prefix has mutated the history embedding somehow and turned it into something else. That mutation is what I’m referring to as a matrix rotation.

Let me try another way:

Have you every tried miracle fruit? It’s a berry with unusual property – it doesn’t have a taste itself, but it changes other tastes. In particular it rotates sour to sweet.

So I had some of these berries and was drinking beer, and the beer tasted like Fanta. Amazing! Then some time later, suddenly as I was crossing a road, the berries wore off and I tasted the inside of my mouth as if for the first time, and good grief it was disgusting.

The horse- prefix is the miracle berry of words.

And also:

New words are addresses to previously unused embeddings in concept space.


The invention of new words provides new scaffolding for thought

I think what I’m convinced by, with horsehistory, is that it’s worth developing new words.

In the sci-fi novel Native Tongue (Bookshop.org) about aliens, linguistics, and a fierce patriarchy, Suzette Haden Elgin supposes a new language for women. The creation of this artificial language is an act of resistance and also way to carve out a space for unique feminist thought and being. (It’s a stunning book.)

In Native Tongue, discovering a new word in this new language is a big deal: a new word, a new concept, a new “Encoding” as Elgin calls it, a new valid embedding in concept phase space, we might say. Finding a new Encoding rarely happens! Each Encoding is hard won. If somebody discovers/invents one or two, that is huge news!

And so it is for us, I think. Discovering a new concept that isn’t simply a metaphorical framing, today, is rare and propels thought. Back to “Brexit” for a second: it derived from “Grexit,” itself a neologism (for Greece leaving the EU), but having been coined it was possible to poke at it the concept, to discuss it, to ask about how it could happen and if so when and what it would mean, and so on, and without the word I believe the process would have unfolded in another way entirely.

Yes we can initially refer to these same concepts in other more cumbersome ways, but as single words it is possible to combine and manipulate more ideas that are more complex, and then they take on their own reality. Cheap referents have value.

(Metaphorically, I’ve long believed that this process is what the Old Testament story of Moses receiving the Commandments is about, at least partially. There is an arduous journey - up a mountain - at which point some simple rules that are received, literally inscribed into rock. The rules are straightforward and lead to a long-term healthy and stable society, at least in this belief framework, but are hard to arrive at from first principles. The foundational ideas are too unwieldy and require a rare perspective. Similarly, great minds dedicate their lives to climbing their own mountains to claim complex insights similarly inscribed in simple terms and, having received the new concept in a graspable formulation and bringing it down the mountain for us, our whole society benefits.)

The question is: is it possible to move beyond Native Tongue? It’s slow. Can we automate the process of concept discovery?


A reward function for the invention of new words

Yes there are ways to invent new words with AI. Take the website ThisWordDoesNotExist.com which generates a random word and a plausible definition (making use of GPT-2, the ancestor of GPT-3 which, as previously discussed is an idea machine). For example here’s one generated word:

tokou

a black wine made from fermented soybeans cooked in molasses and yeast; “various African wines and tokou grape”

The problem is that tokou isn’t as useful as, say, horsehistory (at least on first glance). We could sit there, refreshing the website, trying out each word to see if it’s handy, but that’s a bottleneck in the process and besides I would lack the domain expertise in most cases. So any algorithm will have automate that process too.

Here’s the algorithm I propose.

1. Invent candidate words and their definitions.

Either invent at random, or use the method I ran by hand above: collide multiple embeddings and jiggle the result with semantic drift. That was how the horseroman concept was generated. An AI can do this.

Now we effectively have two languages: English, which is the language we speak, and English-Prime, which is identical save for the addition of this single new word.

2. Translate all of human knowledge into English-Prime.

This isn’t as hard as is sounds.

Google Translate has used, since November 2016, a system called Google Neural Machine Translation. It’s a machine learning method of translation between language pairs.

But look closer at how it works…

“Visual interpretation of the results shows that these models learn a form of interlingua representation for the multilingual model between all involved language pairs,” the researchers wrote in the paper.

An interlingua is a type of artificial language, which is used to fulfil a purpose. In this case, the interlingua was used within the AI to explain how unseen material could be translated.

… The data within the network allowed the team to interpret that the neural network was “encoding something” about the semantics of a sentence rather than comparing phrase-to-phrase translations.

In short, Google Translate has an intermediary language that exists only in the form of embeddings.

So, even without training data already written our new English-Prime, we know how to represent it in language space: it’s identical to English but with a single extra embedding available. And if we know that, Google Translate can translate into it. (Exactly how is left as an exercise for the reader.)

Which means that step 2 is to automatically translate all of Wikipedia into English-Prime.

3. The reward function: test whether the new word is useful

What does useful mean? Machine learning has the idea of a “reward function”: how do we state what good looks like? If we can do that, the process can run automatically.

For horsehistory it meant that the word acted as an intuition pump (philosopher Daniel Dennett’s term): by examining what it could mean, it took me to a place where complex ideas were reached and could then be articulated more efficiently.

So what is our reward function? Simply:

Compare the word count of Wikipedia in English and the word count of Wikipedia in English-Prime. If the latter is shorter, i.e. more efficient, then the proposed new word is useful.

RINSE AND REPEAT.

Each new term is a new place to start thinking. Give each its own journal and academic conference and see what happens.


Let me summarise.

By way of speculating about a single new field of study, horsehistory, I have proposed a general method for the automatic generation of many new fields of study.

In the same way that Magnus Carlsen is a “centaur” chess player, a player of chess greater than any other human because he has trained with a custom AI and benefited from the wisdom of this machine which is effectively 200 data-years old, the method I propose leads to a new discipline of centaur philosophers, thinkers who are able to systemically reveal new scaffoldings for thought, far beyond what would ordinarily be reached in a single human lifetime, to more rapidly develop and examine new ideas for the betterment of society at large.

Or the whole thing is horsefeathers. Take your pick.

More posts tagged:
Follow-up posts:

If you enjoyed this post, please consider sharing it by email or on social media. Here’s the link. Thanks, —Matt.