For month 2 of my residency, I’m looking at how we interact with AIs online. I’m zooming in on this approach: what if an AI was like working with a teammate?

ℹ️ This is one of a series of notes from my summer 2023 invention residency with PartyKit. See all research notes here.

I want to investigate what it feels like to have nonhuman colleagues. And, although I could sketch that on paper, how would you build this kind of system? — perhaps there are lessons from the building that would inform the design, too.

Once there’s you plus one other (human or AI) then you’re in a multiplayer situation. So PartyKit is the ideal platform to prototype with.

These are my first sketches (all working code!) and I went pretty playful...

Summoning NPC teammates

How do we collaborate with AIs? What if they were just like our human colleagues, cursors and all? Well not exactly human

I’ve been sketching NPCs-as-dolphins — you build a little pool, summon them, and that’s where they hang out.

(NPCs = non-player characters, a term from video games)

Issuing commands

Different dolphins do different things! Just like you work with different folks over a project, you'd pick diff NPCs for the task at hand

This one writes haikus (as a quick demo)

It’s an real, coded-up virtual user btw: the infinite canvas makes use of the open source tldraw and I’m using PartyKit to (a) host the tldraw collaboration back-end, and (b) connect to that same back-end as a legit user with presence and the ability to monitor/affect the document state.


A design advantage of cursors is you can play with DISTANCE to show ATTENTION

AI chatbots are tough. They don’t have affordances, you don’t know what to do

But this dolphin (who loves painting stars) can proactively offer its help whenever it sees an opportunity! Better

From building the proto, it’s pretty clear that apps will need serious re-engineering to become natively multiplayer (once you have user + NPC, you’re already multiplayer). Like, do you need a server-side machine API? What else?

Oh yeah, dolphins...

Dolphins! My go-to nonhuman companion species. So smart but very much not us, right? Like AIs obv.

The triangle and circle “embassy” shape I used is a schematic of the original Dolphin Embassy by architects Ant Farm (1974). it's amazing work, a barge floating on the ocean, containing a cetacean pool/portal, a perfect meeting point 🐬

For completeness, my initial sketch (tap to view)

I make software protos for one of three reasons: because I want to illustrate and bottom out an interaction; because I want to learn a new technique; because I can‘t stop laughing every time I think of it.

Sometimes all three? This is my first sketch of what I wanted to build with NPCs. What came out wasn't exactly what I expected - that’s the point! - but the dolphin metaphor gave me a frame and a goal, and I’ve now got decent foundations to build on. The “serious” applications of this pattern can come later.

A few more links

I’ve written about NPCs and the multiplayer web before:

And of course the Dolphin Embassy by Ant Farm...

I’ve talked about the Dolphin Embassy in passing but you really need to check out the architectural sketches linked from that post. Incredible work. Back in the 60s and 70s there was an assumption that we would have opened cetacean diplomatic channels by now.

Not crazy, as it happens! Because here’s a bonus fact for you: even now, humans work closely with dolphins. 25% of the United States nuclear stockpile is defended by trained dolphins today (source). We ought to be taking this relationship more seriously. Best of luck Project CETI, a research org currently attempting to teach AIs how to speak whale.