An overview and details of the technical implementation make up the bulk of the text.

If you want the highlights:

• Jump directly to the video demos.
• Then read my observations and speculations.

Lares is an AI agent operating in a simulation of a smart home.

This section summarises what we'll see in the demos.

Scenario

Lares is based around a toy simulation of a smart home. It's just a simulation. The agent isn't a simulation, it uses AI. The Lares agent interacts with the smart home simulation via an API. It takes instructions via a web UI.

Demo UI

When the demo runs, the interaction is via this web UI.

On the left is the live state of the simulated smart home:

• There are five rooms, each with an ID. The rooms are connected via exits.
• Each room has one or more smart devices. In this case, each room has a light. The light can be on or off, and the room view is light or dark appropriately.
• There are certain moveable items such as a dog and a person. Although these appear in this web UI, these are initially hidden from the agent.
• One of the moveable items is a robot. The agent is able to move and look through the robot, seeing the hidden items.

On the right is the agent interface itself. The user can instruct the agent here and, when running, a transcript of agent actions is shown.

History

I built the first version of Lares in April 2023 at the London AI Hackathon and collaborated with Campbell Orme on user experience and comms. We won two of the five hackathon awards, from Google DeepMind and Amazon AWS.

• Video demonstration 1
• Original technical write-up 2

It served as a proof of concept but didn't operate reliably.

This new version is based on OpenAI's function calling functionality and their GPT-4o model, rather than GPT-3 and the text-based ReAct pattern, and I'm sharing the code for the first time. (See Technical Implementation, below.)

Simple instruction following

We instruct the agent to turn on the light in the kitchen.

It figures out the correct function to call to toggle the light.

Transcript

Problem solving

We instruct the agent to turn on the light for my dog.

The agent doesn't know where the dog is, so it has to look via the robot, moving it into various rooms. However it realises that the robot can't see in the dark, so it turns on the lights. This action is not pre-programmed.

Transcript

Agent-initiated user interaction

We instruct the agent to set the light in the office depending on if it's daytime.

It doesn't know whether it's daytime, so it asks the user a clarifying question. Then it decides to turn off the light because there's daylight outside. (I wish everyone in my home did the same.)

Transcript

What makes an agent an agent is:

1. Tool use. The AI replies using specially structured text that is interpreted and executed by the agent system ("function calls" or "tools") instead of prose aimed at the user.
2. Autonomy. The AI is in charge of deciding what to do next. It loops, calling functions, until it decides to halt.

The emergent behavior, such as problem solving, comes from the large language model (the AI) making use the functions.

The following sections run through the implementation in detail. If you've developed agents before, jump ahead to two tactics that increase reliability: Separating Intent and Action and Managing World Knowledge.

Code

The code can be viewed or downloaded from this GitHub repo: github.com/genmon/lares 3

app/

The user interface is a client-side React app.

party/assistant.ts

The agent runs as a PartyKit 4 server for coordination. (PartyKit is a framework for writing tiny stateful servers, ideal for agent orchestration.) It makes use of the OpenAI Chat Completion API to produce function calls. It communicates with the UI via WebSockets. It interfaces with the smart home simulation via an HTTP API.

party/server.ts

The smart home simulation runs as a PartyKit server that maintains a hidden world state, and offers an HTTP API to query and act on the sim.

Agent Lifecycle

The user gives the agent an instruction.

A single turn of the loop consists of these steps:

1. Setup. A transcript is prepared including:
   • The user instruction
   • The agent's up-to-date world knowledge (it knows which lights are on and off, but not the location of non-networked moveable actors, such as the dog)
   • Historic function calls and return values, over the current loop
2. Intent. The agent is asked to call decideBestFunction to choose which function to call
3. Action. The agent is asked for parameters to call its chosen function.

If the chosen function was halt, the agent stops and replies to the user.

If the chosen function was anything else, the function is dispatched, the return value added to the transcript, and the loop repeats.

OpenAI API and Prior Art

The concept of a chat transcript is standard in OpenAI's Chat Completions API 5.

Function calling is also standard in OpenAI's API, including the architecture that the runtime to handle the function call and append the function return to the transcript. See OpenAI's Function Calling guide 6. Function definitions, calls, and returns are encoded using JSON.

This demo does not use OpenAI's Assistants API 7 as it is simpler to use Chat Completion to match our desired agent lifecycle.

Function calling is a formalisation of the ReAct pattern 8 by Google Research and the Princeton NLP Group, published in November 2022.

Available Functions

Function calls are how the agent interacts with its runtime system, and acts in the world. The result of the function call is returned to the AI.

The agent loop for Lares understands six functions.

(If you're looking at the code, these functions are declared in party/functions.ts.)

• toggleLight
  • Description: Toggle the light in a room, making it light or dark. You can see what's in a room when it's light.
  • Parameters: lightId — ID of the light to toggle
  • This function is handled by the simulation and updates the world state.
• moveRobot
  • Description: Move the robot to an adjacent room. The robot can only move to rooms that are connected to its current location by an exit.
  • Parameters: destinationRoomId — ID of the adjacent room to move the robot to
  • This function is handled by the simulation and updates the world state.
• lookWithRobot
  • Description: Look into the robot's current room, returning the fixed and moving items (only if the room is light).
  • Parameters: None.
  • This function is handled by the simulation and updates the world state. Returns an error and the message "It's too dark to see anything" if there's no light on in the robot's current room.
• askUser
  • Description: Ask the user a question. Use this to get more information or clarify something. Only use this as a last resort
  • Parameters: question — The question to ask the user
  • Handled by the agent runtime and opens a prompt in the user interface. The user's response to given at the agent as the function return value.
• halt
  • Description: Stop operations because you've achieved the goal or can't go any further
  • Parameters: messageToUser — Report back to the user with a message. Never use this to ask for more information.
  • Handled by the agent runtime.
• decideBestFunction
  • Description: Decide which function to call next based on how to best respond to the user
  • Parameters:
    • reasoning — no description provided in the prompt
    • bestFunction — an enum of the above function names
  • This is a special function used by the agent runtime for the Intent step, ahead of the action function call.

Constructing the Transcript

On each turn of the loop, a transcript is constructed. The transcript is the prompt for the AI.

(If you're looking at the code, this is done in the agent code in party/assistant.ts.)

The history of interactions is then appended.

For example, for the instruction turn on the light in the lounge, the first loop may result in the following addition to the transcript.

Separating Intent and Action Increases Reliability

The agent is asked which function to use by first asking it to use the function decideBestFunction.

The following messages are temporarily added to the transcript.

Recall that decideBestFunction has two parameters:

• reasoning
• bestFunction

The reasoning parameter is an opportunity for chain-of-thought 9 — a series of intermediate reasoning steps which improves reliability.

There is no return value for the intent function call. The call is not added to the transcript. This is to avoid distracting the agent in subsequent steps.

Instead a summary is added to the transcript, and a prompt to call the chosen action function.

Only a single action function is provided (e.g. moveRobot). The agent is not allowed to decided between tools at this point, only complete the parameters.

Decoupling intent and action appears to improve reliability.

World Knowledge is Managed Outside the Transcript, Improving Reliability

The transcript in each turn of the loop is not a historic transcript. Although it resembles a chat transcript (which is append-only), it is dynamically constructed.

In particular, the second message is the current state of the home.

This is:

• the agent's knowledge of the home, not the world state (which is hidden). The agent can tell the state of the light in the living room, for example, but not the location of the dog.
• the latest state, taking into account all the function calls that follow in the transcript. This has improved reliability over including just the initial state and asking the agent to construct the current state by looking at its function history.

Specifically: the hidden world state includes moveable items (like people and animals), whereas the world knowledge does not.

If you're running the code, compare:

• True world state, hidden from the agent:
  http://127.0.0.1:1999/parties/main/default/debug
• World knowledge, as supplied to the agent:
  http://127.0.0.1:1999/parties/main/default/world

Supplying the latest state increases reliability, although it does sometimes confuse the agent regarding the order of historic events (e.g. it may say the light in the office was already on when it had turned on the light itself).

The agent approach is widely applicable.

Even this basic approach would be suitable for:

• A real smart home. I would be keen to see the Lares approach scaled to an actual smart home, including the askUser-style opportunity for the agent to disambiguate instructions. A good place to start would be a new interface for an open source assistant such as the Home Assistant 10 app and framework.
• Self-contained devices with voice control. The simplicity of this approach and local models would make it possible to embody sophisticated voice control in devices such as kitchen timers, digital-clock radios, and car entertainment systems. A stove would be an ideal application.
• Simulation toys. Old software toys such as Little Computer People and old home robotics such as Big Trak would both have fresh life with an agentive control and reasoning system.

Considering the potential applications, what I've learned with this Lares demo, and the general applicability of agentive systems, I have a number of observations.

Systems will need an agent-facing API.

Whereas some APIs are fine-grained to allowed for a front-end UI, and some APIs are terse and machine-readable for integration with other systems, an agent-facing API has specific requirements.

• High level: common tasks should be achievable in a single function call, rather than chaining two or three
• Straightforward and consistent naming: names of function calls and parameters are "read" by the agent and should be considered part of the prompt
• Machine input; natural language output. Success messages should be a sentence describing how the system has changed, and error messages should hint at the required resolution (e.g. "It's too dark to see anything"). (Conventional APIs use error codes.)

This is sufficiently different from regular APIs that they will need to be designed and maintained specifically.

Agent runtimes should provide agent abstractions

Chat is not a good abstraction for agents as, although it allows for function calling, it is optimised for an append-only transcript.

An agent-focused system would have abstractions for:

• World knowledge. An up-to-date agent-friendly representation of the world state (excluding hidden state) is essential.
• Separating intent and action. This allows for inline chain-of-thought reasoning which increases reliability.

Also:

Non-JSON encoding. Encoded in JSON, function calls only decode cleanly when complete. This means it is not straightforward to use function calls to stream messages to the user.

Ideally this would move high-performance structured generation 11 (the Python library outlines 12 achieves this by managing the next-token probability according to a specified grammar).

Embedded agents lead to new demands on technology and UI

The simplicity of the agent approach means that devices such as stoves, lamps, and timers could have sophisticated interfaces without being connected to the internet.

It could even include voice control 13.

It provokes some interesting routes for research:

• Embedded models. What is the best model to use in, say, a stove? How can it be updated and how can it be debugged?
• User interface. An overheated stove will spoil the food. What's an appropriate UI to allow a stove to show its interpretation and allow for (a) legibility of actions; and, (b) repair before committing?
• Exploits. Even cursory red teaming will reveal exploits: what if someone yells through a letterbox, "hey stove, turn up to maximum at 4am and then cycle rapidly"? Can physical proximity be used for access control, and are there dependable sensors for this?
• Coordination. Can a kitchen timer tell an oven when to pre-heat, without those devices being configured on the network? How can multiple agents work together?

Pursuing agents in multiple scenarios will reveal more technical and UX challenges.

Function marketplaces and standards

Because of the ease of implementing agents, we may anticipate a future where small agents are widespread, rather than single, monolithic agents.

With a proliferation of agents, the challenge will be in coordination and adapting to new environments, and in implementing and discovering functions.

For example, how does my personal AI pendent (with built-in agent) know how to turn on the TV in an Airbnb that I haven't previously visited? If I ask my HVAC to prepare for tomorrow's weather, what function should it use to fetch the forecast?

We will need search engines or marketplaces for agent functions, and standards for calling conventions and agent-readable function documentation.

Considerable the search engine or marketplace that returns the most appropriate TV remote control function for the Airbnb I'm staying in. This will have to consider signals such as the device model, the interfaces I've used before, what's been used historically by people in this location, and so on. How can these signals can shared and trained against in a privacy-preserving fashion?

I have discussed Marketplaces for AI agents 14 in detail elsewhere. One search engine embodiment is described in a patent: Shared Experiences 15.

What is the origin of the observed emergent problem solving abilities?

Given the simplicity of the agent lifecycle code and functions, the emergent abilities of the agent must arise from the large language model itself. This provokes several research questions:

• Is problem solving inherent in the structure of language?
• Can emergent abilities be quantified?
• Do abilities improve with the availability of user interaction, or other knowledge building "tools"?
• Can the abilities be improved, for example with alternate problem solving strategies?
• Can the above be achieved without increasing the complexity of the agent lifecycle code?

For example, on the strategy question, the problem solving strategy right now is a basic step by step "hill climbing" approach. Could the agent be encouraged to make plans to look further ahead, or test the feasibility of different approaches before engaging?

Even simple agents can exhibit emergent behavior such as basic problem solving. There is very little code in Lares outside the large language model itself. There are many sophisticated AI agents available today, and the simplicity of the code implies that many more will be developed.

So we can anticipate a future proliferation of agents. This has implications and opportunities: for user experience, standards/interop, underlying technology platforms, and business.

Thank you PartyKit 4 for supporting this work.