---
title: "Build a voice AI agent with memory using LiveKit and Supabase"
date: "2026-06-19"
authors: "Jesse Hall"
tags: "guide"
---

# Build a voice AI agent with memory using LiveKit and Supabase

Voice agents get interesting the moment they remember you. An agent that can answer a question, recall what you told it last week, look up your order, and pick up a conversation where it left off feels less like a demo and more like a product. All of that needs a backend. You need somewhere to store knowledge, memories, profiles, and session history, and a way to search it by meaning.

This post walks through a [complete starter](https://github.com/livekit-examples/supabase-hacker-starter) that wires a [LiveKit Agents](https://docs.livekit.io/agents/) voice agent into [Supabase](https://supabase.com). It uses Postgres with [pgvector](https://github.com/pgvector/pgvector) for similarity search, Supabase Auth for real per-visitor identity, Row Level Security as the data-isolation model, and an Edge Function that produces embeddings *inside* your database, with no third-party embedding provider and no extra API key.

We'll lead with the TypeScript (Node.js) agent, which is the primary runtime in this starter, and there's a fully equivalent Python agent alongside it. Every code excerpt below is pulled straight from the starter so you can follow along in the repo.

## What we're building

The agent demonstrates five Supabase-native patterns:

1. **RAG over pgvector.** Semantic search over a shared knowledge base.
2. **Agentic memory with Reciprocal Rank Fusion.** Per-user memory the agent reads and writes during the conversation, retrieved with hybrid vector + full-text search.
3. **Pre-loaded user context.** The agent loads your profile and prior memories before it says a word.
4. **Function-tool CRUD.** Looking up an order by id, the canonical "call my backend" tool.
5. **Session report persistence.** A structured report written to the database when the call ends.

Here's how the pieces fit together:

![Architecture diagram: a Next.js frontend talks to LiveKit Cloud over WebRTC; LiveKit invokes the voice agent through tool calls; the agent reads and writes Supabase with the secret key and calls the embed Edge Function, which runs inside Supabase.](/blog/content/images/2026/06/supabase-voice-agent-architecture.png)

The browser talks to LiveKit Cloud over WebRTC. LiveKit handles speech-to-text, the LLM, and text-to-speech, and your agent is invoked through tool calls. The agent reads and writes Supabase using the project's **secret key**, and every embedding, for both the seed data and live queries, comes from the `embed` Edge Function running the `gte-small` model right inside Supabase.

## The Supabase-native wins

Before the code, it's worth calling out *why* Supabase is a strong fit for a voice agent, because the architecture leans on each of these:

- **Real identity with zero friction.** Supabase Auth [anonymous sign-ins](https://supabase.com/docs/guides/auth/auth-anonymous) give every visitor a genuine `auth.users` row and a JWT without asking for an email. You get a real user id to scope data by, and a clean upgrade path to a permanent account later.
- **RLS is the security model.** Row Level Security policies (`auth.uid() = user_id`) mean the browser physically cannot read another user's memories. The database enforces it, not your application code.
- **Vector search is just Postgres.** pgvector adds a `vector(384)` column type and an `<=>` cosine-distance operator. An HNSW index makes nearest-neighbour lookups fast, and it lives in the same table as the rest of your data.
- **Hybrid search via Reciprocal Rank Fusion.** Postgres ships full-text search (`tsvector`) in the box. Fusing it with vector search inside a single SQL function gives you retrieval that's better than either branch alone.
- **Embeddings inside the database.** A Supabase Edge Function runs `gte-small` on Supabase's own inference runtime, so embeddings are one HTTP call away and there's no separate provider to sign up for or bill.
- **Modern API keys.** The starter uses Supabase's current [publishable / secret keys](https://supabase.com/docs/guides/api/api-keys) (`sb_publishable_…` in the browser, `sb_secret_…` on the server) rather than the legacy `anon` / `service_role` JWTs.

## Prerequisites

- Node.js 22+ and [pnpm](https://pnpm.io/) 10+
- A [Supabase](https://supabase.com) project (pgvector and the `gte-small` inference model are available on the free tier) plus the [Supabase CLI](https://supabase.com/docs/guides/cli)
- A [LiveKit Cloud](https://cloud.livekit.io) project
- Python 3.11+ and [uv](https://docs.astral.sh/uv/), only if you want to run the Python agent

## Identity: anonymous auth + RLS

Most "who is the user?" stories for agents involve a homemade cookie and a lot of trust. This starter uses real auth instead, and the chain of custody is short and verifiable.

**1. The browser signs in anonymously on load.** A small hook checks for an existing session and, if there isn't one, mints a real user with `signInAnonymously()`:

```ts
// frontend/hooks/useSupabaseSession.ts
const { data: { session } } = await supabase.auth.getSession();

if (!cancelled && !session) {
  const { error } = await supabase.auth.signInAnonymously();
  if (error) {
    console.error('Supabase anonymous sign-in failed:', error.message);
  }
}
```

This creates a genuine `auth.users` row and a JWT, with no email and no password, so the rest of the system has a real `user.id` to work with. The browser client uses the **publishable** key and is fully constrained by RLS:

```ts
// frontend/lib/supabase/client.ts
export function createClient() {
  return createBrowserClient(
    process.env.NEXT_PUBLIC_SUPABASE_URL!,
    process.env.NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY!
  );
}
```

**2. The server is the sole authority for identity.** When the frontend asks for a LiveKit token, the route verifies the Supabase session server-side and reads the user id from the *verified* user, never from the request body:

```ts
// frontend/app/api/token/route.ts
const supabase = createServerClient(SUPABASE_URL, SUPABASE_PUBLISHABLE_KEY, {
  cookies: { /* read req cookies, collect rotated ones */ },
});

const { data: { user }, error } = await supabase.auth.getUser();
if (error || !user) {
  return new NextResponse('Unauthorized: no valid Supabase session', { status: 401 });
}

// The verified Supabase user id (= auth.uid()) is the sole identity.
const userId = user.id;
const metadata = JSON.stringify({ user_id: userId });
const roomConfig = AGENT_NAME
  ? new RoomConfiguration({
      agents: [new RoomAgentDispatch({ agentName: AGENT_NAME, metadata })],
    })
  : new RoomConfiguration();
```

The verified id rides to the agent through [agent dispatch metadata](https://docs.livekit.io/agents/server/agent-dispatch/).

**3. The agent scopes every query by that id.** It connects with the **secret** key, which bypasses RLS, so the discipline of always filtering by the server-verified `user_id` is what keeps users isolated on the backend. RLS, meanwhile, protects the browser, where untrusted code runs.

The result is two independent layers of protection. RLS makes it *impossible* for the browser to read someone else's rows, and the token route makes it impossible for a client to claim an id it didn't earn. The policies are plain SQL:

```sql
-- supabase/migrations/0001_init.sql
create policy "memories_select_own"
  on public.memories for select
  to authenticated
  using (auth.uid() = user_id);
```

A signup trigger gives every new auth user a profile row automatically, including anonymous ones:

```sql
create or replace function public.handle_new_user()
returns trigger
language plpgsql
security definer
set search_path = public
as $$
begin
  insert into public.profiles (id, email)
  values (new.id, new.email)
  on conflict (id) do nothing;
  return new;
end;
$$;

create trigger on_auth_user_created
  after insert on auth.users
  for each row execute function public.handle_new_user();
```

## Pattern 1: RAG over pgvector

The knowledge base is a single table with a 384-dimensional embedding column and an HNSW index for fast cosine search:

```sql
create table public.knowledge (
  id         bigint generated always as identity primary key,
  title      text,
  content    text not null,
  category   text,
  embedding  vector(384),
  created_at timestamptz not null default now()
);

create index knowledge_embedding_hnsw
  on public.knowledge
  using hnsw (embedding vector_cosine_ops);
```

Retrieval is a small SQL function that embeds the query, orders by cosine distance (`<=>`), and takes the top *k*:

```sql
create or replace function public.match_knowledge(
  query_embedding vector(384),
  match_count     int default 3
)
returns table (title text, content text)
language sql
security invoker
set search_path = public
as $$
  select k.title, k.content
  from public.knowledge k
  where k.embedding is not null
  order by k.embedding <=> query_embedding asc
  limit greatest(match_count, 1);
$$;
```

On the agent side, the `search_knowledge` tool embeds the user's question and calls the RPC:

```ts
// agent-ts/src/agent.ts
export async function vectorSearchKnowledge(
  supabase: SupabaseClient,
  query: string,
  limit = 3,
): Promise<KnowledgeHit[]> {
  const queryEmbedding = await embedText(query);
  const { data, error } = await supabase.rpc('match_knowledge', {
    query_embedding: queryEmbedding,
    match_count: limit,
  });
  if (error) {
    throw new Error(`Knowledge search failed: ${error.message}`);
  }
  return (data ?? []) as KnowledgeHit[];
}
```

Voice has one nice touch worth calling out. Knowledge lookups can take a beat, and silence feels broken to a user, so the tool kicks off a brief "I'm looking that up" reply if the search runs past 500 ms, keeping the agent conversational while pgvector works.

## Pattern 2: agentic memory with reciprocal rank fusion

Each memory lives in a **slot**, one row per `(user_id, memory_type)`, so writing `favorite_color` twice replaces the old value rather than piling up duplicates. The `UNIQUE` constraint plus an upsert makes that a one-liner. The table carries both an embedding for semantic search and a generated `tsvector` for full-text search:

```sql
create table public.memories (
  id          bigint generated always as identity primary key,
  user_id     uuid not null references auth.users (id) on delete cascade,
  memory_type text not null,
  content     text not null,
  embedding   vector(384),
  fts         tsvector generated always as (
                to_tsvector('english',
                  coalesce(memory_type, '') || ' ' || coalesce(content, ''))
              ) stored,
  created_at  timestamptz not null default now(),
  updated_at  timestamptz not null default now(),
  unique (user_id, memory_type)
);
```

Writing a memory is an upsert keyed on the slot:

```ts
// agent-ts/src/tools/memory.ts
const { error } = await supabase
  .from('memories')
  .upsert(row, { onConflict: 'user_id,memory_type' });
```

The interesting part is retrieval. Pure vector search is great at meaning (asking "what does the user like to drink?" can surface a stored "oat-milk latte") but can whiff on short, literal values. Pure text search nails the literal hits but misses paraphrases. So the starter fuses both with **Reciprocal Rank Fusion**, ranking each branch independently, then combining with `weight / (k + rank)` and sorting by the fused score. It all happens in one SQL function, scoped to the user:

```sql
create or replace function public.match_memories_hybrid(
  query_embedding vector(384),
  query_text      text,
  p_user_id       uuid,
  match_count     int default 5
)
returns table (memory_type text, content text)
language sql
security invoker
set search_path = public
as $$
  with vector_ranked as (
    select m.id,
      row_number() over (order by m.embedding <=> query_embedding asc) as rank
    from public.memories m
    where m.user_id = p_user_id and m.embedding is not null
    order by m.embedding <=> query_embedding asc
    limit least(greatest(match_count, 1) * 4, 50)
  ),
  text_ranked as (
    select m.id,
      row_number() over (
        order by ts_rank_cd(m.fts, websearch_to_tsquery('english', query_text)) desc
      ) as rank
    from public.memories m
    where m.user_id = p_user_id
      and m.fts @@ websearch_to_tsquery('english', query_text)
    order by ts_rank_cd(m.fts, websearch_to_tsquery('english', query_text)) desc
    limit least(greatest(match_count, 1) * 4, 50)
  ),
  fused as (
    select coalesce(v.id, t.id) as id,
      coalesce(0.7 / (60 + v.rank), 0.0)
        + coalesce(0.3 / (60 + t.rank), 0.0) as score
    from vector_ranked v
    full outer join text_ranked t on v.id = t.id
  )
  select m.memory_type, m.content
  from fused f
  join public.memories m on m.id = f.id
  order by f.score desc
  limit greatest(match_count, 1);
$$;
```

The agent exposes five memory tools (`remember_detail`, `recall_detail`, `forget_detail`, `search_memories`, and `list_user_memories`), and `search_memories` calls the hybrid RPC:

```ts
const { data, error } = await supabase.rpc('match_memories_hybrid', {
  query_embedding: queryEmbedding,
  query_text: query,
  p_user_id: userId,
  match_count: limit,
});
```

There's a deliberate bit of resilience here too. If embedding the query ever fails, the tool degrades to a text-only scan over the generated `fts` column instead of dropping the request. Memory should be the last thing to break.

## Pattern 3: pre-loaded user context

A returning user shouldn't have to reintroduce themselves. Before the agent speaks, it stamps `last_seen_at`, reads back the profile, and appends any remembered facts, all as assistant messages so the model sees them on its very first turn:

```ts
// agent-ts/src/preload.ts
export async function preloadUser(userId: string): Promise<llm.ChatContext> {
  const supabase = getSupabase();
  const now = new Date().toISOString();

  const { data: profile } = await supabase
    .from('profiles')
    .upsert({ id: userId, last_seen_at: now }, { onConflict: 'id' })
    .select('id, name, email, preferences')
    .maybeSingle();

  const chatCtx = new llm.ChatContext();
  // ... add a profile summary message ...

  const memories = await listMemories(supabase, userId);
  if (memories.length > 0) {
    const lines = memories.map((m) => `- ${m.memory_type}: ${m.content}`).join('\n');
    chatCtx.addMessage({
      role: 'assistant',
      content: `Remembered facts from prior sessions:\n${lines}`,
    });
  }
  return chatCtx;
}
```

Timing matters for latency. The agent resolves the user id from dispatch metadata *before* connecting to the room, so the preload query runs in parallel with the WebRTC connection rather than after it:

```ts
// agent-ts/src/main.ts
const userId = resolveUserId(ctx.job.metadata);
ctx.proc.userData.user_id = userId;

const initialChatCtx = await preloadUser(userId);
// ... build the AgentSession ...
await ctx.connect();
```

`resolveUserId` reads `user_id` from the metadata and falls back to `DEMO_USER_ID` when there isn't any, which is exactly the case when you're testing locally without the frontend (more on that below).

## Pattern 4: function-tool CRUD

This is the "call my backend" pattern, and it's intentionally plain, just a tool that reads a row by a business key. Orders are shared demo data (readable by any authenticated visitor) and looked up by `order_id`:

```ts
lookup_order: llm.tool({
  description: 'Look up an order by its ID. Returns items, total, and status.',
  parameters: z.object({
    order_id: z.string().describe('The order ID to look up.'),
  }),
  execute: async ({ order_id }) => {
    const supabase = getSupabase();
    const { data: order, error } = await supabase
      .from('orders')
      .select('order_id, items, total, status')
      .eq('order_id', order_id)
      .maybeSingle();
    if (error) {
      throw new llm.ToolError(`Failed to look up order ${order_id}: ${error.message}`);
    }
    if (!order) {
      throw new llm.ToolError(`Order ${order_id} not found.`);
    }
    return JSON.stringify(order);
  },
}),
```

Swap `orders` for any table in your domain and you've got the shape for inventory checks, account lookups, booking status, and anything else the agent needs to fetch on demand.

## Pattern 5: session report persistence

When a call ends, the agent writes a structured report to the `sessions` table from a shutdown callback. Because the agent holds the secret key, the insert succeeds even though `sessions` has *no* insert policy for end users, so only the backend can write session history:

```ts
// agent-ts/src/main.ts
ctx.addShutdownCallback(async () => {
  try {
    const report = ctx.makeSessionReport(session);
    const supabase = getSupabase();
    const { error } = await supabase.from('sessions').insert({
      session_id: ctx.room.name,
      user_id: ctx.proc.userData.user_id,
      room_name: ctx.room.name,
      report: voice.sessionReportToJSON(report),
    });
    if (error) throw new Error(error.message);
    console.info(`Persisted session report for ${ctx.room.name}`);
  } catch (err) {
    console.error('Failed to persist session report', err);
  }
});
```

The owner-only `SELECT` policy means a user can later read their own session history through the browser, but the writes stay locked to the backend.

## Embeddings inside Supabase: the `embed` edge function

Every embedding in the starter, for seed data and for live queries alike, comes from one place. A Supabase Edge Function runs `gte-small` on Supabase's built-in inference runtime. Defining the model and dimensions in exactly one spot keeps the seed script and both agent runtimes in sync.

```ts
// supabase/functions/embed/index.ts
const session = new Supabase.ai.Session('gte-small');

Deno.serve(async (req: Request): Promise<Response> => {
  // ... validate method + auth + body ...
  const inputs = Array.isArray(input) ? input : [input];

  const embeddings: number[][] = [];
  for (const text of inputs) {
    const embedding = (await session.run(text, {
      mean_pool: true,
      normalize: true,
    })) as number[];
    embeddings.push(embedding);
  }
  return json({ embeddings }, 200);
});
```

It takes `{ input: string | string[] }` and always returns `{ embeddings: number[][] }`, one 384-dim, unit-length vector per input, ready for cosine distance. Callers hit it over plain HTTP:

```ts
// agent-ts/src/tools/embeddings.ts
const response = await fetch(functionUrl(), {
  method: 'POST',
  headers: {
    apikey: key,
    Authorization: `Bearer ${key}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({ input }),
});
```

Authorization is worth a note, because it trips people up. Supabase's modern API keys (`sb_publishable_…` / `sb_secret_…`) are **not** JWTs, so the platform's built-in `verify_jwt` gateway check can't validate them. Per the Supabase docs, you deploy the function with `verify_jwt = false` and authorize the request yourself, here by requiring the secret key on the `apikey` header:

```bash
supabase functions deploy embed --no-verify-jwt
```

That keeps the endpoint from being trivially open. For production you'd harden further. The function supports an `EMBED_SECRET` for an exact-match check, and you can layer on network or rate-limit policies.

## Setup

From the repo root:

```bash
pnpm setup                       # installs packages, copies .env.local files

# link the Supabase CLI to your project once
supabase link --project-ref <your-project-ref>

pnpm db:migrate                  # apply schema, RLS, RPCs, indexes, triggers
pnpm db:deploy-functions         # deploy the gte-small `embed` Edge Function
pnpm db:seed                     # knowledge docs, orders, and a demo user
```

There are two things to do in the Supabase dashboard. First, **enable Anonymous sign-ins** under *Authentication → Providers*, since the frontend depends on it. Second, grab your keys from *Project Settings → API Keys* (the **Publishable** and **Secret** tabs). The secret key goes in the agent and `db` env files, and the publishable key goes in the frontend.

`pnpm db:seed` creates a demo auth user and prints its UUID:

```ts
// db/seed.ts
console.info('Copy the following into agent-ts/.env.local and agent-py/.env.local:');
console.info(`DEMO_USER_ID=${demoUserId}`);
```

Copy that into `DEMO_USER_ID` so local runs have a user to preload. Knowledge and orders are shared demo data that work for any visitor. Profiles, memories, and sessions start empty for a real anonymous visitor and fill in as they talk.

## Running and testing

Start the agent and the frontend together:

```bash
pnpm dev:ts             # Node agent (dev mode) + frontend
```

To test the agent on its own, run it in dev mode and connect from the **hosted Agent Console** in the LiveKit Cloud dashboard, a browser-based console that talks to your locally running agent with no custom frontend required:

```bash
pnpm dev:ts             # then, in LiveKit Cloud: Agents → Launch Console
```

Set the **Agent name** to `my-agent` and start talking. Because there's no frontend in this flow, there's no dispatch metadata, so the agent falls back to `DEMO_USER_ID` for the preload path, which is exactly why the seed script prints it. The Agent Console works with locally running agents on the current SDKs (Node ≥ 1.2.4, and this starter uses 1.4.5).

## The Python agent

The starter ships a fully equivalent Python agent at `agent-py/`, built on `supabase-py` and the same Canonical Contract, with identical tables, RPCs, and the shared `embed` function. It registers under the same `my-agent` name, so run one runtime at a time. The patterns map cleanly onto the Python SDK's `@function_tool` decorators:

```python
# agent-py/src/agent.py
@function_tool()
async def lookup_order(self, context: RunContext, order_id: str) -> str:
    client = await get_client()
    # ... select from orders by order_id ...

@function_tool()
async def search_knowledge(self, context: RunContext, query: str) -> str:
    client = await get_client()
    results = await _search_knowledge(client, query, limit=3)
    # ...
```

If you prefer Python, it's a first-class path. As a bonus, the Python SDK has a built-in **terminal console** (`pnpm agent:py:console`) that runs the agent entirely in your terminal, handy for quick iteration without a browser.

## Next steps

This starter is intentionally minimal so it's easy to build on. A few natural directions:

- **Upgrade anonymous users to real accounts.** An anonymous session can be promoted in place with `supabase.auth.updateUser({ email, password })` or `linkIdentity()`, and the `user_id` is preserved, so every memory and profile field carries over. Wire it into the frontend when a visitor wants to come back as themselves.
- **Add multi-tenancy.** Introduce an `org_id` column, extend the RLS policies to scope by organization as well as user, and pass the tenant through the same verified dispatch-metadata path.
- **Harden the `embed` function.** Set a dedicated `EMBED_SECRET` for an exact-match check, restrict who can invoke it, and add rate limiting before you point production traffic at it.
- **Swap in a different embedding model.** Change the model in the `embed` function and `EMBEDDING_DIMENSIONS` in the agent, then update the `vector(N)` columns and HNSW indexes to match. A local or larger model is a one-function change because embeddings are centralized.

## Wrapping up

With Supabase behind it, a voice agent gets real identity (anonymous auth), real isolation (RLS), real retrieval (pgvector + RRF hybrid search), and a place to remember things, all without standing up a separate vector database or embedding service. The five patterns here cover the parts most agents need, namely knowledge, memory, context, backend calls, and history.

Clone the [starter](https://github.com/livekit-examples/supabase-hacker-starter), point it at a Supabase project and a LiveKit Cloud project, run `pnpm dev:ts`, and start talking. Then make it yours. The schema, tools, and instructions are all a few lines of edits away.

## Get started with LiveKit

Ready to build your own voice agent? [Sign up for a free LiveKit Cloud account](https://cloud.livekit.io/signup) to get model inference, agent deployment, and realtime transport in one place, then point this starter at your project and start talking.