Meeting Transcript to Action Items: an LLM Pipeline

A client call runs forty minutes, the account manager writes half a page of notes, and three of the seven commitments made on that call never reach anybody's task board. That is the gap the meeting pipeline in Opaeron exists to close: paste the raw transcript, get back action items that already carry an owner, a due date, the account they belong to, and the verbatim line of the transcript they came from.

#The transcript is usually the only record

Opaeron is the internal operations platform I build at Trilliant Digital, a digital marketing agency. It is a large system by now: measured against the repository today it runs to about 226,000 lines of code, 711 API route handlers across 89 routers, 77 Postgres tables, and 112 agent module files spanning eleven marketing disciplines. The meeting pipeline is one of the smallest features in it and one of the most used, because it sits on the one artefact every client call reliably produces.

The design assumptions came from data, not vibes. Most meetings never generate an agenda you can parse: Microsoft's telemetry puts 57% of meetings as ad hoc calls with no calendar invite at all, and Atlassian found 62% of workers regularly attend meetings where no goal was stated in the invite. So extraction has to infer structure rather than read it off a plan. And the demand is real: Zoom's own research reports that only 39% of employees receive post-meeting summaries and action items while 54% want them.

That last number sets the tone for the whole prompt design. Benchmark speech recognition looks solved from a distance: the Open ASR Leaderboard puts Whisper large-v3 at 7.44% word error rate on its short form English composite and 6.43% on the long form track. Real conference room audio is a different problem. The original Whisper paper measures 16.9% word error rate on the AMI meeting corpus with individual headset mics and 36.4% with a single distant mic. Roughly one word in three is wrong in the realistic case, which means the extraction prompt is not reading clean minutes, it is reading a damaged text and has to stay calm about it.

There is no bot joining the call. Clients would have to consent to a recorder in the room, and half of these conversations happen on the client side platform rather than ours. So the input is whatever transcript the account manager can get hold of: a Meet or Zoom export, a Whisper run someone did locally, sometimes a rough manual write up pasted into the same box. The pipeline has to accept all three, which is another argument for parsing speaker turns defensively and treating an unparsable line as a continuation of the previous speaker instead of an error.

The feature itself is one page and one endpoint. An account manager opens the account, pastes the transcript into a text area, picks the meeting date, and hits import. The endpoint returns a review table rather than writing anything: proposed action items with an owner, a due date and the transcript line each one came from, each with a checkbox. Nothing lands on the task board until a person confirms it. That review step is not a hedge against a weak model, it is the thing that makes people willing to use the feature at all, because the cost of a bad automated write into a shared board is far higher than the cost of one extra click.

#Chunking into model safe segments

Claude Haiku 4.5 has a 200k token context window, so a two hour transcript fits in a single call with room to spare. Fitting is not the same as being read well. When I ran whole transcripts as one prompt, recall sagged in the middle: commitments made in the first ten minutes were quietly dropped in favour of the ones nearest the end of the text. Splitting the transcript and running extraction per segment produced flatter recall across the meeting, and made each call fast enough to fan out.

The split has to respect speaker turns. Cutting mid sentence loses the antecedent for half the pronouns in the chunk, and a commitment like "yeah we'll get that over by Thursday" is worthless without the turn above it. So the chunker parses turns first, packs them into a token budget, and carries a few turns of overlap into the next chunk so a commitment straddling a boundary appears whole in at least one segment.

The 6,000 token budget came out of testing rather than theory. I took a set of client transcripts, hand labelled the commitments in them, and ran the same extraction at several chunk sizes. Recall stayed flat from roughly 3,000 to 8,000 tokens per chunk and fell away above that; below 3,000 the call count climbed and cross turn context started to be lost at the seams. Six thousand sat in the middle of the flat region with the fewest calls. The character to token ratio of 3.6 is likewise measured on our own transcripts, which are conversational English with speaker labels and timestamps, and it is deliberately a cheap estimate: it only has to be accurate enough to keep chunks inside a budget with headroom, not to be a tokenizer.

import re
from dataclasses import dataclass

# "[00:14:02] Priya Nair: let's push the reels batch to next week"
TURN_RE = re.compile(
    r"^\s*(?:\[?\d{1,2}:\d{2}(?::\d{2})?\]?\s*)?"   # optional timestamp
    r"([A-Za-z][\w .'-]{1,48})\s*:\s*(.+)quot;
)

CHARS_PER_TOKEN = 3.6      # measured on our own client transcripts, not a guess
CHUNK_TOKEN_BUDGET = 6000  # far below the 200k window; recall, not capacity, drives this
OVERLAP_TURNS = 4


@dataclass(slots=True)
class Turn:
    speaker: str
    text: str
    raw: str


def parse_turns(transcript: str) -> list[Turn]:
    turns: list[Turn] = []
    for line in transcript.splitlines():
        if not line.strip():
            continue
        m = TURN_RE.match(line)
        if m:
            turns.append(Turn(m.group(1).strip(), m.group(2).strip(), line))
        elif turns:
            # continuation of the previous speaker, keep it attached
            turns[-1].text += " " + line.strip()
            turns[-1].raw += "\n" + line
        else:
            turns.append(Turn("Unknown", line.strip(), line))
    return turns


def chunk_transcript(transcript: str) -> list[dict]:
    turns = parse_turns(transcript)
    chunks, buf, budget = [], [], 0

    def flush(tail: list[Turn]) -> list[Turn]:
        chunks.append({
            "index": len(chunks),
            "speakers": sorted({t.speaker for t in buf}),
            "text": "\n".join(t.raw for t in buf),
        })
        return tail

    for turn in turns:
        cost = len(turn.raw) / CHARS_PER_TOKEN
        if buf and budget + cost > CHUNK_TOKEN_BUDGET:
            carry = buf[-OVERLAP_TURNS:]
            flush(carry)
            buf = list(carry)
            budget = sum(len(t.raw) / CHARS_PER_TOKEN for t in buf)
        buf.append(turn)
        budget += cost

    if buf:
        flush([])
    return chunks

#The extraction call and its schema

Each chunk goes through one extraction call. The output shape is a Pydantic v2 model, which matters because the whole backend is FastAPI on Python 3.11 with Pydantic v2 already, so the same model validates the LLM response and the API payload. Anything the model returns that fails validation is retried once with the validation error appended to the prompt, then dropped.

The single most useful field in that schema is evidence. Every action item must carry a verbatim span copied out of the chunk, and after parsing I check that the span actually appears as a substring of the source text. That one check removed most of the plausible sounding inventions, because a model that has to quote the transcript cannot easily invent a commitment that was never made. It also gives the UI something to show: clicking an action item scrolls the transcript to the line it came from.

Temperature is pinned at zero and the failure handling is boring on purpose. A response that does not parse, or that fails Pydantic validation, is retried once with the validation error appended as an extra user message, which recovers most malformed cases. A second failure logs the chunk id and returns nothing for that chunk rather than raising, because losing one segment of a meeting is survivable and failing the whole import is not. The most common validation failure I saw was the model returning a bare list instead of an object with an items key, and the retry fixes that nearly every time.

from datetime import date
from typing import Literal
from pydantic import BaseModel, Field, field_validator


class ActionItem(BaseModel):
    title: str = Field(max_length=140)
    detail: str = ""
    owner_hint: str = ""                       # a name or role as spoken, not a user id
    due_hint: str = ""                         # "next Thursday", "before the 15th", ""
    due_date: date | None = None               # resolved server side, never by the model
    priority: Literal["low", "medium", "high"] = "medium"
    discipline: Literal[
        "seo", "smm", "abm", "email", "cro",
        "paidads", "content", "brand", "analytics", "ops", "website",
    ] | None = None
    evidence: str = Field(min_length=8)        # verbatim span from the chunk

    @field_validator("title")
    @classmethod
    def no_trailing_period(cls, v: str) -> str:
        return v.strip().rstrip(".")


class ChunkExtraction(BaseModel):
    items: list[ActionItem] = []


SYSTEM = """You read one segment of a client meeting transcript for a marketing agency
and return only the commitments made in it.

Rules:
1. A commitment is something a named person or team agreed to DO. Opinions,
   status recaps and questions are not commitments.
2. Copy the exact words that prove the commitment into "evidence". Do not
   paraphrase that field.
3. The transcript comes from automatic speech recognition and contains
   misheard words. If a line is garbled but the intent is clear, extract it
   and keep the garbled span as evidence. If the intent is not clear, skip it.
4. Never compute a calendar date. Put the spoken phrase in "due_hint".
5. Returning {"items": []} is a correct answer for a segment with no
   commitments in it. Most segments have none.

Team members you may name as owner_hint, and nobody else:
{roster}

Return JSON matching this schema and nothing else:
{schema}"""


async def extract_chunk(client, chunk: dict, roster: str) -> list[ActionItem]:
    resp = await client.chat.completions.create(
        model="anthropic/claude-haiku-4.5",
        temperature=0,
        max_tokens=2000,
        response_format={"type": "json_object"},
        messages=[
            {"role": "system", "content": SYSTEM.format(
                roster=roster,
                schema=ChunkExtraction.model_json_schema(),
            )},
            {"role": "user", "content": chunk["text"]},
        ],
    )
    parsed = ChunkExtraction.model_validate_json(resp.choices[0].message.content)
    source = chunk["text"].lower()
    # drop anything the model could not actually quote
    return [i for i in parsed.items if i.evidence.lower()[:60] in source]

Two prompt rules earn their place. Rule 5 tells the model that an empty array is a correct answer, which removes the quiet pressure to produce something for a segment that is only greetings and small talk. Rule 4 forbids date arithmetic entirely. Models are unreliable at resolving "next Thursday" against a meeting date, and I already have the meeting date in Postgres, so a small deterministic resolver handles it with the account's timezone and gets it right every time.

#Account context injection and owner assignment

An extraction that says "Priya to send the reels calendar" is only useful if the system knows which Priya, on which account, in which pod. Opaeron models the agency as client pods (Beacons, Dyota, Elites, Nexus) that own accounts end to end, and practice pods (Maverick for social, Ajna for paid, Alpha for SEO) that cut across every account. Before any extraction call runs, I build a roster block for the specific account and paste it into the system prompt.

Half of that block is a straight SQL read: who sits on this account, in which pod, with which role. The other half is retrieval. Recent briefs, the last two meeting summaries and open deliverables for the account are embedded with openai/text-embedding-3-small and stored in Postgres 16 running the pgvector image, so I can pull the handful of prior notes closest to the current chunk. That is what lets the model connect "the landing page thing we discussed" to an actual open deliverable instead of creating a vague new task.

That retrieval sits on infrastructure I moved in July 2026, when Opaeron came off Supabase onto self hosted PostgreSQL 16 running the pgvector image. Rather than rewrite hundreds of route handlers, I wrote a Supabase style query builder shim over psycopg3 with a connection pool, so existing call sites kept working unchanged and the vector search simply pointed at a local table. The retrieval query is narrow by design: the top five prior notes for this account only, filtered by account id in SQL before the vector comparison, never across accounts. Client data separation is a hard boundary in an agency platform, and a similarity search that ignores it is a leak waiting to happen.

Owner resolution itself is deliberately not the model's job. The model emits a free text owner_hint and a discipline; a rules layer maps that to a real user id.

from rapidfuzz import process, fuzz

# keyword to discipline, checked when the model gives no discipline at all
KEYWORD_ROUTES = (
    ("smm",     ("instagram", "reel", "carousel", "community", "content calendar")),
    ("paidads", ("google ads", "cpc", "roas", "pmax", "bid", "budget cap")),
    ("seo",     ("backlink", "schema markup", "core web vitals", "serp", "keyword")),
    ("email",   ("newsletter", "drip", "klaviyo", "open rate", "sequence")),
    ("website", ("landing page", "wordpress", "staging", "form", "page speed")),
)

DISCIPLINE_POD = {"smm": "Maverick", "paidads": "Ajna", "seo": "Alpha"}


def route_discipline(item) -> str | None:
    if item.discipline:
        return item.discipline
    blob = f"{item.title} {item.detail} {item.evidence}".lower()
    for discipline, keywords in KEYWORD_ROUTES:
        if any(k in blob for k in keywords):
            return discipline
    return None


def resolve_owner(item, roster: list[dict], actor_id: str) -> str | None:
    """roster rows: {user_id, name, aliases, pod, client_pod, open_tasks}"""
    if item.owner_hint:
        names = [r["name"] for r in roster]
        match = process.extractOne(
            item.owner_hint, names, scorer=fuzz.WRatio, score_cutoff=82
        )
        if match:
            return roster[names.index(match[0])]["user_id"]

    discipline = route_discipline(item)
    pod = DISCIPLINE_POD.get(discipline)
    if pod:
        candidates = [r for r in roster if r["pod"] == pod]
        if candidates:
            # lightest load on the practice pod wins
            return min(candidates, key=lambda r: r["open_tasks"])["user_id"]

    # no confident owner: park it on whoever ran the import, never guess
    return actor_id

Fuzzy name matching at a WRatio cutoff of 82 handles the reality of speech recognition output, where a name is as likely to arrive misspelled as not. When nothing matches confidently the item is parked on the person who pasted the transcript rather than assigned to a plausible stranger. A wrong owner is worse than no owner, because a wrongly assigned task looks handled on the board and is not.

Every write from this path still passes through pod_permissions.py. Opaeron self hosts Postgres without Row Level Security, so authorization lives entirely in application code against a super admin editable permission matrix stored in tdx_app_settings. The extraction pipeline gets no exemption from it: if the importing user cannot create tasks on that account, the items are held in a review state instead of being written.

Every write the pipeline makes is attributed in the audit log to the person who ran the import, not to a service account. When an account manager asks why a task landed on their queue, the answer has to trace back to a specific import and a specific line of transcript, and a generic system actor makes that impossible. Opaeron also has a View-As mode for super admins, and the meeting importer is one of the places where seeing exactly what a given user is permitted to create saved a lot of guessing during rollout.

#Deduplication across chunks

Chunk overlap guarantees duplicates, and long meetings produce a second kind: the same commitment restated near the end of the call as a recap. Both need collapsing, and the recap version is usually the better worded one, so the merge keeps the longest title and the earliest evidence.

Exact string matching catches almost nothing here, because the two phrasings differ. A two stage pass works well: a cheap normalized token set ratio first, then cosine similarity over embeddings only for pairs that the cheap stage rates as borderline. On our transcripts a cosine threshold of 0.86 sat comfortably between restatements of one commitment and genuinely distinct commitments about the same deliverable.

import re
import numpy as np
from rapidfuzz import fuzz

CHEAP_MERGE = 92      # token set ratio above this: same item, no embedding needed
CHEAP_SKIP = 55       # below this: definitely different, no embedding needed
COSINE_MERGE = 0.86   # tuned on our own labelled transcripts

STOP = {"the", "a", "an", "to", "for", "on", "we", "will", "please", "and"}


def norm(text: str) -> str:
    words = re.findall(r"[a-z0-9]+", text.lower())
    return " ".join(w for w in words if w not in STOP)


async def dedupe(items: list, embed) -> list:
    kept: list = []
    keys = [norm(f"{i.title} {i.detail}") for i in items]
    vectors: dict[int, np.ndarray] = {}

    for idx, item in enumerate(items):
        duplicate_of = None
        for k_idx, existing in enumerate(kept):
            cheap = fuzz.token_set_ratio(keys[idx], keys[items.index(existing)])
            if cheap >= CHEAP_MERGE:
                duplicate_of = k_idx
                break
            if cheap <= CHEAP_SKIP:
                continue
            for i in (idx, items.index(existing)):
                if i not in vectors:
                    vectors[i] = await embed(keys[i])   # text-embedding-3-small
            a, b = vectors[idx], vectors[items.index(existing)]
            if float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b))) >= COSINE_MERGE:
                duplicate_of = k_idx
                break

        if duplicate_of is None:
            kept.append(item)
        else:
            winner = kept[duplicate_of]
            # keep the fuller title, the firmest due hint, the earliest evidence
            if len(item.title) > len(winner.title):
                winner.title = item.title
            winner.due_hint = winner.due_hint or item.due_hint
            winner.owner_hint = winner.owner_hint or item.owner_hint
    return kept

Skipping the embedding call for pairs above 92 or below 55 on the cheap stage removes most of the network round trips. On a typical hour long transcript that leaves a few dozen embeddings instead of a few hundred pairwise calls, and the whole dedupe pass finishes well inside the time budget of the synchronous request.

Picking 0.86 was not guesswork either. I labelled a few hundred pairs drawn from real transcripts as same or different, looked at where cosine similarity fell for each group, and took the value that separated them with the fewest false merges. False merges are the expensive error here: two distinct commitments collapsed into one means a task silently disappears, while a duplicate that survives is visible in the review table and deleted in a second. The threshold is biased slightly high for that reason.

#The summary pass runs asynchronously

The narrative summary is a second, separate pass, and it does not block the response. Action items are what the user is standing there waiting for; the summary is what they will read later. The API returns the deduped items immediately and enqueues a Celery task, using the same Celery plus Redis setup that already runs deadline reminders every fifteen minutes, nightly recurrence regeneration and the nightly Postgres to Google Sheets backup.

Mechanically it is a task id returned alongside the items, a row in Postgres holding the summary status, and the frontend (React 19 with Vite 7 and React Router 7) polling that row until it flips to ready. If the worker fails, the meeting keeps its action items and shows the summary as unavailable with a retry button, because the two halves are independent and there is no reason for a slow narrative pass to take the useful output down with it.

The summary model is google/gemini-2.5-flash rather than the extraction model, because summary writing rewards fluency and cheap long context, while extraction rewards obedience to a schema. The summary prompt receives the full transcript plus the already resolved action items, so the narrative and the task list cannot contradict each other. The business case for bothering is direct: in Otter's survey, 71% of workers said they would skip unnecessary meetings if quality notes were shared with them promptly afterward.

The text guard on every generated response

Everything the summary pass produces goes through backend/shared/text_guard.py before a user sees it. It is a global module, applied to every AI response in the platform, not just this feature. It does two jobs: it rewrites the stock marketing cliches that models reach for by default, and it strips em dashes out of the text entirely.

The em dash rule started as a personal irritation and turned out to be the strongest single tell. Client facing copy written by our team almost never contains one; copy written by a model is full of them. Removing them, plus a substitution table for the handful of phrases that recur in every model's output, is enough that a meeting summary reads like an account manager wrote it in a hurry rather than like a press release. It is not a content filter and it does not change meaning, it only removes the register that gives the machine away.

The module is a pure function over a string, applied at the last step before serialization, which keeps it out of every prompt. Pushing style rules into prompts is tempting and it does not hold: the instruction competes with the actual task, it costs tokens on every single call, and models drift back to their defaults on longer outputs. A deterministic pass after generation costs nothing per call, is unit testable, and applies identically across all twenty or so user facing agents in the platform no matter which model produced the text.

#Why a Haiku class model for extraction

All model traffic in Opaeron goes through OpenRouter as a multi-model gateway, so switching the extraction model is a string change. That made it cheap to test the choice properly rather than argue about it. The original build used anthropic/claude-3.5-haiku; that model was retired in February 2026 and the pipeline now runs on anthropic/claude-haiku-4.5, which slotted in without a prompt rewrite.

That retirement turned out to be a useful test of the gateway design. When the old model id started returning 404, the fix was a string change and a re-run of the labelled transcript set to confirm recall had not regressed. Nothing about the prompt, the schema or the validation moved. Routing every model call through one gateway module with a per feature model id held in configuration, rather than scattering client code across thirty two backend feature modules, is why that migration took an afternoon instead of a sprint.

• Cost per meeting is the deciding factor. Extraction is a fan out: one call per chunk, several chunks per meeting, many meetings a week across every account. At Anthropic's published pricing of $1 per million input tokens and $5 per million output tokens, Haiku 4.5 sits at half of Sonnet 5 input pricing and a fifth of Opus 5 input pricing, and a meeting costs cents rather than dollars.
• Schema obedience beats reasoning depth here. The task is not hard reasoning, it is disciplined reading with a strict output contract. Larger models did not find meaningfully more commitments in my comparisons, they just cost more and took longer.
• Latency compounds across chunks. Even fanned out with asyncio, per call latency sets the floor on how fast the user sees a task list, and a small model keeps that floor low.
• Context window is not the binding constraint. Haiku 4.5 offers a 200k window and 64k max output, both far beyond what a 6,000 token chunk needs. Recall behaviour, not capacity, is what drove the chunk size.

The rest of the platform uses different models for different shapes of work for the same reason: google/gemini-2.5-flash for chat and brief writing, google/gemini-2.5-flash-lite for cheap classification, openai/gpt-4o-mini where a second opinion helps, and openai/text-embedding-3-small for embeddings. LangGraph orchestrates the agents that need multi step control flow. The meeting pipeline is simple enough that plain asyncio fan out beat a graph.

#What held up in production

• Force the model to quote. The mandatory verbatim evidence span, verified as a substring after parsing, did more for output trust than any amount of prompt tightening.
• Keep dates and identities out of the model. Date resolution and owner lookup are deterministic code reading the account row and the roster. The model supplies hints, the backend supplies facts.
• Chunk for recall, then pay for it in dedupe. Overlapping chunks plus a cheap-then-embedding merge is a better trade than one long call that quietly forgets the first ten minutes.
• Split synchronous from asynchronous by what the user is waiting on. Tasks now, summary in a minute.
• Write the prompt for damaged input. With word error rates on real meeting audio measured at 16.9% to 36.4%, an extractor that assumes clean text is being tested on a case that does not occur.
• Post-process the register. A shared text guard applied to every generated response is the cheapest quality control in the whole platform.