Extract tasks from meetings

Converting meeting recordings into tasks

The meeting productivity problem

Your team finishes an hour-long meeting. Six people agreed to do twelve things. By next week, three get done, five are forgotten, and four were misunderstood.

Meetings generate lots of work, but without tracking, action items disappear. You’ve seen the cycle - repeated meetings where people ask “wait, who was supposed to do that?” because nobody captured tasks from the last one.

How Tallyfy will solve this

We’re building an integration that connects your meeting platform (Zoom, Teams, Google Meet) to Tallyfy’s task system. Record your meeting as usual - Tallyfy handles the rest.

The system will transcribe your meeting using AI that works with multiple languages, then analyze the conversation to identify who committed to what and when.

Identity recognition

“John, can you send the proposal to the client by Friday?”

Simple for humans. Surprisingly hard for AI.

The planned system will identify both the task creator (the person asking) and the assignee (John) using SSO integration. When your team logs into both the meeting platform and Tallyfy with the same SSO provider, voices get matched to user profiles automatically. John gets assigned the task. The requester becomes the creator.

If someone joins as a guest or SSO isn’t configured, the system will create placeholder assignments you can map to real users during review.

Rich task context, not just titles

A task titled “Send proposal” tells you nothing. The system will capture full context:

Task Title: Send Q4 budget proposal to ACME Corp Description: During the October 15 planning meeting, John committed to sending the revised Q4 budget proposal to ACME Corp’s CFO. Meeting Context:

Quoted from recording at 23:45: “John, can you send the updated proposal with the new pricing structure to their CFO? They need it for their board meeting.”
AI Rationale: Identified as a task due to specific assignee (John), clear deliverable (proposal), explicit deadline (Friday), and confirmation response (“Sure, I’ll get that over by Thursday”). Deadline: October 18, 2024 at 5:00 PM Assignee: John Smith Creator: Sarah Johnson

Every extracted task will include the actual quote from the meeting and the AI’s reasoning for flagging it as a task.

Conversation thread tracking

Conversations aren’t linear. Someone proposes an idea at minute 5. It gets modified at minute 15. By minute 30, the team decides something different. Naive extraction would create three conflicting tasks.

The AI will track conversation threads throughout the meeting and understand when action items evolve, get canceled, or change - delivering only the final agreed-upon version of each task.

The planned Draft state

AI isn’t perfect. That’s why we plan to introduce a new task state: Draft.

Draft tasks would work like a moderation queue - all extracted tasks appear grouped together for review. You’d edit assignments, adjust deadlines, delete irrelevant items, and convert good ones to active tasks.

This human-in-the-loop approach means AI handles extraction and organization while you provide the final quality check.

Technical challenges

Accuracy at scale

Current systems face several hurdles:

Background noise and cross-talk reduces transcription accuracy
Technical jargon causes misinterpretation
Multiple speakers at once creates attribution errors
Long meetings exceed AI context windows

The hallucination problem

Language models sometimes generate tasks that were never discussed - invented from context clues rather than explicit conversation.

The planned approach uses multiple validation layers: requiring explicit transcript evidence for every task, confidence scoring with warning flags on low-confidence items, and human review before tasks go live.

Meeting evolution patterns

Conversations follow common patterns that affect extraction:

Progressive refinement: Ideas evolve through discussion before becoming tasks
Implicit delegation: “Someone should…” becomes “John will…” over time
Conditional commitments: “If X happens, then I’ll do Y” requires tracking conditions
Retroactive cancellation: “Actually, let’s not do that” negates earlier commitments

Planned platform support

Phase 1 - major video conferencing

Zoom, Microsoft Teams, Google Meet, Cisco Webex

Phase 2 - specialized platforms

Slack Huddles, Discord, GoToMeeting, BlueJeans

Phase 3 - recording tools

Otter.ai, Fireflies.ai, Sembly AI, Jamie

Phase 4 - async collaboration

Loom, Vidyard, BombBomb, Soapbox

Current workaround - build it yourself

You can build your own meeting-to-tasks pipeline today using Tallyfy’s API. This requires technical expertise and ongoing maintenance.

What you’ll need

You’re responsible for:

Capturing and storing meeting recordings
Transcribing audio to text (using OpenAI Whisper, Google Speech-to-Text, etc.)
Extracting action items from transcripts (using GPT-4 or Claude)
Mapping speakers to Tallyfy users
Creating tasks via Tallyfy’s API
Handling errors and retries

Pipeline architecture

Capture meeting recording

// Option A: Webhook from Zoom/Teams
app.post('/webhook/meeting-ended', async (req, res) => {
  const recordingUrl = req.body.download_url;
  await processRecording(recordingUrl);
});

// Option B: Poll cloud storage
const recordings = await checkGoogleDrive('/meeting-recordings');

Transcribe audio

import openai

def transcribe_meeting(audio_file):
    with open(audio_file, "rb") as f:
        transcript = openai.Audio.transcribe(
            model="whisper-1",
            file=f,
            response_format="verbose_json"  # Includes timestamps
        )
    return transcript

Extract action items with AI

def extract_tasks(transcript):
    prompt = """
    Extract action items from this transcript.
    Return JSON array with: task_name, assignee, deadline, context
    Only include items with clear owners and deadlines.
    Include quote from transcript as evidence.
    """

    response = openai.ChatCompletion.create(
        model="gpt-4",
        messages=[
            {"role": "system", "content": prompt},
            {"role": "user", "content": transcript}
        ],
        temperature=0.3  # Lower temperature for consistency
    )

    return json.loads(response.choices[0].message.content)

Map to Tallyfy users

const userMapping = {
  "John Smith": "user_abc123",
  "Sarah Johnson": "user_def456",
  "john@company.com": "user_abc123"
};

function resolveAssignee(extractedName) {
  if (userMapping[extractedName]) {
    return userMapping[extractedName];
  }

  const emailMatch = Object.keys(userMapping).find(key =>
    key.includes('@') && key.toLowerCase().includes(extractedName.toLowerCase())
  );

  return emailMatch ? userMapping[emailMatch] : null;
}

Create tasks in Tallyfy

async function createTallyfyTasks(tasks, meetingTitle) {
  const firstTask = tasks[0];
  const response = await fetch(
    `https://go.tallyfy.com/api/organizations/${orgId}/tasks`,
    {
      method: 'POST',
      headers: {
        'Authorization': `Bearer ${accessToken}`,
        'X-Tallyfy-Client': 'APIClient',
        'Content-Type': 'application/json'
      },
      body: JSON.stringify({
        title: `Tasks from: ${meetingTitle}`,
        task_type: 'task',
        owners: { users: [firstTask.assigneeId] },
        separate_task_for_each_assignee: true,
        deadline: firstTask.deadline
      })
    }
  );

  const process = await response.json();
  const processId = process.data.run.id;

  // Add remaining tasks to same process
  for (let i = 1; i < tasks.length; i++) {
    await createTask(tasks[i], processId);
  }
}

Extraction patterns that work

Explicit action patterns:

“[Name] will/to [action] by [date]”
“Action item: [description]”
“Next steps: [list]”
”@[name] please [action]”

Temporal markers:

“by end of day” - today at 5 PM
“next Friday” - calculate from meeting date
“ASAP” - 2 business days default
“before the board meeting” - check calendar API

Assignment indicators:

Direct: “John, can you…”
Confirmation: “I’ll handle…”
Delegation: “Let’s have Sarah…”
Volunteering: “I can take that”

Handling edge cases

# Multiple assignees
if "and" in assignee_text or "," in assignee_text:
    assignees = parse_multiple_assignees(assignee_text)
    create_task_for_each(assignees)

# Conditional tasks
if "if" in task_text or "assuming" in task_text:
    task.add_note("Conditional: " + extract_condition(task_text))

# Rejected tasks
if check_for_cancellation(transcript, task_timestamp):
    skip_task(task)  # Don't create if later rejected

# Vague deadlines
deadline = parse_deadline(deadline_text)
if not deadline:
    deadline = meeting_date + timedelta(days=7)  # Default 1 week

Production considerations

Rate limiting: Use exponential backoff for API calls

async function retryWithBackoff(fn, maxRetries = 3) {
  for (let i = 0; i < maxRetries; i++) {
    try {
      return await fn();
    } catch (error) {
      if (error.status === 429) {
        await sleep(Math.pow(2, i) * 1000);  // 1s, 2s, 4s
      } else {
        throw error;
      }
    }
  }
}

Idempotency: Prevent duplicate task creation

def generate_task_fingerprint(task):
    content = f"{meeting_id}:{task['assignee']}:{task['description']}"
    return hashlib.md5(content.encode()).hexdigest()

if not task_exists(fingerprint):
    create_task(task)

Privacy and compliance:

Store recordings in compliant storage (HIPAA/GDPR)
Redact sensitive information from transcripts
Control access to task creation
Maintain audit logs linking tasks to source recordings

Testing your pipeline

test_cases = [
    {
        "transcript": "John, please send the proposal by Friday",
        "expected": {
            "assignee": "John",
            "task": "Send the proposal",
            "deadline": "Friday"
        }
    },
    {
        "transcript": "Someone should update the dashboard. Sarah: I'll do it tomorrow",
        "expected": {
            "assignee": "Sarah",
            "task": "Update the dashboard",
            "deadline": "tomorrow"
        }
    },
    {
        "transcript": "Let's review the budget next week. Actually, cancel that.",
        "expected": None  # Should not create task
    }
]

Success metrics

Track these to measure pipeline effectiveness:

Extraction accuracy: Percentage of real action items captured
False positive rate: Tasks created that weren’t real action items
Assignment accuracy: Correctly identified task owners
Deadline accuracy: Properly parsed due dates
Processing time: Meeting end to tasks created

What’s next

When the official feature launches, you’ll get:

Zero-configuration setup
Built-in transcription
Conversation thread tracking
The Draft state review workflow
Native integration with major platforms
Automatic user mapping via SSO
No coding or maintenance required

The initial release will target Zoom, Teams, and Google Meet, with more platforms to follow.

Want early access? Contact our support team to join the beta program.

Use Cases > Create tasks from meeting notes

Paste messy meeting notes into any AI assistant connected to Tallyfy. It automatically parses…

Vendors > Microsoft Teams

Microsoft Teams handles communication but not structured process execution. Tallyfy fills that…

Pro > Integrations

Tallyfy connects with your existing business software through twelve methods, from a full REST…

Vendors > Slack

Tallyfy’s native Slack integration turns chat messages into tracked tasks and workflows with…

Was this helpful?

Get in touch

About Tallyfy