Multimodal AI Revolution: Building Apps That See, Hear, and Speak with GPT-4o

📅 December 17, 2025⏱️ 25 min read🏷️ Multimodal AI

Text is so 2023. The world isn't just text—it's images, sounds, and video. Multimodal AI models like GPT-4o ("o" for omni) and Gemini 1.5 Pro natively understand and generate across these modalities. This unlocks a new class of applications: apps that can watch a video and answer questions, listen to a meeting and draw a diagram, or look at a wireframe and write the code.

Native Multimodality vs. Patchwork

Before GPT-4o, "multimodal" meant stitching together separate models: Whisper for speech-to-text -> LLM for reasoning -> TTS for text-to-speech. This was slow and lost emotional nuance (latency was ~5 seconds).

Native Multimodal Models are trained on tokens that represent text, image, and audio interchangeably. They "hear" the tone of your voice and "see" the emotion in your face directly. This brings latency down to ~300ms (human conversational speed).

GPT-4o: Real-time Voice and Vision

Vision

Can analyze video streams in real-time. "What is happening in this security camera feed?"

Audio

Can sing, whisper, and detect sarcasm. It can translate spoken language with preserved intonation.

Code Example: Building a Vision Analysis App

Let's build a Next.js app that lets users upload an image of a UI mockup, and GPT-4o generates the Tailwind CSS code for it.

Next.js API Route

import OpenAI from "openai";

const openai = new OpenAI();

export async function POST(req: Request) {
  const { imageUrl } = await req.json();

  const response = await openai.chat.completions.create({
    model: "gpt-4o",
    messages: [
      {
        role: "system",
        content: "You are an expert frontend developer. Convert images to Tailwind CSS code."
      },
      {
        role: "user",
        content: [
          { type: "text", text: "Convert this UI mockup into HTML/Tailwind code." },
          {
            type: "image_url",
            image_url: {
              "url": imageUrl,
            },
          },
        ],
      },
    ],
    max_tokens: 4096,
  });

  return Response.json({ code: response.choices[0].message.content });
}

Code Example: Real-time Audio Processing

Using the OpenAI Realtime API (WebSockets) to build a voice assistant that can be interrupted.

Node.js (WebSocket)

import WebSocket from "ws";

const url = "wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview";
const ws = new WebSocket(url, {
  headers: {
    "Authorization": "Bearer " + process.env.OPENAI_API_KEY,
    "OpenAI-Beta": "realtime=v1",
  },
});

ws.on("open", () => {
  // Configure the session
  ws.send(JSON.stringify({
    type: "session.update",
    session: {
      modalities: ["text", "audio"],
      voice: "alloy",
    },
  }));
});

ws.on("message", (data) => {
  const event = JSON.parse(data.toString());

  if (event.type === "response.audio.delta") {
    // Play audio chunk to user
    playAudio(event.delta);
  }
});

// Simulate sending user audio
function sendUserAudio(base64Audio) {
  ws.send(JSON.stringify({
    type: "input_audio_buffer.append",
    audio: base64Audio
  }));
}

Killer Use Cases for 2026

Accessibility

Apps like Be My Eyes use GPT-4o to describe the world to blind users in real-time, reading menus, navigating streets, and finding lost objects.

Education

A math tutor that can "see" your handwriting on a tablet and explain where you made a mistake in the equation, encouraging you verbally.

Customer Support

Video support agents that can see the broken product you're holding up to the camera and guide you through the repair process step-by-step.