Concepts / Real-time avatars
Real-time avatars
A Zeli avatar is a live, lip-synced talking head rendered on a GPU server and streamed to your process in real time. The SDK is a thin, fully-typed client. It does not run the models itself.
Two planes
A session runs over two coordinated channels:
- Media plane: a WebRTC connection delivers synchronized audio + video
frames to your process in real time. You consume them as PyAV
VideoFrame/AudioFrameobjects. - Control plane: a control channel (a WebSocket) carries the text you send
(
send_message,talk, talk streams) and the events you receive (transcripts, speech start/stop, emotions, warnings, errors).
Use the hosted base URL and authenticated session. The SDK handles transport; rendering and your selected speech and model providers run outside the client.
What happens on connect
async with client.connect() as session:
...- The SDK negotiates a WebRTC connection via the server's
POST /connectendpoint (audio + video, receive-only), then emitsCONNECTION_ESTABLISHED. - It opens the control channel and, once the handshake completes, emits
SESSION_READYwith aSessionInfo. - Frames begin flowing on
session.video_frames()andsession.audio_frames(); you drive the avatar with text.
The control gateway
Some capabilities require the server's control gateway:
| Capability | Needs gateway? |
|---|---|
send_message (conversational) | Falls back to HTTP |
talk (direct TTS) | ✅ |
create_talk_stream (incremental) | ✅ |
| Streaming events (speech start/stop, emotions) | ✅ |
Check session.has_control_channel to detect a server that predates the gateway.
On such a server, talk / talk streams raise
SessionError and send_message uses a plain-HTTP fallback.
Hosting and processing regions
The hosted service manages avatar rendering. Speech recognition, voice synthesis and model processing depend on the selected conversation mode and providers; they do not necessarily run on the same server or in the same country. Review the provider and region information before selecting a mode.
Self-hosted deployments have separate operator requirements. See the operator guide.