LiveKit的agent介绍

李昂的数字之旅

于 2024-09-05 19:25:59 发布

阅读量1.5k

点赞数 25

分类专栏：聊天系统文章标签： aiagent livekit

本文链接：https://blog.csdn.net/xsgnzb/article/details/141938915

版权

聊天系统专栏收录该内容

6 篇文章 0 订阅

订阅专栏

概念

LiveKit核心概念：

Room（房间）
Participant（参会人）
Track（信息流追踪）

Agent 架构图

订阅信息流

agent交互流程

客户端操作

加入房间

房间创建方式

手动

赋予用户创建房间的权限，在客户的加入并创建房间。

自动

客户的指定ws_url和token，加入指定房间。

room = LiveKit.create(appContext = applicationContext)
room.connect(wsUrl, token)

离开房间

调用 Room.disconnect（） 通知 LiveKit 离开事件。如果应用程序在未通知 LiveKit 的情况下关闭，则将继续显示参与者在 Room 中 15 秒。

Swift上，当应用程序退出时，会自动调用 Room.disconnect。

发送消息

发送方式

客户端通过LocalParticipant.publishData API 向房间中的任何参与者发布任意数据消息。房间数据通过 WebRTC 数据通道发布到SFU；LiveKit 服务器会将该数据转发给聊天室中的一个或多个参与者。

给指定用户发消息，通过设置destinationIdentities ，它表示用户的身份。

// 发送消息
coroutineScope.launch {
  val data: ByteArray = //...

  // 发送有损消息给全员，LOSSY表示数据发送一次，无顺序保证。这对于优先考虑交付速度的实时更新来说是理想的选择。
  room.localParticipant.publishData(data, DataPublishReliability.LOSSY)

  // 发送可靠的消息给指定成员，RELIABLE表示发送数据时最多重试3次并保证顺序。适合优先保证交付而不是抵延迟的场景，例如室内聊天。
  val identities = listOf(
    Participant.Identity("alice"),
    Participant.Identity("bob"),
  )
  room.localParticipant.publishData(data, DataPublishReliability.RELIABLE, identities)
}

// 处理接收到的消息
coroutineScope.launch {
  room.events.collect { event ->
    if(event is RoomEvent.DataReceived) {
        // Process data
    }
  }
}

消息大小限制

由于 SCTP 协议的限制，对大于 16 KiB 的消息使用数据通道是不切实际的，包括 LiveKit 的协议包装器。我们建议将消息大小保持在 15 KiB 以下。详细了解数据通道大小限制。

消息的topic

消息可以指定topic，在接收方通过topic进行过滤出感兴趣的消息。

发送信息流

livekit默认支持摄像头、麦克风、录屏3个流，也支持用户发布自定义流的配置。

音视频流

// Turns camera track on
room.localParticipant.setCameraEnabled(true)

// Turns microphone track on
room.localParticipant.setMicrophoneEnabled(true)

录屏流

// Create an intent launcher for screen capture
// This *must* be registered prior to onCreate(), ideally as an instance val
val screenCaptureIntentLauncher = registerForActivityResult(
    ActivityResultContracts.StartActivityForResult()
) { result ->
    val resultCode = result.resultCode
    val data = result.data
    if (resultCode != Activity.RESULT_OK || data == null) {
        return@registerForActivityResult
    }
    lifecycleScope.launch {
        room.localParticipant.setScreenShareEnabled(true, data)
    }
}

// When it's time to enable the screen share, perform the following
val mediaProjectionManager =
    getSystemService(MEDIA_PROJECTION_SERVICE) as MediaProjectionManager
screenCaptureIntentLauncher.launch(mediaProjectionManager.createScreenCaptureIntent())

自定义流配置

// Option 1: set room defaults
val options = RoomOptions(
  audioTrackCaptureDefaults = LocalAudioTrackOptions(
    noiseSuppression = true,
    echoCancellation = true,
    autoGainControl = true,
    highPassFilter = true,
    typingNoiseDetection = true,
  ),
  videoTrackCaptureDefaults = LocalVideoTrackOptions(
    deviceId = "",
    position = CameraPosition.FRONT,
    captureParams = VideoPreset169.H1080.capture,
  ),
  audioTrackPublishDefaults = AudioTrackPublishDefaults(
    audioBitrate = 20_000,
    dtx = true,
  ),
  videoTrackPublishDefaults = VideoTrackPublishDefaults(
    videoEncoding = VideoPreset169.H1080.encoding,
  )
)
var room = LiveKit.create(
  ...
  roomOptions = options,
)

// Option 2: create tracks manually
val localParticipant = room.localParticipant
val audioTrack = localParticipant.createAudioTrack("audio")
localParticipant.publishAudioTrack(audioTrack)

val videoTrack = localParticipant.createVideoTrack("video", LocalVideoTrackOptions(
  CameraPosition.FRONT,
  VideoPreset169.H1080.capture
))
localParticipant.publishVideoTrack(videoTrack)

订阅信息流

默认用户进入房间，会监听所有信息流。

coroutineScope.launch {
  room.events.collect { event ->
    when(event) {
      is RoomEvent.TrackSubscribed -> {
        // Audio tracks are automatically played.
        val videoTrack = event.track as? VideoTrack ?: return@collect
        videoTrack.addRenderer(videoRenderer)
      }
      else -> {}
    }
  }
}

监听事件

事件分为：room事件和参与者事件。这是事件列表：

EVENT	DESCRIPTION	ROOM EVENT	PARTICIPANT EVENT
ParticipantConnected 参与者Connected	A RemoteParticipant joins after the local participant.	✔️
	RemoteParticipant 在本地参与者之后加入。
ParticipantDisconnected 参与者断开连接	A RemoteParticipant leaves	✔️
	RemoteParticipant 离开
Reconnecting 重新连接	The connection to the server has been interrupted and it's attempting to reconnect.	✔️
	与服务器的连接已中断，它正在尝试重新连接。
Reconnected 重新	Reconnection has been successful	✔️
	重新连接成功
Disconnected 断开	Disconnected from room due to the room closing or unrecoverable failure	✔️
	由于会议室关闭或无法恢复的故障而与会议室断开连接
TrackPublished 轨迹已发布	A new track is published to room after the local participant has joined	✔️	✔️
	本地参加者加入后，新轨道将发布到聊天室
TrackUnpublished TrackUnpublished （未发布）	A RemoteParticipant has unpublished a track	✔️	✔️
	RemoteParticipant 已取消发布轨道
TrackSubscribed	The LocalParticipant has subscribed to a track	✔️	✔️
	LocalParticipant 已订阅跟踪
TrackUnsubscribed 跟踪Unsubscribed	A previously subscribed track has been unsubscribed	✔️	✔️
	之前订阅的曲目已取消订阅
TrackMuted TrackMuted （轨道静音）	A track was muted, fires for both local tracks and remote tracks	✔️	✔️
	轨道已静音，本地轨道和远程轨道均触发
TrackUnmuted TrackUnmuted （轨道未静音）	A track was unmuted, fires for both local tracks and remote tracks	✔️	✔️
	轨道已取消静音，本地轨道和远程轨道均触发
LocalTrackPublished LocalTrack已发布	A local track was published successfully	✔️	✔️
	已成功发布本地轨道
LocalTrackUnpublished	A local track was unpublished	✔️	✔️
	本地曲目未发布
ActiveSpeakersChanged ActiveSpeakers已更改	Current active speakers has changed	✔️
	当前当前活跃的发言人已更改
IsSpeakingChanged	The current participant has changed speaking status		✔️
	当前参与者已更改发言状态
ConnectionQualityChanged 连接质量已更改	Connection quality was changed for a Participant	✔️	✔️
	参与者的连接质量已更改
ParticipantMetadataChanged	A participant's metadata was updated via server API	✔️	✔️
	参与者的元数据已通过服务器 API 更新
RoomMetadataChanged RoomMetadataChanged 的	Metadata associated with the room has changed	✔️
	与聊天室关联的元数据已更改
DataReceived 已接收数据	Data received from another participant or server	✔️	✔️
	从其他参与者或服务器接收的数据
TrackStreamStateChanged TrackStreamStateChanged （已更改）	Indicates if a subscribed track has been paused due to bandwidth	✔️	✔️
	指示订阅的曲目是否因带宽而暂停
TrackSubscriptionPermissionChanged	One of subscribed tracks have changed track-level permissions for the current participant	✔️	✔️
	其中一个已订阅的轨道已更改当前参与者的轨道级别权限
ParticipantPermissionsChanged	When the current participant's permissions have changed	✔️	✔️
ParticipantPermissions已更改	当前参与者的权限发生更改时

服务端操作

生成用户token

需要LiveKit服务的API_KEY和API-SECRET，通过LiveKit API生成JWT令牌。

通过登录JWT获取到用户的信息，identify=user_id+场景，name=用户昵称（默认值），room名称=场景名（user_id）

# server.py
import os
from livekit import api
from flask import Flask

app = Flask(__name__)

@app.route('/getToken')
def getToken():
 token = api.AccessToken(os.getenv('LIVEKIT_API_KEY'), os.getenv('LIVEKIT_API_SECRET')) \
 .with_identity("identity") \
 .with_name("my name") \
 .with_grants(api.VideoGrants(
 room_join=True,
 room="my-room",
 ))
 return token.to_jwt()

开发环境可以通过CLI快速创建token：

livekit-cli token create   --api-key devkey --api-secret secret   --join --room test_room --identity test_user   --valid-for 24h

token属性

基于JWT的令牌，包含用户身份、放假名称、功能、权限等。按照场景颁发token，也就是对应的房间。

聊天室权限在解码的加入令牌的 video 字段中指定。它可能包含以下一个或多个属性：

FIELD	TYPE	DESCRIPTION
roomCreate room创建	bool	Permission to create or delete rooms
		创建或删除聊天室的权限
roomList roomList 会议室	bool	Permission to list available rooms
		列出可用会议室的权限
roomJoin room加入	bool	Permission to join a room
		加入聊天室的权限
roomAdmin roomAdmin 管理员	bool	Permission to moderate a room
		管理聊天室的权限
roomRecord roomRecord （房间记录）	bool	Permissions to use Egress service
		使用 Egress 服务的权限
ingressAdmin 入口管理员	bool 布尔	Permissions to use Ingress service
		Ingress 服务使用权限
room 房间	string 字符串	Name of the room, required if join or admin is set
		聊天室的名称，如果设置了 join 或 admin，则为必填项
canPublish 可以发布	bool 布尔	Allow participant to publish tracks
		允许参与者发布轨迹
canPublishData	bool 布尔	Allow participant to publish data to the room
		允许参与者将数据发布到聊天室
canPublishSources	string[] 字符串[]	When set, only listed source can be published. (camera, microphone, screen_share, screen_share_audio)
		设置后，只能发布列出的源。（摄像头、麦克风、screen_share、screen_share_audio）
canSubscribe canSubscribe 订阅	bool 布尔	Allow participant to subscribe to tracks
		允许参加者订阅曲目
canUpdateOwnMetadata	bool 布尔	Allow participant to update its own metadata
		允许参与者更新自己的元数据
hidden 隐藏	bool 布尔	Hide participant from others in the room
		对聊天室中的其他人隐藏参与者
kind 类	string 字符串	Type of participant (standard, ingress, egress, sip, or agent). this field is typically set by LiveKit internals.
		参与者类型（标准、入口、出口、SIP 或代理）。此字段通常由 LiveKit 内部设置。

session断开操作

用户离开房间后，回话会结束，通过add_shutdown_callback回调，可以处理后续操作。例如：发送聊天结束事件。

async def entrypoint(ctx: JobContext):
    async def my_shutdown_hook():
        # save user state
        ...
    ctx.add_shutdown_callback(my_shutdown_hook)

Agent操作

创建Agent服务节点

LiveKit的Agent框架现在只支持python的SDK，文档地址如下：https://docs.livekit.io/agents/quickstart/

这是官方给的demo：

import asyncio

from livekit.agents import AutoSubscribe, JobContext, WorkerOptions, cli, llm
from livekit.agents.voice_assistant import VoiceAssistant
from livekit.plugins import deepgram, openai, silero


# This function is the entrypoint for the agent.
async def entrypoint(ctx: JobContext):
    # Create an initial chat context with a system prompt
    initial_ctx = llm.ChatContext().append(
        role="system",
        text=(
            "You are a voice assistant created by LiveKit. Your interface with users will be voice. "
            "You should use short and concise responses, and avoiding usage of unpronouncable punctuation."
        ),
    )

    # Connect to the LiveKit room
    # indicating that the agent will only subscribe to audio tracks
    await ctx.connect(auto_subscribe=AutoSubscribe.AUDIO_ONLY)

    # VoiceAssistant is a class that creates a full conversational AI agent.
    # See https://github.com/livekit/agents/tree/main/livekit-agents/livekit/agents/voice_assistant
    # for details on how it works.
    assistant = VoiceAssistant(
        vad=silero.VAD.load(),
        stt=deepgram.STT(),
        llm=openai.LLM(),
        tts=openai.TTS(),
        chat_ctx=initial_ctx,
    )

    # Start the voice assistant with the LiveKit room
    assistant.start(ctx.room)

    await asyncio.sleep(1)

    # Greets the user with an initial message
    await assistant.say("Hey, how can I help you today?", allow_interruptions=True)


if __name__ == "__main__":
    # Initialize the worker with the entrypoint
    cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))

Agent的生命周期

当worker程序启动时，会通过websocket连接到LiveKit服务器，将自己注册成worker。一个worker下会有多个子进程（Agent）来处理请求。
当用户进入房间时，LiveKit服务器通过负载均衡选择一个worker，为用户提供服务。
子进程处理来自用户的消息，并给出回复。
当用户退出房间时，房间滚啊比，并且断开与agent的连接。

Agent内部执行流程

agent在处理请求时，包含几个节点：

request handler：判断能否处理请求，不能请求则LiveKit会讲任务交给其他worker
entrypoint：agent进入房间之前，执行的初始化操作
prewarm function：agent进程启动时调用，可以执行加载模型等耗时的操作

Worker类型

opts = WorkerOptions(
    ...
    # when omitted, the default is JobType.JT_ROOM
    worker_type=JobType.JT_ROOM,
)

JobType 枚举有两个选项：

JT_ROOM：将为每个房间创建一个新的代理实例。
JT_PUBLISHER：将为房间里的每个参与者创建一个新的代理实例。

Agent处理请求

处理音频流

@ctx.room.on("track_subscribed")
def on_track_subscribed(
    track: rtc.Track,
    publication: rtc.TrackPublication,
    participant: rtc.RemoteParticipant,
):
    # 监听音频流
    if track.kind == rtc.TrackKind.KIND_AUDIO:
        audio_stream = rtc.AudioStream(track)
        async for event in audio_stream:
            do_something(event.frame)

发布音频流

发布音频涉及将流拆分为长度固定的音频帧。内部缓冲区保存 50 毫秒长的音频队列，实时发送。用于发送新帧的 capture_frame 方法是阻塞的，在缓冲区接收整个帧之前阻塞在那里。这样可以更轻松地处理中断。

为了发布音轨，需要事先确定采样率和声道数，以及每帧的长度（样本数）。下面的示例是在 10ms 长帧中以 48kHz 传输恒定的 16 位正弦波：

SAMPLE_RATE = 48000
NUM_CHANNELS = 1 # mono audio
AMPLITUDE = 2 ** 8 - 1
SAMPLES_PER_CHANNEL = 480 # 10ms at 48kHz

async def entrypoint(ctx: JobContext):
    await ctx.connect()

    source = rtc.AudioSource(SAMPLE_RATE, NUM_CHANNELS)
    track = rtc.LocalAudioTrack.create_audio_track("example-track", source)
    # since the agent is a participant, our audio I/O is its "microphone"
    options = rtc.TrackPublishOptions(source=rtc.TrackSource.SOURCE_MICROPHONE)
    # ctx.agent is an alias for ctx.room.local_participant
    publication = await ctx.agent.publish_track(track, options)

    frequency = 440
    async def _sinewave():
        audio_frame = rtc.AudioFrame.create(SAMPLE_RATE, NUM_CHANNELS, SAMPLES_PER_CHANNEL)
        audio_data = np.frombuffer(audio_frame.data, dtype=np.int16)

        time = np.arange(SAMPLES_PER_CHANNEL) / SAMPLE_RATE
        total_samples = 0
        while True:
            time = (total_samples + np.arange(SAMPLES_PER_CHANNEL)) / SAMPLE_RATE
            sinewave = (AMPLITUDE * np.sin(2 * np.pi * frequency * time)).astype(np.int16)
            np.copyto(audio_data, sinewave)

            # send this frame to the track
            await source.capture_frame(frame)
            total_samples += samples_per_channel

处理文本消息

监听data_received事件，处理用户发来的消息；通过publish_data()发送消息给用户。

@room.on("data_received")
def on_data_received(data: rtc.DataPacket):
  logging.info("received data from %s: %s", data.participant.identity, data.data)

# string payload will be encoded to bytes with UTF-8
await room.local_participant.publish_data("my payload",
                reliable=True,
                destination_identities=["identity1", "identity2"],
                topic="topic1")