LiveRescue

readme.md

1. Prepare dev env

mkdir LiveRescue
curl -o live_rescue.py https://gist.github.com/fuji246/09c2a3d074c317404253a12c34ae21cd/raw/6eb2eb2bba739151bd86d78756030fa66ed2049d/live_rescue.py
cd LiveRescue/
mkdir venv
python3 -m venv venv
source ./venv/bin/activate

2. Install outspeed

git clone https://github.com/outspeed-ai/outspeed.git 
cd outspeed
pip install -e .
cd ..

3. Create .env file in dir "LiveRescue" with the following content, apply for API key for following services

https://fireworks.ai/account/profile https://console.deepgram.com/signup https://elevenlabs.io/

FIREWORKS_API_KEY=
DEEPGRAM_API_KEY=
ELEVEN_LABS_API_KEY=

4. Run the demo

• Run server

python3 live_rescue.py

• Launch client

open https://playground.outspeed.com/human_avatar_input and run "Human Avatar" using the default settings there.

DisasterLiveHelper.md

Problem Status

• Information lag: During disasters, the dissemination and collection of information often lags behind, making it difficult to convey critical disaster information to relevant departments and affected populations. Thousands Cried for Help as Houston Flooded”

• Emergency responders could not always keep pace with the influx of rescue requests as residents from neighborhoods across the city watched the waters rise higher and higher.

• Unequal resource allocation: Due to the lack of timely and accurate information, the allocation of rescue resources is often uneven, which may result in some heavily affected areas not receiving timely assistance, while some lightly affected areas have excess resources. Distribution of deficient resources in disaster response situation using particle swarm optimization

• The lack of coordination in resource mapping can leads to uneven distribution of resource with one place coping with excess where another place is resource starved. From past studies [8,9], it has been evident that in absence of proper planning and management in resource distribution strategies resulted in subsequent resource wastage in various disaster situations like tsunami 2004, Haiti Earthquake 2010 etc.

• Lack of interaction and guidance: During disasters, affected populations often lack professional guidance and do not know how to self-rescue or help others, while rescue personnel may not be able to reach every location in need of assistance in real-time. Hurricane Helene victims in NC are fending for themselves as response from Biden-Harris’ FEMA underwhelms

Creative Description

Overview

Develop a disaster live streaming platform based on WebRTC. The platform allows users to live stream the current disaster situation in real-time and interact with users through an intelligent assistant via voice and video, providing timely disaster relief advice and guidance. The platform can also classify and analyze different disasters by understanding the video content in real-time, helping to optimize the allocation of rescue resources.

Key Features

• Real-time disaster live streaming: Users can conduct real-time video live streaming through their mobile phones or computers to show the actual situation in disaster areas.

• Intelligent live streaming assistant: Using WebRTC for voice interaction, providing real-time guidance and advice to users through the GPT-4 large language model. The intelligent assistant can answer questions, provide disaster relief guidance, and guide users to perform safety procedures.

• Real-time video content analysis: Analyze the live streaming video content through computer vision technology to identify important information such as disaster locations, severity, and number of injured people. Combined with large models for sentiment analysis and text understanding, the disaster can be classified and evaluated more accurately.

• Interactive map and visualization: Display real-time disaster maps in the platform, marking affected areas and specific situations. Rescue teams and government agencies can view real-time changes in disaster situations and analysis results through the map, making more accurate resource allocation and response.

Potential Value

• Improve the real-time and accuracy of information: Real-time disaster live streaming and video content analysis can greatly improve the speed and accuracy of information dissemination. Governments and rescue agencies can obtain disaster information in a timely manner and respond quickly.

• Optimize resource allocation: Using real-time video analysis and disaster classification, rescue resources can be more effectively allocated to the most needed areas, reducing material waste and rescue delays. $$$$ 3. Enhance public participation and self-rescue capabilities: The real-time guidance provided by the intelligent live broadcast assistant can help disaster-affected people better engage in self-rescue and mutual aid, reducing further threats to life and property from disasters.

• Improve emergency response efficiency: The real-time interaction and automated analysis of the entire platform can significantly enhance the efficiency of emergency response, making rescue operations more organized and efficient.

• Data accumulation and analysis: The large amount of real-time collected data provides valuable experience and data support for subsequent disaster prevention and handling, which can be used to optimize future emergency response strategies.

live_rescue.py

import asyncio
import logging
import os
from typing import Tuple

logging.basicConfig(level=logging.DEBUG)

import outspeed
from outspeed.plugins.deepgram_stt import DeepgramSTT
from outspeed.plugins.eleven_labs_tts import ElevenLabsTTS
from outspeed.plugins.fireworks_llm import FireworksLLM
from outspeed.plugins.token_aggregator import TokenAggregator
from outspeed.streams import AudioStream, VideoStream, Stream, TextStream, VADStream
from outspeed.plugins.silero_vad import SileroVAD

SYSTEM_PROMPT = """
You are Disaster Live Helper, an intelligent assistant designed to provide real-time disaster relief guidance \
and support. Your primary functions include answering user inquiries about disaster situations, offering practical \
advice and instructions for emergency responses, and guiding users through specific safety protocols to ensure their \
well-being. Your goal is to help users navigate and respond effectively during disasters, enhancing their safety and \
the efficiency of rescue operations.

Greeting by introducing yourself Frist. Keep the answer short and concise.
"""

@outspeed.App()
class LiveRescue:
    async def setup(self):
        self.deepgram_node = DeepgramSTT()
        self.llm_node = FireworksLLM(
            system_prompt=SYSTEM_PROMPT,
        )
        self.token_aggregator_node = TokenAggregator()
        self.tts_node = ElevenLabsTTS()
        self.silero_vad_node = SileroVAD()

    @outspeed.streaming_endpoint()
    async def run(
        self, audio_input_stream: AudioStream, video_input_stream: VideoStream
    ) -> Tuple[Stream, ...]:

        deepgram_stream: TextStream = self.deepgram_node.run(audio_input_stream)

        silero_vad_stream: VADStream = self.silero_vad_node.run(
            audio_input_stream.clone()
        )

        llm_token_stream: TextStream
        chat_history_stream: TextStream
        llm_token_stream, chat_history_stream = self.llm_node.run(deepgram_stream)

        token_aggregator_stream: TextStream = self.token_aggregator_node.run(llm_token_stream)
        tts_stream: AudioStream = self.tts_node.run(token_aggregator_stream)

        self.tts_node.set_interrupt_stream(silero_vad_stream.clone())
        self.token_aggregator_node.set_interrupt_stream(silero_vad_stream.clone())

        return tts_stream, chat_history_stream

    async def teardown(self):
        await self.deepgram_node.close()
        await self.llm_node.close()
        await self.token_aggregator_node.close()
        await self.tts_node.close()
        await self.silero_vad_node.close()


if __name__ == "__main__":
    LiveRescue().start()