Hysia Video to Retail Platform
An intelligent multimodal-learning based system for video, product and ads analysis. You can build various downstream applications with the system, such as product recommendation, video retrieval. Several examples are provided.
V2 is under active development currently. You are welcome to create a issue, pull request here. We will credit them into V2.
Showcase
👉 Full list of showcase.
Upload video and process it by selecting different models
Display video processing result
Search scene by image and text
Insert product advertisement and display insertion result
Features
Multimodal learning-based video analysis:
Scene / Object / Face detection and recognition
Multimodality data pre-processing
Results align and store
Downstream applications:
Intelligent ads insertion
Content-product match
Visualized testbed
Visualize multimodality results
Can be installed separately
Setup Environment
1. Download Data
# Make sure this script is run from project root
bash scripts/download-data.sh
2. Installation
👉 Install with Docker
🐳
docker pull hysia/hysia:v2o
Configuration
Change decoder and model server running devices at device_placement.yml:
decoder: CPU
visual_model_server: CUDA:1
audio_model_server: CUDA:2
feature_model_server: CUDA:3
product_search_server: CUDA:2
scene_search_server: CUDA:3
Device value format: cpu, cuda or cuda:.
Demo
Run with docker
🐳
docker run --rm \
--gpus all -d -p 8000:8000 \
-v ${PWD}/server/config/device_placement.yml:/content/server/config/device_placement.yml \
hysia/hysia:v2o
Then you can go to http://localhost:8000. Use username: admin and password: admin to login.
Some Useful Tools
Large dataset preprocessing
Video/audio decoding
Model profiling
Multimodality data testbed
Contributing
You are welcome to contribute to Hysia! Please refer to here to get start.
Paper Citation
Coming soon!
About Us
Maintainers
Huaizheng Zhang
Yuanming Li
Qiming Ai
Previous Contributors
Shengsheng Zhou
Wenbo Jiang (Now, Shopee)
Ziyuan Liu (Now, Tencent)
Yongjie Wang (Now, NTU PhD)