【语义分割 01】Open MMLab

1 Tutorial

https://github.com/TommyZihao/MMSegmentation_Tutorials
https://github.com/TommyZihao/Train_Custom_Dataset
https://github.com/TommyZihao/aidlux_tutorial

OpenMMLab是一个由中国开发者主导的具有世界影响力的人工智能计算机视觉开源算法体系, 至今已经开源了30+视觉算法库。
OpenMMLab is a world-influential open source artificial intelligence computer vision algorithm system led by Chinese developers, and has opened 30+ visual algorithms library so far.

语义分割不能解决三维问题.

2 Introduce

openmmlab介绍视频：Github八万star，开源计算机视觉神器OpenMMLab_哔哩哔哩_bilibili

 MMSegmentation分割：主要应用于医疗影像、遥感图像
 MMSegmentation: Mainly applied to medical images and remote sensing images

 MMPretrain：

  MMHuman3D：

  MMAction2：

MMagic：生成模型+底层视觉+AIGC算法库
MMagic: Generative model + low-level vision +AIGC algorithm library

3 Semantic segmentation annotation format

Integer mask

• Watermelon data set

• When to use semantic segmentation? When to split by instance?

如果不存在“实例”概念，只需要每个像素分类，用语义分割
If there is no "instance" concept, you just need to classify each pixel and use semantic segmentation.
如果需要目标检测预测框(计数、追踪、流量统计)用实例分割
If the target detection prediction box (counting, tracking, traffic statistics) is required, use the instance segmentation

Track-Anything：https://github.com/gaomingqi/Track-Anything
Segment-Anything：Segment Anything | Meta AI

• Development trend of image segmentation.

        Use Transformer architecture
        Unified semantic segmentation, instance segmentation, panoramic segmentation
        Vision Large Model (SAM)
        Multimodal +AIGC(Text-generated picture, Picture-generated picture , Image editing)

• Semantic segmentation evaluation

• Summary

        There are many application scenarios for semantic segmentation, and some time ago I also wanted to use sam on ROS, but it has not been made.

4 labelme

Download link：https://github.com/wkentaro/labelme/releases

使用这种方式会闪退 

   

The resulting json file.

     

可以使用sam大模型去做图像分割
需要在虚拟环境里面使用：pip install labelme

5 Install and configure the MMSegmentation environment

 First，let's create a new virtual environment.

 

 Install Pytorch using the following command.

pip3 install install torch==1.12.1+cu113 torchvision==0.13.1+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html

Follow the second tutorial to make your environment successful!

6 use API to predict

7 Train

40000 Iter
Logs are printed every 100 steps
aux.acc_seg:辅助语义分割损失函数
表格是在测试集上的评估性能（6类）
mIoU:平均IoU

8 visualize log

 

9 Test set evaluates metrics

10 Use the trained model to predict

 

 

 11 Deploy

Netron  -- onnx可视化模型结构

12 ONNX Runtime Deploy

 

13 AidLux图像分割部署-安卓鸿蒙高通ARM

 AidLux Image Segmentation Deployment - Andrew Hung Meng Qualcomm ARM
AidLux智能物联网(AIoT)应用开发和部署平台-阿加犀智能科技
AidLux Docs

 

14 other question

如何使用mmseg框架训练4波段 tif格式的遥感图像 以及 预训练模型载入的时候分配给rgb然后第四波段随机初始化?
多波段用pca主成分取前三个映射rgb

Docshttps://qxi0eyc5s7.feishu.cn/docx/Gx5OdFfa6oKOsBxFfmYcTKLhn1N

先教会大模型分割西瓜，然后把大模型当老师，再教一个轻量模型专门学习分割西瓜…先是迁移再蒸馏：用mmrazor