lanenet模型的复现文章有很多，原文Towards End-to-End Lane Detection: an Instance Segmentation Approach的代码是基于tensorflow1.0版本实现的。本文将在以tensorflow2.0版本为后端的keras上对lanenet模型部分功能进行复现，实现细节和原文有出入，但是实现目的和原文是一致的。希望读者能够为本文提出宝贵的建议。
一、LaneNet论文简要介绍
1.论文主要方法以及实现目的
论文中提出了一种端到端的车道线检测算法，包括LaneNet和H-Net两个网络模型。其中，LaneNet是一种将语义分割和对像素进行向量表示结合起来的多任务模型，目的是将不同车道线实例化；H-Net是由卷积层和全连接层组成的网络模型，负责预测转换矩阵H，目的是对属于同一车道线的像素点进行回归。下图为论文中提到的网络结构，下文将具体介绍每一部分实现的功能。
1.1 LaneNet网络的框架结构
下图为LaneNet的网络结构，是有两个解码分支的多任务模型，Segmentation branch 负责对输入图像进行语义分割（对像素进行二分类，判断像素属于车道线还是背景）；Embedding branch 对像素进行嵌入式表示，论文将嵌入式向量的维度设置为4，训练得到的 embedding 向量用于聚类。最后将两个分支的结果进行结合利用 Mean-Shift 算法进行聚类，得到实例分割的结果。
1.2 H-Net网络结构
LaneNet的输出是每条车道线的像素集合，还需要根据这些像素点回归出一条车道线。传统的做法是将图片投影到 bird’s-eye view 中，然后使用 2 阶或者 3 阶多项式进行拟合。在这种方法中，变换矩阵 H 只被计算一次，所有的图片使用的是相同的变换矩阵，这会导致地平面（山地，丘陵）变化下的误差。
为了解决这个问题，论文训练了一个可以预测变换矩阵 H 的神经网络 H-Net，网络的输入是图片，输出是变换矩阵 H：
由上式可以看出，转置矩阵 H 只有6个参数，因此H-Net的输出是一个 6 维的向量。H-Net 由 6 层普通卷积网络和一层全连接网络构成，其网络结构如图所示：
1.3曲线拟合
Curve fitting的过程就是通过坐标 y 去重新预测坐标 x 的过程：
- 对于包含 N 个像素点的车道线，每个像素点
, 首先使用
H-Net 的预测输出 H 对其进行坐标变换：P′=HP - 随后使用 最小二乘法对 3d 多项式的参数进行拟合：
- 根据拟合出的参数
预测出
- 最后将
投影回去：
注：
LaneNet论文讲解的文章有很多，在这里主要对LaneNet的论文的核心内容进行描述。
想对原文有更多了解的读者可参考：
https://www.jianshu.com/p/c6d38d648509
https://www.cnblogs.com/xuanyuyt/p/11523192.html
二、基于tensorflow2.0-keras的LaneNet网络实现
本节主要讲论文中没有提到的实验细节，主要对LaneNet网络中的LaneNet分支进行复现（思路复现），tensorflow_keras实现内容包括数据集处理、数据预处理、网络搭建、损失函数、学习率调整机制。网络的基础框架没有沿用论文中提到的Enet，复现时用到的基础网络是Unet编解码器网络。
注：文中提到的代码不能直接使用，需根据自己的需求进行简单修改
2.1数据集处理
论文中提到的数据集为TuSimple数据集，此数据集包含三个文件train_set.zip、 test_set.zip 、test_label.json，tusimple数据集是点标注数据集，为保证之后的模型训练，在将数据投入网络之前需要对数据集进行处理。
文件格式为：
2.1.1将点数据转换为线数据
import cv2
import json
import numpy as np
import os
base_path = "F:\\train_set\\"
file = open(base_path + 'label_data_0313.json', 'r')
image_num=0
for line in file.readlines():
data=json.loads(line)
image = cv2.imread(os.path.join(base_path, data['raw_file']))
#image=cv2.imread(data['raw_file'])
binaryimage=np.zeros((image.shape[0],image.shape[1],1),np.uint8)
instanceimage=binaryimage.copy()
arr_width=data['lanes']
arr_height=data['h_samples']
width_num=len(arr_width)
height_num=len(arr_height)
for i in range(height_num):
lane_hist=0
for j in range(width_num):
if arr_width[j][i-1]>0 and arr_width[j][i]>0:
binaryimage[int(arr_height[i]),int(arr_width[j][i])]=255
instanceimage[int(arr_height[i]),int(arr_width[j][i])]=lane_hist
if i>0:
cv2.line(binaryimage, (int(arr_width[j][i-1]),int(arr_height[i-1])), (int(arr_width[j][i]),int(arr_height[i])), 255, 10)
cv2.line(instanceimage,(int(arr_width[j][i-1]),int(arr_height[i-1])), (int(arr_width[j][i]),int(arr_height[i])), lane_hist, 10)
lane_hist+=50
string1="H:\\unet-keras\\TUsimple\\label_instance\\"+data['raw_file'][6:]+str(image_num)+".png"
string2="F:\\train_set\\png1\\"+str(image_num)+".png"
string3="F:\\train_set\\image\\"+str(image_num)+".jpg"
cv2.imwrite(string1,binaryimage)
cv2.imwrite(string2,instanceimage)
cv2.imwrite(string3,image)
image_num=image_num+1
#if image_num==500:
#break
file.close()2.1.2构建数据路径文件
训练时需要通过txt文件提取网络训练数据，根据网络实现具体目的构建txt，由于网络的结构时单输入，两输出，因此三个图片途径为一条数据，分别是真实地面、二值分割标签和车道线实例标签。
生成训练数据文件的代码
import os
import re
base_path = "H:/unet-keras/TUsimple\Data/"
img_path= "H:/unet-keras/TUsimple/Data/image"
label_path='H:/unet-keras/TUsimple/Data/label_instance'
seg_path='H:/unet-keras/TUsimple/Data/label_binary'
with open(base_path + 'train_seg_instance.txt', 'w') as f:
images = os.listdir(img_path)
labels = os.listdir(label_path)
seg_labels=os.listdir(seg_path)
for dir in images: #0313-1 0313-2
#image
dir1=img_path+'/'+dir
Dir= os.listdir(dir1)
Dir.sort(key=lambda i: int(re.match(r'(\d+)', i).group()))
#label
label_dir1 = label_path + '/' + dir
#print(label_dir1) #H:\unet-keras\TUsimple\Data\label_instance\0313-1
label_dir = os.listdir(label_dir1)
label_dir.sort(key=lambda i: int(re.match(r'(\d+)', i).group()))
#print(label_dir)
seg_label_dir1 = seg_path + '/' + dir
# print(label_dir1) #H:\unet-keras\TUsimple\Data\label_instance\0313-1
seg_label_dir = os.listdir(seg_label_dir1)
seg_label_dir.sort(key=lambda i: int(re.match(r'(\d+)', i).group()))
for filenames in Dir: #60.... 20...
#print(filenames)
filenames1=dir1+'/'+filenames
#print(filenames1)
Filenames=os.listdir(filenames1)
Filenames.sort(key=lambda i: int(re.match(r'(\d+)', i).group()))
#print(filenames)
#print(Filenames[15:])
a=[]
for filename in Filenames[15:]:
#print(filename)
out_path=filenames1+'/'+filename #H:\unet-keras\TUsimple\Data\image\0313-1\60\1.jpg
a.append(out_path)
#print(out_path)
label_out_path=label_dir1+'/'+filenames+'.png'
seg_label_out_path=seg_label_dir1+'/'+filenames+'.png'
#print(label_out_path)
a.append(label_out_path)
a.append(seg_label_out_path)
path=' '.join(a)
#print(path)
#print(a)
f.write(path+'\n')可将数据集按8：2划分为训练集和测试集：
import numpy.random
'''划分数据集'''
n=0
from random import shuffle
with open(r"H:\unet-keras\TUsimple\Data\train_seg_instance.txt","r", encoding='UTF-8') as f:
with open(r"H:\unet-keras\TUsimple\Data\train_instance_tusimple0.8.txt", "w", encoding='UTF-8') as f1:
with open(r"H:\unet-keras\TUsimple\Data\train_instance_tusimple0.2.txt", "w",
encoding='UTF-8') as f2:
if n == 0:
train = f.readlines()
shuffle(train)
split=0.8
num = len(train)
train_num = int(num * split)
# for j in train:
# print(j)
#print(len(train))
for line in train[:train_num]:
f1.writelines(line)
for line in train[train_num:]:
f2.writelines(line)
n = n + 1
2.2数据预处理_图片边框处理
数据预处理为网络的输入做准备工作，本文中用到Unet网络做编解码器网络，输入尺寸为
（256，256，3）即（H，W，3），但是原图为256*128*3，为了使图片不失真，这里将图片进行白条边框处理，代码如下
def letterbox_image(self,image, label, instance_label,size):
label = Image.fromarray(np.array(label))
instance_label = Image.fromarray(np.array(instance_label))
'''resize image with unchanged aspect ratio using padding'''
iw, ih = image.size
w, h = size
scale = min(w / iw, h / ih)
nw = int(iw * scale)
nh = int(ih * scale)
image = image.resize((nw, nh), Image.BICUBIC)
new_image = Image.new('RGB', size, (252, 252, 252))
new_image.paste(image, ((w - nw) // 2, (h - nh) // 2))
label = label.resize((nw, nh), Image.NEAREST)
instance_label = instance_label.resize((nw, nh), Image.NEAREST)
new_label = Image.new('L', size, (252))
new_label.paste(label, ((w - nw) // 2, (h - nh) // 2))
new_instance_label = Image.new('L', size, (252))
new_instance_label.paste(instance_label, ((w - nw) // 2, (h - nh) // 2))
return new_image, new_label, new_instance_label 
左边为地面真实图像，右边为二值分割图像标签，即lanenet的输入图像（Input）和二值分割分支的输出标签（y_true）,解释一下这里也对实例分割标签进行了白条调整，但是嵌入分支网络的输出的不是图像，而是每个像素点的多向量表示矩阵。
2.3标签处理
在上边的白条处理部分可以看出，对不同的训练任务，标签处理的方式有很大的差别。二值分割分支标签处理用one_hot处理，例如：[0,1]代表背景像素，[1,0]代表车道线像素，可以看出非0即1，这样表示的好处是各类别之间不存在任何的相关性。
png = np.array(png)
# 转化成one_hot的形式
seg_labels = np.zeros_like(png)
seg_labels[png <= 127.5] = 1
seg_labels = np.eye(self.num_classes + 1)[seg_labels.reshape([-1])]
#把白条部分也加上，在之后的损失函数计算中会将其去除
seg_labels = seg_labels.reshape((int(self.image_size[1]), int(self.image_size[0]), 2 + 1)) 嵌入分支主要目的是用多维向量来表示每一个像素点，能够体现各像素点之间的关系，方便之后对像素点聚类，具体实现在损失函数部分会提到，这里只对该分支的标签处理进行解释，对于需要表示各类别之间关系的数据，one_hot编码就不适用了，我们将用到标签编码，例[1，2，3，4]，图片像素中所有为1的数值作为一类，2，3，4也分别表示一类，这样表示的好处在于能够直接讲同一类的像素点聚为一个类别。
有了这样的想法，我们就要对数据进行特殊的处理，数据集处理部分的代码中，将每隔50像素值作为一个车道线种类，对此可设置标签编码
instance_png=np.array(instance_png)
instance_label = np.zeros_like(instance_png)
instance_label[instance_png < 255] = 4
instance_label[instance_png == 200] = 3
instance_label[instance_png == 150] = 2
instance_label[instance_png == 100] = 1
instance_label[instance_png == 50] = 02.4网络结构搭建
LaneNet网络是多任务网络，基于此本文用到Unet网络为基础网络，单输入（真实路面图像256，256，3），两输出（二值分割图像（256，256，2）和嵌入式矩阵（256，256，5）），了解了网络组成，就很容易搭建了：
from tensorflow.keras.layers import *
from tensorflow.keras.models import Model
import tensorflow.keras.backend as K
class UNET_double(object):
def __init__(self, input_shape, category_classes, color_classes):
# default input_shape = (width, height, channel)
self.input_shape = input_shape
self.category_classes = category_classes
self.color_classes = color_classes
def vgg(self, inputs):
# Conv->ReLU->BN->Pool
bn_axis = 3
x = Conv2D(64, (3, 3), padding='same')(inputs)
x = BatchNormalization(axis=bn_axis)(x)
x = Activation('relu')(x)
x = Conv2D(64, (3, 3), padding='same')(x)
x = BatchNormalization(axis=bn_axis)(x)
x = Activation('relu')(x)
feat1 = x
x = MaxPooling2D((2, 2), strides=(2, 2))(x)
# Block 2
x = Conv2D(128, (3, 3), padding='same')(x)
x = BatchNormalization(axis=bn_axis)(x)
x = Activation('relu')(x)
x = Conv2D(128, (3, 3), padding='same')(x)
x = BatchNormalization(axis=bn_axis)(x)
x = Activation('relu')(x)
feat2 = x
x = MaxPooling2D((2, 2), strides=(2, 2))(x)
# Block 3
x = Conv2D(256, (3, 3), padding='same')(x)
x = BatchNormalization(axis=bn_axis)(x)
x = Activation('relu')(x)
x = Conv2D(256, (3, 3), padding='same')(x)
x = BatchNormalization(axis=bn_axis)(x)
x = Activation('relu')(x)
x = Conv2D(256, (3, 3), padding='same')(x)
x = BatchNormalization(axis=bn_axis)(x)
x = Activation('relu')(x)
feat3 = x
x = MaxPooling2D((2, 2), strides=(2, 2))(x)
# Block 4
x = Conv2D(512, (3, 3), padding='same')(x)
x = BatchNormalization(axis=bn_axis)(x)
x = Activation('relu')(x)
x = Conv2D(512, (3, 3), padding='same')(x)
x = BatchNormalization(axis=bn_axis)(x)
x = Activation('relu')(x)
x = Conv2D(512, (3, 3), padding='same')(x)
x = BatchNormalization(axis=bn_axis)(x)
x = Activation('relu')(x)
feat4 = x
x = MaxPooling2D((2, 2), strides=(2, 2))(x)
# Block 5
x = Conv2D(512, (3, 3), padding='same')(x)
x = BatchNormalization(axis=bn_axis)(x)
x = Activation('relu')(x)
x = Conv2D(512, (3, 3), padding='same')(x)
x = BatchNormalization(axis=bn_axis)(x)
x = Activation('relu')(x)
x = Conv2D(512, (3, 3), padding='same')(x)
x = BatchNormalization(axis=bn_axis)(x)
x = Activation('relu')(x)
return x,feat1,feat2,feat3,feat4
def binery(self, inputs):
x, feat1, feat2, feat3, feat4 = inputs
channels = [64, 128, 256, 512]
# Conv->ReLU->BN->Pool
P5_up = UpSampling2D(size=(2, 2))(x)
P4 = Concatenate(axis=3)([feat4, P5_up]) # 28,28,512 28,28,512
P4 = Conv2D(channels[3], 3, padding='same', kernel_initializer='he_normal')(P4)
P4 = Activation('relu')(P4)
P4 = BatchNormalization()(P4)
P4 = Conv2D(channels[3], 3, padding='same', kernel_initializer='he_normal')(P4)
P4 = Activation('relu')(P4)
P4 = BatchNormalization()(P4)
P4_up = UpSampling2D(size=(2, 2))(P4) # 56,56,512
P3 = Concatenate(axis=3)([feat3, P4_up])
P3 = Conv2D(channels[2], 3, padding='same', kernel_initializer='he_normal')(P3)
P3 = Activation('relu')(P3)
P3 = BatchNormalization()(P3)
P3 = Conv2D(channels[2], 3, padding='same', kernel_initializer='he_normal')(P3)
P3 = Activation('relu')(P3)
P3 = BatchNormalization()(P3)
P3_up = UpSampling2D(size=(2, 2))(P3)
P2 = Concatenate(axis=3)([feat2, P3_up])
P2 = Conv2D(channels[1], 3, padding='same', kernel_initializer='he_normal')(P2)
P2 = Activation('relu')(P2)
P2 = BatchNormalization()(P2)
P2 = Conv2D(channels[1], 3, padding='same', kernel_initializer='he_normal')(P2)
P2 = Activation('relu')(P2)
P2 = BatchNormalization()(P2)
P2_up = UpSampling2D(size=(2, 2))(P2)
P1 = Concatenate(axis=3)([feat1, P2_up])
P1 = Conv2D(channels[0], 3, padding='same', kernel_initializer='he_normal')(P1)
P1 = Activation('relu')(P1)
P1 = BatchNormalization()(P1)
P1 = Conv2D(channels[0], 3, padding='same', kernel_initializer='he_normal')(P1)
P1 = Activation('relu')(P1)
P1 = BatchNormalization()(P1)
# softmax classifier
out1 = Conv2D(self.category_classes, 1, activation="softmax",name='category_output')(P1)
return out1
def instance(self, inputs):
# Conv->ReLU->BN->Pool
x, feat1, feat2, feat3, feat4=inputs
channels = [64, 128, 256, 512]
P5_up = UpSampling2D(size=(2, 2))(x)
P4 = Concatenate(axis=3)([feat4, P5_up]) # 28,28,512 28,28,512
P4 = Conv2D(channels[3], 3, padding='same', kernel_initializer='he_normal')(P4)
P4 = Activation('relu')(P4)
P4 = BatchNormalization()(P4)
P4 = Conv2D(channels[3], 3, padding='same', kernel_initializer='he_normal')(P4)
P4 = Activation('relu')(P4)
P4 = BatchNormalization()(P4)
P4_up = UpSampling2D(size=(2, 2))(P4) # 56,56,512
P3 = Concatenate(axis=3)([feat3, P4_up])
P3 = Conv2D(channels[2], 3, padding='same', kernel_initializer='he_normal')(P3)
P3 = Activation('relu')(P3)
P3 = BatchNormalization()(P3)
P3 = Conv2D(channels[2], 3, padding='same', kernel_initializer='he_normal')(P3)
P3 = Activation('relu')(P3)
P3 = BatchNormalization()(P3)
P3_up = UpSampling2D(size=(2, 2))(P3)
P2 = Concatenate(axis=3)([feat2, P3_up])
P2 = Conv2D(channels[1], 3, padding='same', kernel_initializer='he_normal')(P2)
P2 = Activation('relu')(P2)
P2 = BatchNormalization()(P2)
P2 = Conv2D(channels[1], 3, padding='same', kernel_initializer='he_normal')(P2)
P2 = Activation('relu')(P2)
P2 = BatchNormalization()(P2)
P2_up = UpSampling2D(size=(2, 2))(P2)
P1 = Concatenate(axis=3)([feat1, P2_up])
P1 = Conv2D(channels[0], 3, padding='same', kernel_initializer='he_normal')(P1)
P1 = Activation('relu')(P1)
P1 = BatchNormalization()(P1)
P1 = Conv2D(channels[0], 3, padding='same', kernel_initializer='he_normal')(P1)
P1 = Activation('relu')(P1)
P1 = BatchNormalization()(P1)
out2 = Conv2D(self.color_classes, 1,name='color_output')(P1) #256,256,5
return out2
def build_model(self):
input_shape = self.input_shape
inputs = Input(shape=input_shape)
vgg = self.vgg(inputs)
binery=self.binery(vgg)
instance=self.instance(vgg)
model = Model(inputs=inputs, outputs=[binery,instance])
#model.summary()
return model
#UNET_double(input_shape=(256,256,3),category_classes=2,color_classes=5).build_model()2.5损失函数
两个任务需要构建两个损失函数，本文中二值分割分支用到的损失函数是dice_loss损失函数，车道线和背景属于极度不均衡的样本分类，此函数能够很好的解决这样的问题并且不需要设置任何权重超参数。
def dice_loss(beta=1, smooth=1e-5):
def _dice_loss_with_CE(y_true, y_pred):
#print(y_pred) #2,256,256,2
#print(y_true) #2,256,256,3
y_pred = K.clip(y_pred, K.epsilon(), 1.0 - K.epsilon())
CE_loss = - y_true[..., :-1] * K.log(y_pred)
CE_loss = K.mean(K.sum(CE_loss, axis=-1))
tp = K.sum(y_true[..., :-1] * y_pred, axis=[0, 1, 2])
fp = K.sum(y_pred, axis=[0, 1, 2]) - tp
fn = K.sum(y_true[..., :-1], axis=[0, 1, 2]) - tp
score = ((1 + beta ** 2) * tp + smooth) / ((1 + beta ** 2) * tp + beta ** 2 * fn + fp + smooth)
score = tf.reduce_mean(score)
dice_loss = 1 - score
return CE_loss + dice_loss
return _dice_loss像素嵌入分支的损失函数保留论文中的损失函数：
为了区分车道线上的像素属于哪条车道，embedding_branch为每个像素初始化一个embedding向量，并且在设计loss时，使得属于同一条车道线的像素向量距离很小，属于不同车道线的像素向量距离很大。
方差loss(L_var) ：当像素向量(pixel embedding)x_i与对应车道线均值向量μ_c的距离大于δ_v时，模型会进行更新，使得x_i 靠近μ_c ；
距离loss(L_dist) ：当不同车道线均值向量 μ_ca和μ_cb之间的距离小于δ_d 时，模型会进行更新，使得μ_ca与μ_cb远离彼此；
方差loss(L_var)使得像素向量向车道线的均值向量 μ_c 靠近，距离loss(L_dist)则会推动聚类中心远离彼此。
tf.config.experimental_run_functions_eagerly(True)#调试时用于将静态图转换为动态图
def discriminative_loss_single(
prediction,
correct_label,
feature_dim,
label_shape,
delta_v,
delta_d,
param_var,
param_dist,
param_reg):
"""
鉴别损失
：参数预测：网络推理
：param correct_label:实例标签
：param feature_dim：预测的特征维度
：param label_shape：标签的形状
：param delta_v：截止方差距离 0.5, 3.0, 1.0, 1.0, 0.001
：param delta_d：截止簇距离
：param param_var：簇内方差的权重
：param param_dist：簇间距离的权重
：param param_reg：权重正则化
"""
correct_label = tf.reshape(
correct_label, [label_shape[1] * label_shape[0]] #可认为是网络的输入形状（h*w）
)
reshaped_pred = tf.reshape(
prediction, [label_shape[1] * label_shape[0], feature_dim] #可认为是网络的输出形状（h*w,特征维度）
)
# calculate instance nums 计算实例的个数
unique_labels, unique_id, counts = tf.unique_with_counts(correct_label)
#unique_with_counts函数
# tensor 'x' is [1, 1, 2, 4, 4, 4, 7, 8, 8]
# y, idx, count = unique_with_counts(x)
# y == > [1, 2, 4, 7, 8]
# idx == > [0, 0, 1, 2, 2, 2, 3, 4, 4]
# count == > [2, 1, 3, 1, 2]
#以上返回的都是tensor
counts = tf.cast(counts, tf.float32) #将counets的张量形式转换维float32 counts有几个元素代表几类，元素值代表那一类的数量
num_instances = tf.size(unique_labels) #返回unique_labels的形状
# calculate instance pixel embedding mean vec计算实例像素嵌入平均向量
segmented_sum = tf.math.unsorted_segment_sum(
reshaped_pred, unique_id, num_instances) #将相同类别的像素加在一起
#a = [[1 2 3],[4 5 6],[7 8 9]]
#tf.unsorted_segment_sum(data=a,segment_ids=[0,1,0],num_segments=2))
#输出为#[[ 8 10 12],[ 4 5 6]
mu = tf.divide(segmented_sum, tf.reshape(counts, (-1, 1))) #segmented_sum里的每一个元素除以依次counets中的每个元素
mu_expand = tf.gather(mu, unique_id) #根据unique_id将mu进行扩充
distance = tf.norm(tf.subtract(mu_expand, reshaped_pred), axis=1, ord=1)
distance = tf.subtract(distance, delta_v)
distance = tf.clip_by_value(distance, 0., distance)
distance = tf.square(distance)
l_var = tf.math.unsorted_segment_sum(distance, unique_id, num_instances)
l_var = tf.divide(l_var, counts)
l_var = tf.reduce_sum(l_var)
l_var = tf.divide(l_var, tf.cast(num_instances, tf.float32))
mu_interleaved_rep = tf.tile(mu, [num_instances, 1])
mu_band_rep = tf.tile(mu, [1, num_instances])
mu_band_rep = tf.reshape(
mu_band_rep,
(num_instances *
num_instances,
feature_dim))
mu_diff = tf.subtract(mu_band_rep, mu_interleaved_rep)
intermediate_tensor = tf.reduce_sum(tf.abs(mu_diff), axis=1)
zero_vector = tf.zeros(1, dtype=tf.float32)
bool_mask = tf.not_equal(intermediate_tensor, zero_vector)
mu_diff_bool = tf.boolean_mask(mu_diff, bool_mask)
mu_norm = tf.norm(mu_diff_bool, axis=1, ord=1)
mu_norm = tf.subtract(2. * delta_d, mu_norm)
mu_norm = tf.clip_by_value(mu_norm, 0., mu_norm)
mu_norm = tf.square(mu_norm)
l_dist = tf.reduce_mean(mu_norm)
l_reg = tf.reduce_mean(tf.norm(mu, axis=1, ord=1))
param_scale = 1.
l_var = param_var * l_var
l_dist = param_dist * l_dist
l_reg = param_reg * l_reg
loss = param_scale * (l_var + l_dist + l_reg)
return loss, l_var, l_dist, l_reg
def instance_loss(feature_dim=5, image_shape=(256,256),
delta_v=0.5, delta_d=3.0, param_var=1.0, param_dist=1.0, param_reg=0.001):
"""
0.5, 3.0, 1.0, 1.0, 0.001
:return: discriminative loss and its three components 判别损失及其三个分量
"""
def loss(y_true,y_pred):
def cond(label, batch, out_loss, out_var, out_dist, out_reg, i):
return tf.less(i, tf.shape(batch)[0])
def body(label, batch, out_loss, out_var, out_dist, out_reg, i):
disc_loss, l_var, l_dist, l_reg = discriminative_loss_single(
y_pred[i], y_true[i], feature_dim, image_shape, delta_v, delta_d, param_var, param_dist, param_reg)
out_loss = out_loss.write(i, disc_loss)
out_var = out_var.write(i, l_var)
out_dist = out_dist.write(i, l_dist)
out_reg = out_reg.write(i, l_reg)
return label, batch, out_loss, out_var, out_dist, out_reg, i + 1
# TensorArray is a data structure that support dynamic writing
output_ta_loss = tf.TensorArray(
dtype=tf.float32, size=0, dynamic_size=True)
output_ta_var = tf.TensorArray(
dtype=tf.float32, size=0, dynamic_size=True)
output_ta_dist = tf.TensorArray(
dtype=tf.float32, size=0, dynamic_size=True)
output_ta_reg = tf.TensorArray(
dtype=tf.float32, size=0, dynamic_size=True)
_, _, out_loss_op, out_var_op, out_dist_op, out_reg_op, _ = tf.while_loop(
cond, body, [
y_true, y_pred, output_ta_loss, output_ta_var, output_ta_dist, output_ta_reg, 0])
out_loss_op = out_loss_op.stack()
out_var_op = out_var_op.stack()
out_dist_op = out_dist_op.stack()
out_reg_op = out_reg_op.stack()
dice_loss = tf.reduce_mean(out_loss_op)
l_var = tf.reduce_mean(out_var_op)
l_dist = tf.reduce_mean(out_dist_op)
l_reg = tf.reduce_mean(out_reg_op)
return dice_loss, l_var, l_dist, l_reg
return loss因为有背景像素的存在，此问题也是不均衡分类问题，可以看出以上代码中也用到了dice_loss。
2.6网络学习率下降策略
本文用到了两种学习率下降策略，
阶段性下降：
reduce_lr = ReduceLROnPlateau(monitor='category_output_loss', factor=0.5, patience=3, verbose=1)
# 是否需要早停，当val_loss一直不下降的时候意味着模型基本训练完毕，可以停止
early_stopping = EarlyStopping(monitor='category_output_loss', min_delta=0, patience=10, verbose=1)
余弦退火学习率下降：
def cosine_decay_with_warmup(global_step,
learning_rate_base,
total_steps,
warmup_learning_rate=0.0,
warmup_steps=0,
hold_base_rate_steps=0,
min_learn_rate=0,
):
"""
参数：
global_step: 上面定义的Tcur，记录当前执行的步数。
learning_rate_base：预先设置的学习率，当warm_up阶段学习率增加到learning_rate_base，就开始学习率下降。
total_steps: 是总的训练的步数，等于epoch*sample_count/batch_size,(sample_count是样本总数，epoch是总的循环次数)
warmup_learning_rate: 这是warm up阶段线性增长的初始值
warmup_steps: warm_up总的需要持续的步数
hold_base_rate_steps: 这是可选的参数，即当warm up阶段结束后保持学习率不变，知道hold_base_rate_steps结束后才开始学习率下降
"""
if total_steps < warmup_steps:
raise ValueError('total_steps must be larger or equal to '
'warmup_steps.')
#这里实现了余弦退火的原理，设置学习率的最小值为0，所以简化了表达式
learning_rate = 0.5 * learning_rate_base * (1 + np.cos(np.pi *
(global_step - warmup_steps - hold_base_rate_steps) / float(total_steps - warmup_steps - hold_base_rate_steps)))
#如果hold_base_rate_steps大于0，表明在warm up结束后学习率在一定步数内保持不变
if hold_base_rate_steps > 0:
learning_rate = np.where(global_step > warmup_steps + hold_base_rate_steps,
learning_rate, learning_rate_base)
if warmup_steps > 0:
if learning_rate_base < warmup_learning_rate:
raise ValueError('learning_rate_base must be larger or equal to '
'warmup_learning_rate.')
#线性增长的实现
slope = (learning_rate_base - warmup_learning_rate) / warmup_steps
warmup_rate = slope * global_step + warmup_learning_rate
#只有当global_step 仍然处于warm up阶段才会使用线性增长的学习率warmup_rate，否则使用余弦退火的学习率learning_rate
learning_rate = np.where(global_step < warmup_steps, warmup_rate,
learning_rate)
learning_rate = max(learning_rate,min_learn_rate)
return learning_rateclass WarmUpCosineDecayScheduler(keras.callbacks.Callback):
"""
继承Callback，实现对学习率的调度
"""
def __init__(self,
learning_rate_base,
total_steps,
global_step_init=0,
warmup_learning_rate=0.0,
warmup_steps=0,
hold_base_rate_steps=0,
min_learn_rate=0,
# interval_epoch代表余弦退火之间的最低点
interval_epoch=[1],
verbose=0):
super(WarmUpCosineDecayScheduler, self).__init__()
# 基础的学习率
self.learning_rate_base = learning_rate_base
# 热调整参数
self.warmup_learning_rate = warmup_learning_rate
# 参数显示
self.verbose = verbose
# learning_rates用于记录每次更新后的学习率，方便图形化观察
self.min_learn_rate = min_learn_rate
self.learning_rates = []
self.interval_epoch = interval_epoch
# 贯穿全局的步长
self.global_step_for_interval = global_step_init
# 用于上升的总步长
self.warmup_steps_for_interval = warmup_steps
# 保持最高峰的总步长
self.hold_steps_for_interval = hold_base_rate_steps
# 整个训练的总步长
self.total_steps_for_interval = total_steps
self.interval_index = 0
# 计算出来两个最低点的间隔
self.interval_reset = [self.interval_epoch[0]]
for i in range(len(self.interval_epoch)-1):
self.interval_reset.append(self.interval_epoch[i+1]-self.interval_epoch[i])
self.interval_reset.append(1-self.interval_epoch[-1])
#更新global_step，并记录当前学习率
def on_batch_end(self, batch, logs=None):
self.global_step = self.global_step + 1
self.global_step_for_interval = self.global_step_for_interval + 1
lr = K.get_value(self.model.optimizer.lr)
self.learning_rates.append(lr)
#更新学习率
def on_batch_begin(self, batch, logs=None):
# 每到一次最低点就重新更新参数
if self.global_step_for_interval in [0]+[int(i*self.total_steps_for_interval) for i in self.interval_epoch]:
self.total_steps = self.total_steps_for_interval * self.interval_reset[self.interval_index]
self.warmup_steps = self.warmup_steps_for_interval * self.interval_reset[self.interval_index]
self.hold_base_rate_steps = self.hold_steps_for_interval * self.interval_reset[self.interval_index]
self.global_step = 0
self.interval_index += 1
lr = cosine_decay_with_warmup(global_step=self.global_step,
learning_rate_base=self.learning_rate_base,
total_steps=self.total_steps,
warmup_learning_rate=self.warmup_learning_rate,
warmup_steps=self.warmup_steps,
hold_base_rate_steps=self.hold_base_rate_steps,
min_learn_rate = self.min_learn_rate)
K.set_value(self.model.optimizer.lr, lr)
if self.verbose > 0:
print('\nBatch %05d: setting learning '
'rate to %s.' % (self.global_step + 1, lr))2.7模型编译
losses = {'category_output': dice_loss_with_CE(),
'color_output': instance_loss(feature_dim=5, image_shape=(256,256),delta_v=0.5, delta_d=3.0, param_var=1.0, param_dist=1.0, param_reg=0.001)}
model.compile(loss=losses ,
optimizer=Adam(lr=learning_rate_base),loss_weights=(0.5, 0.5),
metrics=[f_score(),'acc',recall(),precision()])2.8超参数设置，开始训练：
学习率：0.01；轮次：100；批处理：16
model.fit_generator(generator=gen,
steps_per_epoch=max(1, len(train_lines) // batch_size),
epochs=epochs,verbose=2,
callbacks=[checkpoint_period, tensorboard, early_stopping,
reduce_lr])2.9后处理
训练结束后，获得的数据需要进行后处理才能得出最终的输出图，本文只将laneNet网络两分支的输出图进行后处理展示，H-Net部分没有复现（H-Net后处理过程为一个特征提取网络，输入为真是地面图像，输出为坐标转换矩阵用来处理上下坡车道线位置变化情况的）。
import copy
import matplotlib.pyplot as plt
import numpy as np
from PIL import Image
from Model.model import UNET_double as lanenet
from dataload import postprocess
import os
#os.environ["CUDA_VISIBLE_DEVICES"]="-1"
def minmax_scale(input_arr):
min_val = np.min(input_arr)
max_val = np.max(input_arr)
output_arr = (input_arr - min_val) * 255.0 / (max_val - min_val)
return output_arr
def letterbox_image(image, size):
image = image.convert("RGB")
iw, ih = image.size
w, h = size
scale = min(w / iw, h / ih)
nw = int(iw * scale)
nh = int(ih * scale)
image = image.resize((nw, nh), Image.BICUBIC)
new_image = Image.new('RGB', size, (128, 128, 128))
new_image.paste(image, ((w - nw) // 2, (h - nh) // 2))
return new_image, nw, nh
def test_lanenet(image_path, weights_path,input_size=(256,256,3)):
model = lanenet(input_shape=input_size,category_classes=2,color_classes=5).build_model()
model.load_weights(weights_path)
image = Image.open(image_path)
old_img = copy.deepcopy(image)
image_vis = old_img
img, nw, nh = letterbox_image(image, (input_size[0],input_size[1]))
image_vis1=img
img = [np.array(img) / 255]
image = np.asarray(img)
binary_seg_image, instance_seg_image = model.predict(image)
'''二值图后处理'''
binary_seg_image = binary_seg_image.argmax(axis=-1).reshape([input_size[0], input_size[1]])
# 去除灰条
pr = binary_seg_image[int((input_size[0]- nh) // 2):int((input_size[0] - nh) // 2 + nh),
int((input_size[1] - nw) // 2):int((input_size[1] - nw) // 2 + nw)]
seg_img = np.zeros((np.shape(pr)[0], np.shape(pr)[1], 3))
colors = [(255, 255, 255), (0, 0, 0)]
for c in range(2):
seg_img[:, :, 0] += ((pr[:, :] == c) * (colors[c][0])).astype('uint8')
seg_img[:, :, 1] += ((pr[:, :] == c) * (colors[c][1])).astype('uint8')
seg_img[:, :, 2] += ((pr[:, :] == c) * (colors[c][2])).astype('uint8')
seg_img = np.array(Image.fromarray(np.uint8(seg_img)).resize((256, 128)))
postprocessor = postprocess.LaneNetPostProcessor()
postprocess_result = postprocessor.postprocess(
binary_seg_result=binary_seg_image,
instance_seg_result=instance_seg_image[0],
source_image=image_vis1
)
mask_image = postprocess_result['mask_image']
for i in range(5):
instance_seg_image[0][:, :, i] = minmax_scale(instance_seg_image[0][:, :, i])
embedding_image = np.array(instance_seg_image[0], np.uint8)
#去灰条
embedding_image = embedding_image[int((input_size[0] - nh) // 2):int((input_size[0] - nh) // 2 + nh),
int((input_size[1] - nw) // 2):int((input_size[1] - nw) // 2 + nw)]
plt.figure('mask_image')
plt.imshow(mask_image)
plt.figure('src_image')
plt.imshow(image_vis)
plt.figure('instance_image')
plt.imshow(embedding_image[:, :, (3, 1, 0)])
plt.figure('seg_image')
plt.imshow(seg_img)
plt.show()
return
if __name__ == '__main__':
image_path = r'D:\dataset\trainset\image\origin\0601\1494452579506899721\9.jpg'
weights_path = r'H:\unet-keras\多任务训练\aspp_convlstm_cos\lanenet_loss0.14.h5'
test_lanenet(image_path, weights_path)最终输出图：
总结
我刚开始复现的时候是一头雾水，因为没有对论文中的内容进行细致理解，因此对一篇论文完成复现，首先明确论文的研究目的，熟悉整个论文框架，再对每一块的内容进行学习理解。
635
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
