python实现keras多模态（语音文字图像）情绪识别【文末源码】

1、注：源码放置文末

1.1 环境配置要求：

https://blog.csdn.net/qq_42279468/article/details/124987801

2、展示效果：

https://www.bilibili.com/video/BV1bh4y1n7bw/

3、代码

  本项目通过python实现多模态情绪识别，使用keras框架搭建网络，包括语音、文字和图像三种处理后的数据。
  算法使用LayerNormBasicLSTMCell+注意力机制构建网络

3.1 数据集展示

3.2 训练过程

Training epoch 1
3it [00:03,  1.58s/it]	 	Epoch 1:, loss 0.868626, accuracy 0.467928
4it [00:03,  1.03it/s]

0it [00:00, ?it/s]Training epoch 2
3it [00:01,  2.13it/s]	 	Epoch 2:, loss 0.688236, accuracy 0.584177
4it [00:01,  2.33it/s]

0it [00:00, ?it/s]Training epoch 3
4it [00:01,  2.44it/s]
	 	Epoch 3:, loss 0.623478, accuracy 0.65178

0it [00:00, ?it/s]Training epoch 4
3it [00:01,  2.10it/s]	 	Epoch 4:, loss 0.472371, accuracy 0.801504
4it [00:01,  2.40it/s]

Training epoch 5
3it [00:01,  2.03it/s]	 	Epoch 5:, loss 0.429814, accuracy 0.834892
4it [00:01,  2.43it/s]

0it [00:00, ?it/s]Training epoch 6
3it [00:01,  2.09it/s]	 	Epoch 6:, loss 0.400317, accuracy 0.847018
4it [00:01,  2.44it/s]

0it [00:00, ?it/s]Training epoch 7
4it [00:01,  2.36it/s]
	 	Epoch 7:, loss 0.340432, accuracy 0.853922
0it [00:00, ?it/s]
Training epoch 8
4it [00:01,  2.24it/s]
	 	Epoch 8:, loss 0.304842, accuracy 0.884431
0it [00:00, ?it/s]
Training epoch 9
3it [00:01,  2.03it/s]	 	Epoch 9:, loss 0.275721, accuracy 0.908767
4it [00:01,  2.44it/s]

0it [00:00, ?it/s]Training epoch 10
4it [00:01,  2.54it/s]
	 	Epoch 10:, loss 0.242068, accuracy 0.931539

0it [00:00, ?it/s]Training epoch 11
3it [00:01,  2.03it/s]	 	Epoch 11:, loss 0.246808, accuracy 0.910064
4it [00:01,  2.36it/s]
0it [00:00, ?it/s]
Training epoch 12
4it [00:01,  2.29it/s]
	 	Epoch 12:, loss 0.206893, accuracy 0.939517

0it [00:00, ?it/s]Training epoch 13
4it [00:01,  2.57it/s]
	 	Epoch 13:, loss 0.185851, accuracy 0.944807
0it [00:00, ?it/s]
Training epoch 14
4it [00:01,  2.29it/s]
	 	Epoch 14:, loss 0.157054, accuracy 0.95869

0it [00:00, ?it/s]Training epoch 15
4it [00:01,  2.42it/s]
	 	Epoch 15:, loss 0.166478, accuracy 0.952298
0it [00:00, ?it/s]
Training epoch 16
3it [00:01,  2.09it/s]	 	Epoch 16:, loss 0.149369, accuracy 0.971128
4it [00:01,  2.35it/s]

0it [00:00, ?it/s]Training epoch 17
4it [00:01,  2.31it/s]
	 	Epoch 17:, loss 0.124102, accuracy 0.975202
0it [00:00, ?it/s]
Training epoch 18
4it [00:01,  2.31it/s]
	 	Epoch 18:, loss 0.127283, accuracy 0.965784

0it [00:00, ?it/s]Training epoch 19
4it [00:01,  2.14it/s]
	 	Epoch 19:, loss 0.114086, accuracy 0.972137

0it [00:00, ?it/s]Training epoch 20
3it [00:01,  1.97it/s]	 	Epoch 20:, loss 0.121704, accuracy 0.973938
4it [00:01,  2.22it/s]

0it [00:00, ?it/s]Training epoch 21
3it [00:01,  2.10it/s]	 	Epoch 21:, loss 0.103751, accuracy 0.969179
4it [00:01,  2.31it/s]
0it [00:00, ?it/s]
Training epoch 22
3it [00:01,  2.17it/s]	 	Epoch 22:, loss 0.114447, accuracy 0.968958
4it [00:01,  2.31it/s]

0it [00:00, ?it/s]Training epoch 23
3it [00:01,  2.16it/s]	 	Epoch 23:, loss 0.0959018, accuracy 0.978891
4it [00:01,  2.34it/s]

0it [00:00, ?it/s]Training epoch 24
4it [00:01,  2.59it/s]
	 	Epoch 24:, loss 0.0853932, accuracy 0.986028

0it [00:00, ?it/s]Training epoch 25
4it [00:01,  2.44it/s]
	 	Epoch 25:, loss 0.0839167, accuracy 0.987315

0it [00:00, ?it/s]Training epoch 26
4it [00:01,  2.42it/s]
	 	Epoch 26:, loss 0.0777089, accuracy 0.986904

Training epoch 27
3it [00:01,  1.93it/s]	 	Epoch 27:, loss 0.074715, accuracy 0.987368
4it [00:01,  2.21it/s]

0it [00:00, ?it/s]Training epoch 28
3it [00:01,  2.09it/s]	 	Epoch 28:, loss 0.0690631, accuracy 0.987052
4it [00:01,  2.41it/s]

0it [00:00, ?it/s]Training epoch 29
4it [00:01,  2.39it/s]
	 	Epoch 29:, loss 0.0754572, accuracy 0.986618

0it [00:00, ?it/s]Training epoch 30
4it [00:01,  2.40it/s]
	 	Epoch 30:, loss 0.0808934, accuracy 0.978521

0it [00:00, ?it/s]Training epoch 31
3it [00:01,  2.03it/s]	 	Epoch 31:, loss 0.0768318, accuracy 0.977635
4it [00:01,  2.35it/s]
0it [00:00, ?it/s]
Training epoch 32
3it [00:01,  2.10it/s]	 	Epoch 32:, loss 0.0907751, accuracy 0.97986
4it [00:01,  2.31it/s]

Training epoch 33
3it [00:01,  2.10it/s]	 	Epoch 33:, loss 0.0650676, accuracy 0.984582
4it [00:01,  2.30it/s]

0it [00:00, ?it/s]Training epoch 34
4it [00:01,  2.32it/s]
	 	Epoch 34:, loss 0.084893, accuracy 0.9796

Training epoch 35
3it [00:01,  2.26it/s]	 	Epoch 35:, loss 0.0657072, accuracy 0.97979
4it [00:01,  2.30it/s]

0it [00:00, ?it/s]Training epoch 36
3it [00:01,  2.25it/s]	 	Epoch 36:, loss 0.0571314, accuracy 0.989951
4it [00:01,  2.51it/s]

0it [00:00, ?it/s]Training epoch 37
4it [00:01,  2.17it/s]
	 	Epoch 37:, loss 0.0639608, accuracy 0.984723
0it [00:00, ?it/s]
Training epoch 38
3it [00:01,  2.01it/s]	 	Epoch 38:, loss 0.0603083, accuracy 0.987819
4it [00:01,  2.13it/s]
0it [00:00, ?it/s]
Training epoch 39
3it [00:01,  2.08it/s]	 	Epoch 39:, loss 0.0597859, accuracy 0.987659
4it [00:01,  2.35it/s]
0it [00:00, ?it/s]
Training epoch 40
4it [00:01,  2.29it/s]
	 	Epoch 40:, loss 0.0657729, accuracy 0.987306
0it [00:00, ?it/s]
Training epoch 41
3it [00:01,  1.99it/s]	 	Epoch 41:, loss 0.0577475, accuracy 0.989388
4it [00:01,  2.38it/s]

0it [00:00, ?it/s]Training epoch 42
3it [00:01,  1.97it/s]	 	Epoch 42:, loss 0.0561785, accuracy 0.985871
4it [00:01,  2.26it/s]

0it [00:00, ?it/s]Training epoch 43
4it [00:01,  2.48it/s]
	 	Epoch 43:, loss 0.0551007, accuracy 0.987269

0it [00:00, ?it/s]Training epoch 44
3it [00:01,  1.92it/s]	 	Epoch 44:, loss 0.0516939, accuracy 0.986137
4it [00:01,  2.24it/s]

Training epoch 45
4it [00:01,  2.34it/s]
	 	Epoch 45:, loss 0.0713419, accuracy 0.978741
0it [00:00, ?it/s]
Training epoch 46
4it [00:01,  2.31it/s]
	 	Epoch 46:, loss 0.078447, accuracy 0.978097
0it [00:00, ?it/s]
Training epoch 47
4it [00:01,  2.20it/s]	 	Epoch 47:, loss 0.0496744, accuracy 0.987609
4it [00:01,  2.23it/s]

0it [00:00, ?it/s]Training epoch 48
4it [00:01,  2.41it/s]
	 	Epoch 48:, loss 0.054719, accuracy 0.98776
0it [00:00, ?it/s]
Training epoch 49
4it [00:01,  2.30it/s]
	 	Epoch 49:, loss 0.0739138, accuracy 0.978141

0it [00:00, ?it/s]Training epoch 50
3it [00:01,  2.07it/s]	 	Epoch 50:, loss 0.0537297, accuracy 0.987054
4it [00:01,  2.12it/s]

0it [00:00, ?it/s]Training epoch 51
4it [00:01,  2.29it/s]
	 	Epoch 51:, loss 0.0486002, accuracy 0.98399

0it [00:00, ?it/s]Training epoch 52
4it [00:01,  2.31it/s]
	 	Epoch 52:, loss 0.0549857, accuracy 0.986394

0it [00:00, ?it/s]Training epoch 53
4it [00:01,  2.21it/s]
	 	Epoch 53:, loss 0.0684459, accuracy 0.982455

0it [00:00, ?it/s]Training epoch 54
3it [00:01,  2.20it/s]	 	Epoch 54:, loss 0.0616392, accuracy 0.981636
4it [00:01,  2.41it/s]

3.3 模型评估

4 源码下载

https://gitcode.net/qq_42279468/python-muti-fuse.git