input自适应_一种Dynamic ReLU：自适应参数化ReLU激活函数（调参记录13）

自适应参数化ReLU，本身就是一种动态的ReLU（Dynamic ReLU，Dy-ReLU）。该论文在2019年5月的时候，投稿到IEEE Transactions on Industrial Electronics，在2020年1月被录用，2020年2月在IEEE官网公布。

从以往的调参记录来看，过拟合是最核心的问题。本文在调参记录12的基础上，将层数减少，减到9个残差块，再试一次。

自适应参数化ReLU原理如下：

自适应参数化ReLU（一种Dynamic ReLU，Dy-ReLU）

Keras程序：

#!/usr/bin/env python3

实验结果如下：

Epoch 2500/5000
12s 151ms/step - loss: 0.1258 - acc: 0.9867 - val_loss: 0.4697 - val_acc: 0.9024
Epoch 2501/5000
12s 151ms/step - loss: 0.1274 - acc: 0.9852 - val_loss: 0.4688 - val_acc: 0.9026
Epoch 2502/5000
12s 151ms/step - loss: 0.1260 - acc: 0.9861 - val_loss: 0.4585 - val_acc: 0.9040
Epoch 2503/5000
12s 152ms/step - loss: 0.1241 - acc: 0.9869 - val_loss: 0.4489 - val_acc: 0.9066
Epoch 2504/5000
12s 152ms/step - loss: 0.1236 - acc: 0.9869 - val_loss: 0.4469 - val_acc: 0.9106
Epoch 2505/5000
12s 151ms/step - loss: 0.1276 - acc: 0.9850 - val_loss: 0.4515 - val_acc: 0.9034
Epoch 2506/5000
12s 151ms/step - loss: 0.1252 - acc: 0.9864 - val_loss: 0.4586 - val_acc: 0.9074
Epoch 2507/5000
12s 151ms/step - loss: 0.1289 - acc: 0.9852 - val_loss: 0.4585 - val_acc: 0.9057
Epoch 2508/5000
12s 151ms/step - loss: 0.1285 - acc: 0.9853 - val_loss: 0.4485 - val_acc: 0.9077
Epoch 2509/5000
12s 151ms/step - loss: 0.1284 - acc: 0.9851 - val_loss: 0.4529 - val_acc: 0.9032
Epoch 2510/5000
12s 151ms/step - loss: 0.1287 - acc: 0.9855 - val_loss: 0.4567 - val_acc: 0.9040
Epoch 2511/5000
12s 151ms/step - loss: 0.1253 - acc: 0.9862 - val_loss: 0.4554 - val_acc: 0.9080
Epoch 2512/5000
12s 152ms/step - loss: 0.1262 - acc: 0.9859 - val_loss: 0.4477 - val_acc: 0.9086
Epoch 2513/5000
12s 151ms/step - loss: 0.1241 - acc: 0.9864 - val_loss: 0.4531 - val_acc: 0.9063
Epoch 2514/5000
12s 150ms/step - loss: 0.1247 - acc: 0.9866 - val_loss: 0.4484 - val_acc: 0.9073
Epoch 2515/5000
12s 151ms/step - loss: 0.1239 - acc: 0.9869 - val_loss: 0.4502 - val_acc: 0.9078
Epoch 2516/5000
12s 151ms/step - loss: 0.1275 - acc: 0.9857 - val_loss: 0.4790 - val_acc: 0.8981
Epoch 2517/5000
12s 152ms/step - loss: 0.1259 - acc: 0.9862 - val_loss: 0.4625 - val_acc: 0.9063
Epoch 2518/5000
12s 151ms/step - loss: 0.1278 - acc: 0.9853 - val_loss: 0.4751 - val_acc: 0.9009
Epoch 2519/5000
12s 151ms/step - loss: 0.1283 - acc: 0.9857 - val_loss: 0.4655 - val_acc: 0.9056
Epoch 2520/5000
12s 151ms/step - loss: 0.1275 - acc: 0.9859 - val_loss: 0.4386 - val_acc: 0.9085
Epoch 2521/5000
12s 151ms/step - loss: 0.1245 - acc: 0.9871 - val_loss: 0.4699 - val_acc: 0.9006
Epoch 2522/5000
12s 151ms/step - loss: 0.1278 - acc: 0.9860 - val_loss: 0.4520 - val_acc: 0.9050
Epoch 2523/5000
12s 151ms/step - loss: 0.1249 - acc: 0.9864 - val_loss: 0.4566 - val_acc: 0.9056
Epoch 2524/5000
12s 152ms/step - loss: 0.1278 - acc: 0.9855 - val_loss: 0.4650 - val_acc: 0.9018
Epoch 2525/5000
12s 151ms/step - loss: 0.1235 - acc: 0.9873 - val_loss: 0.4555 - val_acc: 0.9061
Epoch 2526/5000
12s 151ms/step - loss: 0.1260 - acc: 0.9862 - val_loss: 0.4556 - val_acc: 0.9061
Epoch 2527/5000
12s 152ms/step - loss: 0.1261 - acc: 0.9866 - val_loss: 0.4667 - val_acc: 0.9040
Epoch 2528/5000
12s 152ms/step - loss: 0.1240 - acc: 0.9874 - val_loss: 0.4539 - val_acc: 0.9083
Epoch 2529/5000
12s 152ms/step - loss: 0.1281 - acc: 0.9856 - val_loss: 0.4584 - val_acc: 0.9048
Epoch 2530/5000
12s 151ms/step - loss: 0.1234 - acc: 0.9871 - val_loss: 0.4538 - val_acc: 0.9048
Epoch 2531/5000
12s 151ms/step - loss: 0.1235 - acc: 0.9868 - val_loss: 0.4504 - val_acc: 0.9056
Epoch 2532/5000
12s 151ms/step - loss: 0.1247 - acc: 0.9871 - val_loss: 0.4529 - val_acc: 0.9053
Epoch 2533/5000
12s 150ms/step - loss: 0.1241 - acc: 0.9872 - val_loss: 0.4591 - val_acc: 0.9034
Epoch 2534/5000
12s 152ms/step - loss: 0.1255 - acc: 0.9865 - val_loss: 0.4502 - val_acc: 0.9058
Epoch 2535/5000
12s 151ms/step - loss: 0.1254 - acc: 0.9865 - val_loss: 0.4596 - val_acc: 0.9039
Epoch 2536/5000
12s 152ms/step - loss: 0.1239 - acc: 0.9872 - val_loss: 0.4488 - val_acc: 0.9040
Epoch 2537/5000
12s 151ms/step - loss: 0.1260 - acc: 0.9865 - val_loss: 0.4494 - val_acc: 0.9042
Epoch 2538/5000
12s 150ms/step - loss: 0.1288 - acc: 0.9851 - val_loss: 0.4621 - val_acc: 0.9039
Epoch 2539/5000
12s 152ms/step - loss: 0.1267 - acc: 0.9855 - val_loss: 0.4497 - val_acc: 0.9068
Epoch 2540/5000
12s 151ms/step - loss: 0.1250 - acc: 0.9869 - val_loss: 0.4626 - val_acc: 0.9024
Epoch 2541/5000
12s 152ms/step - loss: 0.1272 - acc: 0.9856 - val_loss: 0.4621 - val_acc: 0.9038
Epoch 2542/5000
12s 151ms/step - loss: 0.1258 - acc: 0.9862 - val_loss: 0.4738 - val_acc: 0.9044
Epoch 2543/5000
12s 152ms/step - loss: 0.1257 - acc: 0.9862 - val_loss: 0.4597 - val_acc: 0.9061
Epoch 2544/5000
12s 151ms/step - loss: 0.1271 - acc: 0.9854 - val_loss: 0.4571 - val_acc: 0.9008
Epoch 2545/5000
12s 151ms/step - loss: 0.1247 - acc: 0.9861 - val_loss: 0.4450 - val_acc: 0.9065
Epoch 2546/5000
12s 152ms/step - loss: 0.1273 - acc: 0.9860 - val_loss: 0.4568 - val_acc: 0.9031
Epoch 2547/5000
12s 151ms/step - loss: 0.1291 - acc: 0.9855 - val_loss: 0.4558 - val_acc: 0.9034
Epoch 2548/5000
12s 152ms/step - loss: 0.1280 - acc: 0.9849 - val_loss: 0.4463 - val_acc: 0.9077
Epoch 2549/5000
12s 151ms/step - loss: 0.1237 - acc: 0.9868 - val_loss: 0.4427 - val_acc: 0.9083
Epoch 2550/5000
12s 151ms/step - loss: 0.1247 - acc: 0.9865 - val_loss: 0.4486 - val_acc: 0.9060
Epoch 2551/5000
12s 152ms/step - loss: 0.1265 - acc: 0.9864 - val_loss: 0.4414 - val_acc: 0.9047
Epoch 2552/5000
12s 151ms/step - loss: 0.1275 - acc: 0.9859 - val_loss: 0.4652 - val_acc: 0.9003
Epoch 2553/5000
12s 151ms/step - loss: 0.1241 - acc: 0.9864 - val_loss: 0.4713 - val_acc: 0.8976
Epoch 2554/5000
12s 152ms/step - loss: 0.1258 - acc: 0.9862 - val_loss: 0.4549 - val_acc: 0.9048
Epoch 2555/5000
12s 151ms/step - loss: 0.1249 - acc: 0.9866 - val_loss: 0.4376 - val_acc: 0.9069
Epoch 2556/5000
12s 152ms/step - loss: 0.1251 - acc: 0.9866 - val_loss: 0.4519 - val_acc: 0.9062
Epoch 2557/5000
12s 151ms/step - loss: 0.1269 - acc: 0.9857 - val_loss: 0.4479 - val_acc: 0.9069
Epoch 2558/5000
12s 151ms/step - loss: 0.1240 - acc: 0.9870 - val_loss: 0.4629 - val_acc: 0.9023
Epoch 2559/5000
12s 151ms/step - loss: 0.1257 - acc: 0.9866 - val_loss: 0.4487 - val_acc: 0.9039
Epoch 2560/5000
12s 151ms/step - loss: 0.1272 - acc: 0.9859 - val_loss: 0.4574 - val_acc: 0.9029
Epoch 2561/5000
12s 152ms/step - loss: 0.1238 - acc: 0.9872 - val_loss: 0.4530 - val_acc: 0.9073
Epoch 2562/5000
12s 152ms/step - loss: 0.1226 - acc: 0.9872 - val_loss: 0.4589 - val_acc: 0.9048
Epoch 2563/5000
12s 151ms/step - loss: 0.1283 - acc: 0.9854 - val_loss: 0.4525 - val_acc: 0.9032
Epoch 2564/5000
12s 151ms/step - loss: 0.1286 - acc: 0.9851 - val_loss: 0.4488 - val_acc: 0.9063
Epoch 2565/5000
12s 150ms/step - loss: 0.1263 - acc: 0.9862 - val_loss: 0.4520 - val_acc: 0.9044
Epoch 2566/5000
12s 152ms/step - loss: 0.1280 - acc: 0.9854 - val_loss: 0.4561 - val_acc: 0.9025
Epoch 2567/5000
12s 151ms/step - loss: 0.1259 - acc: 0.9860 - val_loss: 0.4532 - val_acc: 0.9034
Epoch 2568/5000
12s 156ms/step - loss: 0.1249 - acc: 0.9864 - val_loss: 0.4449 - val_acc: 0.9072
Epoch 2569/5000
12s 152ms/step - loss: 0.1269 - acc: 0.9857 - val_loss: 0.4465 - val_acc: 0.9056
Epoch 2570/5000
12s 153ms/step - loss: 0.1282 - acc: 0.9853 - val_loss: 0.4445 - val_acc: 0.9074
...
Epoch 2877/5000
12s 151ms/step - loss: 0.1217 - acc: 0.9872 - val_loss: 0.4713 - val_acc: 0.9009
Epoch 2878/5000
12s 152ms/step - loss: 0.1227 - acc: 0.9877 - val_loss: 0.4590 - val_acc: 0.9031
Epoch 2879/5000
12s 150ms/step - loss: 0.1247 - acc: 0.9863 - val_loss: 0.4390 - val_acc: 0.9054
Epoch 2880/5000
12s 152ms/step - loss: 0.1251 - acc: 0.9865 - val_loss: 0.4582 - val_acc: 0.9025
Epoch 2881/5000
12s 151ms/step - loss: 0.1261 - acc: 0.9859 - val_loss: 0.4480 - val_acc: 0.9053
Epoch 2882/5000
12s 152ms/step - loss: 0.1228 - acc: 0.9869 - val_loss: 0.4430 - val_acc: 0.9089
Epoch 2883/5000
12s 151ms/step - loss: 0.1215 - acc: 0.9869 - val_loss: 0.4476 - val_acc: 0.9061
Epoch 2884/5000
12s 151ms/step - loss: 0.1272 - acc: 0.9856 - val_loss: 0.4586 - val_acc: 0.9062
Epoch 2885/5000
12s 151ms/step - loss: 0.1243 - acc: 0.9861 - val_loss: 0.4557 - val_acc: 0.9021
Epoch 2886/5000
12s 151ms/step - loss: 0.1240 - acc: 0.9858 - val_loss: 0.4631 - val_acc: 0.9041
Epoch 2887/5000
12s 152ms/step - loss: 0.1234 - acc: 0.9870 - val_loss: 0.4378 - val_acc: 0.9074
Epoch 2888/5000
12s 152ms/step - loss: 0.1245 - acc: 0.9865 - val_loss: 0.4415 - val_acc: 0.9072
Epoch 2889/5000
12s 151ms/step - loss: 0.1252 - acc: 0.9865 - val_loss: 0.4535 - val_acc: 0.9072
Epoch 2890/5000
12s 151ms/step - loss: 0.1218 - acc: 0.9869 - val_loss: 0.4449 - val_acc: 0.9073
Epoch 2891/5000
12s 152ms/step - loss: 0.1268 - acc: 0.9852 - val_loss: 0.4485 - val_acc: 0.9015
Epoch 2892/5000
12s 152ms/step - loss: 0.1248 - acc: 0.9865 - val_loss: 0.4578 - val_acc: 0.9034
Epoch 2893/5000
12s 151ms/step - loss: 0.1236 - acc: 0.9870 - val_loss: 0.4452 - val_acc: 0.9067
Epoch 2894/5000
12s 151ms/step - loss: 0.1251 - acc: 0.9866 - val_loss: 0.4537 - val_acc: 0.9031
Epoch 2895/5000
12s 151ms/step - loss: 0.1268 - acc: 0.9859 - val_loss: 0.4650 - val_acc: 0.9049
Epoch 2896/5000
12s 151ms/step - loss: 0.1245 - acc: 0.9864 - val_loss: 0.4558 - val_acc: 0.9049
Epoch 2897/5000
12s 152ms/step - loss: 0.1225 - acc: 0.9867 - val_loss: 0.4567 - val_acc: 0.9062
Epoch 2898/5000
12s 151ms/step - loss: 0.1227 - acc: 0.9873 - val_loss: 0.4518 - val_acc: 0.9032
Epoch 2899/5000
12s 151ms/step - loss: 0.1224 - acc: 0.9877 - val_loss: 0.4370 - val_acc: 0.9065
Epoch 2900/5000
12s 151ms/step - loss: 0.1232 - acc: 0.9870 - val_loss: 0.4514 - val_acc: 0.9074
Epoch 2901/5000
12s 151ms/step - loss: 0.1230 - acc: 0.9867 - val_loss: 0.4497 - val_acc: 0.9035
Epoch 2902/5000
12s 152ms/step - loss: 0.1242 - acc: 0.9870 - val_loss: 0.4468 - val_acc: 0.9046
Epoch 2903/5000
12s 151ms/step - loss: 0.1241 - acc: 0.9866 - val_loss: 0.4631 - val_acc: 0.9030
Epoch 2904/5000
12s 152ms/step - loss: 0.1232 - acc: 0.9871 - val_loss: 0.4451 - val_acc: 0.9057
Epoch 2905/5000
12s 152ms/step - loss: 0.1217 - acc: 0.9869 - val_loss: 0.4466 - val_acc: 0.9079
Epoch 2906/5000
12s 150ms/step - loss: 0.1249 - acc: 0.9861 - val_loss: 0.4484 - val_acc: 0.9049
Epoch 2907/5000
12s 152ms/step - loss: 0.1253 - acc: 0.9865 - val_loss: 0.4467 - val_acc: 0.9079
Epoch 2908/5000
12s 152ms/step - loss: 0.1250 - acc: 0.9866 - val_loss: 0.4523 - val_acc: 0.9040
Epoch 2909/5000
12s 151ms/step - loss: 0.1261 - acc: 0.9855 - val_loss: 0.4477 - val_acc: 0.9075
Epoch 2910/5000
12s 152ms/step - loss: 0.1247 - acc: 0.9866 - val_loss: 0.4358 - val_acc: 0.9090
Epoch 2911/5000
12s 150ms/step - loss: 0.1222 - acc: 0.9876 - val_loss: 0.4622 - val_acc: 0.9016
Epoch 2912/5000
12s 152ms/step - loss: 0.1246 - acc: 0.9863 - val_loss: 0.4487 - val_acc: 0.9069
Epoch 2913/5000
12s 152ms/step - loss: 0.1231 - acc: 0.9873 - val_loss: 0.4466 - val_acc: 0.9051
Epoch 2914/5000
12s 151ms/step - loss: 0.1240 - acc: 0.9866 - val_loss: 0.4554 - val_acc: 0.9051
Epoch 2915/5000
12s 152ms/step - loss: 0.1232 - acc: 0.9872 - val_loss: 0.4558 - val_acc: 0.9063
Epoch 2916/5000
12s 151ms/step - loss: 0.1217 - acc: 0.9870 - val_loss: 0.4533 - val_acc: 0.9055
Epoch 2917/5000
12s 151ms/step - loss: 0.1242 - acc: 0.9866 - val_loss: 0.4584 - val_acc: 0.9017
Epoch 2918/5000
12s 152ms/step - loss: 0.1233 - acc: 0.9871 - val_loss: 0.4594 - val_acc: 0.9025
Epoch 2919/5000
12s 151ms/step - loss: 0.1213 - acc: 0.9876 - val_loss: 0.4582 - val_acc: 0.9034
Epoch 2920/5000
12s 151ms/step - loss: 0.1226 - acc: 0.9872 - val_loss: 0.4494 - val_acc: 0.9047
Epoch 2921/5000
12s 151ms/step - loss: 0.1218 - acc: 0.9872 - val_loss: 0.4576 - val_acc: 0.9066
Epoch 2922/5000
12s 151ms/step - loss: 0.1243 - acc: 0.9863 - val_loss: 0.4597 - val_acc: 0.9055
Epoch 2923/5000
12s 152ms/step - loss: 0.1269 - acc: 0.9852 - val_loss: 0.4563 - val_acc: 0.9054
Epoch 2924/5000
12s 151ms/step - loss: 0.1242 - acc: 0.9865 - val_loss: 0.4465 - val_acc: 0.9038
Epoch 2925/5000
12s 152ms/step - loss: 0.1218 - acc: 0.9872 - val_loss: 0.4531 - val_acc: 0.9027
Epoch 2926/5000
12s 151ms/step - loss: 0.1247 - acc: 0.9863 - val_loss: 0.4551 - val_acc: 0.9046
Epoch 2927/5000
12s 151ms/step - loss: 0.1242 - acc: 0.9866 - val_loss: 0.4591 - val_acc: 0.9019
Epoch 2928/5000
12s 151ms/step - loss: 0.1232 - acc: 0.9867 - val_loss: 0.4550 - val_acc: 0.9037
Epoch 2929/5000
12s 151ms/step - loss: 0.1216 - acc: 0.9879 - val_loss: 0.4495 - val_acc: 0.9054
Epoch 2930/5000
12s 152ms/step - loss: 0.1228 - acc: 0.9871 - val_loss: 0.4478 - val_acc: 0.9043
Epoch 2931/5000
12s 152ms/step - loss: 0.1243 - acc: 0.9859 - val_loss: 0.4601 - val_acc: 0.9025
Epoch 2932/5000
12s 152ms/step - loss: 0.1238 - acc: 0.9865 - val_loss: 0.4561 - val_acc: 0.9050
Epoch 2933/5000
12s 152ms/step - loss: 0.1233 - acc: 0.9873 - val_loss: 0.4625 - val_acc: 0.9024
Epoch 2934/5000
12s 152ms/step - loss: 0.1245 - acc: 0.9859 - val_loss: 0.4558 - val_acc: 0.9025
Epoch 2935/5000
12s 152ms/step - loss: 0.1252 - acc: 0.9862 - val_loss: 0.4648 - val_acc: 0.9030
Epoch 2936/5000
12s 151ms/step - loss: 0.1229 - acc: 0.9872 - val_loss: 0.4648 - val_acc: 0.9024
Epoch 2937/5000
12s 152ms/step - loss: 0.1233 - acc: 0.9867 - val_loss: 0.4577 - val_acc: 0.9015
Epoch 2938/5000
12s 151ms/step - loss: 0.1266 - acc: 0.9854 - val_loss: 0.4721 - val_acc: 0.9011
Epoch 2939/5000
12s 151ms/step - loss: 0.1236 - acc: 0.9868 - val_loss: 0.4562 - val_acc: 0.9050
Epoch 2940/5000
12s 152ms/step - loss: 0.1221 - acc: 0.9868 - val_loss: 0.4583 - val_acc: 0.9046
Epoch 2941/5000
12s 152ms/step - loss: 0.1244 - acc: 0.9863 - val_loss: 0.4601 - val_acc: 0.9039
Epoch 2942/5000
12s 151ms/step - loss: 0.1234 - acc: 0.9873 - val_loss: 0.4710 - val_acc: 0.9021
Epoch 2943/5000
12s 152ms/step - loss: 0.1227 - acc: 0.9869 - val_loss: 0.4574 - val_acc: 0.9057
Epoch 2944/5000
12s 151ms/step - loss: 0.1245 - acc: 0.9862 - val_loss: 0.4752 - val_acc: 0.9023
Epoch 2945/5000
12s 151ms/step - loss: 0.1228 - acc: 0.9866 - val_loss: 0.4870 - val_acc: 0.8995
Epoch 2946/5000
12s 151ms/step - loss: 0.1230 - acc: 0.9870 - val_loss: 0.4680 - val_acc: 0.9030
Epoch 2947/5000
12s 151ms/step - loss: 0.1240 - acc: 0.9870 - val_loss: 0.4631 - val_acc: 0.9043
Epoch 2948/5000
12s 151ms/step - loss: 0.1244 - acc: 0.9863 - val_loss: 0.4531 - val_acc: 0.9046
Epoch 2949/5000
12s 151ms/step - loss: 0.1265 - acc: 0.9857 - val_loss: 0.4524 - val_acc: 0.9047
Epoch 2950/5000
12s 151ms/step - loss: 0.1242 - acc: 0.9870 - val_loss: 0.4612 - val_acc: 0.9021
Epoch 2951/5000
12s 152ms/step - loss: 0.1252 - acc: 0.9861 - val_loss: 0.4594 - val_acc: 0.9056
Epoch 2952/5000
12s 151ms/step - loss: 0.1238 - acc: 0.9865 - val_loss: 0.4678 - val_acc: 0.9039
Epoch 2953/5000
12s 151ms/step - loss: 0.1195 - acc: 0.9883 - val_loss: 0.4595 - val_acc: 0.9059
Epoch 2954/5000
12s 151ms/step - loss: 0.1219 - acc: 0.9870 - val_loss: 0.4533 - val_acc: 0.9056
Epoch 2955/5000
12s 152ms/step - loss: 0.1266 - acc: 0.9854 - val_loss: 0.4631 - val_acc: 0.9023
Epoch 2956/5000
12s 151ms/step - loss: 0.1252 - acc: 0.9856 - val_loss: 0.4567 - val_acc: 0.9050
Epoch 2957/5000
12s 151ms/step - loss: 0.1263 - acc: 0.9862 - val_loss: 0.4424 - val_acc: 0.9104
Epoch 2958/5000
12s 151ms/step - loss: 0.1221 - acc: 0.9871 - val_loss: 0.4534 - val_acc: 0.9059
Epoch 2959/5000
12s 152ms/step - loss: 0.1227 - acc: 0.9869 - val_loss: 0.4523 - val_acc: 0.9097
Epoch 2960/5000
12s 150ms/step - loss: 0.1237 - acc: 0.9874 - val_loss: 0.4554 - val_acc: 0.9057
Epoch 2961/5000
12s 151ms/step - loss: 0.1246 - acc: 0.9860 - val_loss: 0.4488 - val_acc: 0.9077
Epoch 2962/5000
12s 152ms/step - loss: 0.1235 - acc: 0.9872 - val_loss: 0.4559 - val_acc: 0.9021
Epoch 2963/5000
12s 151ms/step - loss: 0.1226 - acc: 0.9873 - val_loss: 0.4650 - val_acc: 0.9019
Epoch 2964/5000
12s 151ms/step - loss: 0.1259 - acc: 0.9858 - val_loss: 0.4653 - val_acc: 0.9009
Epoch 2965/5000
12s 151ms/step - loss: 0.1259 - acc: 0.9861 - val_loss: 0.4566 - val_acc: 0.9026
Epoch 2966/5000
12s 151ms/step - loss: 0.1221 - acc: 0.9873 - val_loss: 0.4626 - val_acc: 0.9038
Epoch 2967/5000
12s 152ms/step - loss: 0.1251 - acc: 0.9861 - val_loss: 0.4591 - val_acc: 0.9049
Epoch 2968/5000
12s 151ms/step - loss: 0.1242 - acc: 0.9861 - val_loss: 0.4526 - val_acc: 0.9073
Epoch 2969/5000
12s 152ms/step - loss: 0.1235 - acc: 0.9865 - val_loss: 0.4467 - val_acc: 0.9064
Epoch 2970/5000
12s 151ms/step - loss: 0.1272 - acc: 0.9854 - val_loss: 0.4561 - val_acc: 0.9058
Epoch 2971/5000
12s 152ms/step - loss: 0.1233 - acc: 0.9869 - val_loss: 0.4655 - val_acc: 0.9031
Epoch 2972/5000
12s 152ms/step - loss: 0.1221 - acc: 0.9871 - val_loss: 0.4414 - val_acc: 0.9078
Epoch 2973/5000
12s 151ms/step - loss: 0.1232 - acc: 0.9867 - val_loss: 0.4539 - val_acc: 0.9062
Epoch 2974/5000
12s 152ms/step - loss: 0.1253 - acc: 0.9860 - val_loss: 0.4650 - val_acc: 0.9040
Epoch 2975/5000
12s 151ms/step - loss: 0.1249 - acc: 0.9865 - val_loss: 0.4454 - val_acc: 0.9089
Epoch 2976/5000
12s 151ms/step - loss: 0.1219 - acc: 0.9878 - val_loss: 0.4514 - val_acc: 0.9044
Epoch 2977/5000
12s 152ms/step - loss: 0.1255 - acc: 0.9853 - val_loss: 0.4585 - val_acc: 0.9056
Epoch 2978/5000
12s 151ms/step - loss: 0.1242 - acc: 0.9862 - val_loss: 0.4536 - val_acc: 0.9058
Epoch 2979/5000
12s 152ms/step - loss: 0.1248 - acc: 0.9864 - val_loss: 0.4638 - val_acc: 0.9048
Epoch 2980/5000
12s 150ms/step - loss: 0.1275 - acc: 0.9854 - val_loss: 0.4498 - val_acc: 0.9050
Epoch 2981/5000
12s 152ms/step - loss: 0.1260 - acc: 0.9856 - val_loss: 0.4647 - val_acc: 0.9033
Epoch 2982/5000
12s 151ms/step - loss: 0.1245 - acc: 0.9865 - val_loss: 0.4649 - val_acc: 0.9033
Epoch 2983/5000
12s 151ms/step - loss: 0.1265 - acc: 0.9857 - val_loss: 0.4429 - val_acc: 0.9103
Epoch 2984/5000
12s 151ms/step - loss: 0.1242 - acc: 0.9866 - val_loss: 0.4585 - val_acc: 0.9023
Epoch 2985/5000
12s 152ms/step - loss: 0.1257 - acc: 0.9858 - val_loss: 0.4490 - val_acc: 0.9038
Epoch 2986/5000
12s 151ms/step - loss: 0.1252 - acc: 0.9863 - val_loss: 0.4498 - val_acc: 0.9043
Epoch 2987/5000
12s 151ms/step - loss: 0.1248 - acc: 0.9864 - val_loss: 0.4459 - val_acc: 0.9058
Epoch 2988/5000
12s 152ms/step - loss: 0.1226 - acc: 0.9868 - val_loss: 0.4559 - val_acc: 0.9032
Epoch 2989/5000
12s 151ms/step - loss: 0.1237 - acc: 0.9868 - val_loss: 0.4560 - val_acc: 0.9051
Epoch 2990/5000
12s 151ms/step - loss: 0.1242 - acc: 0.9870 - val_loss: 0.4648 - val_acc: 0.9033
Epoch 2991/5000
12s 152ms/step - loss: 0.1245 - acc: 0.9863 - val_loss: 0.4604 - val_acc: 0.9018
Epoch 2992/5000
12s 151ms/step - loss: 0.1240 - acc: 0.9865 - val_loss: 0.4575 - val_acc: 0.9036
Epoch 2993/5000
13s 157ms/step - loss: 0.1226 - acc: 0.9872 - val_loss: 0.4626 - val_acc: 0.9028
Epoch 2994/5000
12s 152ms/step - loss: 0.1255 - acc: 0.9860 - val_loss: 0.4777 - val_acc: 0.8998
Epoch 2995/5000
12s 151ms/step - loss: 0.1256 - acc: 0.9860 - val_loss: 0.4574 - val_acc: 0.9023
Epoch 2996/5000
12s 151ms/step - loss: 0.1232 - acc: 0.9866 - val_loss: 0.4663 - val_acc: 0.9024
Epoch 2997/5000
12s 151ms/step - loss: 0.1210 - acc: 0.9881 - val_loss: 0.4663 - val_acc: 0.9046
Epoch 2998/5000
12s 152ms/step - loss: 0.1201 - acc: 0.9876 - val_loss: 0.4492 - val_acc: 0.9065
Epoch 2999/5000
12s 152ms/step - loss: 0.1260 - acc: 0.9861 - val_loss: 0.4677 - val_acc: 0.9031
Epoch 3000/5000
12s 151ms/step - loss: 0.1256 - acc: 0.9861 - val_loss: 0.4517 - val_acc: 0.9044
Epoch 3001/5000
lr changed to 0.0009999999776482583
12s 151ms/step - loss: 0.1226 - acc: 0.9877 - val_loss: 0.4332 - val_acc: 0.9071
Epoch 3002/5000
12s 151ms/step - loss: 0.1123 - acc: 0.9911 - val_loss: 0.4282 - val_acc: 0.9088
Epoch 3003/5000
12s 152ms/step - loss: 0.1072 - acc: 0.9926 - val_loss: 0.4277 - val_acc: 0.9110
Epoch 3004/5000
12s 152ms/step - loss: 0.1051 - acc: 0.9938 - val_loss: 0.4253 - val_acc: 0.9108
Epoch 3005/5000
12s 152ms/step - loss: 0.1041 - acc: 0.9941 - val_loss: 0.4242 - val_acc: 0.9101
Epoch 3006/5000
12s 151ms/step - loss: 0.1021 - acc: 0.9945 - val_loss: 0.4259 - val_acc: 0.9098
Epoch 3007/5000
12s 151ms/step - loss: 0.1034 - acc: 0.9940 - val_loss: 0.4255 - val_acc: 0.9100
Epoch 3008/5000
12s 152ms/step - loss: 0.1018 - acc: 0.9949 - val_loss: 0.4252 - val_acc: 0.9100
Epoch 3009/5000
12s 152ms/step - loss: 0.1029 - acc: 0.9945 - val_loss: 0.4276 - val_acc: 0.9103
Epoch 3010/5000
12s 151ms/step - loss: 0.1018 - acc: 0.9947 - val_loss: 0.4275 - val_acc: 0.9102
Epoch 3011/5000
12s 152ms/step - loss: 0.1004 - acc: 0.9951 - val_loss: 0.4237 - val_acc: 0.9106
Epoch 3012/5000
12s 152ms/step - loss: 0.0996 - acc: 0.9954 - val_loss: 0.4213 - val_acc: 0.9120
Epoch 3013/5000
12s 151ms/step - loss: 0.0997 - acc: 0.9953 - val_loss: 0.4247 - val_acc: 0.9112
Epoch 3014/5000
12s 151ms/step - loss: 0.0998 - acc: 0.9956 - val_loss: 0.4249 - val_acc: 0.9111
Epoch 3015/5000
12s 152ms/step - loss: 0.0999 - acc: 0.9953 - val_loss: 0.4261 - val_acc: 0.9103
Epoch 3016/5000
12s 151ms/step - loss: 0.0984 - acc: 0.9958 - val_loss: 0.4285 - val_acc: 0.9102
Epoch 3017/5000
12s 151ms/step - loss: 0.0999 - acc: 0.9954 - val_loss: 0.4284 - val_acc: 0.9098
Epoch 3018/5000
12s 154ms/step - loss: 0.0997 - acc: 0.9952 - val_loss: 0.4290 - val_acc: 0.9105
Epoch 3019/5000
12s 152ms/step - loss: 0.0992 - acc: 0.9955 - val_loss: 0.4273 - val_acc: 0.9118
Epoch 3020/5000
12s 151ms/step - loss: 0.0988 - acc: 0.9953 - val_loss: 0.4270 - val_acc: 0.9110
Epoch 3021/5000
12s 152ms/step - loss: 0.0988 - acc: 0.9957 - val_loss: 0.4298 - val_acc: 0.9104
Epoch 3022/5000
12s 151ms/step - loss: 0.0984 - acc: 0.9957 - val_loss: 0.4317 - val_acc: 0.9103
Epoch 3023/5000
12s 151ms/step - loss: 0.0976 - acc: 0.9960 - val_loss: 0.4282 - val_acc: 0.9107
Epoch 3024/5000
12s 152ms/step - loss: 0.0984 - acc: 0.9959 - val_loss: 0.4283 - val_acc: 0.9111
Epoch 3025/5000
12s 152ms/step - loss: 0.0969 - acc: 0.9960 - val_loss: 0.4288 - val_acc: 0.9090

过拟合依然严重，还是得继续减小网络。

Minghang Zhao, Shisheng Zhong, Xuyun Fu, Baoping Tang, Shaojiang Dong, Michael Pecht, Deep Residual Networks with Adaptively Parametric Rectifier Linear Units for Fault Diagnosis, IEEE Transactions on Industrial Electronics, 2020, DOI: 10.1109/TIE.2020.2972458, Date of Publication: 13 Feb. 2020

Deep Residual Networks with Adaptively Parametric Rectifier Linear Units for Fault Diagnosis​ieeexplore.ieee.org

转自：

深度残差网络+自适应参数化ReLU激活函数（调参记录13）_人工智能_dangqing1988的博客-CSDN博客​blog.csdn.net