本文质疑了基于均匀收敛的泛化界在解释深度学习泛化能力上的充分性，指出即便是考虑SGD的隐含偏置，泛化界在某些情况下也会变得无效。文章通过构造实例证明，即使训练误差和测试误差很小，深度学习的决策边界复杂度也可能高到足以记住训练数据点位置的偏差，从而导致最紧致的均匀收敛界变得空洞。这强调了泛化界应至少能合理反映参数数量和训练集大小的影响。
本文价值：understand the limitations of u.c. based bounds / cast doubt on the power of u.c. bounds to fully explain generalization in DL
- highlight that explaining the training-set-size dependence of the generalization error is apparently just as non-trivial as explaining its parameter-count dependence.
- show that there are scenarios where all uniform convergence bounds, however cleverly applied, become vacuous.
Development of generalisation bound
stage 1：conventional u.c. bound
generalisation gap ≤ O ( representational complexity of whole hypothesis class training set size ) \text{generalisation gap} \leq O\Big(\sqrt{\frac{\text{representational complexity of whole hypothesis class}}{\text{training set size}}}\Big) generalisation gap≤O(training set sizerepresentational complexity of whole hypothesis class
最低0.47元/天 解锁文章
扫一扫
286
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
