MTandHJ的博客

文章目录概主要内容Shimodaira H. and Ltd N. M. C. A weight value initialization method for improving learning performance of the backpropagation algorithm in neural networks.概考虑f(x)=σ(∑i=1nwixi+w0)f(x) = \sigma (\sum_{i=1}^n w_i x_i + w_0)f(x)=σ(i=1∑n​wi​xi​

文章目录概主要内容输入层-隐藏层隐藏层-输出层Wessels L. F. A. and Barnard E. Avoiding False local minima by proper initialization of connections. In IEEE Transactions on Neural Networks, 1992.概避免局部最优的一种初始化方法, 文中给出的‘合适的’初始化方法的准则还挺有道理.主要内容本文主要考虑单隐层的情形, 即f(x)=∑j=1Hvjh(∑i=1

文章目录概主要内容一维情形如何加速多维情形代码Nguyen D. and Widrow B. Improving the learning speed of 2-layer neural networks by choosing initial values of the adaptive weights. In International Joint Conference on Neural Networks (IJCNN), 1990.概本文提出了一种关于两层网络的权重初始化方法.主要内容

文章目录概主要内容Diffusion modelsreverse processforward process变分界损失求解LtL_{t}Lt​L0L_0L0​最后的算法参数代码Ho J., Jain A. and Abbeel P. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NIPS), 2020.[Page E. Approximating to

文章目录概主要内容Langevin dynamicsScore MatchingDenoising Score MatchingNoise Conditional Score NetworksSlow mixing of Langevin dynamics损失函数Annealed Langevin dynamics细节代码[Song Y. and Ermon S. Generative modeling by estimating gradients of the data distribution.

文章目录概主要内容ImageNet-RStreetView StoreFronts (SVSF)DeepFashion RemixedDeepAugment实验结论代码Hendrycks D., Basart S., Mu N., Kadavath S., Wang F., Dorundo E., Desai R., Zhu T., Parajuli S., Guo M., Song D., Steinhardt J. Gilmer J. The many faces of robustness: a

文章目录概主要内容Chen E. and Lee C. LTD: Low temperature distillation for robust adversarial training. arXiv preprint arXiv:2111.02331, 2021.概本文利用distillation来提高网络鲁棒性.主要内容如上图所示, 作者认为, 如果我们用one-hot的标签进行训练, 结果会导致图(b)中的情形, 于是两个分布中间的空袭部分均可以作为分类边界, 从而导致存在大量的对抗样

文章目录Girshick R., Donahue J., Darrel T. and Malik J. Rich feature hierarchies for accurate object detection and semantic segmentation Tech report. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014Girshick R. Fast R-CNN. In IEEE

文章目录概主要内容Wang Z., Zhang W., Liu N. and Wang J. Scalable rule-based representation learning for interpretable classification. In Advances in Neural Information Processing Systems (NIPS), 2021.概传统的诸如决策树之类的机器学习方法具有很强的结构性, 也因此具有很好的可解释性. 和深度学习方法相比, 这类方法比较难

文章目录概主要内容Croce F. and Hein M. Mind the box: ℓ1\ell_1ℓ1​-APGD for sparse adversarial attacks on image classifiers. In International Conference on Machine Learning (ICML), 2021.概以往的ℓ1\ell_1ℓ1​攻击, 为了保证∥x′−x∥1≤ϵ,x′∈[0,1]d,\|x' - x\|_1 \le \epsilon, x' \

文章目录概主要内容Huster T., Chiang C. J. and Chadha R. Limitations of the lipschitz constant as a defense against adversarial examples. In European Conference on Machine Learning and Data Mining (ECML PKDD), 2018.概本文是想说明现有的依赖Lipschitz常数的以获得可验证的鲁棒性存在很大局限性.主要内

文章目录概主要内容深度宽度代码Huang H., Wang Y., Erfani S., Gu Q., Bailey J. and Ma X. Exploring architectural ingredients of adversarially robust deep neural networks. In Advances in Neural Information Processing Systems (NIPS), 2021概本文是对现有的残差网络结构的探索, grid search一个

文章目录概主要内容代码Rade R. and Moosavi-Dezfooli S. Helper-based adversarial training: reducing excessive margin to achieve a better accuracy vs. robustness trade-off. In International Conference on Machine Learning (ICML), 2021概本文认为普通的对抗训练会导致不必要的adversarial m

文章目录概主要内容代码Zhang J., Xu X., Han B., Niu G., Cui L., Sugiyama M., Kankanhalli M. Attacks which do not kill training make adversarial learning stronger. In International Conference on Machine Learning (ICML), 2020.概本文提出了一种early-stopped PGD, 通过一种逐渐增强的方法

文章目录概主要内容Random Step的作用线性性质gradient alignment代码Andriushchenko M. and Flammarion N. Understanding and improving fast adversarial training. In Advances in Neural Information Processing Systems (NIPS), 2020.概本文主要探讨:为什么简单的FGSM不能够提高鲁棒性;为什么FGSM-RS(即加了随机扰

文章目录概主要内容positional encoding额外的细节代码Mildenhall B., Srinivasan P. P., Tancik M., Barron J. T., Ramamoorthi R. and Ng R. NeRF: representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision (ECCV), 2020.概通过MLP

文章目录概主要内容初始化策略其它的好处Sitzmann V., Martel J. N. P., Bergman A. W., Lindell D. B., Wetzstein G. Implicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems (NIPS), 2020.概本文提出用sin⁡\sinsin作为激活函数, 并分析

文章目录概主要内容如何扩展?实验细节Martins P., Marinho Z. and Martins A. ∞\infty∞-former: Infinite Memory Transformer. arXiv preprint arXiv:2109.00301, 2021.概在transformer中引入一种长期记忆机制.主要内容假设X∈RL×dX \in \mathbb{R}^{L \times d}X∈RL×d, 即每一行xix_ixi​代表一个token对应的特征.Attenti

文章目录Kernel Density (KD)Local Intrinsic Dimensionality (LID)Gaussian Discriminant Analysis (GDA)Gaussian Mixture Model (GMM)SelectiveNetCombined Abstention Robustness Learning (CARL)Adversarial Training with a Rejection OptionEnergy-based Out-of-distributio

文章目录概主要内容代码Leino K., Wang Z. and Fredrikson M. Globally-robust neural networks. In International Conference on Machine Learning (ICML), 2021.概本文是一种可验证的鲁棒方法, 并且提出了一种globally-robust的概念, 但是实际看下来并不觉得有特别出彩的地方.主要内容对于网络f:Rn→Rmf : \mathbb{R}^{n} \rightarrow

文章目录概主要内容代码Bai Y., Zeng Y., Jiang Y., Xia S., Ma X., Wang Y. Improving adversarial robustness via channel-wise activation suppressing. In International Conference on Learning Representations (ICLR), 2021.Yan H., Zhang J., Niu G., Feng J., Tan V., Sugi

文章目录概主要内容代码Foret P., Kleiner A., Mobahi H., Neyshabur B. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations.概在训练的时候对权重加扰动能增强泛化性.主要内容如上图所示, 一般的训练方法虽然能够收敛到一个不错的局部最优点, 但是往往这个局

文章目录概主要内容代码Tolstlkhin I., Houlsby N., Kolesnikov A., Beyer L., Zhai X., Unterthiner T., Yung J., Steiner A., Keysers D., Uszkoreit J., Lucic M., Dosovitskly A. MLP-mixer: an all-mlp architecture for vision. In International Conference on Learning Represe

文章目录概主要内容Hu H., Zhang Z., Xie Z., Lin S. Local relation networks for image recognition. In International Conference on Computer Vision (ICCV), 2019.概一种特殊的卷积?主要内容CNN通过许许多多的filters进行模式匹配(a pattern matching process), 非常低效, 本文提出利用局部相关性来替代这些卷积层.输入特征图

文章目录概主要内容Hu H., Zhang Z., Xie Z., Lin S. Local relation networks for image recognition. In International Conference on Computer Vision (ICCV), 2019.概一种特殊的卷积?主要内容CNN通过许许多多的filters进行模式匹配(a pattern matching process), 非常低效, 本文提出利用局部相关性来替代这些卷积层.输入特征图

文章目录概主要内容iGPTViT代码Chen M., Radford A., Child R., Wu J., Jun H., Dhariwal P., Luan D., Sutskever I. Generative pretraining from pixels. In International Conference on Machine Learning (ICML), 2020.Dosovitskiy A., Beyer L., Kolesnikov A., Weissenborn D.

文章目录概主要内容GPTBERTRadford A., Narasimhan K., Salimans T. and Sutskever I. Improving language understanding by generative pre-training. 2018.Devlin J., Chang M., Lee K. and Toutanova K. BERT: Pre-training of deep bidirectional transformers for language u

文章目录Heuristics-drivenPad and CropCutoutMixUpCutMixAugMixData-drivenAutoAugmentRandAugmentDeepAugmentHeuristics-drivenPad and CropHe K., Zhang X., Ren S. and Sun J. Deep residual learning for image recognition. In IEEE Conference on Computer Vision an

文章目录概主要内容Positional Encodingauto_regressive额外的细节代码Vaswani A., Shazeer N., Parmar N., Uszkoreit J., Jones L., Gomez A. N., and Kaiser L. Attention is all you need. In Advances in Neural Information Processing Systems (NIPS), 2017.概Transformer.主要内容[外链

文章目录概主要内容模型ELBO关于AAAYang M., Liu F., Chen Z., Shen X., Hao J. and Wang J. CausalVAE: disentangled representation learning via neural structural causal models. arXiv preprint arXiv:2004.086975, 2020.概隐变量的因果表示.主要内容我们通常希望隐变量zzz能够表示一些特别的特征, 通过改变zzz使得生成的

文章目录概主要内容本文的模型IdentifiabilityKhemakhem I., Kingma D. P., Monti R. P. and Hyv"{a}rinen A. Variational autoencoders and nonlinear ICA: a unifying framework. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2020.概本文讨论ident

文章目录概主要内容代码Yu Y., Chen J., Gao T. and Yu M. DAG-GNN: DAG structure learning with graph neural networks. In International Conference on Machine Learning (ICML), 2019.概有向无环图 + GNN + VAE.主要内容先前已经有工作(NOTEARS)讨论了如何处理线性SEM模型X=ATX+Z,X = A^TX + Z,X=ATX+

文章目录概主要内容Gumbel distributionJang E., Gu S. and Poole B. Categorical reparameterization with gumbel-softmax. In International Conference On Learning Representations (ICLR), 2017.概利用梯度反向传播训练网咯几乎是深度学习的不二法门, 但是这往往要求保证梯度的存在, 这在一定程度上限制了一些扩展. 比如在VAE中, 虽然当qϕ(

文章目录概主要内容Locatello F., Bauer S., Lucic M., R"{a}tsch G., Gelly S. Sch"{o}lkopf and Bachem Olivier. Challenging common assumptions in the unsupervised learning of disentangled representations. In International Conference on Machine Leaning (ICML), 2018.

文章目录概主要内容Batch NormalizationLayer NormalizationInstance NormalizationGroup NormalizationIoffe S. and Szegedy C. Batch normalization: accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning

文章目录概主要内容proxy distribution如何利用构造的数据Sehwag V., Mahloujifar S., Handina T., Dai S., Xiang C., Chiang M. and Mittal P. Improving adversarial robustness using proxy Distributions. arXiv preprint arXiv: 2104.09425, 2021.概本文利用GAN生成数据, 并利用这些数据进行对抗训练, 无论是自然精

文章目录概主要内容代码Gowal S., Dvijotham K., Stanforth R., Bunel R., Qin C., Uesato J., Arandjelovic R., Mann T. & Kohli P. Scalable verified training for provably robust image classification. In IEEE International Conference on Computer Vision (ICCV), 2019.

文章目录概主要内容Zhang H., Zhang Z., Odena A. and Lee H. CONSISTENCY REGULARIZATION FOR GENERATIVE ADVERSARIAL NETWORKS. ICLR, 2020.Zhao Z., Singh S., Lee H., Zhang Z., Odena A. and Zhang H. Improved Consistency Regularization for GANs. AAAI, 2020.概让GAN训练稳

文章目录概主要内容Differentiable AugmentationAdaptive Augmentation代码Zhao S., Liu Z., Lin J., Zhu J. and Han S. Differentiable Augmentation for Data-Efficient GAN Training. NIPS, 2020.Karras T., Aittala M., Hellsten J., Laine S., Lehtinen J. and Alia T. Trainin

文章目录概主要内容实现代码[Niu Y., Tang K., Zhang H., Lu Z., Hua X. and Wen J. Counterfactual VQA: A Cause-Effect Look at Language Bias. CVPR, 2021.]概利用因果分析消除VQA(Visual Question Answering (VQA))中的language bias.主要内容如上图所示,QQQ: question;VVV: image;KKK: multi-mo