科研小随笔 2024-2-27

深度学习科研的一点点感悟

最近，做深度学习科研工作并没有刚开始接触这个领域时的哪种热情了，因为好像自己能够把握的东西太少了。数据量越堆越多，模型越来越大，算力要求越来越大。似乎只要有足够的数据量，然后用足够的算力，训练一个大模型，就能够解决绝大多数问题。

你自己的在模型上的一点点改进带来的性能提升，除了能够水论文，似乎没有一点点应用价值。但是现在真正能够搞大模型的人有几个呢，连bat这样的大厂做起来都吃力，更别说一个小小的课题组了。没有钱搞大模型，就没有办法去追踪前沿技术、前沿论文，每次看到那些NLP、CV的大模型的文章，一个个都很惊艳，非常吸引人，总是有一种研究下去的冲动。

但是，转念一想，即便我认真研究了这些文章，我又没法复现，没有办法做实验验证自己的创新点，即便研究了好像也没有太大的意义。

深度学习这一最新兴起的学科，感觉是靠“资本”和“人才”共同驱动的，两者缺一不可，离开哪一个都不行。缺少了“资本”，你只能搞一些模型的修修补补，在一些小数据集上做做实验，拿着仅有的几个百分点的性能提升去发论文。但是，在真实应用中根本行不通，甚至完全用不上。看看近几年引起整个社会震惊的AI成果，有几个是离开资本，仅靠“人才”做出来的。而那些没有资本的研究人员，那些每年发表的成百上千的文章，有哪些是真正能够经得起实际检验的。

深度学习，确实是一门非常有趣的学科，让我们见识到了数据的威力，让人不禁感叹，这才是真正的数据科学啊！致敬Hiton老爷子！

神经网络，能够拟合一切的强大函数。仅仅是大脑神经网络的冰山一角，如果有一天能够真正揭开人的大脑工作的秘密，我觉得那才是真正的技术奇点。那时，才是真正的技术大爆发的时代，真正的技术革命，那时的人们就会惊奇的发现，现在的这些技术真的是不堪一击，渺小如尘埃！

现在人们对大脑了解的太少太少，对其的工作原理似懂非懂，一知半解。而且，在这一点上去搞研究的人好像并没有多少，感觉优点舍本逐末了，未来绝对是大脑的时代，谁弄懂了大脑的工作原理，谁就站在了人类科技的制高点。

Hinton老爷子在一次采访中说的很好，现在人们对大脑的了解，对大脑工作方式的了解，就像是远古时期的人们对生命的了解一样。在远古时期，你问一个人，什么是生命？他可能会说出一大串文字，给出一大串定义，你听了之后，可能还是一头雾水。但是随着技术的发展，当人们真正弄懂了人身体的每一个结构，每一个功能之后，什么是生命，这个问题似乎就迎刃而解了。这个时候，就很少有人再去纠结什么是生命这个问题了。引用我的偶像费曼的观点就是，what i cannot create, i do not understand.

“卷王”Lex Fridman和技术大佬Andrej Karpathy的对话，对我这样刚入AI大门，左瞧右看之后，一脸懵逼的人有很大的启发：

Lex Fridman: What advice would you give to beginners interested in getting into machine learning?

Andrej Karpathy: Beginners are often focused on like what to do, and I think the focus should be more like how much you do. So I'm a believer on the high-level, in this 10,000 hours concept where you just have to just pick the things where you can spend time and you care about and you are interested in. You leterally have to put in 10,000 hours of work. It doesn't even matter as much where you put it, you'll iterate and you'll improve and you'll waste some time. I dunno if there's a better way. You need to put in 10,000 hours. But I think it's actually really nice, cause I feel like there's some sense of determinsm about being an expert at a thing if you spend 10,000 hours. You can literally pick an arbitrary thing, and I think if you spend 10,000 hours of deliberate effort and work, you actually will become an expert at it. And so I think it's like a nice thought. And so basically I would focus more on like, are you spending 10,000 hours? That what I focus on.

Lex Fridman: And then thinking about what kind of machanisms maximize your likelihood of getting to 10,000 hours. Which for us silly humans means probably forming a daily habit of every single day actually doing thing.

Andrej Karpathy: Whatever helps you. So I do think to a large extent, it's a psychological problem for yourself. One other thing that I think is helpful, for the psychology of it, is many times people compare themselves to others in the area, I think this very harmful. Only compare yourself to you from some time age. Like say a year age,  are you better than you a year ago? This is the only way to think. And I think then you can see your progress, and it's very motivationg. 

Lex Fridman: That's so interesting. That focus on the quantity of hous. Cause I think a lot of people in the beginner stage but actually throughout get paralyzed by the choice. Like which one do I pick, this path or this path? They'll literally get paralyzed by which IED to use?

Andrej Karpathy: Well, they're worried, yeah, they'll worried about all these things. But the thing is, you will waste time doing something wrong. You will eventually figure out it's not right. You will accumulate scar tissue and next time you'll grow stronger, because next time you'll have the scar tissue, and next time you will learn from it. And now next time you come to a similar situation, you'll be like, oh, I messed up. I've spent a lot of time working on things that never materialized into anything, and I have all that scar tissue, and I have some intuitions about what was useful, what wasn't useful, how things turned out. So all those mistakes were not dead work. So I just think they should just focus on working. What have you done, what have you done last week?

...

Lex Fridman: What advice would you give to researchers trying to develop and publish an idea that have a big impact in the world of AI? So maybe undergrads, maybe early-graduates students.

Andrej Karpathy: I mena I would say they definitely have to be a little bit more strategic than I had to be as a PhD student because of the way AI is evolving, it's going the way of physics. Where in physics you used to be able to do experiments on your bench-top and everything was great and you could make progress, and now you have to work in like LHC or like CERN， and so AI is going in that direction as well. So the。。。