海量数据(又称大数据)已经成为各大互联网企业面临的最大问题,如何处理海量数据,提供更好的解决方案,是目前相当热门的一个话题。类似MapReduce、 Hadoop等架构的普遍推广,大家都在构建自己的大数据处理,大数据分析平台。
相应之下,目前对于海量数据处理人才的需求也在不断增多,此类人才可谓炙手可热!越来越多的开发者把目光转移到海量数据的处理上。但是不是所有人都能真正接触到,或者有机会去处理海量数据的,所以就需要一些公开的海量数据集来研究。
在Quora上有人就问到,如何获取海量数据集。此问题得到了很多人的关注。具体可以看看回答,数据集的种类多种多样,有化学分析,基因遗传等等,从中你肯定能得到自己想要个数据集。
[url=http://www.quora.com/Data/Where-can-I-get-large-datasets-open-to-the-public]Where can I get large datasets open to the public?[/url]
[b]首先说说几个收集数据集的网站:[/b]
1、Public Data Sets on Amazon Web Services (AWS)
[url=http://aws.amazon.com/datasets]http://aws.amazon.com/datasets[/url]
Amazon从2008年开始就为开发者提供几十TB的开发数据。
2、Yahoo! Webscope
[url=http://webscope.sandbox.yahoo.com/index.php]http://webscope.sandbox.yahoo.com/index.php[/url]
3、Konect is a collection of network datasets
[url=http://konect.uni-koblenz.de/]http://konect.uni-koblenz.de/[/url]
4、Stanford Large Network Dataset Collection
[url=http://snap.stanford.edu/data/index.html]http://snap.stanford.edu/data/index.html[/url]
[b]再就是说说几个跟互联网有关的数据集:[/b]
1、Dataset for "Statistics and Social Network of YouTube Videos"
[url=http://netsg.cs.sfu.ca/youtubedata/]http://netsg.cs.sfu.ca/youtubedata/[/url]
2、1998 World Cup Web Site Access Logs
[url=http://ita.ee.lbl.gov/html/contrib/WorldCup.html]http://ita.ee.lbl.gov/html/contrib/WorldCup.html[/url]
这个是1998年世界杯期间的数据集。从1998/04/26 到 1998/07/26 的92天中,发生了 1,352,804,107次请求。
3、Page view statistics for Wikimedia projects
[url=http://dammit.lt/wikistats/]http://dammit.lt/wikistats/[/url]
4、AOL Search Query Logs - RP
[url=http://www.researchpipeline.com/mediawiki/index.php?title=AOL_Search_Query_Logs]http://www.researchpipeline.com/mediawiki/index.php?title=AOL_Search_Query_Logs[/url]
5、livedoor gourmet
[url=http://blog.livedoor.jp/techblog/archives/65836960.html]http://blog.livedoor.jp/techblog/archives/65836960.html[/url]
[b]海量图像数据集:[/b]
1、ImageNet
[url=http://www.image-net.org/]http://www.image-net.org/[/url]
包含1400万的图像。
2、Tiny Images Dataset
[url=http://horatio.cs.nyu.edu/mit/tiny/data/index.html]http://horatio.cs.nyu.edu/mit/tiny/data/index.html[/url]
包含8000万的32x32图像。
3、 MirFlickr1M
[url=http://press.liacs.nl/mirflickr/]http://press.liacs.nl/mirflickr/[/url]
Flickr中的100万的图像集。
4、 CoPhIR
[url=http://cophir.isti.cnr.it/whatis.html]http://cophir.isti.cnr.it/whatis.html[/url]
Flickr中的1亿600万的图像
5、SBU captioned photo dataset
[url=http://dsl1.cewit.stonybrook.edu/~vicente/sbucaptions/]http://dsl1.cewit.stonybrook.edu/~vicente/sbucaptions/[/url]
Flickr中的100万的图像集。
6、Large-Scale Image Annotation using Visual Synset(ICCV 2011)
[url=http://cpl.cc.gatech.edu/projects/VisualSynset/]http://cpl.cc.gatech.edu/projects/VisualSynset/[/url]
包含2亿图像
7、NUS-WIDE
[url=http://lms.comp.nus.edu.sg/research/NUS-WIDE.htm]http://lms.comp.nus.edu.sg/research/NUS-WIDE.htm[/url]
Flickr中的27万的图像集。
8、SUN dataset
[url=http://people.csail.mit.edu/jxiao/SUN/]http://people.csail.mit.edu/jxiao/SUN/[/url]
包含13万的图像
9、MSRA-MM
[url=http://research.microsoft.com/en-us/projects/msrammdata/]http://research.microsoft.com/en-us/projects/msrammdata/[/url]
包含100万的图像,23000视频
10、TRECVID
[url=http://trecvid.nist.gov/]http://trecvid.nist.gov/[/url]
[b]Stack Overflow Dump Files[/b]
7.3G [url=https://archive.org/download/stackexchange/stackoverflow.com-Posts.7z]stackoverflow.com-Posts.7z[/url]
573.1K [url=https://archive.org/download/stackexchange/stackoverflow.com-Tags.7z]stackoverflow.com-Tags.7z[/url]
153.0M [url=https://archive.org/download/stackexchange/stackoverflow.com-Users.7z]stackoverflow.com-Users.7z[/url]
2.2G [url=https://archive.org/download/stackexchange/stackoverflow.com-Comments.7z]stackoverflow.com-Comments.7z[/url]
截止目前好像还没有国内的企业或者组织开放自己的数据集。希望也能有企业开发自己的数据集给研究人员使用,从而推动海量数据处理在国内的发展!
2014/07/07 雅虎发布超大Flickr数据集 1亿的图片+视频
[url=http://yahoolabs.tumblr.com/post/89783581601/one-hundred-million-creative-commons-flickr-images-for]http://yahoolabs.tumblr.com/post/89783581601/one-hundred-million-creative-commons-flickr-images-for[/url]
100多个有趣的数据集
[url=http://www.csdn.net/article/2014-06-06/2820111-100-Interesting-Data-Sets-for-Statistics]http://www.csdn.net/article/2014-06-06/2820111-100-Interesting-Data-Sets-for-Statistics[/url]
[b]城市[/b]
Public Data EU http://publicdata.eu
Open Data Europe http://data.europa.eu/euodp/en/home
UK Government Data https://data.gov.uk
Africa Open Data https://africaopendata.org
Code for South Africa http://code4sa.org
Code for Africa https://codeforafrica.org
Open Cities Project http://www.opencitiesproject.org
Open Nepal http://data.opennepal.net
National Bureau of Statistics of China http://www.stats.gov.cn
Open Data Hong Kong https://opendatahk.com
Indonesian Government Open Data http://data.go.id
Japan http://www.data.go.jp
Russian Government Data Website http://data.gov.ru
PakReport - Pakistan Open Data and Maps http://pakreport.org
Open Data India https://data.gov.in
Turkey Open Statistics http://www.turkstat.gov.tr
Canada statistics https://crdcn.org/data
Open Canada http://open.canada.ca/en
Open Data Brazil http://dados.gov.br
Open Data Mexico https://datos.gob.mx
Open Data Latin America http://opendatalatinoamerica.org
[b]机构[/b]
United Nations Open Data http://data.un.org
United Nations Development Program Data http://open.undp.org
Open Knowledge Foundation https://okfn.org
World Bank https://data.worldbank.org
WikiLeaks https://wikileaks.org
International Aid and Transparency Datasets https://www.iatiregistry.org
DataHub http://datahub.io
Population Reference Bureau http://www.prb.org/DataFinder.aspx
[b]教育[/b]
Lexis Nexis http://lexisnexis.com
Google Scholar search http://scholar.google.com
Cornell University arXiv https://arxiv.org
UCI Machine Learning Datasets https://archive.ics.uci.edu/ml/index.php
Common Data Set Initiative http://www.commondataset.org
[b]医疗[/b]
Open Science Data Cloud https://www.opensciencedatacloud.org/publicdata/
Open Science Directory http://www.opensciencedirectory.net
WHO http://www.who.int/gho/database/en/
Broad Institute Open Data https://www.broadinstitute.org/data-software-and-tools
Human Connectome Project http://www.humanconnectomeproject.org
UNC's Psychiatric Genomics http://www.med.unc.edu/pgc
Social Science Datasets http://3stages.org/idata/
CDC Medical Data https://www.cdc.gov/nchs/fastats/
相应之下,目前对于海量数据处理人才的需求也在不断增多,此类人才可谓炙手可热!越来越多的开发者把目光转移到海量数据的处理上。但是不是所有人都能真正接触到,或者有机会去处理海量数据的,所以就需要一些公开的海量数据集来研究。
在Quora上有人就问到,如何获取海量数据集。此问题得到了很多人的关注。具体可以看看回答,数据集的种类多种多样,有化学分析,基因遗传等等,从中你肯定能得到自己想要个数据集。
[url=http://www.quora.com/Data/Where-can-I-get-large-datasets-open-to-the-public]Where can I get large datasets open to the public?[/url]
[b]首先说说几个收集数据集的网站:[/b]
1、Public Data Sets on Amazon Web Services (AWS)
[url=http://aws.amazon.com/datasets]http://aws.amazon.com/datasets[/url]
Amazon从2008年开始就为开发者提供几十TB的开发数据。
2、Yahoo! Webscope
[url=http://webscope.sandbox.yahoo.com/index.php]http://webscope.sandbox.yahoo.com/index.php[/url]
3、Konect is a collection of network datasets
[url=http://konect.uni-koblenz.de/]http://konect.uni-koblenz.de/[/url]
4、Stanford Large Network Dataset Collection
[url=http://snap.stanford.edu/data/index.html]http://snap.stanford.edu/data/index.html[/url]
[b]再就是说说几个跟互联网有关的数据集:[/b]
1、Dataset for "Statistics and Social Network of YouTube Videos"
[url=http://netsg.cs.sfu.ca/youtubedata/]http://netsg.cs.sfu.ca/youtubedata/[/url]
2、1998 World Cup Web Site Access Logs
[url=http://ita.ee.lbl.gov/html/contrib/WorldCup.html]http://ita.ee.lbl.gov/html/contrib/WorldCup.html[/url]
这个是1998年世界杯期间的数据集。从1998/04/26 到 1998/07/26 的92天中,发生了 1,352,804,107次请求。
3、Page view statistics for Wikimedia projects
[url=http://dammit.lt/wikistats/]http://dammit.lt/wikistats/[/url]
4、AOL Search Query Logs - RP
[url=http://www.researchpipeline.com/mediawiki/index.php?title=AOL_Search_Query_Logs]http://www.researchpipeline.com/mediawiki/index.php?title=AOL_Search_Query_Logs[/url]
5、livedoor gourmet
[url=http://blog.livedoor.jp/techblog/archives/65836960.html]http://blog.livedoor.jp/techblog/archives/65836960.html[/url]
[b]海量图像数据集:[/b]
1、ImageNet
[url=http://www.image-net.org/]http://www.image-net.org/[/url]
包含1400万的图像。
2、Tiny Images Dataset
[url=http://horatio.cs.nyu.edu/mit/tiny/data/index.html]http://horatio.cs.nyu.edu/mit/tiny/data/index.html[/url]
包含8000万的32x32图像。
3、 MirFlickr1M
[url=http://press.liacs.nl/mirflickr/]http://press.liacs.nl/mirflickr/[/url]
Flickr中的100万的图像集。
4、 CoPhIR
[url=http://cophir.isti.cnr.it/whatis.html]http://cophir.isti.cnr.it/whatis.html[/url]
Flickr中的1亿600万的图像
5、SBU captioned photo dataset
[url=http://dsl1.cewit.stonybrook.edu/~vicente/sbucaptions/]http://dsl1.cewit.stonybrook.edu/~vicente/sbucaptions/[/url]
Flickr中的100万的图像集。
6、Large-Scale Image Annotation using Visual Synset(ICCV 2011)
[url=http://cpl.cc.gatech.edu/projects/VisualSynset/]http://cpl.cc.gatech.edu/projects/VisualSynset/[/url]
包含2亿图像
7、NUS-WIDE
[url=http://lms.comp.nus.edu.sg/research/NUS-WIDE.htm]http://lms.comp.nus.edu.sg/research/NUS-WIDE.htm[/url]
Flickr中的27万的图像集。
8、SUN dataset
[url=http://people.csail.mit.edu/jxiao/SUN/]http://people.csail.mit.edu/jxiao/SUN/[/url]
包含13万的图像
9、MSRA-MM
[url=http://research.microsoft.com/en-us/projects/msrammdata/]http://research.microsoft.com/en-us/projects/msrammdata/[/url]
包含100万的图像,23000视频
10、TRECVID
[url=http://trecvid.nist.gov/]http://trecvid.nist.gov/[/url]
[b]Stack Overflow Dump Files[/b]
7.3G [url=https://archive.org/download/stackexchange/stackoverflow.com-Posts.7z]stackoverflow.com-Posts.7z[/url]
573.1K [url=https://archive.org/download/stackexchange/stackoverflow.com-Tags.7z]stackoverflow.com-Tags.7z[/url]
153.0M [url=https://archive.org/download/stackexchange/stackoverflow.com-Users.7z]stackoverflow.com-Users.7z[/url]
2.2G [url=https://archive.org/download/stackexchange/stackoverflow.com-Comments.7z]stackoverflow.com-Comments.7z[/url]
截止目前好像还没有国内的企业或者组织开放自己的数据集。希望也能有企业开发自己的数据集给研究人员使用,从而推动海量数据处理在国内的发展!
2014/07/07 雅虎发布超大Flickr数据集 1亿的图片+视频
[url=http://yahoolabs.tumblr.com/post/89783581601/one-hundred-million-creative-commons-flickr-images-for]http://yahoolabs.tumblr.com/post/89783581601/one-hundred-million-creative-commons-flickr-images-for[/url]
100多个有趣的数据集
[url=http://www.csdn.net/article/2014-06-06/2820111-100-Interesting-Data-Sets-for-Statistics]http://www.csdn.net/article/2014-06-06/2820111-100-Interesting-Data-Sets-for-Statistics[/url]
[b]城市[/b]
Public Data EU http://publicdata.eu
Open Data Europe http://data.europa.eu/euodp/en/home
UK Government Data https://data.gov.uk
Africa Open Data https://africaopendata.org
Code for South Africa http://code4sa.org
Code for Africa https://codeforafrica.org
Open Cities Project http://www.opencitiesproject.org
Open Nepal http://data.opennepal.net
National Bureau of Statistics of China http://www.stats.gov.cn
Open Data Hong Kong https://opendatahk.com
Indonesian Government Open Data http://data.go.id
Japan http://www.data.go.jp
Russian Government Data Website http://data.gov.ru
PakReport - Pakistan Open Data and Maps http://pakreport.org
Open Data India https://data.gov.in
Turkey Open Statistics http://www.turkstat.gov.tr
Canada statistics https://crdcn.org/data
Open Canada http://open.canada.ca/en
Open Data Brazil http://dados.gov.br
Open Data Mexico https://datos.gob.mx
Open Data Latin America http://opendatalatinoamerica.org
[b]机构[/b]
United Nations Open Data http://data.un.org
United Nations Development Program Data http://open.undp.org
Open Knowledge Foundation https://okfn.org
World Bank https://data.worldbank.org
WikiLeaks https://wikileaks.org
International Aid and Transparency Datasets https://www.iatiregistry.org
DataHub http://datahub.io
Population Reference Bureau http://www.prb.org/DataFinder.aspx
[b]教育[/b]
Lexis Nexis http://lexisnexis.com
Google Scholar search http://scholar.google.com
Cornell University arXiv https://arxiv.org
UCI Machine Learning Datasets https://archive.ics.uci.edu/ml/index.php
Common Data Set Initiative http://www.commondataset.org
[b]医疗[/b]
Open Science Data Cloud https://www.opensciencedatacloud.org/publicdata/
Open Science Directory http://www.opensciencedirectory.net
WHO http://www.who.int/gho/database/en/
Broad Institute Open Data https://www.broadinstitute.org/data-software-and-tools
Human Connectome Project http://www.humanconnectomeproject.org
UNC's Psychiatric Genomics http://www.med.unc.edu/pgc
Social Science Datasets http://3stages.org/idata/
CDC Medical Data https://www.cdc.gov/nchs/fastats/