大数据工程人员知识图谱

http://yanbohappy.sinaapp.com/?cat=32

大数据工程人员知识图谱

3 Replies

在企业里面从事大数据相关的工作到底需要掌握哪些知识呢？我认为需要从两个角度来看：一个是技术；一个是业务。技术上主要涉及到概率和数理统计，计算机系统、算法和编程等；而业务的角度呢则是因公司业务的不同而异。对于从事大数据的工程人员来说，需要学会使用数据挖掘方法在计算机系统和编程工具的帮助下解决实际的问题，这样才能够在海量数据中挖掘出业务增长的助推剂，才能在激烈的市场竞争中为企业创造更多的价值。

因为业务会因公司的不同而不同，但是技术点是想通的。我在这里简单总结了一下大数据相关工程人员需要掌握的技术相关知识点。主要涉及到数据库、数据仓库、编程、分布式系统、Hadoop生态系统相关、数据挖掘和机器学习相关的基础知识点。当然我这里列出来的应该是一个team的人员汇集在一起所具备的，每个人会因在团队中的角色不同而有所侧重。在此剖砖引玉，欢迎大家发表意见。

Topic	Content		Key points	Reference
DB/OLTP & DW/OLAP	Database/OLTP basic		The relational model, SQL, index/secondary index, inner join/left join/right join/full join, transaction/ACID	Ramakrishnan, Raghu, and Johannes Gehrke. Database Management Systems.
	Database internal & implementation		Architecture, memory management, storage/B+ tree, query parse /optimization/execution, hash join/sort-merge join
	Distributed and parallel database		Sharding, database proxy
	Data warehouse/OLAP		Materialized views, ETL, column-oriented storage, reporting, BI tools
Basic programming	Programming language		Java, Python (Pandas/NumPy/SciPy/scikit-learn), SQL, Functional programming, R/SAS/SPSS	Wes McKinney. Python for Data Analysis: Agile Tools for Real World Data.
	OS		Linux
	DB & DW system		MySQL/ Hive/Impala
	Text format and process		JSON/XML, regex
	Tool		Git/SVN, Maven
Distributed system & Hadoop ecosystem & NoSQL	Distributed system principal theory		CAP theorem, RPC (Protocol Buffer/Thrift/Avro), Zookeeper, Metadata management (HCatalog)
	Distributed storage & computing framework & resource management		Hadoop/HDFS/MapReduce/YARN	Tom White. Hadoop : The Definitive Guide. Donald Miner, Adam Shook. MapReduce Design Patterns : Building Effective Algorithm and Analytics for Hadoop and Other Systems.
	SQL on Hadoop	Data (log) acquisition/integration/fusion, normalization, feature extraction	Sqoop, Flume/Scribe/Chukwa,SerDe	Edward Capriolo, Dean Wampler, Jason Rutherglen. Programming Hive.
		Query & In-database analytics	Hive, Impala, UDF/UDAF
	Large scale data mining & machine learning framework		Spark/MLbase, MR/Mahout
	Streaming process		Storm
	NoSQL		HBase/Cassandra (column oriented database)	Lars George. HBase: The Definitive Guide.
			Mongodb (Document database)
			Neo4j (graph database)
			Redis (cache)
Data mining & Machine learning	DM & ML basic		Numerical/Categorical variable, training/test data, over fitting, bias/variance, precision/recall, tagging
	Statistic		Data exploration (mean, median/range/standard deviation/variance/histogram), Continues distributions (Normal/ Poisson/Gaussian), covariance, correlation coefficient, distance and similarity computing, Bayes theorem, Monte Carlo Method, Hypothesis testing
	Supervised learning		Classifier, boosting, prediction, regression analysis	Han, Jiawei,Micheline Kamber, and Jian Pei. Data mining: concepts and techniques.
	Unsupervised learning		Cluster, deep learning
	Collaborative filtering		Item based CF, user based CF
	Algorithm	Classifier	Decision trees, KNN (K-Nearest neighbor), SVM (support vector machines), SVD (Singular Value Decomposition), naïve Bayes classifiers, neural networks,
		Regression	Linear regression, logistic regression, ranking, perception
		Cluster	Hierarchical cluster, K-means cluster, Spectral Cluster
		Dimensionality reduction	PCA (Principal Component Analysis), LDA (Linear discriminant Analysis), MDS (Multidimensional scaling)
	Text mining & Information retrieval		Corpus, term document matrix, term frequency & weight, association rules, market based analysis, vocabulary mapping, sentiment analysis, tagging, PageRank, VSM (Vector Space Model), inverted index	Jimmy Lin and Chris Dyer. Data-Intensive Text Processing with MapReduce.

This entry was posted in Data Mining, Data Warehouse, Database, Hadoop, HBase, Hive, Impala, Machine Learning, NewSQL, NoSQL, PostgreSQL and tagged BigData, Data Mining, Data Warehouse, Database, Hadoop, HBase, Machine Learning on 2013 年 11 月 5 日 by ybliang8@gmail.com.