One pupolar application of Hadoop is data-warehousing.
Data-warehousing have been implemented through relational databases, particularly those optimized for a specific workload.
Database Workload
- OLTP(online transaction processing)
- Typical applications: e-commerce, banking, airline reservations
- User facing: real-time(process in memory), low latency, highly-concurrent
- Tasks: relatively small set of “standard” transactional queries
- Data access pattern: random reads, updates, writes
- OLAP(online analytical processing)
- Typical applications: business intelligence, data mining
- Back-end processing: **(process in disk)**batch workloads, less concurrent
- Tasks: complex analytical queries, often ad hoc
- Data access pattern: table scans, large amounts of data per query
Database solution: Separate databases
- User-facing OLTP database for high-volume transactions
- Data warehousing for OLAP workloads
How do we connect the OLTP and OLAP?
OLTP/OLAP Integration
- Extract-Transform-Load(ETL)
- Extract records from sourse(OLTP)
- Transform: clean data, check integrity, aggregate
- Load into OLAP database
- ETL Bottleneck
- ETL typically a nightly task
- limited by the speed of communication
Given successful applications of Hadoop to data-warehousing and complex analytical queries that are prevalent in such an environment, it makes sense to examine MapReduce algorithms for manipulating relational data.
**Hadoop is not a database!**Hadoop to ETL(actually, first L, then E and T)
What’s changed?
- Dropping cost of disks: cheaper to store everything
- Types of data collected: even collect the data whose value is less apparent
- Rise of social media and user-generated content: large increase in data volume
- Growing maturity of data mining techniques: demonstrates value of data analytics