Cloud Computing(5)_Big Data Infrastructure

One pupolar application of Hadoop is data-warehousing.
Data-warehousing have been implemented through relational databases, particularly those optimized for a specific workload.

Database Workload

• OLTP(online transaction processing)
  
  • Typical applications: e-commerce, banking, airline reservations
  • User facing: real-time(process in memory), low latency, highly-concurrent
  • Tasks: relatively small set of “standard” transactional queries
  • Data access pattern: random reads, updates, writes
• OLAP(online analytical processing)
  
  • Typical applications: business intelligence, data mining
  • Back-end processing: **(process in disk)**batch workloads, less concurrent
  • Tasks: complex analytical queries, often ad hoc
  • Data access pattern: table scans, large amounts of data per query

Database solution: Separate databases

• User-facing OLTP database for high-volume transactions
• Data warehousing for OLAP workloads

How do we connect the OLTP and OLAP?

OLTP/OLAP Integration

• Extract-Transform-Load(ETL)
  
  • Extract records from sourse(OLTP)
  • Transform: clean data, check integrity, aggregate
  • Load into OLAP database
• ETL Bottleneck
  
  • ETL typically a nightly task
  • limited by the speed of communication

Given successful applications of Hadoop to data-warehousing and complex analytical queries that are prevalent in such an environment, it makes sense to examine MapReduce algorithms for manipulating relational data.

**Hadoop is not a database!**Hadoop to ETL(actually, first L, then E and T)

What’s changed?

• Dropping cost of disks: cheaper to store everything
• Types of data collected: even collect the data whose value is less apparent
• Rise of social media and user-generated content: large increase in data volume
• Growing maturity of data mining techniques: demonstrates value of data analytics