Solr: realtime ingest data from mysql to solr using flume

 

 

 

 

 

 

• Create an easy ETL job to load data from MySQL to Hadoop and vice versa (using Apache Flume and other tools)
• Create a near-real-time replication channel between MySQL and Hadoop (using a MySQL replication API and Flume)

 

 

References

http://mvnrepository.com/artifact/org.apache.flume.flume-ng-sinks/flume-ng-morphline-solr-sink/1.5.2#mvn

https://cwiki.apache.org/confluence/display/FLUME/How+to+Setup+Solr+Sink+for+Flume

https://github.com/mvalleavila/flume-ng-sql-source

http://dev.mysql.com/tech-resources/articles/mysql-hadoop-applier.html

http://dev.mysql.com/doc/refman/5.0/en/mysqlbinlog.html

 https://github.com/Flipkart/MySQL-replication-listener

 http://code.google.com/p/open-replicator/

https://github.com/whitesock/open-replicator

http://intuitive-search.blogspot.com/2011/07/binary-log-api-and-replication-listener.html

https://launchpad.net/mysql-replication-listener

https://bitbucket.org/winebarrel/mysql-replication-listener

 

http://www.prosysopc.com/blog/hadoop-hive-integration-with-opc-ua/

https://docs.continuent.com/tungsten-replicator-3.0/deployment-hadoop.html

 

http://blog.andreamostosi.name/tag/apache-hadoop/

http://www.jb51.net/article/53542.htm

 

http://dev.mysql.com/doc/internals/en/binary-log.html