1、DDL
Hbase 存储
用户行为数据的存储 用户在平台上做的操作
浏览:15个字段
订单:20
支付:18
userid orderid paytype resultprice …… 收藏:12
添加购物车 不同的操作产生的日志的字段不一样 所有的日志都存储在一个文件中 ETL以后将每一行的数据写入Hbase,一天一张表
每天处理昨天的数据 程序中获取昨天的日期,创建一张Hbase表 问题:Java Client如何知道Hbase的地址呢?
目标:让Congratulation对象获取服务端地址 服务端地址是谁:Zookeeper的地址
方式一:要么将hbase-site.xml放入resources目录中
方式二:在configuration对象中定义服务端地址
conf.set(“hbase.zookeeper.quorum”,“node-01:2181,node-02:2181,node-03:2181”); 代码实现
public class HbaseClientDDL {
private String nsName = "nstest" ;
private String tbName = "hanjiaxiaozhi01:tbtest" ;
public static void main ( String[ ] args) throws IOException {
HbaseClientDDL clientDDL = new HbaseClientDDL ( ) ;
Configuration conf = HBaseConfiguration. create ( ) ;
Connection conn = clientDDL. createHbaseConn ( conf) ;
HBaseAdmin admin = clientDDL. createHbaseAdmin ( conn) ;
clientDDL. deleteHbaseTB ( admin) ;
clientDDL. listHbaseTB ( admin) ;
admin. close ( ) ;
conn. close ( ) ;
}
private void deleteHbaseTB ( HBaseAdmin admin) throws IOException {
if ( admin. tableExists ( tbName) ) {
admin. disableTable ( tbName) ;
admin. deleteTable ( tbName) ;
}
}
private void listHbaseTB ( HBaseAdmin admin) throws IOException {
TableName[ ] tableNames = admin. listTableNames ( ) ;
for ( TableName tableName : tableNames) {
System. out. println ( tableName. getNameAsString ( ) ) ;
}
}
private void createHbaseTB ( HBaseAdmin admin) throws IOException {
if ( admin. tableExists ( tbName) ) {
admin. disableTable ( tbName) ;
admin. deleteTable ( tbName) ;
}
HTableDescriptor desc = new HTableDescriptor ( TableName. valueOf ( tbName) ) ;
HColumnDescriptor family = new HColumnDescriptor ( "family" ) ;
HColumnDescriptor info = new HColumnDescriptor ( "info" ) ;
info. setMaxVersions ( 3 ) ;
desc. addFamily ( family) ;
desc. addFamily ( info) ;
admin. createTable ( desc) ;
}
private void deleteHbaseNS ( HBaseAdmin admin) throws IOException {
admin. deleteNamespace ( nsName) ;
}
private void listHbaseNS ( HBaseAdmin admin) throws IOException {
NamespaceDescriptor[ ] descriptors = admin. listNamespaceDescriptors ( ) ;
for ( NamespaceDescriptor descriptor : descriptors) {
System. out. println ( descriptor. getName ( ) ) ;
}
}
private void createHbaseNS ( HBaseAdmin admin) throws IOException {
NamespaceDescriptor descriptor = NamespaceDescriptor. create ( nsName) . build ( ) ;
admin. createNamespace ( descriptor) ;
}
private HBaseAdmin createHbaseAdmin ( Connection conn) throws IOException {
HBaseAdmin admin = ( HBaseAdmin) conn. getAdmin ( ) ;
return admin;
}
private Connection createHbaseConn ( Configuration conf) throws IOException {
Connection conn = ConnectionFactory. createConnection ( conf) ;
return conn;
}
}
2、DML
Hbase的数据类型:没有数据类型 所有数据都是以字节的形式存储 put
private void putData ( Table table) throws IOException {
Put put = new Put ( Bytes. toBytes ( "20200304_000" ) ) ;
put. addColumn ( Bytes. toBytes ( "basic" ) , Bytes. toBytes ( "name" ) , Bytes. toBytes ( "laowu" ) ) ;
table. put ( put) ;
}
private void getData ( Table table) throws IOException {
Get get = new Get ( Bytes. toBytes ( "20200101_001" ) ) ;
Result result = table. get ( get) ;
for ( Cell cell : result. rawCells ( ) ) {
System. out. println (
Bytes. toString ( CellUtil. cloneRow ( cell) ) + "\t" +
Bytes. toString ( CellUtil. cloneFamily ( cell) ) + "\t" +
Bytes. toString ( CellUtil. cloneQualifier ( cell) ) + "\t" +
Bytes. toString ( CellUtil. cloneValue ( cell) ) + "\t" +
cell. getTimestamp ( )
) ;
}
}
private void deleteData ( Table table) throws IOException {
Delete del = new Delete ( Bytes. toBytes ( "20200103_002" ) ) ;
del. addColumn ( Bytes. toBytes ( "basic" ) , Bytes. toBytes ( "name" ) ) ;
table. delete ( del) ;
}
scan
private void scanData ( Table table) throws IOException {
Scan scan = new Scan ( ) ;
ResultScanner scanner = table. getScanner ( scan) ;
for ( Result result : scanner) {
System. out. println ( "==========================================" ) ;
System. out. println ( Bytes. toString ( result. getRow ( ) ) ) ;
for ( Cell cell : result. rawCells ( ) ) {
System. out. println (
Bytes. toString ( CellUtil. cloneRow ( cell) ) + "\t" +
Bytes. toString ( CellUtil. cloneFamily ( cell) ) + "\t" +
Bytes. toString ( CellUtil. cloneQualifier ( cell) ) + "\t" +
Bytes. toString ( CellUtil. cloneValue ( cell) ) + "\t" +
cell. getTimestamp ( )
) ;
}
}
}
重点
Scan的使用 Cell:就一列的内容 Result:就是一个Rowkey的内容,里面有一个Cell数组 ResultScanner:就是多个Rowkey的返回值,里面是多个Result对象
3、Filter
Scan + Filter查询是工作中最常用的方式 命令行
scan 't1' , { ROWPREFIXFILTER = > 'row2' , FILTER = > "
( QualifierFilter ( >= , 'binary:xyz' ) ) AND ( TimestampsFilter ( 123 , 456 ) ) "}
ROWPREFIXFILTER:rowkey的前缀过滤器
QualifierFilter:列标签过滤器
TimestampsFilter:时间戳过滤器
STARTROW:从哪个rowkey可以开始
STOPROW:到哪个rowkey结束,不包含stoprow
从某个位置开始查询
scan 'student:stu' , { STARTROW= > '20200102_003' }
用于定义范围查询:前闭后开区间
scan 'student:stu' , { STARTROW= > '20200102_003' , STOPROW= > '20200304_000' }
请记住:
Hbase中scan查询对Rowkey的匹配是前缀匹配 只要rowkey前缀符合就可以进行查询,不需要给定完整的rowkey
scan ‘student:stu’,{STARTROW=>‘2020’} scan ‘student:stu’,{STARTROW=>‘20200103’} 因为底层的rowkey按照asc码进行比较排序的 应用
如果我的rowkey是这么设计的
年月日_用户id 20200101_001 20200102_002 20200803_003 20210101_001 我想走索引查询所有2020年的数据
scan ‘student:stu’,{STARTROW=>‘2020’,STOPROW=>‘2021’} 我想走是索引查询所有2020年8月和9月的数据
scan ‘student:stu’,{STARTROW=>‘20200801’,STOPROW=>‘20201001’} Rowkey的前缀是什么至关重要:决定了可以按照什么 条件做索引查询
如果我想查询用户id为001的用户的数据 不能走索引,只能全表扫描 前缀是日期,只能按照日期做索引检索 hbase按照rowkey的前缀进行检索匹配 实现
public class HbaseClientFilter {
public static void main ( String[ ] args) throws IOException {
HbaseClientFilter clientDML = new HbaseClientFilter ( ) ;
Configuration conf = HBaseConfiguration. create ( ) ;
Connection conn = clientDML. createHbaseConn ( conf) ;
Table table = clientDML. createHTable ( conn) ;
clientDML. scanData ( table) ;
table. close ( ) ;
conn. close ( ) ;
}
private void scanData ( Table table) throws IOException {
Scan scan = new Scan ( ) ;
RowFilter rowFilter = new RowFilter ( CompareFilter. CompareOp. NOT_EQUAL, new BinaryComparator ( Bytes. toBytes ( "20200101_001" ) ) ) ;
FamilyFilter familyFilter = new FamilyFilter ( CompareFilter. CompareOp. EQUAL, new SubstringComparator ( "sic" ) ) ;
QualifierFilter qualifierFilter = new QualifierFilter ( CompareFilter. CompareOp. NOT_EQUAL, new BinaryComparator ( Bytes. toBytes ( "name" ) ) ) ;
ValueFilter valueFilter = new ValueFilter ( CompareFilter. CompareOp. EQUAL, new SubstringComparator ( "lao" ) ) ;
PrefixFilter prefixFilter = new PrefixFilter ( Bytes. toBytes ( "20200102" ) ) ;
SingleColumnValueFilter singleColumnValueFilter = new SingleColumnValueFilter (
Bytes. toBytes ( "basic" ) ,
Bytes. toBytes ( "name" ) ,
CompareFilter. CompareOp. EQUAL,
Bytes. toBytes ( "laoda" )
) ;
byte [ ] [ ] prefixes = {
Bytes. toBytes ( "name" )
} ;
MultipleColumnPrefixFilter multipleColumnPrefixFilter = new MultipleColumnPrefixFilter ( prefixes) ;
FilterList filterList = new FilterList ( ) ;
filterList. addFilter ( singleColumnValueFilter) ;
filterList. addFilter ( multipleColumnPrefixFilter) ;
scan. setStartRow ( Bytes. toBytes ( "20200102_003" ) ) ;
PageFilter pageFilter = new PageFilter ( 2 ) ;
scan. setFilter ( pageFilter) ;
ResultScanner scanner = table. getScanner ( scan) ;
for ( Result result : scanner) {
System. out. println ( "==========================================" ) ;
System. out. println ( Bytes. toString ( result. getRow ( ) ) ) ;
for ( Cell cell : result. rawCells ( ) ) {
System. out. println (
Bytes. toString ( CellUtil. cloneRow ( cell) ) + "\t" +
Bytes. toString ( CellUtil. cloneFamily ( cell) ) + "\t" +
Bytes. toString ( CellUtil. cloneQualifier ( cell) ) + "\t" +
Bytes. toString ( CellUtil. cloneValue ( cell) ) + "\t" +
cell. getTimestamp ( )
) ;
}
}
}
private Table createHTable ( Connection conn) throws IOException {
Table table = conn. getTable ( TableName. valueOf ( "student:stu" ) ) ;
return table;
}
private Connection createHbaseConn ( Configuration conf) throws IOException {
Connection conn = ConnectionFactory. createConnection ( conf) ;
return conn;
}
}
重要
SingleColumnValueFilter:列值过滤器
用户行为分析中,用于过滤用户不同的操作 event=pay/collect/add 用于事件分析 MultipleColumnPrefixFilter:多列前缀过滤器
这个是最常用的 分布式计算程序中需要处理某些列,但不需要这张表的所有列 在Hbase读取时直接过滤,然后在程序中直接处理