对接大数据接口涉及到的知识以及 Hbase的使用小结

###

什么是RestTemplate?

RestTemplate是Spring提供的用于访问Rest服务的客户端,RestTemplate提供了多种便捷访问远程Http服务的方法,能够大大提高客户端的编写效率。
调用RestTemplate的默认构造函数,RestTemplate对象在底层通过使用java.net包下的实现创建HTTP 请求,可以通过使用ClientHttpRequestFactory指定不同的HTTP请求方式。
ClientHttpRequestFactory接口主要提供了两种实现方式

一种是SimpleClientHttpRequestFactory,使用J2SE提供的方式(既java.net包提供的方式)创建底层的Http请求连接。
一种方式是使用HttpComponentsClientHttpRequestFactory方式,底层使用HttpClient访问远程的Http服务,使用HttpClient可以配置连接池和证书等信息

@Component
public class RestTemplateTest {

    @Value("${hbaseURL}")
    private String hbaseURL;
    @Value("${userName}")
    private String userName1;
    @Value("${departmentName}")
    private String departmentName1;

    private static String ip ;
    private static String userName;
    private static String departmentName ;

    @PostConstruct
    public void Sys(){ ;
        ip=hbaseURL;
        userName=userName1;
        departmentName=departmentName1;
    }

 方法1:从大数据获取列值:

例如获取姓名,直接调用: xx.setName(GetValueUtil.getColumValue(MREGISTEREDEntity,MREGISTERED.Colums.REC_NAME.name()));// 姓名

 /**
     * method for get some values by colums from hbase
     * @param tableName  hbase table's name
     * @param rowKeyReg  rowkey's regex value
     * @param colums  pair's keys seted colums, like  pairs.add(new Pair("AdverseReactions",""));
     * @param familyName like "CF"
     * @return List<Pair> pair's key is same as  @param colums, pair'value is colums's values
     */
    public static List<ResaultEntity> getColumsValues(String tableName,String rowKeyReg, List<ResaultEntity>colums,String familyName,String columName,VisitorAttribute v){
        v.setUserName(userName);
        v.setDepartmentName(departmentName);
        List<ResaultEntity> getRes = new ArrayList<>();
	    try {
            Map<String, NavigableSet<String>> family = new TreeMap<>();
            NavigableSet<String> set1 = new TreeSet<>();
            set1.add(columName);
            family.put(familyName, set1);
            String resaultScan = new RestTemplateTest().doScan(tableName,rowKeyReg,family,v);
            Set<String>rowkeyLst = getValueForRow(resaultScan);//获取rowkey集合
            if (rowkeyLst.size()!=0){
                for (String rowkey:rowkeyLst) {
                    if (!StringUtils.isEmpty(rowkey)){
                        Map<String, NavigableSet<String>> familyMap
                                = new TreeMap<>();
                        NavigableSet<String> set = new TreeSet<>();
                        colums.forEach(pair -> {
                            set.add(pair.getColumn());
                        });
                        familyMap.put(familyName, set);//设置列族
                        String resault = new RestTemplateTest().doGet(tableName,rowkey,familyMap,v);//通过完整的rowkey+字段名集合查找
                        getRes.addAll(getValueForRow(resault,colums));//全部结果整合
                    }
                }
            }
        }catch (Exception e){
	        e.printStackTrace();
            return getRes;
        }
        return getRes;
    }

 方法2:通过正则去获取rowkey集合,例如:

String rowkeyReg = ".x.";//rowkey表达式
Set<String> rowkeys = MREGISTERED.getRowKeys(rowkeyReg,hbaseFront);

 String rowkey = new ArrayList<>(rowkeys).get(0);//拿取第一条数据即是rowkey

   public static Set<String> getRowKeys(String tableName,String rowKeyReg,String familyName,String columName,VisitorAttribute v) throws Exception{
        v.setUserName(userName);
        v.setDepartmentName(departmentName);
        Map<String, NavigableSet<String>> family = new TreeMap<>();
        NavigableSet<String> set1 = new TreeSet<>();
        set1.add(columName);
        family.put(familyName, set1);
        String resaultScan = new RestTemplateTest().doScan(tableName,rowKeyReg,family,v);
        Set<String>rowkeyLst = getValueForRow(resaultScan);
        return rowkeyLst;
    }

方法3:doGet方法:

    /**
     * method for resault from hbase
     * @param tableName  hbase table's name
     * @param rowkey  rowkey value
     * @param v  VisitorAttribute
     * @param familyMap
     * @return resault String
     */
    public String doGet(String tableName,String rowkey,Map familyMap,VisitorAttribute v) throws Exception{
        // 设置地址
        String url = toUrl(tableName, "get");
        HttpHeaders headers = new HttpHeaders();
        MediaType type = MediaType.parseMediaType("application/json; charset=UTF-8");
        headers.setContentType(type);
        headers.add("Accept", MediaType.APPLICATION_JSON.toString());

        JsonBaseGet baseGet = new JsonBaseGet();
        baseGet.setFamilyMap(familyMap);
        baseGet.setRowKey(rowkey);

        JsonGet get = new JsonGet();
        get.setVisitorAttribute(v);
        get.setJsonBaseGet(baseGet);

        HttpEntity<JsonGet> formEntity = new HttpEntity<>(get, headers);
        RestTemplate restTemplate = new RestTemplate();
        List<HttpMessageConverter<?>> lst = restTemplate.getMessageConverters();
        HttpMessageConverter<?> converter = new StringHttpMessageConverter();
        lst.add(0,converter);
        restTemplate.setMessageConverters(lst);
        String reult = restTemplate.postForObject(url, formEntity, String.class);
        return reult;
    }

方法4:doScan方法

    /**
     * method for resault from hbase
     * @param tableName  hbase table's name
     * @param RowKeyReg  rowkey's regex value
     * @param v  VisitorAttribute
     * @return resault String
     */
	public String doScan(String tableName,String RowKeyReg,Map familyMap,VisitorAttribute v) throws Exception{
		// 设置地址
		String url = toUrl(tableName, "scan");
		HttpHeaders headers = new HttpHeaders();
		MediaType type = MediaType.parseMediaType("application/json");
		headers.setContentType(type);
		headers.add("Accept", MediaType.APPLICATION_JSON.toString());

//        //设置正则比较器
        ByteArrayComparableModel byteArrayComparableModel = new ByteArrayComparableModel();
        byteArrayComparableModel.setType("RegexStringComparator");
        byteArrayComparableModel.setValue(RowKeyReg);
        byteArrayComparableModel.setOp("EQUAL");
        //设置过滤器
        JsonBaseFilter jsonBaseFilter = new JsonBaseFilter();
        jsonBaseFilter.setComparator(byteArrayComparableModel);
        jsonBaseFilter.setType("RowFilter");//行过滤
        jsonBaseFilter.setOp("EQUAL");
        jsonBaseFilter.setLatestVersion(true);
        jsonBaseFilter.setIfMissing(true);

        JsonBaseScan baseScan = new JsonBaseScan();
        baseScan.setJsonBaseFilter(jsonBaseFilter);
        baseScan.setFamilyMap(familyMap);

        JsonScan scan = new JsonScan();
        scan.setVisitorAttribute(v);
        scan.setJsonBaseScan(baseScan);

		HttpEntity<JsonScan> formEntity = new HttpEntity<>(scan, headers);
        RestTemplate restTemplate = new RestTemplate();
        List<HttpMessageConverter<?>> lst = restTemplate.getMessageConverters();
        HttpMessageConverter<?> converter = new StringHttpMessageConverter();
        lst.add(0,converter);
        restTemplate.setMessageConverters(lst);
		String reult = restTemplate.postForObject(url, formEntity, String.class);
        System.out.println("url----->"+url);
		System.out.println("scan res:   "+reult);
		return reult;
	}

方法5: URl拼接,type为“get”或者“post”,对用是doGet方法和doScan方法

	private String toUrl(String tableName, String type) {
		return ip + "/" + tableName + "/" + type;
	}

方法6:结果处理获取rowkey集合,正则的处理

public static Set<String> getValueForRow(String resaultStr){
        Set<String> returns = new HashSet<>();
        try {
            JSONObject jsStr = JSONObject.parseObject(resaultStr);
            String list = jsStr.getString("list")
                    .replace("\\", "")
                    .replace("[\"", "")
                    .replace("\"]", "")
                    .replace("\"keyvalues\":", "")
                    .replace("[", "")
                    .replace("]", "")
                    .replace("/", "-");
            System.out.println("list--->"+list);
            if (StringUtils.isEmpty(list)){
                return returns;
            }
            String[] entitysStrs = list.split("},");
            List<JSONObject>entitysJSONObjects = new ArrayList<>();
            for (String entitysStr:entitysStrs) {
                if (!entitysStr.endsWith("}")){
                    entitysStr+="}";
                }
                entitysJSONObjects.add(JSONObject.parseObject(entitysStr.replace("\\","")));
            }
            entitysJSONObjects.forEach(jsonObject -> {
                String rowkey = jsonObject.getString("row");
                returns.add(rowkey);
            });
            return returns;
        }catch (Exception e){
            e.printStackTrace();
            return returns;
        }
    }

方法7:结果处理获取字段值集合,正则处理

 public static List<ResaultEntity> getValueForRow(String resaultStr, List<ResaultEntity>colums){
        List<ResaultEntity> returns = new ArrayList<>();
	    try {
            JSONObject jsStr = JSONObject.parseObject(resaultStr);
            String list = jsStr.getString("list")
                    .replace("\\", "")
                    .replace("[\"", "")
                    .replace("\"]", "")
                    .replace("\"keyvalues\":", "")
                    .replace("[", "")
                    .replace("]", "")
                    .replace("/", "-");
            System.out.println("list--->"+list);
            if (StringUtils.isEmpty(list)){
                return returns;
            }
            String[] entitysStrs = list.split("},");
            List<JSONObject>entitysJSONObjects = new ArrayList<>();
            for (String entitysStr:entitysStrs) {
                if (!entitysStr.endsWith("}")){
                    entitysStr+="}";
                }
                entitysJSONObjects.add(JSONObject.parseObject(entitysStr.replace("\\","")));
            }
            if (colums.size()!=0){
                entitysJSONObjects.forEach(jsonObject -> {
                    colums.forEach(colum->{
                        if (colum.getColumn().equals(jsonObject.get("Column").toString())){
                            ResaultEntity entity = new ResaultEntity();
                            entity.setRow(jsonObject.getString("row"));
                            entity.setColumn(colum.getColumn());
                            entity.setValue(jsonObject.get("value").toString());
                            returns.add(entity);
                        }
                    });
                });
            }
            return returns;
        }catch (Exception e){
            e.printStackTrace();
            return returns;
        }
    }
  • 3
    点赞
  • 2
    收藏
    觉得还不错? 一键收藏
  • 1
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 1
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值