2019年08月_SeasonRun

原创 linux配置yum源

wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repowget -O /etc/yum.repos.d/CentOS.repo http://mirrors.163.com/.help/CentOS7-Base-163.repo (2选1)rpm --import ...

2019-08-31 21:35:03 90

原创 Hadoop 2.6.1安装配置详解

1.配置Java，我这里是用的1.7版本，配置好环境变量；2.下载Hadoop（https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/）并上传至服务器上；3.进入到/usr/local/src/hadoop-2.6.1/etc/hadoop目录下，开始配置：（1）hadoop-env.sh -> JAVA_HOME...

2019-08-31 09:16:50 375

原创 ssh免密连接服务器

我这里有三台机器做集群，需要设置免密连接，步骤如下：1. 在三台机器上分别执行ssh-keygen；2.cd /root/.ssh目录，会看到id_rsa、id_rsa.pub两个文件，此时再其中一台服务器上该目录下新建authorized_keys；3.将三台机器的id_rsa.pub文件内容全部复制到authorized_keys中；4.将该文件分发到另外两台服务器 /root...

2019-08-30 11:18:37 130

原创 workstation新建虚机报错：您已输入用户名，客户机操作系统将保留此用户名

在VMware Workstation中新建虚机时，提示“您已输入用户名，客户机操作系统将保留此用户名”，原因是输入的用户名root和系统原有的账户冲突，此时只修要修改用户名为其他用户即可。...

2019-08-30 09:19:18 1834

原创监控识别初探

基于人体形状识别：基于人体形状的人体目标检测，这种方法一般是通过将人体进行建模，建模的时候重点考虑人体的形状特征，然后得到用来检测人体目标的模板，通过将模板应用到图像帧中进行人体目标检测。可分为基于人体全身形状建模和基于局部特征建模，如常见的头部识别、脸部识别、上下肢识别等，具体有Hough变换法，“凸”字形模板法，最小负曲率法，人体高度比例等。基于人体肤色识别：采用人体肤色进行构建模板，...

2019-08-29 22:22:27 430

原创 The OVF package requires unsupported hardware

VSphere Client导入OVF文件时提示以下信息：The OVF packages uses features that are not supported:The OVF package requires unsupported hardware.这是因为导入文件的hardware version不支持，于是想到要修改vmx文件里的参数配置，这里需要用到ovftool（参照h...

2019-08-29 21:35:41 709

原创 The following manifest file entry (line 1) is invalid: SHA256

今天从本地的VMware Workstation导出的VM ovf文件在导入到VSphere Client集群时报错：The OVF package is invalid and cannot be deployed.The following manifest file entry (line 1) is invalid: SHA256 (xxxxxxx.ovf).原因是vSphere...

2019-08-29 20:08:51 1637 1

原创 flume-kafka配置详解

目标：通过flume抓取日志消息传递给kafkaflume和kafka相关配置见前序文章，本文主要讲解如何实现flume将消息传递给kafka。1、配置flume conf文件，在flume/conf目录下创建conf文件，flume_kafka.conf# Name the components on this agenta1.sources = r1a1.sinks = k1...

2019-08-27 14:22:04 3263

原创 kafka集群配置

配置kafka集群只需要在单节点基础上，将kafka安装包分发到各个节点，并修改config/server.properties文件中的broker.id，从0开始，如有三台机器，则依次修改为0,1,2；分别在三台机器上启动kafka，即可进行测试：./bin/kafka-server-start.sh config/server.properties ##启动kafka./bin/...

2019-08-27 12:52:51 158

原创 kafka安装配置详解

1.启动zookeeper，jps---> QuorumPeerMain即启动成功，详见https://blog.csdn.net/haiziccc/article/details/994688312.download kafka from http://mirrors.hust.edu.cn/apache/kafka/;3.将压缩文件上传至server目录，解压；4.进入ka...

2019-08-27 12:48:09 148 1

原创 python-setuptool-pip install

Download python from https://www.python.org/ftp/python/Download setuptools from https://pypi.org/project/setuptools/#filesDownload pip from https://pypi.org/project/pip/#filesYou can use:wge...

2019-08-26 23:22:04 226

This article describes how to config flume load balance, Master is the data generator, flume agent get logserver.log then transmit to slave1 and slave2, which is the collector, and save log to hdfs....

2019-08-25 08:36:44 178

原创 flume failover

Architecture like that:Master conf:# Name the components on this agenta1.sources = r1a1.sinks = k1 k2a1.channels = c1#describe/configure the sourcea1.sources.r1.type = execa1.sources...

2019-08-25 00:19:34 154

原创 flume to hdfs

With flume, you can collect log and storage to hdfs.First, you should create a hdfs path, like "hadoop fs -mkdir /flume_data_pool";Second, create a conf file in flume conf path,# Name the compon...

2019-08-24 23:55:32 188

原创 Flume deploy

Flume provide powerful functions to collect and move large logs from different source to storage like hdfs or hbase.Data collect flow like that:Every agent contains source, channel and sink....

2019-08-24 23:36:40 88

原创 Hive-Hbase

1. you can create a table and insert data to it on Hbase first;2. Create table links on hive,–create external table classes(id int, name string, age int)–STORED BY 'org.apache.hadoop.hive.hbase.H...

2019-08-23 06:54:43 100

原创 Hive commands notes

hive> insert overwrite directory '/result' select * from userbehavior;hive> insert overwrite local directory '/hive_test/result' select * from userbehavior;hive> create table userbehavior...

2019-08-22 20:45:40 108

原创 Mysql install

1. Install mysqlyum install mysql -yyum install mysql-server -y2.Start mysql/etc/init.d/mysqld start3.Config user and passwdmysqladmin -u root password ****4.Login mysqlmysql -u root -...

2019-08-21 23:19:30 279

原创 Hive Install

1.Download hive package and uncompress it.2.Create hive-site.xml under conf path, config params like that:<configuration> <property> <name>javax.jdo.opti...

2019-08-21 23:11:22 314

原创 Hbase command - shell

hbase shell:进入shell模式create 'user_table', 'user','age','sex'：创建表describe 'user_table'：查看表alter 'user_table', {NAME=>'age', METHOD=>'delete'}:删除表alter 'user_table', {NAME=>'flags', VER...

2019-08-19 14:20:41 170

原创 HBase Install

1.Download hbase package, like hbase-0.98.0-hadoop.2.Upload package to server.3.Edit hbase-env.sh,# The java implementation to use. Java 1.6 required.# export JAVA_HOME=/usr/java/jdk1.6.0/ex...

2019-08-19 12:54:49 153

原创 supervisors on storm ui display zero

supervisors on storm ui display zero, means slave node cannot monitor by zookeeper, so check the zookeeper status, then find it in standalone mode, not follower.Check and find something missed in ...

2019-08-14 13:47:51 96

原创 zookeeper Install

I have download the zookeeper from https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/zookeeper-3.5.5/,then I created myid, modify the zoo.cfg, and add the env variable. When I start the service, i...

2019-08-13 20:13:26 154

原创 WARN LoadSnappy: Snappy native library not loaded

[root@master mapreduce_wordcount_python]# ./run.shDeleted hdfs://192.168.183.10:9000/outputpackageJobJar: [./map.py, ./red.py, /usr/local/src/hadoop-1.2.1/tmp/hadoop-unjar3556020200698085095/...

2019-08-12 16:25:15 248

原创 Word segment based on Jieba

First, u should install jieba, pip install jieba or download in http://pypi.python.org/pypi/jieba/, unpack and cd the storage directory install. or put it on site-packages directory.Second, u can u...

2019-08-07 20:00:22 174

原创 Similarity Computing based on LCS

# -*- coding: utf-8 -*-#!/usr/bin/pythonimport sysdef cal_score(first_str, second_str): len_vv = [[0]*50]*50 len1 = len(first_str) len2 = len(second_str) for i in range(1, len1...

2019-08-07 14:47:29 100

转载中文分词词性名称注解

词性编码词性名称注解 Ag 形语素形容词性语素。形容词代码为 a，语素代码ｇ前面置以A。 a 形容词取英语形容词 adjective的第1个字母。 ad 副形词 ...

2019-08-06 20:32:37 3742 1

原创 Similarity Computing based on TF-IDF and Cosine theorem

This is my first program for similarity computing, and will continuous optimization.Steps:1.Enough articles and split to words, at the same time you can get a word_dict(I think it's better with ...

2019-08-06 15:53:20 174

原创 TF-IDF(2)

#!/usr/bin/pythonimport sysimport mathcurrent_word = Nonecount_pool = []sum = 0#Total article docs_cnt = 524#the result of split wordsfile1 ='d:/share/testresult2'file = open(file1, 'r', ...

2019-08-06 10:38:43 98

原创 TF-IDF(1)

#!/usr/bin/pythonimport sys#split wordfile1 ='d:/share/articlelib'file = open(file1, 'r', encoding='UTF-8')for line in file: ss = line.strip().split('\t', 1) if len(ss) != 2: co...

2019-08-06 10:36:53 103

原创 TF-IDF(3)

#!/usr/bin/pythonimport osimport sys import gzip#suppose testresult1 contains two article which you want compare, testresult2 is the result of idffile_input_fd = 'd:/share/testresult1'idf_dict...

2019-08-06 09:42:20 99

原创 Countword

#!/usr/bin/pythonimport syslist = []current_word = Nonecount_pool = []sum = 0file1 = 'd:/share/1business.seg.txt'file = open(file1, 'r', encoding='UTF-8')for line in file: ss = line....

2019-08-06 08:53:34 210

原创 Linux set row number

1.Temporarywhen use vim edit file, you can use ":set number" display row num.2.Permanentyou can edit /etc/vimrc, add "set number" at the end, then you'll see row number when you edit file.

2019-08-05 11:19:50 250

原创 BinarySearch

#!/usr/bin/python#define initial index 0, and max index length-1#find ip_num in ip_lib_listip_num = 15ip_lib_list = [[2, 2, 3], [5, 3, 4], [7, 4, 5], [8, 5, 6], [12, 7, 8],[15, 9, 0], [19, 0, ...

2019-08-01 20:13:57 603

haiziccc的专栏