qq_18532033的博客

原创 ubuntu 上使用 aptitude 安装 pythondev报错

aptitude: relocation error: aptitude: symbol_ZN3URIcvNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEB5cxx11Ev version APTPKG_5.0 not defined in file libapt-pkg.so.5.0 with linktime reference

原创 ubutun 找不到 apt-get 指令

ubutun18 上执行 sudo apt-get install aptitude后找不到 apt-get 指令

原创 遍历reids中的所有key、value、ttl

public static void main(String[] args) throws InterruptedException { // 连接redis服务器，localhost:6379 Jedis redis = new Jedis("dev-machine", 6379); while (true) { Set<String> keys = redi...

原创 kafka数据丢失问题

1）消费端弄丢了数据唯一可能导致消费者弄丢数据的情况，就是说，你那个消费到了这个消息，然后消费者那边自动提交了offset，让kafka以为你已经消费好了这个消息，其实你刚准备处理这个消息，你还没处理，你自己就挂了，此时这条消息就丢咯。这不是一样么，大家都知道kafka会自动提交offset，那么只要关闭自动提交offset，在处理完之后自己手动提交offset，就可以保证数据不会丢。但是此时...

原创 读取RDD的数据加载DataFrame

package cn.com.bigData0427import org.apache.spark.sql.{DataFrame, Dataset, Row, SparkSession}case class Person(id: Int, name: String, age: Int, score: Double)/** * 通过反射的方式将DataSet(RDD)转换成DataFr...

原创 Runnable 线程

/** * 创建线程方式二: * 1、创建：实现Runnable+重写run * 2、启动: 创建实现类对象 +Thread对象+ start * * 推荐: 避免单继承的局限性，优先使用接口 * 方便共享资源 * @author 裴新 QQ:3401997271 * */public class StartRun implements Runnable{ /** * ...

原创 静态代理

package com.sxt.thread;/** * 静态代理 * 公共接口: * 1、真实角色 * 2、代理角色 * * @author 裴新 QQ:3401997271 * */public class StaticProxy { public static void main(String[] args) { new WeddingCompany(new Yo...

转载 Spark Core 核心知识--RDD

Spark Core 核心知识--RDD 1、Spark核心功能1.1 SparkContext1.2 存储体系1.3 计算引擎1.4 部署模式2、Spark扩展功能2.1 Spark2.2 Spark Streaming2.3 Spark Graphx2.4 Spark MLlib3、Spark核心概念4、Spark 基本架构 ...

转载 Spark中文指南(入门篇)-Spark编程模型(一)

Spark中文指南(入门篇)-Spark编程模型(一) 前言 本章将对Spark做一个简单的介绍，更多教程请参考：Spark教程 本章知识点概括Apache Spark简介Spark的四种运行模式Spark基于Standlone的运行流程Spark基于YARN的运行流程Apache Spark是什么？Spark是一个用来实现...

原创 hadoop 任务提交源码分析

原创 storm报错

C:\cn\com\org\net\yys\program\jdk1.7.0_51\bin\java.exe "-javaagent:C:\cn\com\org\net\yys\program\IntelliJ IDEA 2018.3.3\lib\idea_rt.jar=12452:C:\cn\com\org\net\yys\program\IntelliJ IDEA 2018.3.3\bin" ...

原创 pip --help

E:\BaiduYunDownload\资料\1\爬虫第三节\day03>pipUsage: pip <command> [options]Commands: install Install packages. download Download packages. uninst...

原创 Mr源码分析

Job.class@InterfaceAudience.Public@InterfaceStability.Evolvingpublic class Job extends JobContextImpl implements JobContext { 省略1000多行代码。。。 public boolean waitForCompletion(boolean verbose ) ...

原创 MapTask.java

@SuppressWarnings("unchecked") private <INKEY,INVALUE,OUTKEY,OUTVALUE> void runNewMapper(final JobConf job, final TaskSplitIndex splitIndex, final T...

原创 Partitioner 源码学习

Partitions the key space.Partitioner controls the partitioning of the keys of the intermediate map-outputs. The key (or a subset of the key) is used to derivethe partition, typically by a hash func...

原创 Job Class

Set the Jar by finding where a given class came from.@param cls the example class public void setJarByClass(Class<?> cls) { ensureState(JobState.DEFINE); conf.setJarByClass(cls); ...

原创 Reducer源码学习

import org.apache.hadoop.io.LongWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Reducer;import java.io.IOException;public class MyReducer2 extends Reducer<Text, L...

原创 Mapper源码

public class MyMapper2 extends Mapper<LongWritable, Text,Text,LongWritable> { @Override protected void map(LongWritable key, Text value, Context context) throws IOException, Interrupted...

原创 MapReduce Demo

MainClasspackage com.bjsxt.mr.wordcount;import java.io.IOException;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.LongWritable;import ...

原创 mvn archetype:generate

C:\Users\Administrator>mvn archetype:generate[WARNING][WARNING] Some problems were encountered while building the effective settings[WARNING] expected START_TAG or END_TAG not TEXT (position: TE...

原创 java

C:\Users\Administrator\consumerBanking>java用法: java [-options] class [args...] (执行类) 或 java [-options] -jar jarfile [args...] (执行 jar 文件)其中选项包括: -d32 使用 32 ...

原创 git --help

C:\Users\Administrator>git --helpusage: git [--version] [--help] [-C <path>] [-c name=value] [--exec-path[=<path>]] [--html-path] [--man-path] [--info-path] [-p ...

原创 maven

mvn archetype:generate -DgroupId=com.companyname.bank -DartifactId=consumerBanking -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false

原创 mvn --help

C:\Users\Administrator>mvn --helpusage: mvn [options] [<goal(s)>] [<phase(s)>]Options: -am,--also-make If project list is specified, also ...

原创 flume-ng agent --conf-file option --name a1 -Dflume.root.logger=INFO,console

[root@node002 flume]# flume-ng agent --conf-file option --name a1 -Dflume.root.logger=INFO,console Warning: No configuration directory set! Use --conf <dir> to override.Info: Includ...

原创 flume-conf.properties.template

[root@node002 conf]# vim flume-conf.properties.template # The configuration file needs to define the sources, # the channels and the sinks.# Sources, channels and sinks are defined per agent, #...

原创 flume-ng help

[root@node002 flume]# flume-ng helpUsage: /opt/flume-1.6.0/bin/flume-ng <command> [options]...commands: help display this help text agent run a Fl...

原创 flume-ng version

[root@node002 bin]# ./flume-ng versionFlume 1.6.0Source code repository: https://git-wip-us.apache.org/repos/asf/flume.gitRevision: 2561a23240a71ba20bf288c7c2cda88f443c2080Compiled by hshreedharan...

原创 flume-env.sh

# Licensed to the Apache Software Foundation (ASF) under one# or more contributor license agreements. See the NOTICE file# distributed with this work for additional information# regarding copyrigh...

原创 nginx.conf

[root@node002 conf]# cat nginx.conf#user nobody;worker_processes 1;#error_log logs/error.log;#error_log logs/error.log notice;#error_log logs/error.log info;#pid logs/nginx.pid;...

原创 chkconfig --help

[root@node002 init.d]# chkconfig --helpchkconfig version 1.3.49.3 - Copyright (C) 1997-2000 Red Hat, Inc.This may be freely redistributed under the terms of the GNU Public License.usage: chkconf...

原创 hive官网建表语法

CREATE [TEMPORARY] [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name -- (Note: TEMPORARY available in Hive 0.14.0 and later) [(col_name data_type [COMMENT col_comment], ... [constraint_specif...

原创 tar --help

[root@node002 ~]# tar --helpUsage: tar [OPTION...] [FILE]...GNU `tar' saves many files together into a single tape or disk archive, and canrestore individual files from the archive.Examples: ta...

原创 hdfs前提和设计目标

硬件错误硬件错误是常态而不是异常。HDFS可能由成百上千的服务器所构成，单机故障概率的存在意味着总有一部分服务器不工作的。加粗样式错误检测和快速自动恢复是HDFS最核心架构目标。流式数据访问运行在HDFS上的应用需要流式访问它们的数据集。HDFS的设计重点是批处理，而不是交互处理。是高吞吐量而不是低延迟。为了提高数据的吞吐量，在关键方面修改POSIX的语义。大规模数据集HDFS上...

原创 hadoop fedaration

&lt;configuration&gt; &lt;property&gt; &lt;name&gt;fs.defaultFS&lt;/name&gt; &lt;value&gt;viewfs://ClusterX&lt;/value&gt; &lt;/property&gt; &lt;property&gt; &lt;name&

原创 hive 查看表结构

desc psn;desc formatted psn;

原创 专业英语

verify

原创 shell脚本 启动关闭 hadoop2.6.5集群

启动集群的脚本#/bin/bashfor node in node002 node003 node004do ssh $node &amp;amp;amp;quot;source /etc/profile; zkServer.sh start;&amp;amp;amp;quot;donestart-dfs.shssh node003 &amp;amp;amp;quot;source /etc/profile; start-yarn.sh;&amp;amp;a

原创 Hive DML

Hive DMLLOAD DATA [LOCAL] INPATH 'filepath' [OVERWRITE] INTO TABLE tablename [PARTITION (partcol1=val1, partcol2=val2 ...)]

原创 hive 分区

Hive 分区partition必须在表定义时指定对应的partition字段a、单分区建表语句：create table day_table (id int, content string) partitioned by (dt string);单分区表，按天分区，在表结构中存在id，content，dt三列。以dt为文件夹区分b、 双分区建表语句：create table da...