先自我介绍一下,小编浙江大学毕业,去过华为、字节跳动等大厂,目前阿里P7
深知大多数程序员,想要提升技能,往往是自己摸索成长,但自己不成体系的自学效果低效又漫长,而且极易碰到天花板技术停滞不前!
因此收集整理了一份《2024年最新大数据全套学习资料》,初衷也很简单,就是希望能够帮助到想自学提升又不知道该从何学起的朋友。
既有适合小白学习的零基础资料,也有适合3年以上经验的小伙伴深入学习提升的进阶课程,涵盖了95%以上大数据知识点,真正体系化!
由于文件比较多,这里只是将部分目录截图出来,全套包含大厂面经、学习笔记、源码讲义、实战项目、大纲路线、讲解视频,并且后续会持续更新
如果你需要这些资料,可以添加V获取:vip204888 (备注大数据)
正文
// System.out.println(hashMap);
String appKey = hashMap.get("appKey");
String appVersion = hashMap.get("appVersion");
String deviceId = hashMap.get("deviceId");
String phone_no = hashMap.get("phone_no");
Mail mail = new Mail(appKey, appVersion, deviceId, phone_no);
// System.out.println(mail);
return mail;
}
});
dataStream.print();
// sink
String sql = "INSERT INTO testmaxwell1.ods_countlyV2 (appKey, appVersion, deviceId, phone_no) " +
"VALUES (?, ?, ?, ?)";
MyClickHouseUtil ckSink = new MyClickHouseUtil(sql);
dataStream.addSink(ckSink);
env.execute();
}
}
2.工具类写入clickhouse
package com.kszx;
import com.kszx.Mail;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.streaming.api.functions.sink.RichSinkFunction;
import ru.yandex.clickhouse.ClickHouseConnection;
import ru.yandex.clickhouse.ClickHouseDataSource;
import ru.yandex.clickhouse.settings.ClickHouseProperties;
import ru.yandex.clickhouse.settings.ClickHouseQueryParam;
import java.sql.PreparedStatement;
import java.util.HashMap;
import java.util.Map;
public class MyClickHouseUtil extends RichSinkFunction {
private ClickHouseConnection conn = null;
String sql;
public MyClickHouseUtil(String sql) {
this.sql = sql;
}
@Override
public void open(Configuration parameters) throws Exception {
super.open(parameters);
return ;
}
@Override
public void close() throws Exception {
super.close();
if (conn != null)
{
conn.close();
}
}
@Override
public void invoke(Mail mail, Context context) throws Exception {
String url = "jdbc:clickhouse://172.xx.xxx.xxx:8123/testmaxwell1";
ClickHouseProperties properties = new ClickHouseProperties();
properties.setUser("default");
properties.setPassword("xxxxxx");
properties.setSessionId("default-session-id2");
ClickHouseDataSource dataSource = new ClickHouseDataSource(url, properties);
Map<ClickHouseQueryParam, String> additionalDBParams = new HashMap<>();
additionalDBParams.put(ClickHouseQueryParam.SESSION_ID, "new-session-id2");
try {
conn = dataSource.getConnection();
PreparedStatement preparedStatement = conn.prepareStatement(sql);
preparedStatement.setString(1,mail.getAppKey());
preparedStatement.setString(2, mail.getAppVersion());
preparedStatement.setString(3, mail.getDeviceId());
preparedStatement.setString(4, mail.getPhone_no());
preparedStatement.execute();
}
catch (Exception e){
e.printStackTrace();
}
}
}
3.表属性类
package com.kszx;
//package com.demo.flink.pojo;
public class Mail {
private String appKey;
private String appVersion;
private String deviceId;
private String phone_no;
public Mail(String appKey, String appVersion, String deviceId, String phone_no) {
this.appKey = appKey;
this.appVersion = appVersion;
this.deviceId = deviceId;
this.phone_no = phone_no;
}
public String getAppKey() {
return appKey;
}
public void setAppKey(String appKey) {
this.appKey = appKey;
}
public String getAppVersion() {
return appVersion;
}
public void setAppVersion(String appVersion) {
this.appVersion = appVersion;
}
public String getDeviceId() {
return deviceId;
}
public void setDeviceId(String deviceId) {
this.deviceId = deviceId;
}
public String getPhone_no() {
return phone_no;
}
public void setPhone_no(String phone_no) {
this.phone_no = phone_no;
}
@Override
public String toString() {
return "Mail{" +
"appKey='" + appKey + '\'' +
", appVersion='" + appVersion + '\'' +
", deviceId='" + deviceId + '\'' +
", phone_no='" + phone_no + '\'' +
'}';
}
public Mail of(String appKey, String appVersion, String deviceId, String phone_no)
{
return new Mail(appKey, appVersion, deviceId, phone_no);
}
}
4.pom依赖(注意打包在服务器上运行时会和flink的lib目录下的log4j依赖冲突的问题,如果在服务器上执行jar包时依赖冲突报错的话,最好屏幕代码里的依赖,保留flink原版lib下的依赖)
<?xml version="1.0" encoding="UTF-8"?>
4.0.0
<groupId>com.kszx</groupId>
<artifactId>flink1kc</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<maven.compiler.source>8</maven.compiler.source>
<maven.compiler.target>8</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-java</artifactId>
<version>1.11.1</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-clients_2.11</artifactId>
<version>1.11.1</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-kafka_2.11</artifactId>
<version>1.11.1</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table-api-java-bridge_2.11</artifactId>
<version>1.11.1</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java_2.10</artifactId>
<version>1.3.2</version>
</dependency>
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
<version>1.2.59</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.11</artifactId>
<version>1.0.2</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>1.0.2</version>
</dependency>
<!-- 写入数据到clickhouse -->
<dependency>
<groupId>ru.yandex.clickhouse</groupId>
<artifactId>clickhouse-jdbc</artifactId>
<version>0.1.54</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.1</version> <!-- 这里指定了插件版本 -->
<configuration>
<!-- 在这里可以配置编译器参数 -->
<source>8</source>
<target>8</target>
</configuration>
</plugin>
<!-- 可能还有其他插件配置 -->
</plugins>
</build>
```
#小技巧
查看flink正在执行的任务,如果提交作业后,再查看flink任务时,没有job正在运行,说明作业未成功运行,看报错或看日志。具体在哪里出问题就在那个范围内解决
./flink list
**网上学习资料一大堆,但如果学到的知识不成体系,遇到问题时只是浅尝辄止,不再深入研究,那么很难做到真正的技术提升。**
**需要这份系统化的资料的朋友,可以添加V获取:vip204888 (备注大数据)**
![img](https://img-blog.csdnimg.cn/img_convert/815dd92a8b925b55f3243cb293e941af.png)
**一个人可以走的很快,但一群人才能走的更远!不论你是正从事IT行业的老鸟或是对IT行业感兴趣的新人,都欢迎加入我们的的圈子(技术交流、学习资源、职场吐槽、大厂内推、面试辅导),让我们一起学习成长!**
行,看报错或看日志。具体在哪里出问题就在那个范围内解决
./flink list
网上学习资料一大堆,但如果学到的知识不成体系,遇到问题时只是浅尝辄止,不再深入研究,那么很难做到真正的技术提升。
需要这份系统化的资料的朋友,可以添加V获取:vip204888 (备注大数据)
[外链图片转存中…(img-G6BjXLuX-1713375364168)]
一个人可以走的很快,但一群人才能走的更远!不论你是正从事IT行业的老鸟或是对IT行业感兴趣的新人,都欢迎加入我们的的圈子(技术交流、学习资源、职场吐槽、大厂内推、面试辅导),让我们一起学习成长!