Flink踩坑之DEBUG日志配置与Sink到Mysql

Flink日志配置

log4j，debug需要落盘的话，代码中配置没用，需要在flink/conf/log4j.properties进行修改
代码中使用；

Logger logger = LoggerFactory.getLogger(XXXXX.class);
logger.debug("This message contains {} placeholders. {}", "origin", JSON.toJSONString(xxxx));

自用如下：

#
#  Unless required by applicable law or agreed to in writing, software
#  distributed under the License is distributed on an "AS IS" BASIS,
#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#  See the License for the specific language governing permissions and
# limitations under the License.
################################################################################
# This affects logging for both user code and Flink
log4j.rootLogger=DEBUG, file

# Uncomment this if you want to _only_ change Flink's logging
#log4j.logger.org.apache.flink=INFO
# The following lines keep the log level of common libraries/connectors on
# log level INFO. The root logger does not override this. You have to manually
# change the log levels here.
log4j.logger.akka=INFO
log4j.logger.org.apache.kafka=INFO
log4j.logger.org.apache.hadoop=INFO
log4j.logger.org.apache.zookeeper=INFO
# Log all infos in the given file

log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.file=/opt/frame/debuglogs/flink.log
log4j.appender.file.append=true
log4j.appender.file.Threshold =DEBUG
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n
# Suppress the irrelevant (wrong) warnings from the Netty channel handler
log4j.logger.org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline=DEBUG, file

Sink到Mysql

笔者使用德鲁伊+mybatis结合sink到mysql中的时候，发现数据不对，达不到需求，不确定原因是什么，有大佬知道也可以告知下。需求是：只保留第一批数据，第二批数据不希望插入成功。然而第一批数据写入不成功，第二批数据数据成功（存在主键冲突，不知道是否是框架给优化掉了）。最终采用手写的方式插入成功，没问题。
Sink方法如下：

new RichSinkFunction<List<OrderAndTask>>() {
                    PreparedStatement ps;
                    private Connection connection;
                    @Override
                    public void open(Configuration parameters) throws Exception {
                        connection = MysqlUtils.getSinkMysqlConnection();
                        String sql = "insert ignore into order_task_relevance(id, task_id, create_day) values(?, ?, ?);";
                        ps = this.connection.prepareStatement(sql);
                    }



                    @Override
                    public void invoke(List<OrderAndTask> value, Context context) throws Exception {
                        for (OrderAndTask v : value) {
                            ps.setString(1,v.getId());
                            ps.setString(2,v.getTaskId());
                            ps.setString(3,v.getCreateDay());
                            ps.executeUpdate();
                        }
                    }

                    @Override
                    public void close() throws Exception {
                        //关闭连接和释放资源
                        if (connection != null) {
                            connection.close();
                        }
                        if (ps != null) {
                            ps.close();
                        }
                    }
                }