OpenMLDB Weekly Update(2021.9.5-2021.9.12)

OpenMLDB

Summary

本周合并 Pull requests 11个，新增Pull requests 9个，关闭 Issues 14个，新增 Issues 12个。总计57个文件修改，新增3002行代码，删除539行代码。

Merged Pull Requests

• feat: refine the description of benchmark in readme#405
• feat: update the benchmark images for readme#404
• feat: remove junit dependencies and covert java cases as scala#396
• feat: bump hadoop-common to 2.8.5 to avoid vulnerability#388
• fix: fix a bug in union with instance not in window#381
• feat: add batchjob module#359
• feat: add task manager module#361
• feat: enhance plan optimization for group and filter#350
• feat: support column query with the same name in window skew optimization#349
• feat: add java common lib#347
• test: modify test case#368

Open Pull Requests

• feat: bump junit from 4.11 to 4.13.1 in /java/openmldb-batchjob#382
• WIP : create message table#383
• feat: add batch, batchjob and taskmanager as java submodules#386
• fix: fix some code implementation in window skew optimization#392
• feat: support aggregation over the whole table#393
• feat: add integration test#395
• feat: support insert multiple rows into a table using a single SQL insert statement#399
• feat: add kubernetes java dependencies for taskmanager#400
• fix: python test and cicd#401

Close Issues

• Remove junit since of the EPL license for OpenMLDB Batch#390
• Avoid vulnerability by upgrading hadoop-common dependency#387
• rtidb磁盘占用#389
• feat: engine plan optimization for where and group with the same partition#317
• 是否支持批量插入#177
• 字段含义#378
• 表使用大小#373
• Fix the issue to enable window skew optimization for window union case#374
• Enable WindowSkewOpt by default and resolve the running issues#335
• Add module for OpenMLDB Batch to run custom SQL and submit by TaskManager#351
• Add TaskManager service to submit OpenMLDB Batch jobs#360
• scripts: package java sdk set cmake type to release#372
• Multiple columns with the same name can't execute when last join and over window#356
• Add java common lib#342

Open Issues

• feat: improve error message system.#406
• feat: support insert multiple rows into a table using a single SQL insert statement.#398
• feat: api server support parameterized query#397
• Add engine test on performance insensitive mode#394
• Bug: SQL INSERT Statement with multi rows does not work as expected#391
• Make openmldb-batchjob and openmldb-taskmanager as submodules of openmldb-parent#385
• Support AWS S3 for offline data lake storage#384
• Support creating database API for NearlineTablet#380
• Support hive metastore for NearlineTablet#379
• Support submit and manage Flink jobs for TaskManager#376
• Support submit and manage Kubernetes jobs for TaskManager#375
• scripts: package java sdk set cmake type to release#371

Contributors

• dl239 (dl239@126.com)
• tobe (tobeg3oogle@gmail.com)
• Wang ZeKai (1290561498@qq.com)
• wuyou10206 (zw1020688@163.com)

Highlights

本周新的三个Java模块，分别是openmldb-common、openmldb-batchjob和openmldb-taskmanager。通过可重用的Java模块抽象，可以快速实现新的子模块。BatchJob模块和TaskManager实现了第一个最小功能集版本，提供了基础的批处理任务管理功能。

本周通过拓展SQL语法解析器和物理计划优化器，实现了在线执行引擎的全表分组聚合和过滤功能，此功能难点在于需要在SQL语法解析阶段就区分SQL中的UDF和UDAF函数，并最终生成对应的逻辑计划和优化后的物理计划。

欢迎更多开发者关注和参与OpenMLDB开源项目。