数仓项目问题的不完全记录

任务调度

使用DolphinScheduler进行数据仓库工作流调度时，由数据仓库ods层向dim层调度时，任务实例运行失败，查看实例日志，发现无法识别sql语句连接表中的字段。打开shell脚本发现dim_user_zip该字段其他部分可以识别，不存在当前位置不能识别的问题。

于是仔细检查shell脚本，发现shell语句的开端出现了错误，修改之后运行成功。（该脚本是从虚拟机的bin目录下传输过来的，如果当前出现问题，之前的shell脚本按理也无法执行，于是去虚拟机检查脚本发现语句没有问题，大概是传输过程中出现了失误）

使用DolphinScheduler进行数据仓库工作流调度，定义好工作流并确保各节点及调度线定义没有问题的情况下，工作流实例显示正在执行，但是在任务实例中没有创建任何新的任务实例。

等了一段时间之后，还是没有任务实例被创建，估计调度过程出了问题。于是在监控中心查看各个架构管理是否有问题。于是在Master中发现memoryUsage已经高达98%。查阅资料后发现，如果达到95%以上说明是虚拟机内存太小了，不足以创建实例。于是将hadoop102的这台虚拟机内存分配调至16G，成功解决任务调度的问题。

可视化

(superset) [atguigu@hadoop102 ~]$ superset db upgrade
Traceback (most recent call last):
  File "/opt/module/miniconda3/envs/superset/bin/superset", line 5, in <module>
    from superset.cli.main import superset
  File "/opt/module/miniconda3/envs/superset/lib/python3.7/site-packages/superset/__init__.py", line 21, in <module>
    from superset.app import create_app
  File "/opt/module/miniconda3/envs/superset/lib/python3.7/site-packages/superset/app.py", line 23, in <module>
    from superset.initialization import SupersetAppInitializer
  File "/opt/module/miniconda3/envs/superset/lib/python3.7/site-packages/superset/initialization/__init__.py", line 33, in <module>
    from superset.extensions import (
  File "/opt/module/miniconda3/envs/superset/lib/python3.7/site-packages/superset/extensions/__init__.py", line 116, in <module>
    celery_app = celery.Celery()
  File "/opt/module/miniconda3/envs/superset/lib/python3.7/site-packages/celery/local.py", line 509, in __getattr__
    module = __import__(self._object_origins[name], None, None, [name])
  File "/opt/module/miniconda3/envs/superset/lib/python3.7/site-packages/celery/app/__init__.py", line 5, in <module>
    from celery import _state
  File "/opt/module/miniconda3/envs/superset/lib/python3.7/site-packages/celery/_state.py", line 17, in <module>
    from celery.utils.threads import LocalStack
  File "/opt/module/miniconda3/envs/superset/lib/python3.7/site-packages/celery/utils/__init__.py", line 19, in <module>
    from .nodenames import nodename, nodesplit, worker_direct
  File "/opt/module/miniconda3/envs/superset/lib/python3.7/site-packages/celery/utils/nodenames.py", line 9, in <module>
    from kombu.entity import Exchange, Queue
  File "/opt/module/miniconda3/envs/superset/lib/python3.7/site-packages/kombu/entity.py", line 9, in <module>
    from .serialization import prepare_accept_content
  File "/opt/module/miniconda3/envs/superset/lib/python3.7/site-packages/kombu/serialization.py", line 456, in <module>
    for ep, args in entrypoints('kombu.serializers'):  # pragma: no cover
  File "/opt/module/miniconda3/envs/superset/lib/python3.7/site-packages/kombu/utils/compat.py", line 93, in entrypoints
    for ep in importlib_metadata.entry_points().get(namespace, [])
AttributeError: 'EntryPoints' object has no attribute 'get'

把python环境换成3.8之后报错

ModuleNotFoundError: No module named 'cryptography.hazmat.backends.openssl.x509

解决方案：pip install cryptography==3.3.2 -i https://repo.huaweicloud.com/repository/pypi/simple

然后报错

(superset) [atguigu@hadoop102 ~]$ superset db upgrade
Error: Could not locate a Flask application. Use the 'flask --app' option, 'FLASK_APP' environment variable, or a 'wsgi.py' or 'app.py' file in the current directory.

Error: Could not locate a Flask application. Use the 'flask --app' option, 'FLASK_APP' environment variable, or a 'wsgi.py' or 'app.py' file in the current directory.

Usage: superset [OPTIONS] COMMAND [ARGS]...
Try 'superset --help' for help.

Error: No such command 'db'.

说是因为没有配置管理员用户，执行export FLASK_APP=superset

然后又报错

ModuleNotFoundError: No module named 'wtforms.ext'

解决方案

# 这个是因为WTForms 3.0的版本去掉了ext，需要降低WTForms的版本到2.3.3
python -m pip uninstall -y WTForms
python -m pip install WTForms==2.3.3

之后就是我所期待的info日志

(superset) [atguigu@hadoop102 ~]$ superset db upgrade
--------------------------------------------------------------------------------
                                    WARNING
--------------------------------------------------------------------------------
A Default SECRET_KEY was detected, please use superset_config.py to override it.
Use a strong complex alphanumeric string and use a tool to help you generate 
a sufficiently random sequence, ex: openssl rand -base64 42
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
logging was configured successfully
2023-04-16 11:02:43,247:INFO:superset.utils.logging_configurator:logging was configured successfully
2023-04-16 11:02:43,255:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'>
We haven't found any Content Security Policy (CSP) defined in the configurations. Please make sure to configure CSP using the TALISMAN_CONFIG key or any other external software. Failing to configure CSP have serious security implications. Check https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP for more information. You can disable this warning using the CONTENT_SECURITY_POLICY_WARNING key.
2023-04-16 11:02:43,256:WARNING:superset.initialization:We haven't found any Content Security Policy (CSP) defined in the configurations. Please make sure to configure CSP using the TALISMAN_CONFIG key or any other external software. Failing to configure CSP have serious security implications. Check https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP for more information. You can disable this warning using the CONTENT_SECURITY_POLICY_WARNING key.
Falling back to the built-in cache, that stores data in the metadata database, for the following cache: `FILTER_STATE_CACHE_CONFIG`. It is recommended to use `RedisCache`, `MemcachedCache` or another dedicated caching backend for production deployments
2023-04-16 11:02:43,258:WARNING:superset.utils.cache_manager:Falling back to the built-in cache, that stores data in the metadata database, for the following cache: `FILTER_STATE_CACHE_CONFIG`. It is recommended to use `RedisCache`, `MemcachedCache` or another dedicated caching backend for production deployments
Falling back to the built-in cache, that stores data in the metadata database, for the following cache: `EXPLORE_FORM_DATA_CACHE_CONFIG`. It is recommended to use `RedisCache`, `MemcachedCache` or another dedicated caching backend for production deployments
2023-04-16 11:02:43,261:WARNING:superset.utils.cache_manager:Falling back to the built-in cache, that stores data in the metadata database, for the following cache: `EXPLORE_FORM_DATA_CACHE_CONFIG`. It is recommended to use `RedisCache`, `MemcachedCache` or another dedicated caching backend for production deployments
No PIL installation found
2023-04-16 11:02:43,384:INFO:superset.utils.screenshots:No PIL installation found
WARNI [alembic.env] SQLite Database support for metadata databases will         be removed in a future version of Superset.
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> 4e6a06bad7a8, Init
INFO  [alembic.runtime.migration] Running upgrade 4e6a06bad7a8 -> 5a7bad26f2a7, empty message
INFO  [alembic.runtime.migration] Running upgrade 5a7bad26f2a7 -> 1e2841a4128, empty message
INFO  [alembic.runtime.migration] Running upgrade 1e2841a4128 -> 2929af7925ed, TZ offsets in data sources
INFO  [alembic.runtime.migration] Running upgrade 2929af7925ed -> 289ce07647b, Add encrypted password field
INFO  [alembic.runtime.migration] Running upgrade 289ce07647b -> 1a48a5411020, adding slug to dash
INFO  [alembic.runtime.migration] Running upgrade 1a48a5411020 -> 315b3f4da9b0, adding log model
INFO  [alembic.runtime.migration] Running upgrade 315b3f4da9b0 -> 55179c7f25c7, sqla_descr
INFO  [alembic.runtime.migration] Running upgrade 55179c7f25c7 -> 12d55656cbca, is_featured
/opt/module/miniconda3/envs/superset/lib/python3.8/site-packages/alembic/ddl/sqlite.py:81: UserWarning: Skipping unsupported ALTER for creation of implicit constraint. Please refer to the batch mode feature which allows for SQLite migrations using a copy-and-move strategy.
  util.warn(
INFO  [alembic.runtime.migration] Running upgrade 12d55656cbca -> 2591d77e9831, user_id
INFO  [alembic.runtime.migration] Running upgrade 2591d77e9831 -> 8e80a26a31db, empty message
INFO  [alembic.runtime.migration] Running upgrade 8e80a26a31db -> 7dbf98566af7, empty message
INFO  [alembic.runtime.migration] Running upgrade 7dbf98566af7 -> 43df8de3a5f4, empty message
INFO  [alembic.runtime.migration] Running upgrade 43df8de3a5f4 -> d827694c7555, css templates
INFO  [alembic.runtime.migration] Running upgrade d827694c7555 -> 430039611635, log more
INFO  [alembic.runtime.migration] Running upgrade 430039611635 -> 18e88e1cc004, making audit nullable
INFO  [alembic.runtime.migration] Running upgrade 18e88e1cc004 -> 836c0bf75904, cache_timeouts
INFO  [alembic.runtime.migration] Running upgrade 18e88e1cc004 -> a2d606a761d9, adding favstar model
INFO  [alembic.runtime.migration] Running upgrade a2d606a761d9, 836c0bf75904 -> d2424a248d63, empty message
INFO  [alembic.runtime.migration] Running upgrade d2424a248d63 -> 763d4b211ec9, fixing audit fk
INFO  [alembic.runtime.migration] Running upgrade d2424a248d63 -> 1d2ddd543133, log dt
INFO  [alembic.runtime.migration] Running upgrade 1d2ddd543133, 763d4b211ec9 -> fee7b758c130, empty message
INFO  [alembic.runtime.migration] Running upgrade fee7b758c130 -> 867bf4f117f9, Adding extra field to Database model
INFO  [alembic.runtime.migration] Running upgrade 867bf4f117f9 -> bb51420eaf83, add schema to table model
INFO  [alembic.runtime.migration] Running upgrade bb51420eaf83 -> b4456560d4f3, change_table_unique_constraint
INFO  [alembic.runtime.migration] Running upgrade b4456560d4f3 -> 4fa88fe24e94, owners_many_to_many
INFO  [alembic.runtime.migration] Running upgrade 4fa88fe24e94 -> c3a8f8611885, Materializing permission
INFO  [alembic.runtime.migration] Running upgrade c3a8f8611885 -> f0fbf6129e13, Adding verbose_name to tablecolumn
INFO  [alembic.runtime.migration] Running upgrade f0fbf6129e13 -> 956a063c52b3, adjusting key length
INFO  [alembic.runtime.migration] Running upgrade 956a063c52b3 -> 1226819ee0e3, Fix wrong constraint on table columns
INFO  [alembic.runtime.migration] Running upgrade 1226819ee0e3 -> d8bc074f7aad, Add new field 'is_restricted' to SqlMetric and DruidMetric
INFO  [alembic.runtime.migration] Running upgrade d8bc074f7aad -> 27ae655e4247, Make creator owners
INFO  [alembic.runtime.migration] Running upgrade 27ae655e4247 -> 960c69cb1f5b, add dttm_format related fields in table_columns
INFO  [alembic.runtime.migration] Running upgrade 960c69cb1f5b -> f162a1dea4c4, d3format_by_metric
INFO  [alembic.runtime.migration] Running upgrade f162a1dea4c4 -> ad82a75afd82, Update models to support storing the queries.
INFO  [alembic.runtime.migration] Running upgrade ad82a75afd82 -> 3c3ffe173e4f, add_sql_string_to_table
INFO  [alembic.runtime.migration] Running upgrade 3c3ffe173e4f -> 41f6a59a61f2, database options for sql lab
INFO  [alembic.runtime.migration] Running upgrade 41f6a59a61f2 -> 4500485bde7d, allow_run_sync_async
INFO  [alembic.runtime.migration] Running upgrade 4500485bde7d -> 65903709c321, allow_dml
INFO  [alembic.runtime.migration] Running upgrade 41f6a59a61f2 -> 33d996bcc382, update slice model
INFO  [alembic.runtime.migration] Running upgrade 33d996bcc382, 65903709c321 -> b347b202819b, empty message
INFO  [alembic.runtime.migration] Running upgrade b347b202819b -> 5e4a03ef0bf0, Add access_request table to manage requests to access datastores.
INFO  [alembic.runtime.migration] Running upgrade 5e4a03ef0bf0 -> eca4694defa7, sqllab_setting_defaults
INFO  [alembic.runtime.migration] Running upgrade eca4694defa7 -> ab3d66c4246e, add_cache_timeout_to_druid_cluster
INFO  [alembic.runtime.migration] Running upgrade eca4694defa7 -> 3b626e2a6783, Sync DB with the models.py.
Constraint must have a name
INFO  [alembic.runtime.migration] Running upgrade 3b626e2a6783, ab3d66c4246e -> ef8843b41dac, empty message
INFO  [alembic.runtime.migration] Running upgrade ef8843b41dac -> b46fa1b0b39e, Add json_metadata to the tables table.
INFO  [alembic.runtime.migration] Running upgrade b46fa1b0b39e -> 7e3ddad2a00b, results_key to query
INFO  [alembic.runtime.migration] Running upgrade 7e3ddad2a00b -> ad4d656d92bc, Add avg() to default metrics
INFO  [alembic.runtime.migration] Running upgrade ad4d656d92bc -> c611f2b591b8, dim_spec
INFO  [alembic.runtime.migration] Running upgrade c611f2b591b8 -> e46f2d27a08e, materialize perms
INFO  [alembic.runtime.migration] Running upgrade e46f2d27a08e -> f1f2d4af5b90, Enable Filter Select
INFO  [alembic.runtime.migration] Running upgrade e46f2d27a08e -> 525c854f0005, log_this_plus
INFO  [alembic.runtime.migration] Running upgrade 525c854f0005, f1f2d4af5b90 -> 6414e83d82b7, empty message
INFO  [alembic.runtime.migration] Running upgrade 6414e83d82b7 -> 1296d28ec131, Adds params to the datasource (druid) table
INFO  [alembic.runtime.migration] Running upgrade 1296d28ec131 -> f18570e03440, Add index on the result key to the query table.
INFO  [alembic.runtime.migration] Running upgrade f18570e03440 -> bcf3126872fc, Add keyvalue table
INFO  [alembic.runtime.migration] Running upgrade f18570e03440 -> db0c65b146bd, update_slice_model_json
INFO  [alembic.runtime.migration] Running upgrade db0c65b146bd -> a99f2f7c195a, rewriting url from shortner with new format
INFO  [alembic.runtime.migration] Running upgrade a99f2f7c195a, bcf3126872fc -> d6db5a5cdb5d, empty message
INFO  [alembic.runtime.migration] Running upgrade d6db5a5cdb5d -> b318dfe5fb6c, adding verbose_name to druid column
INFO  [alembic.runtime.migration] Running upgrade d6db5a5cdb5d -> 732f1c06bcbf, add fetch values predicate
INFO  [alembic.runtime.migration] Running upgrade 732f1c06bcbf, b318dfe5fb6c -> ea033256294a, empty message
INFO  [alembic.runtime.migration] Running upgrade b318dfe5fb6c -> db527d8c4c78, Add verbose name to DruidCluster and Database
INFO  [alembic.runtime.migration] Running upgrade db527d8c4c78, ea033256294a -> 979c03af3341, empty message
INFO  [alembic.runtime.migration] Running upgrade 979c03af3341 -> a6c18f869a4e, query.start_running_time
INFO  [alembic.runtime.migration] Running upgrade a6c18f869a4e -> 2fcdcb35e487, saved_queries
INFO  [alembic.runtime.migration] Running upgrade 2fcdcb35e487 -> a65458420354, add_result_backend_time_logging
INFO  [alembic.runtime.migration] Running upgrade a65458420354 -> ca69c70ec99b, tracking_url
INFO  [alembic.runtime.migration] Running upgrade ca69c70ec99b -> a9c47e2c1547, add impersonate_user to dbs
INFO  [alembic.runtime.migration] Running upgrade ca69c70ec99b -> ddd6ebdd853b, annotations
INFO  [alembic.runtime.migration] Running upgrade a9c47e2c1547, ddd6ebdd853b -> d39b1e37131d, empty message
INFO  [alembic.runtime.migration] Running upgrade ca69c70ec99b -> 19a814813610, Adding metric warning_text
INFO  [alembic.runtime.migration] Running upgrade 19a814813610, a9c47e2c1547 -> 472d2f73dfd4, empty message
INFO  [alembic.runtime.migration] Running upgrade 472d2f73dfd4, d39b1e37131d -> f959a6652acd, empty message
INFO  [alembic.runtime.migration] Running upgrade f959a6652acd -> 4736ec66ce19, empty message
/opt/module/miniconda3/envs/superset/lib/python3.8/site-packages/sqlalchemy/dialects/sqlite/base.py:2011: SAWarning: WARNING: SQL-parsed foreign key constraint '('datasource_name', 'datasources', 'datasource_name')' could not be located in PRAGMA foreign_keys for table metrics
  util.warn(
INFO  [alembic.runtime.migration] Running upgrade 4736ec66ce19 -> 67a6ac9b727b, update_spatial_params
INFO  [alembic.runtime.migration] Running upgrade 67a6ac9b727b -> 21e88bc06c02, migrate_old_annotation_layers
INFO  [alembic.runtime.migration] Running upgrade 21e88bc06c02 -> e866bd2d4976, smaller_grid
INFO  [alembic.runtime.migration] Running upgrade e866bd2d4976 -> e68c4473c581, allow_multi_schema_metadata_fetch
INFO  [alembic.runtime.migration] Running upgrade e68c4473c581 -> f231d82b9b26, empty message
INFO  [alembic.runtime.migration] Running upgrade f231d82b9b26 -> bf706ae5eb46, cal_heatmap_metric_to_metrics
INFO  [alembic.runtime.migration] Running upgrade f231d82b9b26 -> 30bb17c0dc76, empty message
INFO  [alembic.runtime.migration] Running upgrade 30bb17c0dc76, bf706ae5eb46 -> c9495751e314, empty message
INFO  [alembic.runtime.migration] Running upgrade f231d82b9b26 -> 130915240929, is_sqllab_view
INFO  [alembic.runtime.migration] Running upgrade 130915240929, c9495751e314 -> 5ccf602336a0, empty message
INFO  [alembic.runtime.migration] Running upgrade 5ccf602336a0 -> e502db2af7be, add template_params to tables
INFO  [alembic.runtime.migration] Running upgrade e502db2af7be -> c5756bec8b47, Time grain SQLA
INFO  [alembic.runtime.migration] Running upgrade c5756bec8b47 -> afb7730f6a9c, remove empty filters
INFO  [alembic.runtime.migration] Running upgrade afb7730f6a9c -> 80a67c5192fa, single pie chart metric
INFO  [alembic.runtime.migration] Running upgrade 80a67c5192fa -> bddc498dd179, adhoc filters
INFO  [alembic.runtime.migration] Running upgrade bddc498dd179 -> 4451805bbaa1, remove double percents
INFO  [alembic.runtime.migration] Running upgrade bddc498dd179 -> 3dda56f1c4c6, Migrate num_period_compare and period_ratio_type
INFO  [alembic.runtime.migration] Running upgrade 3dda56f1c4c6 -> 1d9e835a84f9, empty message
INFO  [alembic.runtime.migration] Running upgrade 4451805bbaa1, 1d9e835a84f9 -> e3970889f38e, empty message
INFO  [alembic.runtime.migration] Running upgrade 4451805bbaa1, 1d9e835a84f9 -> 705732c70154, empty message
INFO  [alembic.runtime.migration] Running upgrade 4451805bbaa1, 1d9e835a84f9 -> fc480c87706c, empty message
INFO  [alembic.runtime.migration] Running upgrade fc480c87706c -> bebcf3fed1fe, Migrate dashboard position_json data from V1 to V2
INFO  [alembic.runtime.migration] Running upgrade bebcf3fed1fe, 705732c70154 -> ec1f88a35cc6, empty message
INFO  [alembic.runtime.migration] Running upgrade 705732c70154, e3970889f38e -> 46ba6aaaac97, empty message
INFO  [alembic.runtime.migration] Running upgrade 46ba6aaaac97, ec1f88a35cc6 -> c18bd4186f15, empty message
INFO  [alembic.runtime.migration] Running upgrade c18bd4186f15 -> 7fcdcde0761c, Reduce position_json size by remove extra space and component id prefix
INFO  [alembic.runtime.migration] Running upgrade 7fcdcde0761c -> 0c5070e96b57, add user attributes table
INFO  [alembic.runtime.migration] Running upgrade 0c5070e96b57 -> 1a1d627ebd8e, position_json
INFO  [alembic.runtime.migration] Running upgrade 1a1d627ebd8e -> 55e910a74826, add_metadata_column_to_annotation_model.py
INFO  [alembic.runtime.migration] Running upgrade 55e910a74826 -> 4ce8df208545, empty message
INFO  [alembic.runtime.migration] Running upgrade 4ce8df208545 -> 46f444d8b9b7, remove_coordinator_from_druid_cluster_model.py
INFO  [alembic.runtime.migration] Running upgrade 46f444d8b9b7 -> a61b40f9f57f, remove allow_run_sync
INFO  [alembic.runtime.migration] Running upgrade a61b40f9f57f -> 6c7537a6004a, models for email reports
INFO  [alembic.runtime.migration] Running upgrade 6c7537a6004a -> 3e1b21cd94a4, change_owner_to_m2m_relation_on_datasources.py
INFO  [alembic.runtime.migration] Running upgrade 6c7537a6004a -> cefabc8f7d38, Increase size of name column in ab_view_menu
INFO  [alembic.runtime.migration] Running upgrade 55e910a74826 -> 0b1f1ab473c0, Add extra column to Query
INFO  [alembic.runtime.migration] Running upgrade 0b1f1ab473c0, cefabc8f7d38, 3e1b21cd94a4 -> de021a1ca60d, empty message
INFO  [alembic.runtime.migration] Running upgrade de021a1ca60d -> fb13d49b72f9, better_filters
INFO  [alembic.runtime.migration] Running upgrade fb13d49b72f9 -> a33a03f16c4a, Add extra column to SavedQuery
INFO  [alembic.runtime.migration] Running upgrade 4451805bbaa1, 1d9e835a84f9 -> c829ff0b37d0, empty message
INFO  [alembic.runtime.migration] Running upgrade c829ff0b37d0 -> 7467e77870e4, remove_aggs
INFO  [alembic.runtime.migration] Running upgrade 7467e77870e4, de021a1ca60d -> fbd55e0f83eb, empty message
INFO  [alembic.runtime.migration] Running upgrade fbd55e0f83eb, fb13d49b72f9 -> 8b70aa3d0f87, empty message
INFO  [alembic.runtime.migration] Running upgrade 8b70aa3d0f87, a33a03f16c4a -> 18dc26817ad2, empty message
INFO  [alembic.runtime.migration] Running upgrade 18dc26817ad2 -> c617da68de7d, form nullable
INFO  [alembic.runtime.migration] Running upgrade c617da68de7d -> c82ee8a39623, Add implicit tags
INFO  [alembic.runtime.migration] Running upgrade 18dc26817ad2 -> e553e78e90c5, add_druid_auth_py.py
INFO  [alembic.runtime.migration] Running upgrade e553e78e90c5, c82ee8a39623 -> 45e7da7cfeba, empty message
INFO  [alembic.runtime.migration] Running upgrade 45e7da7cfeba -> 80aa3f04bc82, Add Parent ids in dashboard layout metadata
INFO  [alembic.runtime.migration] Running upgrade 80aa3f04bc82 -> d94d33dbe938, form strip
INFO  [alembic.runtime.migration] Running upgrade d94d33dbe938 -> 937d04c16b64, update datasources
INFO  [alembic.runtime.migration] Running upgrade 937d04c16b64 -> 7f2635b51f5d, update base columns
INFO  [alembic.runtime.migration] Running upgrade 7f2635b51f5d -> e9df189e5c7e, update base metrics
INFO  [alembic.runtime.migration] Running upgrade e9df189e5c7e -> afc69274c25a, update the sql, select_sql, and executed_sql columns in the
   query table in mysql dbs to be long text columns
INFO  [alembic.runtime.migration] Running upgrade afc69274c25a -> d7c1a0d6f2da, Remove limit used from query model
INFO  [alembic.runtime.migration] Running upgrade d7c1a0d6f2da -> ab8c66efdd01, resample
INFO  [alembic.runtime.migration] Running upgrade ab8c66efdd01 -> b4a38aa87893, deprecate database expression
INFO  [alembic.runtime.migration] Running upgrade b4a38aa87893 -> d6ffdf31bdd4, Add published column to dashboards
INFO  [alembic.runtime.migration] Running upgrade d6ffdf31bdd4 -> 190188938582, Remove duplicated entries in dashboard_slices table and add unique constraint
INFO  [alembic.runtime.migration] Running upgrade 190188938582 -> def97f26fdfb, Add index to tagged_object
INFO  [alembic.runtime.migration] Running upgrade def97f26fdfb -> 11c737c17cc6, deprecate_restricted_metrics
INFO  [alembic.runtime.migration] Running upgrade 11c737c17cc6 -> 258b5280a45e, form strip leading and trailing whitespace
INFO  [alembic.runtime.migration] Running upgrade 258b5280a45e -> 1495eb914ad3, time range
INFO  [alembic.runtime.migration] Running upgrade 1495eb914ad3 -> b6fa807eac07, make_names_non_nullable
INFO  [alembic.runtime.migration] Running upgrade b6fa807eac07 -> cca2f5d568c8, add encrypted_extra to dbs
INFO  [alembic.runtime.migration] Running upgrade cca2f5d568c8 -> c2acd2cf3df2, alter type of dbs encrypted_extra
INFO  [alembic.runtime.migration] Running upgrade c2acd2cf3df2 -> 78ee127d0d1d, reconvert legacy filters into adhoc
INFO  [alembic.runtime.migration] Running upgrade 78ee127d0d1d -> db4b49eb0782, Add tables for SQL Lab state
INFO  [alembic.runtime.migration] Running upgrade db4b49eb0782 -> 5afa9079866a, serialize_schema_permissions.py
INFO  [alembic.runtime.migration] Running upgrade 5afa9079866a -> 89115a40e8ea, Change table schema description to long text
INFO  [alembic.runtime.migration] Running upgrade 89115a40e8ea -> 817e1c9b09d0, add_not_null_to_dbs_sqlalchemy_url
INFO  [alembic.runtime.migration] Running upgrade 817e1c9b09d0 -> e96dbf2cfef0, datasource_cluster_fk
INFO  [alembic.runtime.migration] Running upgrade e96dbf2cfef0 -> 3325d4caccc8, empty message
INFO  [alembic.runtime.migration] Running upgrade 3325d4caccc8 -> 0a6f12f60c73, add_role_level_security
INFO  [alembic.runtime.migration] Running upgrade 0a6f12f60c73 -> 72428d1ea401, Add tmp_schema_name to the query object.
INFO  [alembic.runtime.migration] Running upgrade 72428d1ea401 -> b5998378c225, add certificate to dbs
INFO  [alembic.runtime.migration] Running upgrade b5998378c225 -> f9a30386bd74, cleanup_time_grainularity
INFO  [alembic.runtime.migration] Running upgrade f9a30386bd74 -> 620241d1153f, update time_grain_sqla
INFO  [alembic.runtime.migration] Running upgrade 620241d1153f -> 743a117f0d98, Add slack to the schedule
INFO  [alembic.runtime.migration] Running upgrade 743a117f0d98 -> e557699a813e, add_tables_relation_to_row_level_security
INFO  [alembic.runtime.migration] Running upgrade e557699a813e -> ea396d202291, Add ctas_method to the Query object
INFO  [alembic.runtime.migration] Running upgrade ea396d202291 -> a72cb0ebeb22, deprecate dbs.perm column
INFO  [alembic.runtime.migration] Running upgrade a72cb0ebeb22 -> 2f1d15e8a6af, add_alerts
INFO  [alembic.runtime.migration] Running upgrade 2f1d15e8a6af -> f2672aa8350a, add_slack_to_alerts
INFO  [alembic.runtime.migration] Running upgrade f2672aa8350a -> f120347acb39, Add extra column to tables and metrics
INFO  [alembic.runtime.migration] Running upgrade f2672aa8350a -> 978245563a02, Migrate iframe in dashboard to markdown component
INFO  [alembic.runtime.migration] Running upgrade 978245563a02, f120347acb39 -> f80a3b88324b, empty message
INFO  [alembic.runtime.migration] Running upgrade f80a3b88324b -> 2e5a0ee25ed4, refractor_alerting
INFO  [alembic.runtime.migration] Running upgrade f80a3b88324b -> 175ea3592453, Add cache to datasource lookup table.
INFO  [alembic.runtime.migration] Running upgrade 175ea3592453, 2e5a0ee25ed4 -> ae19b4ee3692, empty message
INFO  [alembic.runtime.migration] Running upgrade ae19b4ee3692 -> e5ef6828ac4e, add rls filter type and grouping key
INFO  [alembic.runtime.migration] Running upgrade e5ef6828ac4e -> 3fbbc6e8d654, fix data access permissions for virtual datasets
INFO  [alembic.runtime.migration] Running upgrade 3fbbc6e8d654 -> 18532d70ab98, Delete table_name unique constraint in mysql
INFO  [alembic.runtime.migration] Running upgrade 18532d70ab98 -> b56500de1855, add_uuid_column_to_import_mixin

Cleaning up slice uuid from dashboard position json.. Done.      

INFO  [alembic.runtime.migration] Running upgrade b56500de1855 -> af30ca79208f, Collapse alerting models into a single one
INFO  [alembic.runtime.migration] Running upgrade af30ca79208f -> 585b0b1a7b18, add exec info to saved queries
INFO  [alembic.runtime.migration] Running upgrade 585b0b1a7b18 -> 96e99fb176a0, add_import_mixing_to_saved_query
INFO  [alembic.runtime.migration] Running upgrade 96e99fb176a0 -> 49b5a32daba5, add report schedules
INFO  [alembic.runtime.migration] Running upgrade 49b5a32daba5 -> a8173232b786, Add path to logs
INFO  [alembic.runtime.migration] Running upgrade a8173232b786 -> e38177dbf641, security converge saved queries
INFO  [alembic.runtime.migration] Running upgrade e38177dbf641 -> 8ee129739cf9, security converge css templates
INFO  [alembic.runtime.migration] Running upgrade 8ee129739cf9 -> 811494c0cc23, Remove path, path_no_int, and ref from logs
INFO  [alembic.runtime.migration] Running upgrade 811494c0cc23 -> 5daced1f0e76, reports add working_timeout column
INFO  [alembic.runtime.migration] Running upgrade 5daced1f0e76 -> 40f16acf1ba7, security converge reports
INFO  [alembic.runtime.migration] Running upgrade 40f16acf1ba7 -> ccb74baaa89b, security converge charts
INFO  [alembic.runtime.migration] Running upgrade ccb74baaa89b -> c25cb2c78727, security converge annotations
INFO  [alembic.runtime.migration] Running upgrade c25cb2c78727 -> 45731db65d9c, security converge datasets
INFO  [alembic.runtime.migration] Running upgrade 45731db65d9c -> 4b84f97828aa, security converge logs
INFO  [alembic.runtime.migration] Running upgrade 4b84f97828aa -> 1f6dca87d1a2, security converge dashboards
INFO  [alembic.runtime.migration] Running upgrade 1f6dca87d1a2 -> 42b4c9e01447, security converge databases
INFO  [alembic.runtime.migration] Running upgrade 42b4c9e01447 -> e37912a26567, security converge queries
INFO  [alembic.runtime.migration] Running upgrade e37912a26567 -> ab104a954a8f, reports alter crontab size
INFO  [alembic.runtime.migration] Running upgrade ab104a954a8f -> 73fd22e742ab, add_dynamic_plugins.py
INFO  [alembic.runtime.migration] Running upgrade 73fd22e742ab -> c878781977c6, alert reports shared uniqueness
INFO  [alembic.runtime.migration] Running upgrade c878781977c6 -> 260bf0649a77, migrate [x dateunit] to [x dateunit ago/later]
INFO  [alembic.runtime.migration] Running upgrade 260bf0649a77 -> e11ccdd12658, add roles relationship to dashboard
INFO  [alembic.runtime.migration] Running upgrade e11ccdd12658 -> 41ce8799acc3, rename pie label type
Updated 0 pie chart labels.
INFO  [alembic.runtime.migration] Running upgrade 41ce8799acc3 -> 070c043f2fdb, add granularity to charts where missing
0 slices altered
INFO  [alembic.runtime.migration] Running upgrade 070c043f2fdb -> c501b7c653a3, add missing uuid column

Cleaning up slice uuid from dashboard position json.. Done.      

INFO  [alembic.runtime.migration] Running upgrade c501b7c653a3 -> 1412ec1e5a7b, legacy force directed to echart
INFO  [alembic.runtime.migration] Running upgrade 1412ec1e5a7b -> 67da9ef1ef9c, add hide_left_bar to tabstate
INFO  [alembic.runtime.migration] Running upgrade 67da9ef1ef9c -> 989bbe479899, rename_filter_configuration_in_dashboard_metadata.py
Updated 0 native filter configurations.
INFO  [alembic.runtime.migration] Running upgrade 989bbe479899 -> 301362411006, add_execution_id_to_report_execution_log_model.py
INFO  [alembic.runtime.migration] Running upgrade 301362411006 -> 134cea61c5e7, remove dataset health check message
INFO  [alembic.runtime.migration] Running upgrade 134cea61c5e7 -> 085f06488938, Country map use lowercase country name
INFO  [alembic.runtime.migration] Running upgrade 085f06488938 -> fc3a3a8ff221, migrate filter sets to new format
Updated 0 filter sets with 0 filters.
INFO  [alembic.runtime.migration] Running upgrade fc3a3a8ff221 -> 19e978e1b9c3, add_report_format_to_report_schedule_model.py
INFO  [alembic.runtime.migration] Running upgrade 19e978e1b9c3 -> d416d0d715cc, add_limiting_factor_column_to_query_model.py
INFO  [alembic.runtime.migration] Running upgrade d416d0d715cc -> f1410ed7ec95, migrate native filters to new schema
Upgraded 0 filters and 0 filter sets.
INFO  [alembic.runtime.migration] Running upgrade f1410ed7ec95 -> 453530256cea, add_save_form_column_to_db_model
INFO  [alembic.runtime.migration] Running upgrade 453530256cea -> 3317e9248280, add_creation_method_to_reports_model
INFO  [alembic.runtime.migration] Running upgrade 3317e9248280 -> 030c840e3a1c, Add query context to slices
INFO  [alembic.runtime.migration] Running upgrade 030c840e3a1c -> ae1ed299413b, add_timezone_to_report_schedule
INFO  [alembic.runtime.migration] Running upgrade ae1ed299413b -> 31b2a1039d4a, drop tables constraint
INFO  [alembic.runtime.migration] Running upgrade 31b2a1039d4a -> e323605f370a, fix schemas_allowed_for_csv_upload
INFO  [alembic.runtime.migration] Running upgrade e323605f370a -> 143b6f2815da, migrate pivot table v2 heatmaps to new format
Upgraded 0 slices.
INFO  [alembic.runtime.migration] Running upgrade 143b6f2815da -> f6196627326f, update chart permissions
INFO  [alembic.runtime.migration] Running upgrade f6196627326f -> 6d20ba9ecb33, add_last_saved_at_to_slice_model
INFO  [alembic.runtime.migration] Running upgrade 6d20ba9ecb33 -> 07071313dd52, change_fetch_values_predicate_to_text
INFO  [alembic.runtime.migration] Running upgrade 07071313dd52 -> 021b81fe4fbb, Add type to native filter configuration
INFO  [alembic] [AddTypeToNativeFilter] Starting upgrade
INFO  [alembic] [AddTypeToNativeFilter] Done!
INFO  [alembic.runtime.migration] Running upgrade 021b81fe4fbb -> 181091c0ef16, add_extra_column_to_columns_model
INFO  [alembic.runtime.migration] Running upgrade 181091c0ef16 -> 3ebe0993c770, add filter set model
INFO  [alembic.runtime.migration] Running upgrade 3ebe0993c770 -> 60dc453f4e2e, migrate timeseries_limit_metric to legacy_order_by in pivot_table_v2
INFO  [alembic.runtime.migration] Running upgrade 60dc453f4e2e -> 32646df09c64, update time grain SQLA
INFO  [alembic.runtime.migration] Running upgrade 32646df09c64 -> f9847149153d, add_certifications_columns_to_slice
INFO  [alembic.runtime.migration] Running upgrade f9847149153d -> aea15018d53b, add_certifications_columns_to_dashboard
INFO  [alembic.runtime.migration] Running upgrade aea15018d53b -> b92d69a6643c, rename_csv_to_file
INFO  [alembic.runtime.migration] Running upgrade b92d69a6643c -> 0ca9e5f1dacd, rename to schemas_allowed_for_file_upload in dbs.extra
INFO  [alembic.runtime.migration] Running upgrade 0ca9e5f1dacd -> abe27eaf93db, add_extra_config_column_to_alerts
INFO  [alembic.runtime.migration] Running upgrade abe27eaf93db -> 3ba29ecbaac5, Change datatype of type in BaseColumn
INFO  [alembic.runtime.migration] Running upgrade 3ba29ecbaac5 -> fe23025b9441, rename_big_viz_total_form_data_fields
INFO  [alembic.runtime.migration] Running upgrade fe23025b9441 -> 31bb738bd1d2, move_pivot_table_v2_legacy_order_by_to_timeseries_limit_metric
INFO  [alembic.runtime.migration] Running upgrade 31bb738bd1d2 -> bb38f40aa3ff, Add force_screenshot to alerts/reports
INFO  [alembic.runtime.migration] Running upgrade bb38f40aa3ff -> c53bae8f08dd, add_saved_query_foreign_key_to_tab_state
INFO  [alembic.runtime.migration] Running upgrade c53bae8f08dd -> 5fd49410a97a, Add columns for external management
INFO  [alembic.runtime.migration] Running upgrade 5fd49410a97a -> 5afbb1a5849b, add_embedded_dahshoard_table
INFO  [alembic.runtime.migration] Running upgrade 5afbb1a5849b -> b8d3a24d9131, New dataset models
INFO  [alembic.runtime.migration] Running upgrade b8d3a24d9131 -> b5a422d8e252, fix query and saved_query null schema
INFO  [alembic.runtime.migration] Running upgrade b5a422d8e252 -> ab9a9d86e695, deprecate time_range_endpoints
INFO  [alembic.runtime.migration] Running upgrade ab9a9d86e695 -> 7293b0ca7944, change_adhoc_filter_b_from_none_to_empty_array
INFO  [alembic.runtime.migration] Running upgrade 7293b0ca7944 -> 6766938c6065, add key-value store
INFO  [alembic.runtime.migration] Running upgrade 6766938c6065 -> 58df9d617f14, add_on_saved_query_delete_tab_state_null_constraint"
INFO  [alembic.runtime.migration] Running upgrade 58df9d617f14 -> 2ed890b36b94, rm_time_range_endpoints_from_qc
INFO  [alembic.runtime.migration] Running upgrade 2ed890b36b94 -> b0d0249074e4, deprecate time_range_endpoints v2
INFO  [alembic.runtime.migration] Running upgrade 2ed890b36b94 -> 8b841273bec3, sql_lab_models_database_constraint_updates
/opt/module/miniconda3/envs/superset/lib/python3.8/site-packages/sqlalchemy/dialects/sqlite/base.py:2011: SAWarning: WARNING: SQL-parsed foreign key constraint '('database_id', 'dbs', 'id')' could not be located in PRAGMA foreign_keys for table tab_state
  util.warn(
/opt/module/miniconda3/envs/superset/lib/python3.8/site-packages/sqlalchemy/dialects/sqlite/base.py:2011: SAWarning: WARNING: SQL-parsed foreign key constraint '('latest_query_id', 'query', 'client_id')' could not be located in PRAGMA foreign_keys for table tab_state
  util.warn(
INFO  [alembic.runtime.migration] Running upgrade 8b841273bec3, b0d0249074e4 -> 9d8a8d575284, merge point
INFO  [alembic.runtime.migration] Running upgrade 9d8a8d575284 -> cecc6bf46990, rm_time_range_endpoints_2
INFO  [alembic.runtime.migration] Running upgrade cecc6bf46990 -> ad07e4fdbaba, rm_time_range_endpoints_from_qc_3
slices updated with no time_range_endpoints: 0
INFO  [alembic.runtime.migration] Running upgrade ad07e4fdbaba -> a9422eeaae74, new_dataset_models_take_2
>> Assign new UUIDs to tables...
>> Drop intermediate columns...
INFO  [alembic.runtime.migration] Running upgrade a9422eeaae74 -> cbe71abde154, fix report schedule and execution log
INFO  [alembic.runtime.migration] Running upgrade cbe71abde154 -> 6f139c533bea, adding_advanced_data_type.py
Revision ID: 6f139c533bea
Revises: cbe71abde154
Create Date: 2021-05-27 16:10:59.567684
INFO  [alembic.runtime.migration] Running upgrade 6f139c533bea -> e786798587de, Delete None permissions
INFO  [alembic.runtime.migration] Running upgrade e786798587de -> e09b4ae78457, Resize key_value blob