一键安装Redash并连接Impala

问题描述

1、 缺陷描述（场景、步骤、预期、实际）

场景：需求描述

使用redash连接impala。

操作步骤：详细描述发现bug的操作步骤

在redash中配置impala并进行查询测试。

预期结果：详细写明预期结果

连接测试成功。

实际结果：详细写明实际结果

• 连接测试报错：expecting list of size 2 for struct args；
• 查询筛选词带中文时报错：Error running query: ‘ascii’ codec can’t encode characters in position 290-291: ordinal not in range(128)

2、 解决方案

总体思路

1. 定位问题可能是由于thrift和pyhive版本造成的，通过python脚本进行测试，确认解决问题的思路正确；
2. 通过Dockerfile对镜像进行修改，生成新镜像；
3. 通过docker-compose + 自定义镜像，搭建redash服务；

处理细节

expecting list of size 2 for struct args

• 搜索该错误，发现原因可能是thrift及pyhive版本问题导致；
• 通过Dockerfile更改redash镜像相关配置

Error running query: ‘ascii’ codec can’t encode characters in position 290-291: ordinal not in range(128)

• 搜索该错误，原因肯定是编码问题；
• 通过docker-compose日志，定位到问题源码为/app/redash/query_runner/impala_ds.py
• 修改源码，添加指定编码方式为utf-8，不优雅但有效

过程

直接运行setup.sh
setup.sh

#!/usr/bin/env bash
# This script sets up dockerized Redash on CentOS 7

set -u

REDASH_BASE_PATH=/opt/redash
COMPOSE_PATH=/usr/local/bin/docker-compose
COMPOSE_VERSION="1.25.5"

distro=centos7

install_docker(){
    # Install Docker
    #sudo yum -y update
    sudo yum -y install https://mirrors.aliyun.com/epel/epel-release-latest-7.noarch.rpm
    sudo curl -L -o /etc/yum.repos.d/docker-ce.repo https://download.docker.com/linux/centos/docker-ce.repo
    sudo chown root:root /etc/yum.repos.d/docker-ce.repo
    sudo restorecon -Fv /etc/yum.repos.d/docker-ce.repo
    sudo yum -y install docker-ce pwgen yajl unzip
    sudo systemctl start docker

    # Install Docker Compose
    #sudo curl -L https://github.com/docker/compose/releases/download/${COMPOSE_VERSION}/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose
    # 改成国内镜像
    sudo curl -L https://get.daocloud.io/docker/compose/releases/download/${COMPOSE_VERSION}/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose
    sudo chmod +x ${COMPOSE_PATH}

    sudo restorecon -Fv ${COMPOSE_PATH}

    # Allow current user to run Docker commands
    sudo usermod -aG docker $USER
}

create_directories() {
    if [[ ! -e $REDASH_BASE_PATH ]]; then
        sudo mkdir -p $REDASH_BASE_PATH
        sudo chown $USER:$USER $REDASH_BASE_PATH
    fi

    if [[ ! -e $REDASH_BASE_PATH/postgres-data ]]; then
        mkdir $REDASH_BASE_PATH/postgres-data
    fi
}

create_config() {
    if [[ -e $REDASH_BASE_PATH/env ]]; then
        rm $REDASH_BASE_PATH/env
        touch $REDASH_BASE_PATH/env
    fi

    COOKIE_SECRET=$(pwgen -1s 32)
    SECRET_KEY=$(pwgen -1s 32)
    POSTGRES_PASSWORD=$(pwgen -1s 32)
    REDASH_DATABASE_URL="postgresql://postgres:${POSTGRES_PASSWORD}@postgres/postgres"

    echo "PYTHONUNBUFFERED=0" >> $REDASH_BASE_PATH/env
    echo "REDASH_LOG_LEVEL=INFO" >> $REDASH_BASE_PATH/env
    echo "REDASH_REDIS_URL=redis://redis:6379/0" >> $REDASH_BASE_PATH/env
    echo "POSTGRES_PASSWORD=$POSTGRES_PASSWORD" >> $REDASH_BASE_PATH/env
    echo "REDASH_COOKIE_SECRET=$COOKIE_SECRET" >> $REDASH_BASE_PATH/env
    echo "REDASH_SECRET_KEY=$SECRET_KEY" >> $REDASH_BASE_PATH/env
    echo "REDASH_DATABASE_URL=$REDASH_DATABASE_URL?client_encoding=utf-8" >> $REDASH_BASE_PATH/env
}

setup_compose() {
    REQUESTED_CHANNEL=stable
    LATEST_VERSION=`curl -s "https://version.redash.io/api/releases?channel=$REQUESTED_CHANNEL"  | json_reformat  | grep "docker_image" | head -n 1 | awk 'BEGIN{FS=":"}{print $3}' | awk 'BEGIN{FS="\""}{print $1}'`
    cd $REDASH_BASE_PATH

    if [ -n "$0" ] ; then
      docker build -t redash-ctm/redash-ctm:8.0.0.b32245 .
    else
      
      GIT_BRANCH="${REDASH_BRANCH:-master}" # Default branch/version to master if not specified in REDASH_BRANCH env var
      wget https://raw.githubusercontent.com/getredash/setup/${GIT_BRANCH}/data/docker-compose.yml
      sed -ri "s/image: redash\/redash:([A-Za-z0-9.-]*)/image: redash\/redash:$LATEST_VERSION/" docker-compose.yml
    fi

    echo "export COMPOSE_PROJECT_NAME=redash" >> ~/.profile
    echo "export COMPOSE_PROJECT_NAME=redash" >> ~/.bashrc
    echo "export COMPOSE_FILE=/opt/redash/docker-compose.yml" >> ~/.profile
    echo "export COMPOSE_FILE=/opt/redash/docker-compose.yml" >> ~/.bashrc
    export COMPOSE_PROJECT_NAME=redash
    export COMPOSE_FILE=/opt/redash/docker-compose.yml
    /usr/local/bin/docker-compose run --rm server create_db
    /usr/local/bin/docker-compose up -d
}

install_docker
create_directories
create_config
setup_compose

# Make the new docker user group effective, so the user doesn't need to re-login
exec sg docker newgrp `id -gn`

docker-compose.yml

version: "2"
x-redash-service: &redash-service
  image: redash-ctm/redash-ctm:8.0.0.b32245
  depends_on:
    - postgres
    - redis
  env_file: /opt/redash/env
  restart: always
services:
  server:
    <<: *redash-service
    command: server
    ports:
      - "5000:5000"
    environment:
      REDASH_WEB_WORKERS: 4
  scheduler:
    <<: *redash-service
    command: scheduler
    environment:
      QUEUES: "celery"
      WORKERS_COUNT: 1
  scheduled_worker:
    <<: *redash-service
    command: worker
    environment:
      QUEUES: "scheduled_queries,schemas"
      WORKERS_COUNT: 1
  adhoc_worker:
    <<: *redash-service
    command: worker
    environment:
      QUEUES: "queries"
      WORKERS_COUNT: 2
  redis:
    image: redis:5.0-alpine
    restart: always
  postgres:
    image: postgres:9.6-alpine
    env_file: /opt/redash/env
    volumes:
      - /opt/redash/postgres-data:/var/lib/postgresql/data
    restart: always
  nginx:
    image: redash/nginx:latest
    ports:
      - "80:80"
    depends_on:
      - server
    links:
      - server:redash
    restart: always

Dockerfile

FROM redash/redash:8.0.0.b32245

USER root

RUN pip uninstall thrift -y
RUN pip uninstall pyhive -y
RUN ["pip", "install", "thrift==0.9.3", "-i", "https://pypi.tuna.tsinghua.edu.cn/simple/"]
RUN ["pip", "install", "pyhive==0.2.1", "-i", "https://pypi.tuna.tsinghua.edu.cn/simple/"]

RUN sed -i 's/postgresql:\/\/\/postgres/postgresql:\/\/\/postgres?client_encoding=utf-8/g' /app/redash/settings/__init__.py
RUN sed -i -e '/run_query(self, query, user):/a\        query = query.encode("utf-8")' /app/redash/query_runner/impala_ds.py
RUN echo "alias ll='ls -l --color=auto'" >> ~/.bashrc

3、参考

一键安装Redash（支持Oracle）
python 连接 hive 遇到的问题
Python ASCII codec error for queries with characters not in ASCII​