Spark 读取多套环境的配置文件数据

最新推荐文章于 2024-08-19 09:48:42 发布

江畔独步

最新推荐文章于 2024-08-19 09:48:42 发布

阅读量709

点赞数 1

分类专栏： Spark

本文链接：https://blog.csdn.net/liuwei0376/article/details/118611309

版权

JVM参数配置管理 Maven Profile 环境隔离应用配置

关键词由CSDN通过智能技术生成

Spark 专栏收录该内容

12 篇文章 1 订阅

订阅专栏

思路一

通过jvm 参数传入环境信息, 并根据环境信息, 拼接出适配该环境的配置文件.
.
-Dactive_env=test -DXss=100m

1.1 工具类清单

import com.typesafe.config.{Config, ConfigFactory}

/**
 * 项目环境对应的配置信息获取工具类
 */
object ConfigTools {

  /**
   * 获取用户自定义的jvm环境配置参数
   *
   * test环境: -Dactive_env=test
   * prod环境: -Dactive_env=prod
   */
  def getConfigFactory(): Config = {
    val activeEnv = System.getProperty("active_env")

    ConfigFactory.load("application-" +activeEnv +".conf")
  }

}

1.2 测试类清单

import com.david.tools.ConfigTools

object TestEnv {

  private val config = ConfigTools.getConfigFactory()

  def main(args: Array[String]): Unit = {

    val url = config.getString("db.default.url")
    val user = config.getString("db.default.user")
    val pwd = config.getString("db.default.password")
    val driver = config.getString("db.default.driver")

    println("url = " + url)
    println("user = " + user)
    println("pwd = " + pwd)
    println("driver = " + driver)
  }

}

1.3 参数解析

同时需要配置如下参数:

-Dactive_env=test 获取test环境的标识test
-DXms=100m 防止堆栈溢出

项目结构:
在这里插入图片描述

思路二

通过maven插件, 根据提取激活的profile 环境信息, 将模板配置文件 application.conf 信息动态替换为包含此环境的配置文件.

2.1 添加过滤筛选器:

maven的pom.xml中,在build配置项中添加如下信息:

        <resources>
            <resource>
                <directory>src/main/resources/</directory>
                <filtering>true</filtering>
                <includes>
                    <include>**/*.conf</include>
                </includes>
            </resource>
        </resources>

2.2 设置激活的环境:

maven的pom.xml中,在根层级下添加如下信息:

<profiles>
        <profile>
            <id>test</id>
            <properties>
                <env>test</env>
            </properties>
            <activation>
                <activeByDefault>true</activeByDefault>
            </activation>
            <build>
                <filters>
                    <filter>src/main/resources/application-test.conf</filter>
                </filters>
            </build>
        </profile>
        <profile>
            <id>prod</id>
            <properties>
                <env>prod</env>
            </properties>
            <activation>
                <activeByDefault>false</activeByDefault>
            </activation>
            <build>
                <filters>
                    <filter>src/main/resources/application-prod.conf</filter>
                </filters>
            </build>
        </profile>
    </profiles>

整体maven 配置文件 pom.xml清单:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>david_offlinedata</groupId>
    <artifactId>david_offlinedata</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <scala.version>2.11.8</scala.version>
        <spark.version>2.2.0</spark.version>
        <encoding>UTF-8</encoding>
    </properties>

    <profiles>
        <profile>
            <id>test</id>
            <properties>
                <env>test</env>
            </properties>
            <activation>
                <activeByDefault>true</activeByDefault>
            </activation>
            <build>
                <filters>
                    <filter>src/main/resources/application-test.conf</filter>
                </filters>
            </build>
        </profile>
        <profile>
            <id>prod</id>
            <properties>
                <env>prod</env>
            </properties>
            <activation>
                <activeByDefault>false</activeByDefault>
            </activation>
            <build>
                <filters>
                    <filter>src/main/resources/application-prod.conf</filter>
                </filters>
            </build>
        </profile>
    </profiles>

    <dependencies>
        <!--sparksql需要导入sparksql的依赖jar包-->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <!--mysql依赖 pom-->
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>5.1.32</version>
        </dependency>

        <!-- 导入spark的依赖 -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <dependency>
            <groupId>com.typesafe</groupId>
            <artifactId>config</artifactId>
            <version>1.0.1</version>
        </dependency>

        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.36</version>
        </dependency>

        <dependency>
            <groupId>redis.clients</groupId>
            <artifactId>jedis</artifactId>
            <version>2.9.0</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-hive_2.11</artifactId>
            <version>2.1.0</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-service</artifactId>
            <version>1.2.1</version>
        </dependency>

        <dependency>
            <groupId>net.sourceforge.jexcelapi</groupId>
            <artifactId>jxl</artifactId>
            <version>2.6.10</version>
        </dependency>

        <dependency>
            <groupId>org.apache.poi</groupId>
            <artifactId>poi</artifactId>
            <version>3.17</version>
        </dependency>
        <dependency>
            <groupId>org.apache.poi</groupId>
            <artifactId>poi-ooxml-schemas</artifactId>
            <version>3.17</version>
        </dependency>

    </dependencies>
    <build>
        <sourceDirectory>src/main/scala</sourceDirectory>

        <resources>
            <resource>
                <directory>src/main/resources/</directory>
                <filtering>true</filtering>
                <includes>
                    <include>**/*.conf</include>
                </includes>
            </resource>
        </resources>

        <plugins>
            <plugin>
                <!-- see http://davidb.github.com/scala-maven-plugin -->
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <version>3.1.3</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>compile</goal>
                            <goal>testCompile</goal>
                        </goals>
                        <configuration>
                            <args>
                                <arg>-dependencyfile</arg>
                                <arg>${project.build.directory}/.scala_dependencies</arg>
                            </args>
                            <jvmArgs>-Xss10m</jvmArgs>
                        </configuration>

                    </execution>

                </executions>


            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.13</version>
                <configuration>
                    <useFile>false</useFile>
                    <disableXmlReport>true</disableXmlReport>
                    <!-- If you have classpath issue like NoDefClassError,... -->
                    <!-- useManifestOnlyJar>false</useManifestOnlyJar -->
                    <includes>
                        <include>**/*Test.*</include>
                        <include>**/*Suite.*</include>
                    </includes>
                </configuration>
            </plugin>

            <!-- 打包依赖包到jar中 -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>2.4.1</version>
                <configuration>
                    <!-- get all project dependencies -->
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                    <!-- MainClass in mainfest make a executable jar -->
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id>
                        <!-- bind to the packaging phase -->
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>


    </build>


</project>

2.3 模板配置项

resources目录中application.conf的配置项:

#jdbc
db.default.driver=${db.default.driver}
db.default.url=${db.default.url}
db.default.user=${db.default.user}
db.default.password=${db.default.password}

2.4 测试类

jvm在启动时, 会自动动态将该环境变量(如test)下的配置文件(如application-test.conf)中的 value 值, 替换掉.
application.conf 模板中的value值.

import com.typesafe.config.ConfigFactory

object JustTest {

  private val config = ConfigFactory.load()

  def main(args: Array[String]): Unit = {

    println("----------- from offline -----------")

    val url = config.getString("db.default.url")
    val user = config.getString("db.default.user")
    val pwd = config.getString("db.default.password")
    val driver = config.getString("db.default.driver")
    //如果该配置项在配置文件中不存在,将会报错(这里是从application.conf文件中读取数据)
    //val k2 = config.getString("k2") 

    println("url = " + url)
    println("user = " + user)
    println("pwd = " + pwd)
    println("driver = " + driver)
    //println("k2 = " + k2)
  }

}