Flink-1.5.0-源码分析系列2：buildProgram

最新推荐文章于 2023-10-11 10:31:40 发布

weixin_34115824

最新推荐文章于 2023-10-11 10:31:40 发布

阅读量411

点赞数

文章标签：大数据 python java

原文链接：https://my.oschina.net/qiangzigege/blog/2120205

版权

2019独角兽企业重金招聘Python工程师标准>>>

下面我们来看看buildProgram是怎么实现的

stop in org.apache.flink.client.cli.CliFrontend.buildProgram

里面有1个细节，就是如果未指定entrypointClassName,就会自己找

Step completed: "thread=main", org.apache.flink.client.program.PackagedProgram.<init>(), line=190 bci=85
190    			entryPointClassName = getEntryPointClassNameFromJar(jarFileUrl);

main[1] step

我们看怎么找！

-----------------

		// if no entryPointClassName name was given, we try and look one up through the manifest
		if (entryPointClassName == null) {
			entryPointClassName = getEntryPointClassNameFromJar(jarFileUrl);
		}

进去看实现

获取manifest文件

// Read from jar manifest
			try {
				manifest = jar.getManifest();
			} catch (IOException ioex) {
				throw new ProgramInvocationException("The Manifest in the jar file could not be accessed '"
					+ jarFile.getPath() + "'. " + ioex.getMessage(), ioex);
			}

Attributes attributes = manifest.getMainAttributes();

			// check for a "program-class" entry first
			className = attributes.getValue(PackagedProgram.MANIFEST_ATTRIBUTE_ASSEMBLER_CLASS);
			if (className != null) {
				return className;
			}

			// check for a main class
			className = attributes.getValue(PackagedProgram.MANIFEST_ATTRIBUTE_MAIN_CLASS);
			if (className != null) {
				return className;
			} else {
				throw new ProgramInvocationException("Neither a '" + MANIFEST_ATTRIBUTE_MAIN_CLASS + "', nor a '" +
						MANIFEST_ATTRIBUTE_ASSEMBLER_CLASS + "' entry was found in the jar file.");
			}

先从manifest文件中看有没有，经过2次查找，有了

Step completed: "thread=main", org.apache.flink.client.program.PackagedProgram.getEntryPointClassNameFromJar(), line=592 bci=276
592    			if (className != null) {

main[1] print className
 className = "org.apache.flink.streaming.examples.kafka.Kafka010Example"

Manifest-Version: 1.0
Implementation-Title: flink-examples-streaming
Implementation-Version: 1.5.0
Archiver-Version: Plexus Archiver
Built-By: admin
Specification-Vendor: The Apache Software Foundation
Specification-Title: flink-examples-streaming
Implementation-Vendor-Id: org.apache.flink
Implementation-Vendor: The Apache Software Foundation
Main-Class: org.apache.flink.streaming.examples.kafka.Kafka010Example
Created-By: Apache Maven 3.0.5
Build-Jdk: 1.8.0_171
Specification-Version: 1.5.0

好，找到了！

然后会提取依赖包，如果这个大jar包里有的话，原理就是

public static List<File> extractContainedLibraries(URL jarFile) throws ProgramInvocationException {
	    //看到这里了//看到这里了//看到这里了
		Random rnd = new Random();
		//看到这里了
		JarFile jar = null;
		try {
			jar = new JarFile(new File(jarFile.toURI()));
			final List<JarEntry> containedJarFileEntries = new ArrayList<JarEntry>();
			//看到这里了
			Enumeration<JarEntry> entries = jar.entries();
			while (entries.hasMoreElements()) {
				JarEntry entry = entries.nextElement();
				String name = entry.getName();
				//看到这里了
				if (name.length() > 8 && name.startsWith("lib/") && name.endsWith(".jar")) {
					containedJarFileEntries.add(entry);
				}//看到这里了
			}

			if (containedJarFileEntries.isEmpty()) {
				return Collections.emptyList();
			}//看到这里了
			else {
				// go over all contained jar files
				final List<File> extractedTempLibraries = new ArrayList<File>(containedJarFileEntries.size());
				final byte[] buffer = new byte[4096];

				boolean incomplete = true;

				try {
					for (int i = 0; i < containedJarFileEntries.size(); i++) {
						final JarEntry entry = containedJarFileEntries.get(i);
						String name = entry.getName();
						name = name.replace(File.separatorChar, '_');

						File tempFile;
						try {
							tempFile = File.createTempFile(rnd.nextInt(Integer.MAX_VALUE) + "_", name);
							tempFile.deleteOnExit();
						}
						catch (IOException e) {
							throw new ProgramInvocationException(
								"An I/O error occurred while creating temporary file to extract nested library '" +
										entry.getName() + "'.", e);
						}

						extractedTempLibraries.add(tempFile);

						// copy the temp file contents to a temporary File
						OutputStream out = null;
						InputStream in = null;
						try {

							out = new FileOutputStream(tempFile);
							in = new BufferedInputStream(jar.getInputStream(entry));

							int numRead = 0;
							while ((numRead = in.read(buffer)) != -1) {
								out.write(buffer, 0, numRead);
							}
						}
						catch (IOException e) {
							throw new ProgramInvocationException("An I/O error occurred while extracting nested library '"
									+ entry.getName() + "' to temporary file '" + tempFile.getAbsolutePath() + "'.");
						}
						finally {
							if (out != null) {
								out.close();
							}
							if (in != null) {
								in.close();
							}
						}
					}

					incomplete = false;
				}
				finally {
					if (incomplete) {
						deleteExtractedLibraries(extractedTempLibraries);
					}
				}

				return extractedTempLibraries;
			}
		}
		catch (Throwable t) {
			throw new ProgramInvocationException("Unknown I/O error while extracting contained jar files.", t);
		}
		finally {
			if (jar != null) {
				try {
					jar.close();
				} catch (Throwable t) {}
			}
		}
	}

然后需要构建大一统的buildUserCodeClassLoader

原理就是

	public static ClassLoader buildUserCodeClassLoader(List<URL> jars, List<URL> classpaths, ClassLoader parent) {
		URL[] urls = new URL[jars.size() + classpaths.size()];//看到这里了
		for (int i = 0; i < jars.size(); i++) {
			urls[i] = jars.get(i);//填充jar包里的所有jar，包含主jar包本身及内部的提取jar包
		}
		for (int i = 0; i < classpaths.size(); i++) {
			urls[i + jars.size()] = classpaths.get(i);//包含参数指定的jar包
		}
		return FlinkUserCodeClassLoaders.parentFirst(urls, parent);
	}

注意这里有一个return FlinkUserCodeClassLoaders.parentFirst(urls, parent);，也就是本身的classLoader优先
===

	public static URLClassLoader parentFirst(URL[] urls, ClassLoader parent) {
		return new ParentFirstClassLoader(urls, parent);
	}

是不是超级简单，后面我们也可以用这种方式来使用了，很棒！！！

然后通过这个新的classLoader来加载主类

注意加载主类的细节，加载完后要还原的

private static Class<?> loadMainClass(String className, ClassLoader cl) throws ProgramInvocationException {
		ClassLoader contextCl = null;
		try {//看到这里了//看到这里了//看到这里了//看到这里了//看到这里了//看到这里了//看到这里了//看到这里了
			contextCl = Thread.currentThread().getContextClassLoader();
			Thread.currentThread().setContextClassLoader(cl);
			return Class.forName(className, false, cl);
		}
		catch (ClassNotFoundException e) {
			throw new ProgramInvocationException("The program's entry point class '" + className
				+ "' was not found in the jar file.", e);
		}
		catch (ExceptionInInitializerError e) {
			throw new ProgramInvocationException("The program's entry point class '" + className
				+ "' threw an error during initialization.", e);
		}
		catch (LinkageError e) {
			throw new ProgramInvocationException("The program's entry point class '" + className
				+ "' could not be loaded due to a linkage failure.", e);
		}
		catch (Throwable t) {
			throw new ProgramInvocationException("The program's entry point class '" + className
				+ "' caused an exception during initialization: " + t.getMessage(), t);
		} finally {//需要复原
			if (contextCl != null) {
				Thread.currentThread().setContextClassLoader(contextCl);
			}
		}
	}

好，然后判断这个主类是不是if (Program.class.isAssignableFrom(this.mainClass)) {

当前我们的卡夫卡例子不是这个类的实现类，所以走另外一个分支

然后判断有没有这个mainClass有main方法

} else if (hasMainMethod(mainClass)) {
			this.program = null;
		} else {
			throw new ProgramInvocationException("The given program class neither has a main(String[]) method, nor does it implement the " +
					Program.class.getName() + " interface.");
		}

private static boolean hasMainMethod(Class<?> entryClass) {
		Method mainMethod;
		try {//直接提取main方法
			mainMethod = entryClass.getMethod("main", String[].class);
		} catch (NoSuchMethodException e) {
			return false;
		}
		catch (Throwable t) {
			throw new RuntimeException("Could not look up the main(String[]) method from the class " +
					entryClass.getName() + ": " + t.getMessage(), t);
		}
		//确保是static public方法
		return Modifier.isStatic(mainMethod.getModifiers()) && Modifier.isPublic(mainMethod.getModifiers());
	}

转载于:https://my.oschina.net/qiangzigege/blog/2120205