zip是一种归档文件格式,zip可以把若干文件和目录下的文件进行归档,这些归档的文件可以压缩也可以不压缩,并且压缩算法也是可以选择的,目前zip最经常使用的是deflate算法,因为zip中包含若干归档的文件,每个文件都有一个元数据区描述该文件,而这个元数据区域是不能被压缩的,因此如果zip中存在大量文件时,直接存储zip格式的文件并不是很有效率,可以对一个zip格式的文件,使用gzip进行压缩,gzip是http请求中默认的压缩格式,gzip通常也是使用deflate算法,但是gzip有相对小的元数据区域.关于zip和gzip的格式,请参选相关规范.
deflate算法在android系统(或者java系统)中有API可以直接调用,其实现由底层的zlib库负责,java相关的封装为java的类Inflater和Deflater实现.在c/c++环境下,可以直接使用zlib的API,只需包含zlib.h即可.通常更高的压缩比,更快的时间执行效率的算法是更优秀的压缩算法,在内存受限的设备上,内存占用也可能是考虑因素.根据压缩算法固有的属性,解压缩通常比压缩快上好几倍,所以在一个解压缩运行频率高,而压缩运行频率低的上下文环境中,取决于解压缩的时间消耗, 例如android系统中安装APK的解压缩过程.通常而言deflate算法的压缩比是可以的,但是压缩和解压缩的效率不是很高,facebook设计的新的压缩算法zstd,想比较deflate算法,压缩比略有提高,压缩和解压缩效率提高显著,因此较deflate算法更好一些.其github工程链接为https://github.com/facebook/zstd/wiki .
zip格式是个典型的分段结构,包括三部分构成,前面是文件内容段,后面两段都是元数据:
通常对于zip中包含的每个文件,视为一个entry.对于zip格式,其访问机制可以用如下伪码表示:
search the start of End of central directory
start=file.length-minLenEOCD
stop=file.length-maxLenEOCD
for(;start>=stop;start--)
seek(start)
int curr=read()
if (curr==signarure(EOCD)) break
seek(offset of central directory)
load central directory
jump to file entry to access
这段伪码的思想就是先找到EOCD在文件中的偏移,EOCD中记录了central directory在文件中的偏移,从而定位到central directory起始位置,central directory包含了各个file entry的元数据和在文件中的偏移.从而可以定位到指定file entry的位置,每个file entry的起始部分是local file header,描述该文件的元数据.其中第一步中如何定位EOCD在文件中的偏移,其细节是这样的:对于EOCD段,其最后是一个comments字段,该字段为一个short,通常zip文件不会填充comments字段,伪码中minLenEOCD就是comments未填充时(长度为0字节)的EOCD长度,maxLenEOCD为comments为最大填充时(长度为65536字节)的EOCD的长度,所以在start和stop这两个文件偏移区间进行扫描,如果匹配到EOCD的签名(四字节值0x06054B50),则定位到了EOCD在文件中的起始位置.
有的zip实现会把central directory部分装载到内存,这样对file entry的定位就比较快,但是如果zip中file entry比较多,可能对内存消耗有一定影响.
android系统中的jar包和APK文件都是zip格式,jar和APK根据具体的使用场景,增加了一些特殊含义的zip entry, 比如jar和APK都支持签名,增加了META-INF目录下的文件.jar可以创建成executable jar,就需要在MF文件中配置Main-Class,并且需要有签名格式正确的main方法(public static void main(String[] args)).Executable jar运行时,将启动虚拟机,装载jar中的类文件,执行entry point类的main方法,执行完毕后,退出虚拟机进程.如果是在java环境中, 启动java虚拟机, 如果是在android环境中,则启动android虚拟机.java环境中的executable jar,比如proguard.jar(代码混淆),apktool.jar(反编译APK)等.android环境中的executable jar,比如系统自带的pm.jar等.在adb shell环境中,运行pm命令时,其执行流程为:首先会执行pm脚本文件(/system/bin),该脚本文件配置好环境变量以后,会启动app_process可执行文件(/system/bin),该程序启动android虚拟机,运行Pm类中的main方法,解析命令行参数,执行相应逻辑,执行完毕退出虚拟机进程.
Android系统中存在不同的逻辑流程要解析zip文件格式,包括签名验证和安装过程,签名验证使用java实现,安装过程使用c实现,由于二者实现的差异以及不严谨,曾导致过若干个安全漏洞,可以绕过签名验证执行任意代码.Masterkey漏洞是在原有的classes.dex前面放置同名的含有恶意代码的文件,这样会通过签名验证,但是却把含有恶意代码的dex安装进来了.int溢出漏洞同样可以通过签名验证,但是可以执行任意文件名称的恶意代码.
Android系统的APK签名(v1版本实现)基本继承了java的jar签名机制,即在zip格式中引入了META-INF目录,存在的区别有:APK是自签名,而且android系统不会校验证书的有效日期.APK文件既可以使用jarsigner进行签名,也可以使用android系统实现的apksigner进行签名.APK签名后,会在META-INF目录下生成MF文件,signature文件(SF)和signature block文件(RSA/DSA/EC). Signature block文件包含证书信息以及对signature文件的签名,signature文件是MF文件section的摘要,MF文件包含zip entry 文件的摘要.
Android系统安装APK文件时,进行签名验证,需要进行zip解析,包括的主要步骤有:
- 遍历
meta-inf目录下的zip entry, 以文件名和文件内容建立一个HashMap(StrictJarFile.java).
private HashMap<String, byte[]> getMetaEntries() throws IOException {
HashMap<String, byte[]> metaEntries = new HashMap<String, byte[]>();
Iterator<ZipEntry> entryIterator = new EntryIterator(nativeHandle, "META-INF/");
while (entryIterator.hasNext()) {
final ZipEntry entry = entryIterator.next();
metaEntries.put(entry.getName(), Streams.readFully(getInputStream(entry)));
}
return metaEntries;
}
- 确认MF中计算摘要的文件在ZIP中对应的entry是存在的(StrictJarFile.java).其中
StrictJarManifest会解析MF中的项目,获得文件名,该步骤执行完毕后,继续执行verifier.readCertificates继续签名验证.
/**
* @param name of the archive (not necessarily a path).
* @param fd seekable file descriptor for the JAR file.
* @param verify whether to verify the file's JAR signatures and collect the corresponding
* signer certificates.
* @param signatureSchemeRollbackProtectionsEnforced {@code true} to enforce protections against
* stripping newer signature schemes (e.g., APK Signature Scheme v2) from the file, or
* {@code false} to ignore any such protections. This parameter is ignored when
* {@code verify} is {@code false}.
*/
private StrictJarFile(String name,
FileDescriptor fd,
boolean verify,
boolean signatureSchemeRollbackProtectionsEnforced)
throws IOException, SecurityException {
this.nativeHandle = nativeOpenJarFile(name, fd.getInt$());
this.fd = fd;
try {
// Read the MANIFEST and signature files up front and try to
// parse them. We never want to accept a JAR File with broken signatures
// or manifests, so it's best to throw as early as possible.
if (verify) {
HashMap<String, byte[]> metaEntries = getMetaEntries();
this.manifest = new StrictJarManifest(metaEntries.get(JarFile.MANIFEST_NAME), true);
this.verifier =
new StrictJarVerifier(
name,
manifest,
metaEntries,
signatureSchemeRollbackProtectionsEnforced);
Set<String> files = manifest.getEntries().keySet();
for (String file : files) {
if (findEntry(file) == null) {
throw new SecurityException("File " + file + " in manifest does not exist");
}
}
isSigned = verifier.readCertificates() && verifier.isSignedJar();
} else {
isSigned = false;
this.manifest = null;
this.verifier = null;
}
} catch (IOException | SecurityException e) {
nativeClose(this.nativeHandle);
IoUtils.closeQuietly(fd);
closed = true;
throw e;
}
guard.open("close");
}
- 以signature block文件名找到对应的signature文件名,并进行签名验证(StrictJarVerifier.java).
/**
* If the associated JAR file is signed, check on the validity of all of the
* known signatures.
*
* @return {@code true} if the associated JAR is signed and an internal
* check verifies the validity of the signature(s). {@code false} if
* the associated JAR file has no entries at all in its {@code
* META-INF} directory. This situation is indicative of an invalid
* JAR file.
* <p>
* Will also return {@code true} if the JAR file is <i>not</i>
* signed.
* @throws SecurityException
* if the JAR file is signed and it is determined that a
* signature block file contains an invalid signature for the
* corresponding signature file.
*/
synchronized boolean readCertificates() {
if (metaEntries.isEmpty()) {
return false;
}
Iterator<String> it = metaEntries.keySet().iterator();
while (it.hasNext()) {
String key = it.next();
if (key.endsWith(".DSA") || key.endsWith(".RSA") || key.endsWith(".EC")) {
verifyCertificate(key);
it.remove();
}
}
return true;
}
/**
* @param certFile
*/
private void verifyCertificate(String certFile) {
// Found Digital Sig, .SF should already have been read
String signatureFile = certFile.substring(0, certFile.lastIndexOf('.')) + ".SF";
byte[] sfBytes = metaEntries.get(signatureFile);
if (sfBytes == null) {
return;
}
byte[] manifestBytes = metaEntries.get(JarFile.MANIFEST_NAME);
// Manifest entry is required for any verifications.
if (manifestBytes == null) {
return;
}
byte[] sBlockBytes = metaEntries.get(certFile);
try {
Certificate[] signerCertChain = verifyBytes(sBlockBytes, sfBytes);
if (signerCertChain != null) {
certificates.put(signatureFile, signerCertChain);
}
} catch (GeneralSecurityException e) {
throw failedVerification(jarName, signatureFile, e);
}
// Verify manifest hash in .sf file
Attributes attributes = new Attributes();
HashMap<String, Attributes> entries = new HashMap<String, Attributes>();
try {
StrictJarManifestReader im = new StrictJarManifestReader(sfBytes, attributes);
im.readEntries(entries, null);
} catch (IOException e) {
return;
}
// If requested, check whether APK Signature Scheme v2 signature was stripped.
if (signatureSchemeRollbackProtectionsEnforced) {
String apkSignatureSchemeIdList =
attributes.getValue(
ApkSignatureSchemeV2Verifier.SF_ATTRIBUTE_ANDROID_APK_SIGNED_NAME);
if (apkSignatureSchemeIdList != null) {
// This field contains a comma-separated list of APK signature scheme IDs which
// were used to sign this APK. If an ID is known to us, it means signatures of that
// scheme were stripped from the APK because otherwise we wouldn't have fallen back
// to verifying the APK using the JAR signature scheme.
boolean v2SignatureGenerated = false;
StringTokenizer tokenizer = new StringTokenizer(apkSignatureSchemeIdList, ",");
while (tokenizer.hasMoreTokens()) {
String idText = tokenizer.nextToken().trim();
if (idText.isEmpty()) {
continue;
}
int id;
try {
id = Integer.parseInt(idText);
} catch (Exception ignored) {
continue;
}
if (id == ApkSignatureSchemeV2Verifier.SF_ATTRIBUTE_ANDROID_APK_SIGNED_ID) {
// This APK was supposed to be signed with APK Signature Scheme v2 but no
// such signature was found.
v2SignatureGenerated = true;
break;
}
}
if (v2SignatureGenerated) {
throw new SecurityException(signatureFile + " indicates " + jarName
+ " is signed using APK Signature Scheme v2, but no such signature was"
+ " found. Signature stripped?");
}
}
}
// Do we actually have any signatures to look at?
if (attributes.get(Attributes.Name.SIGNATURE_VERSION) == null) {
return;
}
boolean createdBySigntool = false;
String createdBy = attributes.getValue("Created-By");
if (createdBy != null) {
createdBySigntool = createdBy.indexOf("signtool") != -1;
}
// Use .SF to verify the mainAttributes of the manifest
// If there is no -Digest-Manifest-Main-Attributes entry in .SF
// file, such as those created before java 1.5, then we ignore
// such verification.
if (mainAttributesEnd > 0 && !createdBySigntool) {
String digestAttribute = "-Digest-Manifest-Main-Attributes";
if (!verify(attributes, digestAttribute, manifestBytes, 0, mainAttributesEnd, false, true)) {
throw failedVerification(jarName, signatureFile);
}
}
// Use .SF to verify the whole manifest.
String digestAttribute = createdBySigntool ? "-Digest" : "-Digest-Manifest";
if (!verify(attributes, digestAttribute, manifestBytes, 0, manifestBytes.length, false, false)) {
Iterator<Map.Entry<String, Attributes>> it = entries.entrySet().iterator();
while (it.hasNext()) {
Map.Entry<String, Attributes> entry = it.next();
StrictJarManifest.Chunk chunk = manifest.getChunk(entry.getKey());
if (chunk == null) {
return;
}
if (!verify(entry.getValue(), "-Digest", manifestBytes,
chunk.start, chunk.end, createdBySigntool, false)) {
throw invalidDigest(signatureFile, entry.getKey(), jarName);
}
}
}
metaEntries.put(signatureFile, null);
signatures.put(signatureFile, entries);
}
- 确认MF文件中section的完整性.摘要算法已经从SHA1升级到SHA2系列(StrictJarVerifier.java).
// Use .SF to verify the mainAttributes of the manifest
// If there is no -Digest-Manifest-Main-Attributes entry in .SF
// file, such as those created before java 1.5, then we ignore
// such verification.
if (mainAttributesEnd > 0 && !createdBySigntool) {
String digestAttribute = "-Digest-Manifest-Main-Attributes";
if (!verify(attributes, digestAttribute, manifestBytes, 0, mainAttributesEnd, false, true)) {
throw failedVerification(jarName, signatureFile);
}
}
// Use .SF to verify the whole manifest.
String digestAttribute = createdBySigntool ? "-Digest" : "-Digest-Manifest";
if (!verify(attributes, digestAttribute, manifestBytes, 0, manifestBytes.length, false, false)) {
Iterator<Map.Entry<String, Attributes>> it = entries.entrySet().iterator();
while (it.hasNext()) {
Map.Entry<String, Attributes> entry = it.next();
StrictJarManifest.Chunk chunk = manifest.getChunk(entry.getKey());
if (chunk == null) {
return;
}
if (!verify(entry.getValue(), "-Digest", manifestBytes,
chunk.start, chunk.end, createdBySigntool, false)) {
throw invalidDigest(signatureFile, entry.getKey(), jarName);
}
}
}
- 执行函数collectCertificates,确认zip entry文件内容的完整性,从代码逻辑可以判断出,如果是系统签名的包,只检查
AndroidManifest.xml文件的完整性,否则需要检查zip中所有entry文件的完整性(PackageParser.java).
// APK's integrity needs to be verified using JAR signature scheme.
Trace.traceBegin(TRACE_TAG_PACKAGE_MANAGER, "verifyV1");
final List<ZipEntry> toVerify = new ArrayList<>();
toVerify.add(manifestEntry);
// If we're parsing an untrusted package, verify all contents
if ((parseFlags & PARSE_IS_SYSTEM_DIR) == 0) {
final Iterator<ZipEntry> i = jarFile.iterator();
while (i.hasNext()) {
final ZipEntry entry = i.next();
if (entry.isDirectory()) continue;
final String entryName = entry.getName();
if (entryName.startsWith("META-INF/")) continue;
if (entryName.equals(ANDROID_MANIFEST_FILENAME)) continue;
toVerify.add(entry);
}
}
// Verify that entries are signed consistently with the first entry
// we encountered. Note that for splits, certificates may have
// already been populated during an earlier parse of a base APK.
for (ZipEntry entry : toVerify) {
final Certificate[][] entryCerts = loadCertificates(jarFile, entry);
if (ArrayUtils.isEmpty(entryCerts)) {
throw new PackageParserException(INSTALL_PARSE_FAILED_NO_CERTIFICATES,
"Package " + apkPath + " has no certificates at entry "
+ entry.getName());
}
final Signature[] entrySignatures = convertToSignatures(entryCerts);
if (pkg.mCertificates == null) {
pkg.mCertificates = entryCerts;
pkg.mSignatures = entrySignatures;
pkg.mSigningKeys = new ArraySet<PublicKey>();
for (int i=0; i < entryCerts.length; i++) {
pkg.mSigningKeys.add(entryCerts[i][0].getPublicKey());
}
} else {
if (!Signature.areExactMatch(pkg.mSignatures, entrySignatures)) {
throw new PackageParserException(
INSTALL_PARSE_FAILED_INCONSISTENT_CERTIFICATES, "Package " + apkPath
+ " has mismatched certificates at entry "
+ entry.getName());
}
}
}
在android系统中, 有时候需要获得jar包或者APK的公钥.对于已经安装的API,可以通过android系统的PackageInfo对象获得.而对于jar包或者没有安装的APK文件,就需要进行zip解析来获得公钥.下面是解析zip格式获得公钥的典型实现:
通过解析zip文件格式获取公钥时,需要注意的是,可以使用zip中的任何entry,但是不能是目录类型,也不能是META-INF目录下的文件entry.因为在jar的initEntry方法中不会为这两类entry附加证书.另外,必须是验证过zip entry的完整性以后,才能读取该entry的公钥,这是jar的实现规范.因此,必须先把entry的内容读取完毕后,才能获取公钥.
android系统的签名算法v1版本的实现是基于zip格式设计的,存在着固有的缺陷:
- 安全性.
尽管v1版本的算法对zip entry类型为文件的删除,增加,修改会有感知,但是不能检测到zip entry类型为目录的更改,也不能检测zip元数据的更改,比如有些APP在comments字段添加数据,来控制一些代码逻辑. - 性能.
APK在安装的时候,v1签名算法需要读出所有的zip entry文件进行完整性验证,而读取的过程中需要解压缩,严重影响安装过程.
android系统在7.0引入了v2版本的签名算法,该算法对zip格式进行了扩展,即在zip中增加了一个APK Signing Block段.v2签名算法通过扩展zip文件格式,实现zip文件全文摘要的效果,相比较v1签名算法,v2算法既快又安全.v2算法扩展后的zip分段:
同时,v2算法在进行zip文件的全文摘要的时候,对zip文件进行分块,实现并行计算,有可以增加更多控制.zip文件的并行分块:
需要注意的是,android系统采用v2签名算法以后,基于java平台的jarsigner将无法支持v2算法,只能使用android特有的签名程序apksigner. 关于android系统的v2签名机制,请参考android官网https://source.android.com/security/apksigning/v2 .
扫一扫
99
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
