url 转码 java_URL 转码遇到的坑

1. 背景

做图片中心时, 需要根据图片URL去下载图片, 发现部分URL浏览器里面能访问,但是代码无法下载

原因: 需要对URL进行转码

2. 使用UrlEncode

一开始直接使用UrlEncode对url转码, 如下

原始链接

https://s3.amazonaws.com/fromfactory.club.image/9f/0f/9fbd598c4681773f6371c27dfb64180f.webp

UrlEncode后

https%3a%2f%2fs3.amazonaws.com%2ffromfactory.club.image%2f9f%2f0f%2f9fbd598c4681773f6371c27dfb64180f.webp

2.1 问题

斜杠,冒号等字符也被UrlEncode进行了转码, 显然这样的转码后是不能下载的

3. 原因分析

编码的意义在于，假如URL的参数中的中文或特殊字符在发送到服务端时，服务端无法解析它的真正意义，会导致服务端不能理解客户端的请求, 此时, 需要对它进行编码

3.1 哪些字符需要编码

RFC3986文档规定，URL中只允许包含英文字母(a-zA-Z)、数字(0-9)、-_.~4个特殊字符以及所有保留字符。

保留字符：Url可以划分成若干个组件，协议、主机、路径等。有一些字符(:/?#[]@)是用作分隔不同组件的。

当组件中的普通数据包含这些特殊字符时，需要对其进行编码。

RFC3986中指定了以下字符为==保留字符：! * ’ ( ) ; : @ & = + $ , / ? # [ ]==

对于图片下载中, 如下都不转码

英文字母 a-zA-Z

数字 0-9

4个特殊字符 _.~、

保留字符 ! * ’ ( ) ; : @ & = + $ , / ? # [ ]==

3.2 UrlEncode不转码字符

UrlEncode中维护了dontNeedEncoding, 但是dontNeedEncoding只包含了

英文字母(a-zA-Z)

数字(0-9)

空格 - _ . *

ab8f20ac3c5a4b0f9beb02bd9be627fb_image.png

如下字符也需要不转码: ! ~ ’ ( ) ; : @ & = + $ , / ? # [ ]==

4. 解决方案

重写UrlEncode添加不转码字符

7be93685d8544476aa88d21c35ec3dd1_image.png

4.1 代码

package com.clubfactory.center.product.util;

import sun.security.action.GetPropertyAction;

import java.io.CharArrayWriter;

import java.io.UnsupportedEncodingException;

import java.nio.charset.Charset;

import java.nio.charset.IllegalCharsetNameException;

import java.nio.charset.UnsupportedCharsetException;

import java.security.AccessController;

import java.util.BitSet;

/**

* Utility class for HTML form encoding. This class contains static methods

* for converting a String to the application/x-www-form-urlencoded MIME

* format. For more information about HTML form encoding, consult the HTML

* specification.

*

*

* When encoding a String, the following rules apply:

*

*

*

The alphanumeric characters "{@code a}" through

* "{@code z}", "{@code A}" through

* "{@code Z}" and "{@code 0}"

* through "{@code 9}" remain the same.

*

The special characters "{@code .}",

* "{@code -}", "{@code *}", and

* "{@code _}" remain the same.

*

The space character "   " is

* converted into a plus sign "{@code +}".

*

All other characters are unsafe and are first converted into

* one or more bytes using some encoding scheme. Then each byte is

* represented by the 3-character string

* "{@code %xy}", where xy is the

* two-digit hexadecimal representation of the byte.

* The recommended encoding scheme to use is UTF-8. However,

* for compatibility reasons, if an encoding is not specified,

* then the default encoding of the platform is used.

*

*

*

* For example using UTF-8 as the encoding scheme the string "The

* string ü@foo-bar" would get converted to

* "The+string+%C3%BC%40foo-bar" because in UTF-8 the character

* ü is encoded as two bytes C3 (hex) and BC (hex), and the

* character @ is encoded as one byte 40 (hex).

*

* @author Herb Jellinek

* @since JDK1.0

*/

public class MyURIEncoder {

static BitSet dontNeedEncoding;

static final int caseDiff = ('a' - 'A');

static String dfltEncName = null;

static {

/* The list of characters that are not encoded has been

* determined as follows:

*

* RFC 2396 states:

* -----

* Data characters that are allowed in a URI but do not have a

* reserved purpose are called unreserved. These include upper

* and lower case letters, decimal digits, and a limited set of

* punctuation marks and symbols.

*

* unreserved = alphanum | mark

*

* mark = "-" | "_" | "." | "!" | "~" | "*" | "'" | "(" | ")"

*

* Unreserved characters can be escaped without changing the

* semantics of the URI, but this should not be done unless the

* URI is being used in a context that does not allow the

* unescaped character to appear.

* -----

*

* It appears that both Netscape and Internet Explorer escape

* all special characters from this list with the exception

* of "-", "_", ".", "*". While it is not clear why they are

* escaping the other characters, perhaps it is safest to

* assume that there might be contexts in which the others

* are unsafe if not escaped. Therefore, we will use the same

* list. It is also noteworthy that this is consistent with

* O'Reilly's "HTML: The Definitive Guide" (page 164).

*

* As a last note, Intenet Explorer does not encode the "@"

* character which is clearly not unreserved according to the

* RFC. We are being consistent with the RFC in this matter,

* as is Netscape.

*

*/

dontNeedEncoding = new BitSet(256);

int i;

for (i = 'a'; i <= 'z'; i++) {

dontNeedEncoding.set(i);

}

for (i = 'A'; i <= 'Z'; i++) {

dontNeedEncoding.set(i);

}

for (i = '0'; i <= '9'; i++) {

dontNeedEncoding.set(i);

}

//dontNeedEncoding.set(' '); /* encoding a space to a + is done * in the encode() method */

dontNeedEncoding.set('-');

dontNeedEncoding.set('_');

dontNeedEncoding.set('.');

dontNeedEncoding.set('*');

//对以下在 URI 中具有特殊含义的 ASCII 标点符号 ;/?:@&=+$,# 不需要转义

dontNeedEncoding.set(';');

dontNeedEncoding.set('/');

dontNeedEncoding.set('?');

dontNeedEncoding.set(':');

dontNeedEncoding.set('@');

dontNeedEncoding.set('&');

dontNeedEncoding.set('=');

dontNeedEncoding.set('+');

dontNeedEncoding.set('$');

dontNeedEncoding.set(',');

dontNeedEncoding.set('#');

dontNeedEncoding.set('!');

dontNeedEncoding.set('\'');

dontNeedEncoding.set('(');

dontNeedEncoding.set(')');

dontNeedEncoding.set('%');

dontNeedEncoding.set('[');

dontNeedEncoding.set(']');

dfltEncName = AccessController.doPrivileged(

new GetPropertyAction("file.encoding")

);

}

/**

* You can't call the constructor.

*/

private MyURIEncoder() { }

/**

* Translates a string into {@code x-www-form-urlencoded}

* format. This method uses the platform's default encoding

* as the encoding scheme to obtain the bytes for unsafe characters.

*

* @param s {@code String} to be translated.

* @deprecated The resulting string may vary depending on the platform's

* default encoding. Instead, use the encode(String,String)

* method to specify the encoding.

* @return the translated {@code String}.

*/

@Deprecated

public static String encode(String s) {

String str = null;

try {

str = encode(s, dfltEncName);

} catch (UnsupportedEncodingException e) {

// The system should always have the platform default

}

return str;

}

/**

* Translates a string into {@code application/x-www-form-urlencoded}

* format using a specific encoding scheme. This method uses the

* supplied encoding scheme to obtain the bytes for unsafe

* characters.

*

* Note: The

* "http://www.w3.org/TR/html40/appendix/notes.html#non-ascii-chars">

* World Wide Web Consortium Recommendation

states that

* UTF-8 should be used. Not doing so may introduce

* incompatibilities.

*

* @param s {@code String} to be translated.

* @param enc The name of a supported

* character

* encoding.

* @return the translated {@code String}.

* @exception UnsupportedEncodingException

* If the named encoding is not supported

* @since 1.4

*/

public static String encode(String s, String enc)

throws UnsupportedEncodingException {

boolean needToChange = false;

StringBuffer out = new StringBuffer(s.length());

Charset charset;

CharArrayWriter charArrayWriter = new CharArrayWriter();

if (enc == null)

throw new NullPointerException("charsetName");

try {

charset = Charset.forName(enc);

} catch (IllegalCharsetNameException e) {

throw new UnsupportedEncodingException(enc);

} catch (UnsupportedCharsetException e) {

throw new UnsupportedEncodingException(enc);

}

for (int i = 0; i < s.length();) {

int c = (int) s.charAt(i);

if (dontNeedEncoding.get(c)) {

out.append((char)c);

i++;

} else {

// convert to external encoding before hex conversion

do {

charArrayWriter.write(c);

/*

* If this character represents the start of a Unicode

* surrogate pair, then pass in two characters. It's not

* clear what should be done if a bytes reserved in the

* surrogate pairs range occurs outside of a legal

* surrogate pair. For now, just treat it as if it were

* any other character.

*/

if (c >= 0xD800 && c <= 0xDBFF) {

/*

System.out.println(Integer.toHexString(c)

+ " is high surrogate");

*/

if ( (i+1) < s.length()) {

int d = (int) s.charAt(i+1);

/*

System.out.println("\tExamining "

+ Integer.toHexString(d));

*/

if (d >= 0xDC00 && d <= 0xDFFF) {

/*

System.out.println("\t"

+ Integer.toHexString(d)

+ " is low surrogate");

*/

charArrayWriter.write(d);

i++;

}

}

}

i++;

} while (i < s.length() && !dontNeedEncoding.get((c = (int) s.charAt(i))));

charArrayWriter.flush();

String str = new String(charArrayWriter.toCharArray());

byte[] ba = str.getBytes(charset);

for (int j = 0; j < ba.length; j++) {

out.append('%');

char ch = Character.forDigit((ba[j] >> 4) & 0xF, 16);

// converting to use uppercase letter as part of

// the hex value if ch is a letter.

if (Character.isLetter(ch)) {

ch -= caseDiff;

}

out.append(ch);

ch = Character.forDigit(ba[j] & 0xF, 16);

if (Character.isLetter(ch)) {

ch -= caseDiff;

}

out.append(ch);

}

charArrayWriter.reset();

needToChange = true;

}

}

return (needToChange? out.toString() : s);

}

}