背景
使用python3的urllib调用spring cloud服务接口,一直报错
File "E:\github\workspace\dbfree\src\test\common\test_paas_api_base.py", line 49, in test_zjk
resp = urllib.request.urlopen(req, timeout=5)
File "C:\Users\zhangjikuan\AppData\Local\Programs\Python\Python36\lib\urllib\request.py", line 223, in urlopen
return opener.open(url, data, timeout)
File "C:\Users\zhangjikuan\AppData\Local\Programs\Python\Python36\lib\urllib\request.py", line 532, in open
response = meth(req, response)
File "C:\Users\zhangjikuan\AppData\Local\Programs\Python\Python36\lib\urllib\request.py", line 642, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Users\zhangjikuan\AppData\Local\Programs\Python\Python36\lib\urllib\request.py", line 570, in error
return self._call_chain(*args)
File "C:\Users\zhangjikuan\AppData\Local\Programs\Python\Python36\lib\urllib\request.py", line 504, in _call_chain
result = func(*args)
File "C:\Users\zhangjikuan\AppData\Local\Programs\Python\Python36\lib\urllib\request.py", line 650, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request
使用浏览器直接调用返回的是封装后的数据,虽然结果错误,但至少是服务中封装的结构
{“RequestId”:“b8e3f811-de16-4693-9084-44cb9aa7cc34”,“Data”:null,“Success”:false,“Code”:500,“Message”:null}
python3 urllib
与python2的不同,包的import有所不同
import urllib.parse
import urllib.request
url = "http://localhost:8787/rm/QueryResource"
k = "param1"
v = urllib.parse.quote("abcdef-123")
param = "{}={}".format(k, v)
data = bytes(param .encode("utf-8"))
headers = dict()
req = urllib.request.Request(url, data, headers)
resp = urllib.request.urlopen(req, timeout=5)
body = resp.read()
排查过程
- 1.看上面的现象,被严重误导,既然浏览器和shell curl都正常,那就是python使用的问题
因为参数值abcdef-123中有中划线,怀疑是编码问题,而且恰巧的是去掉中划线就好了,加上中划线就报400
所以去看了urllib.parse.quote编码代码,如下:
def quote(string, safe='/', encoding=None, errors=None):
"""quote('abc def') -> 'abc%20def'
Each part of a URL, e.g. the path info, the query, etc., has a
different set of reserved characters that must be quoted.
RFC 2396 Uniform Resource Identifiers (URI): Generic Syntax lists
the following reserved characters.
reserved = ";" | "/" | "?" | ":" | "@" | "&" | "=" | "+" |
"$" | ","
Each of these characters is reserved in some component of a URL,
but not necessarily in all of them.
上面代码很明显标注保留以上字符,但是-并不是要编码的字符,不影响才对
- 2 继续debug urllib 在http_response中发现如下代码
def http_response(self, request, response):
code, msg, hdrs = response.code, response.msg, response.info()
# According to RFC 2616, "2xx" code indicates that the client's
# request was successfully received, understood, and accepted.
if not (200 <= code < 300):
response = self.parent.error(
'http', request, response, code, msg, hdrs)
return response
上面代码看到在http code不在[200-300)范围时候就会抛错误,错误就在这里抛出来的
-
3 被排查节奏有点带歪,以至于又走了弯路,以为返回值的code是500导致的,如果把server端错误code都改在2xx也有点不能忍,毕竟定义了很多code,超过100个该怎么办,但是还是先把500改为250看下效果
很遗憾改完后依旧报错 -
4 进一步debug urllib代码,看下这个code在哪里获取到的,一直到了最底层client的代码,发现不是那么回事
def _read_status(self):
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
if len(line) > _MAXLINE:
raise LineTooLong("status line")
if self.debuglevel > 0:
print("reply:", repr(line))
上面的read代码是直接拿的http结果,用的iso-8859-1,那也就是说这个400是http状态,不是我代码返回的code500,此时恍然醒悟,怎么能范这么大的错误,很明显报错信息就是HTTP Error 400: Bad Request,这是HTTP的状态码,浏览器返回的数据是自己在server端定义的用户数据,两个code完全不是一回事
- 5得出上面结论后排查就很顺利了,去server端排查日志发现了明显的堆栈报错
java.lang.NullPointerException
at com..rm.service.K8sOpService.queryPod(K8sOpService.java:322)
at com.rm.service.K8sResourceService.queryResource(K8sResourceService.java:135)
at com.rm.controller.K8sResourceController.queryResource(K8sResourceController.java:68)
at com.rm.controller.K8sResourceController$$FastClassBySpringCGLIB$$8882770f.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
也就是说当参数中有中划线时候server抛异常了,这也导致上面走弯路的原因(加了中划线就抛异常,不加就正常,两次让现象误导的排查方向)。
抛的异常会被controller的globalException获取到,封装为自己的数据返回
- 6 server的RestControllerAdvice 参考 https://blog.csdn.net/zhangjikuan/article/details/115004774
文章中我的@ResponseStatus(HttpStatus.OK) 是修改过的,在排查此问题时是 @ResponseStatus(HttpStatus.BAD_REQUEST) 也就是400的报错
-
7 根据6中的结论,只要所有RestControllerAdvice 获取到的异常都会报400,修改下调用参数,让其抛参数异常,进一步验证是不是还会报400,发现果然是这里。
-
8 最后修改spring cloud端RestControllerAdvice 异常时仍然报200,全部解决,这也是在上一篇文章中最后添加注意的点
-
9 至此所有问题解决
结论
- python3 urllib与python2.7使用上稍有不同
- urllib.parse.quote() 有保留一些字符不编码
- 自己封装接口返回code时要注意与http status的区别
- 一些客户端会判断http status的code值,就算有用户数据也会直接抛异常
- RestControllerAdvice 中 @ResponseStatus可以设置http返回码