日萌社
人工智能AI：Keras PyTorch MXNet TensorFlow PaddlePaddle 深度学习实战（不定时更新）
nginx+keepalived：局域网内网和公网外网 搭建nginx HA 双机热备 高可用
nginx 自动解压gzip压缩数据的两种方式：nginx自动解压、后台java程序解压
nginx安装、nginx反向代理/负载均衡、Lua、LuaJIT、openresty、lua-nginx-module、ngx_devel_kit 的安装
nginx 配置获取GET请求参数、POST请求参数、nginx配置开启跨域访问、nginx+keepalived配置主备切换/双机热备、nginx优化配置
注意：在配置nginx 自动解压gzip压缩数据之前，需要安装以下软件、模块
链接：nginx、Lua、LuaJIT、openresty、lua-nginx-module、ngx_devel_kit 的安装
=============================================
文章参考：
https://huoding.com/2013/09/02/283
https://blog.csdn.net/qingchezhanlandetian/article/details/50327111
http://www.pataliebre.net/howto-make-nginx-decompress-a-gzipped-request.html#.XMKhhjMzaHv
https://blog.csdn.net/chenglian1987/article/details/78502246
=============第一种方式：nginx自动解压====================
vim /root/softs/nginx/conf/nginx.conf
server
{
location /unzip/ {
proxy_pass http://backend;
proxy_set_header REMOTE_ADDR $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_next_upstream http_500 http_502 http_504;
limit_req zone=java burst=5 nodelay;
#proxy_next_upstream_tries 1;
set $max_chunk_size 10240; # Chunks of 10 KB
set $max_body_size 524288; # Max inflated body size of 512 KB
rewrite_by_lua_file /root/b.lua;
}
}b.lua
local zlib = require "zlib"
local encoding = ngx.req.get_headers()["Content-Encoding"]
if encoding == "gzip" then
local body = ngx.req.get_body_data()
if body then
local stream = zlib.inflate()
ngx.req.set_body_data(stream(body))
end
endcurl -v -s --trace-ascii http_post_trace.log --data-binary @gzip_request_test.gz -H "Content-Type: text/html" -H "Content-Encoding: gzip" -o response.txt -X POST http://192.168.88.102:80/unzip/
参数解析：
curl --trace-ascii output.text www.taobao.com # 显示ascii细节
curl -o baidu.html www.baidu.com # 文件保存
curl --header "Content-Type:application/json" http://example.com # 添加头
-X 这个参数可以配置HTTP的方法：
curl -X POST 127.0.0.1:7001/api/material/update
curl -X POST --data "name=kk&name2=gg" 127.0.0.1:7001/api/material/create # 带参数的post提交
默认get：curl http://xxxx
post 文件：curl --data-binary @/tmp/xxx http://xxxx
更新文件（url中指定需要更新的文件名）：curl -X PUT --data-binary @/tmp/xxx http://xxxx
删除（url中指定需要删除的文件名）：curl -X DELETE http://xxxx
Curl命令提供了若干个设置HTTP POST数据的选项，这里比较如下：
官方文档：https://curl.haxx.se/docs/manpage.html
网友介绍：https://blog.csdn.net/taiyangdao/article/details/77020762
-d,--data <key=value>
将HTTP POST请求中的数据发送给HTTP服务器，与用户提交HTML表单时浏览器的行为完全一样。
默认Content-type为application/x-www-form-urlencoded
@file_name：表示数据来自一个文件，文件中的回车符和换行符将被转换
--data-ascii <key=value>
等价于-d
========================================================
zlib库不能直接压缩gzip格式，使用lua-ffi-zlib
源码路径：https://github.com/hamishforbes/lua-ffi-zlib
调用：
local ffi_zlib = require "lib.ffi-zlib"
local chunk = 16384
local count = 0
local input = function(bufsize)
local start = count > 0 and bufsize*count or 1
local data = str:sub(start, (bufsize*(count+1)-1))
if data == "" then
data = nil
end
print(data)
count = count + 1
return data
end
local output_table = {}
local output = function(data)
insert(output_table, data)
end
local ok, err = ffi_zlib.deflateGzip(input, output, chunk)
if not ok then
print(err)
end
local compress = concat(output_table,'')
ngx.header["Content-Encoding"] = "gzip"
ngx.print(compress)
========================================================
方案
第一个选择是使用lua-zlib：
local zlib = require "zlib"
local encoding = ngx.req.get_headers()["Content-Encoding"]
if encoding == "gzip" then
local body = ngx.req.get_body_data()
if body then
local stream = zlib.inflate()
ngx.req.set_body_data(stream(body))
end
end第二个选择是通过LuaJIT的FFI库来包装ZLIB模块，官方教程里有一些现成的可供参考的的例子，不过例子里介绍的是Deflate，而不是Gzip，自己用FFI封装Gzip的话又有点小复杂，好在别人已经做了相关的工作，那就是lua-files：
local ffi = require "ffi"
local zlib = require "zlib"
local function reader(s)
local done
return function()
if done then return end
done = true
return s
end
end
local function writer()
local t = {}
return function(data, sz)
if not data then return table.concat(t) end
t[#t + 1] = ffi.string(data, sz)
end
end
local encoding = ngx.req.get_headers()["Content-Encoding"]
if encoding == "gzip" then
local body = ngx.req.get_body_data()
if body then
local write = writer()
zlib.inflate(reader(body), write, nil, "gzip")
ngx.req.set_body_data(write())
end
end
如上例子代码源自zlib_test.lua，乍看上去，代码里的reader和writer可能会令人费解，其实你可以把它们理解成输入输出接口，可以修改成文件，数据库等等形式。
别高兴太早，当你运行时，很可能会遇到如下错误：
libzlib.so: cannot open shared object file.
实际上这是因为如下zlib.lua代码的缘故：
local C = ffi.load 'zlib'
运行时，ffi.load会自动补全文件名，如果是Windows，则加载zlib.dll文件，如果是Linux，则加载libzlib.so，但实际上在Linux下，ZLIB扩展的名字是libz.so，而非libzlib.so。
知道的问题的原委，我们自然就知道如何修改代码了：
local C
if ffi.os == "Windows" then
C = ffi.load "zlib"
else
C = ffi.load "z"
end
有时候我们不推荐直接修改第三方库的代码，因为这样的话，每次第三库更新代码，我们都要做对应的修改，一旦忘记就会出错，这时候可以考虑做一个软连接别名。
测试
开篇说过，接口都是用PHP做的，不过请求里的Gzip数据是用LUA处理的，如何让PHP使用LUA处理后的数据呢？不同的语言似乎是个难题，好在Nginx有Phases一说，PHP作为FastCGI模块工作在content阶段，LUA可以工作在access阶段，这样它们就和谐了：
location ~ \.php$ {
access_by_lua_file /path/to/lua/file;
include fastcgi.conf;
fastcgi_pass 127.0.0.1:9000;
}
=============第一种方式：后台java程序解压====================
java gzip压缩/ungzip解压缩、base64 encode编码/decode编码、URLEncode.encode编码/URLDecoder.decode编码
代码文件下载链接: https://pan.baidu.com/s/19GIva6_AKlFdbVDxbYqLEw
提取码: m6xj
package com.nginxlog;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.net.URLDecoder;
import java.util.zip.GZIPInputStream;
import java.util.zip.GZIPOutputStream;
public class GZIPUtils
{
public static final String GZIP_ENCODE_UTF_8 = "UTF-8";
public static final String GZIP_ENCODE_ISO_8859_1 = "ISO-8859-1";
public static byte[] compress(String str, String encoding)
// public static byte[] compress(String str, String encoding)
{
if (str == null || str.length() == 0) {
return null;
}
ByteArrayOutputStream out = new ByteArrayOutputStream(str.getBytes().length);
GZIPOutputStream gzip;
try {
gzip = new GZIPOutputStream(out);
gzip.write(str.getBytes(encoding));
// byte[] bytes = out.toByteArray();
gzip.close();
} catch ( Exception e) {
e.printStackTrace();
}
byte[] bytes = out.toByteArray();
try
{
out.close();
}
catch (IOException e) {
e.printStackTrace();
}
return bytes;
// return new String(Base64Coder.encode(bytes));//Base64Coder 对 压缩数据 进行 编码
}
public static byte[] compress(String str) throws IOException
{
return compress(str, GZIP_ENCODE_UTF_8);
}
public static byte[] uncompress(byte[] bytes)
{
int bufferSize = 4096;
if (bytes == null || bytes.length == 0) {
return null;
}
ByteArrayOutputStream out = new ByteArrayOutputStream(bytes.length);
ByteArrayInputStream in = new ByteArrayInputStream(bytes);
try {
GZIPInputStream ungzip = new GZIPInputStream(in,bufferSize);
byte[] buffer = new byte[bufferSize];
int n;
while ((n = ungzip.read(buffer)) >= 0) {
out.write(buffer, 0, n);
}
ungzip.close();
} catch (Exception e) {
e.printStackTrace();
}
byte[] bytes1 = out.toByteArray();
try
{
in.close();
out.close();
}
catch (IOException e)
{
e.printStackTrace();
}
return bytes1;
}
public static String uncompressToString(byte[] bytes, String encoding) throws IOException {
int bufferSize = 4096;
if (bytes == null || bytes.length == 0) {
return null;
}
ByteArrayOutputStream out = new ByteArrayOutputStream(bytes.length);
ByteArrayInputStream in = new ByteArrayInputStream(bytes);
GZIPInputStream ungzip = null;
ungzip = new GZIPInputStream(in,bufferSize);
byte[] buffer = new byte[bufferSize];
int n;
while ((n = ungzip.read(buffer)) > 0)
{
out.write(buffer, 0, n);
out.flush();
}
if (ungzip!=null)
{
ungzip.close();
}
in.close();
String s = out.toString(encoding);
out.close();
return s;
}
public static String uncompressToString(byte[] bytes) throws IOException {
return uncompressToString(bytes, GZIP_ENCODE_UTF_8);
}
// public static void main(String[] args) throws IOException
// {
// String s = "hello world";
// System.out.println("字符串长度："+s.length());
// System.out.println("压缩后：："+compress(s));
// System.out.println("解压后："+uncompress(compress(s)));
// System.out.println("解压字符串后："+uncompressToString(compress(s)));
// System.out.println("解压字符串后的字符串长度："+uncompressToString(compress(s)).length());
//
//
// String tmp = "H4sIAAAAAAAAAI2Tv47UMBDGXwWZK5fI/xLb221FRYWEkBCKbM+YWLebhMSb1ep0BdUVFIgKHoGWhzqJewuc3J5YEIdIE83P+Wa+8UzeXJE6DdZf1hHI+hkrKWXMyEquSIo7JGtWlpWhUkrKK5rhsc+QLBKyIhDHFFufFjUx3nFhgdMscT4HhjtDQQQqmWCcM4FOaA/KciiNsR5LLx1w44wCqgx41FKL0nihUILiPASqBNcqGD8nARRSW3BaOMXLoLiVQA1jFE3lwYCujDAWjFBQBaOsE9l7cCJU3gSRy6BgUlShcpUMNmjUIZvwnjGlfWVyQ9voyPqKXCxvsmlh6HJrqwXUEw5j7Np8IApalDO2fX+GWcEfvt1harr5UnwH+AABk43bBe4KZ1PhG5u2ccL3hfUpTjEdixc2tptT8HR5yPWK4IRtysKLTd+/inh46QfENufth67HIUUc/+67G8/86YIVdKaAU/R4PzXQxgHKUJZUehvYP7rd5V5m+1OcuievOdvcV/i94rhYqw8RUpP3h2r6CzYY3zW5D84XurPtPk8g7QccTmkfu9SZtXbeSHJ38/n2+8cfXz7dfru5+/phVhxii";
// byte[] bytes = tmp.getBytes("ISO8859-1");
// String s1 = new String(bytes, "UTF-8");
// String urldecode = URLDecoder.decode(s1, "UTF-8");
// byte[] decode = Base64Coder.decode(urldecode);
// String s2 = GZIPUtils.uncompressToString(decode);
// System.out.println(s2);
// }
}
----------------------------base64转码------------------------------
package com.nginxlog;
public class Base64Coder
{
// Mapping table from 6-bit nibbles to Base64 characters.
private static char[] map1 = new char[64];
static {
int i = 0;
for (char c = 'A'; c <= 'Z'; c++)
map1[i++] = c;
for (char c = 'a'; c <= 'z'; c++)
map1[i++] = c;
for (char c = '0'; c <= '9'; c++)
map1[i++] = c;
map1[i++] = '+';
map1[i++] = '/';
}
// Mapping table from Base64 characters to 6-bit nibbles.
private static byte[] map2 = new byte[128];
static {
for (int i = 0; i < map2.length; i++)
map2[i] = -1;
for (int i = 0; i < 64; i++)
map2[map1[i]] = (byte) i;
}
/**
* Encodes a string into Base64 format.
* No blanks or line breaks are inserted.
*
* @param s a String to be encoded.
* @return A String with the Base64 encoded data.
*/
public static String encodeString(String s) {
return new String(encode(s.getBytes()));
}
/**
* Encodes a byte array into Base64 format.
* No blanks or line breaks are inserted.
*
* @param in an array containing the data bytes to be encoded.
* @return A character array with the Base64 encoded data.
*/
public static char[] encode(byte[] in) {
return encode(in, in.length);
}
/**
* Encodes a byte array into Base64 format.
* No blanks or line breaks are inserted.
*
* @param in an array containing the data bytes to be encoded.
* @param iLen number of bytes to process in <code>in</code>.
* @return A character array with the Base64 encoded data.
*/
public static char[] encode(byte[] in, int iLen) {
int oDataLen = (iLen * 4 + 2) / 3; // output length without padding
int oLen = ((iLen + 2) / 3) * 4; // output length including padding
char[] out = new char[oLen];
int ip = 0;
int op = 0;
while (ip < iLen) {
int i0 = in[ip++] & 0xff;
int i1 = ip < iLen ? in[ip++] & 0xff : 0;
int i2 = ip < iLen ? in[ip++] & 0xff : 0;
int o0 = i0 >>> 2;
int o1 = ((i0 & 3) << 4) | (i1 >>> 4);
int o2 = ((i1 & 0xf) << 2) | (i2 >>> 6);
int o3 = i2 & 0x3F;
out[op++] = map1[o0];
out[op++] = map1[o1];
out[op] = op < oDataLen ? map1[o2] : '=';
op++;
out[op] = op < oDataLen ? map1[o3] : '=';
op++;
}
return out;
}
/**
* Decodes a string from Base64 format.
*
* @param s a Base64 String to be decoded.
* @return A String containing the decoded data.
* @throws IllegalArgumentException if the input is not valid Base64 encoded data.
*/
public static String decodeString(String s) {
return new String(decode(s));
}
/**
* Decodes a byte array from Base64 format.
*
* @param s a Base64 String to be decoded.
* @return An array containing the decoded data bytes.
* @throws IllegalArgumentException if the input is not valid Base64 encoded data.
*/
public static byte[] decode(String s) throws IllegalArgumentException
{
return decode(s.toCharArray());
}
/**
* Decodes a byte array from Base64 format.
* No blanks or line breaks are allowed within the Base64 encoded data.
*
* @param in a character array containing the Base64 encoded data.
* @return An array containing the decoded data bytes.
* @throws IllegalArgumentException if the input is not valid Base64 encoded data.
*/
public static byte[] decode(char[] in) {
int iLen = in.length;
if (iLen % 4 != 0)
throw new IllegalArgumentException("Length of Base64 encoded input string is not a multiple of 4。base64编码输入字符串的长度不是4的倍数");
while (iLen > 0 && in[iLen - 1] == '=')
iLen--;
int oLen = (iLen * 3) / 4;
byte[] out = new byte[oLen];
int ip = 0;
int op = 0;
while (ip < iLen) {
int i0 = in[ip++];
int i1 = in[ip++];
int i2 = ip < iLen ? in[ip++] : 'A';
int i3 = ip < iLen ? in[ip++] : 'A';
if (i0 > 127 || i1 > 127 || i2 > 127 || i3 > 127)
throw new IllegalArgumentException("Illegal character in Base64 encoded data.");
int b0 = map2[i0];
int b1 = map2[i1];
int b2 = map2[i2];
int b3 = map2[i3];
if (b0 < 0 || b1 < 0 || b2 < 0 || b3 < 0)
throw new IllegalArgumentException("Illegal character in Base64 encoded data.");
int o0 = (b0 << 2) | (b1 >>> 4);
int o1 = ((b1 & 0xf) << 4) | (b2 >>> 2);
int o2 = ((b2 & 3) << 6) | b3;
out[op++] = (byte) o0;
if (op < oLen)
out[op++] = (byte) o1;
if (op < oLen)
out[op++] = (byte) o2;
}
return out;
}
} // end class Base64Coder
1797
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
