Lighttpd+Python实现在线数据处理-CSDN博客

本文链接：https://blog.csdn.net/lpwmm/article/details/83451652

先吐槽一下Apache的配置文件,好长时间没怎么配置过,发现各种版本之间的语法乱七八糟的,懒得去研究了,还是用简单粗暴有效的Lighttpd吧,性能基本不用考虑.

之前用python+pandas做了个数据处理的脚本,想放到服务器上远程调用,B/S模式就可以省的装环境了.还是头一次做这种实现,其实lighttpd就提供个基本的http服务,具体的数据操作通过cgi扔给python脚本来处理,实现的过程中也踩了不少坑,把过程整理记录一下吧:

1.安装配置lighttpd

CentOS7下面搞这个还是很简单的

# yum install -y lighttpd
# systemctl enable lighttpd
# systemctl start lighttpd

站点文件我放在了/www下面,所以需要修改一下lighttpd的配置

# vim /etc/lighttpd/lighttpd.conf
修改部分:
var.server_root = "/www"
server.document-root = server_root
server.use-ipv6 = "disable"
server.max-fds = 2048

配置启用CGI,其中python的脚本放在/www/cgi-bin下面.这里重点是后面增加的$HTTP那一堆

# vim /etc/lighttpd/modules.conf
修改内容:
server.modules = (
  "mod_access",
  "mod_cgi",
)
$HTTP["url"] =~ "^/cgi-bin/" {
        cgi.assign = ( ".py" => "/usr/bin/python" )

配置改好后,重启一下lighttpd服务

# systemctl restart lighttpd

如果服务启动不成功,可以使用控制台直接输出的方式运行lighttpd

[root@pptp lighttpd]# lighttpd -f /etc/lighttpd/lighttpd.conf -D
2018-10-28 12:19:32: (server.c.1346) can't have more connections than fds/2:  1024 1024

出现这种错误是由于lighttpd.conf里面server.max-fds = 2048这一行没有生效,检查一下配置文件内容.

配置防火墙开放80端口

# firewall-cmd --add-service=http --permanent

2.静态页面代码/www/index.html

<!DOCTYPE html>
<html lang="en">

<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta http-equiv="X-UA-Compatible" content="ie=edge">
    <title>数据处理</title>
</head>

<body>
    <h1>请上传要处理的文件:</h1>
    <form enctype="multipart/form-data" action="/cgi-bin/my.py" method="POST">
        <input type="file" name="filename" />
        <input type="submit" value="确认上传" />
    </form>
</body>

</html>

3.python脚本代码/www/cgi-bin/my.py

#!/usr/bin/python
#-*- coding:utf-8 -*-
import cgi
import os
import cgitb
cgitb.enable()
import pandas as pd

form = cgi.FieldStorage()
# Get filename here.
fileitem = form['filename']
# print(fileitem.filename)
# Test if the file was uploaded
err = 'No error'
if fileitem.filename:
    # strip leading path from file name to avoid
    # directory traversal attacks
    fn = os.path.basename(fileitem.filename)
    open('/tmp/' + fn, 'wb').write(fileitem.file.read())
    try:
        df = pd.read_excel('/tmp/' + fn)
        gp = df.groupby('县区').count()
        gp.to_excel('/www/xls/new.xlsx')
    except Exception as e:
        #err = e + '\n  fn:  ' + fn
        err = e
    #message = message + df.head(3)
    message = 'The file "' + fn + '" was uploaded successfully'
else:
    message = 'No file was uploaded'

print """\
Content-Type: text/html\n
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<title>数据处理</title>
</head>
<body>
   <h1>%s</h1>
   <hr>
   <h1>%s</h1>
   <a href='/xls/new.xlsx'><h1>下载Download</h1></a>
</body>
</html>
""" % (message, err)

这里需要补充说明一下,系统默认带的Python 2.7对中文很不友好,上面的代码运行可能会出问题,这个坑随后再用Python3.x填吧

2018.10.28更新

基本上等于全部重构了一遍,使用Bootstrap3做的前端,使用ajax异步提交表单让后台的python脚本执行,效果感觉还是棒棒哒的.bia一下效果图和代码

/www/index.html

<!DOCTYPE html>
<html lang="zh-CN">

<head>
  <meta charset="utf-8">
  <meta http-equiv="X-UA-Compatible" content="IE=edge">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>数据处理平台</title>
  <link href="https://cdn.jsdelivr.net/npm/bootstrap@3.3.7/dist/css/bootstrap.min.css" rel="stylesheet">

</head>

<body>

  <nav class="navbar navbar-default navbar-fixed-top" role="navigation">
    <a class="navbar-brand" href="#">CATI电话调查系统数据处理平台</a>
    <ul class="nav navbar-nav">
      <li class="active">
        <a href="#">首页</a>
      </li>
      <li>
        <a id="aboutBtn" href="#">关于</a>
      </li>
    </ul>
  </nav>

  <div class="container" style="padding-top:60px">

    <div class="jumbotron">
      <form id="mainform" enctype="multipart/form-data">
        <div class="container">
          <h2>平安建设公众安全感和政法机关执法满意度调查</h2>
          <p>问卷进度数据分县区汇总</p>
          <p>
            <div class="form-group">
              <label for="filename">打开xls文件</label>
              <input type="file" id="filename" name="filename">
              <p class="help-block">请选择从CATI平台导出的问卷数据xls文件</p>
            </div>
          </p>
          <button type="submit" class="btn btn-primary btn-lg">开始处理</button>
          <a id="download" href="/xls/new.xlsx" class="btn btn-default btn-lg" style="display:none">点击下载</a>

          <div id="progressbar" style="margin-top:1em;display:none" class="progress progress-striped active">
            <div class="progress-bar progress-bar-success" role="progressbar" aria-valuenow="60" aria-valuemin="0"
              aria-valuemax="100" style="width: 100%;">正在处理...
            </div>
          </div>


          <div id="alert" class="alert alert-success alert-dismissible fade in" style="display:none;margin-top:1em">
            <a href="#" class="close" data-dismiss="alert" aria-label="close">&times;</a>
            <strong>汇总完成</strong>请点击下载
          </div>
        </div>
      </form>
    </div>

  </div>

  <div class="modal fade" id="about" tabindex="-1" role="dialog" aria-labelledby="myModalLabel">
    <div class="modal-dialog" role="document">
      <div class="modal-content">
        <div class="modal-header">
          <button type="button" class="close" data-dismiss="modal" aria-label="Close">
            <span aria-hidden="true">×</span>
          </button>
          <h4 class="modal-title" id="myModalLabel">关于本站点</h4>
        </div>
        <div class="modal-body">
          <p>基于Lighttpd/Python CGI/Pandas开发</p>
          <p>连鹏伟 2018.10.28</p>
        </div>
        <div class="modal-footer">
          <button type="button" class="btn btn-default" data-dismiss="modal">关闭</button>
        </div>
      </div>
    </div>
  </div>


  <script src="https://cdn.jsdelivr.net/npm/jquery@1.12.4/dist/jquery.min.js"></script>
  <script src="https://cdn.jsdelivr.net/npm/bootstrap@3.3.7/dist/js/bootstrap.min.js"></script>
  <script>
    $("#aboutBtn").click(function () {
      $("#about").modal("toggle");
    });


    $("#mainform").submit(function (event) {
      event.preventDefault();
      $("#progressbar").show();

      var formData = new FormData();
      formData.append('filename', $("#filename")[0].files[0]);

      $.ajax({
        async: true,
        type: "POST",
        url: "/cgi-bin/my.py",
        data: formData,
        processData: false,
        contentType: false,
        dataType: "json",
        success: function (data) {
          $("#progressbar").hide();
          $("#alert").show();
          $("#download").show();
        },
        error: function (data) {
          alert('error:' + JSON.stringify(data))
        }
      })
    })
  </script>
</body>

</html>

/www/cgi-bin/my.py

#!/usr/bin/python
# -*- coding:utf-8 -*-
import pandas as pd
import cgi
import os
import cgitb
import json

cgitb.enable()

form = cgi.FieldStorage()
# Get filename here.
fileitem = form['filename']
# print(fileitem.filename)
# Test if the file was uploaded
err = 'Success!'
result = {}
if fileitem.filename:
    # strip leading path from file name to avoid
    # directory traversal attacks
    fn = os.path.basename(fileitem.filename)
    open('/tmp/' + fn, 'wb').write(fileitem.file.read())
    try:
        df = pd.read_excel('/tmp/' + fn)
        df.to_excel('/www/xls/new.xlsx')
    except Exception as e:
        err = e

    result['success'] = True
    message = 'The file "' + fn + '" was uploaded successfully'
else:
    result['success'] = False
    message = 'No file was uploaded'

#result = {'success':'true','msg': message}
#result['success'] = True
result['msg'] = message
print("Content-Type: application/json\n\n")
print(json.dumps(result))