Python csv保存进数据库错误的一种原因

现象:linux上运行Python脚本出错 

这里的问题是csv文件保存不进csv文件,下面的代码是我之前写的,这里我把user-agent删除了,你可以自己加上

# -*- coding: utf-8 -*-
import os
import json
import pandas as pd
import pymysql
import requests
import jsonpath


from datetime import datetime


# 定义文件名为每日的日期
filename = datetime.now().strftime("%Y-%m-%d") + '.csv'



# 请求数据
def get_data(url):
    headers_value = {
        'User-Agent': ""}  # 请求头,自行更改
    try:
        resp = requests.get(url, headers=headers_value)
    except requests.exceptions.ConnectionError as e:
        print("请求错误", e)
        resp = None
    print("china数据请求成功")
    return resp


# 爬取数据
def parse_data(resp):
    data = json.loads(resp.text)
    name = jsonpath.jsonpath(data, "$.data.areaTree[2].children[*].name")  # 省份名
    confirm = jsonpath.jsonpath(data, "$.data.areaTree[2].children[*].today.confirm")  # 现有确诊
    confirm_all = jsonpath.jsonpath(data, "$.data.areaTree[2].children[*].total.confirm")  # 累计确诊
    dead = jsonpath.jsonpath(data, "$.data.areaTree[2].children[*].total.dead")  # 累计死亡
    heal = jsonpath.jsonpath(data, "$.data.areaTree[2].children[*].total.heal")  # 累计治愈
    last_update_time = jsonpath.jsonpath(data, "$.data.areaTree[2].children[*].lastUpdateTime")#更新时间
    # 数据处理,空值设置为0
    for i in range(len(confirm)):
        if confirm[i] == None:
            confirm[i] = 0
        elif confirm_all[i] == None:
            confirm_all[i] = 0
        elif dead[i] == None:
            dead[i] == 0
        elif heal[i] == None:
            heal[i] = 0
        elif last_update_time[i]:
            print(last_update_time[i])
            arr = str(last_update_time[i]).split(" ")
            last_update_time[i] = arr[0]
    print("china 数据爬取成功!")
    print("name:", name)
    print("confirm:", confirm)
    print("confirm_all:", confirm_all)
    print("dead:", dead)
    print("heal:", heal)
    print("last_update_time", last_update_time)
    return name, confirm, confirm_all, dead, heal,last_update_time


# 数据储存csv
def save_csv(name, confirm, confirm_all, dead, heal,last_update_time):
    result = pd.DataFrame()
    result['省份'] = name
    result['新增确诊'] = confirm
    result['累计确诊'] = confirm_all
    result['死亡'] = dead
    result['治愈'] = heal
    result['更新日期'] = last_update_time
    result.to_csv("./" + filename, encoding='utf_8_sig', index=None)  # csv文件保存位置,自行更改
    print("中国数据保存csv成功")



# 数据存入mysql
def save_sql():

    data = pd.read_csv('./' + filename)  # scv文件读取位置,自行更改
    rows_nums = data.shape[0]
    db = pymysql.connect(host='127.0.0.1', port=3306, user='root', password='123456', db='mydb',
                         charset='utf8')  # 数据库用户名、密码,自行更改
    cursor = db.cursor()
    try:
        cursor.execute("drop table if exists now_Crawling")
        cursor.execute("CREATE TABLE now_Crawling("
                             "name VARCHAR (100), "
                             "confirm VARCHAR (100),"
                             "confirm_all VARCHAR (100),"
                             "dead VARCHAR (100),"
                             "heal VARCHAR (200),"
                             "last_update_time VARCHAR (100));")
        for i in range(rows_nums):
            sql = "INSERT INTO now_Crawling(name, confirm,confirm_all,dead,heal,last_update_time) VALUES (%s,%s,%s,%s,%s,%s);"
            cursor.execute(sql, (
                (str)(data.iloc[i, 0]), (str)(data.iloc[i, 1]), (str)(data.iloc[i, 2]), (str)(data.iloc[i, 3]),
                (str)(data.iloc[i, 4]),(str)(data.iloc[i, 5])))
        cursor.close()
        db.commit()
        print("中国数据保存已保存到mysql")
    except:
        db.rollback()
        print("ERROR")
        cursor.close()
    finally:
        cursor.close()





if __name__ == '__main__':
    url = "https://c.m.163.com/ug/api/wuhan/app/data/list-total?t=318268778643"  # 网易提供的疫情数据网址
    resp = get_data(url)
    name, confirm, confirm_all, dead, heal,last_update_time = parse_data(resp)

    save_csv(name, confirm, confirm_all, dead, heal,last_update_time)

    save_sql()

解决方法:初步排查create table语句已经执行,只是值未注入,所以不妨试一试修改数据库的表为utf-8

   cursor.execute("CREATE TABLE now_Crawling("
                             "name VARCHAR (100), "
                             "confirm VARCHAR (100),"
                             "confirm_all VARCHAR (100),"
                             "dead VARCHAR (100),"
                             "heal VARCHAR (200),"
                             "last_update_time VARCHAR (100))default charset = utf8mb4;")

 

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
Python编程,我们常常需要将数据保存到不同的储存方式。其种常见的数据保存方式是使用csv文件。CSV文件是一种以逗号分隔数据的文本文件,非常适合存储表格数据。你可以使用pandas库来保存数据为csv格式。比如,你可以使用以下代码将数据保存csv文件: import pandas as pd data = pd.DataFrame({"姓名":["mary","jack"],"年龄":[22,21]}) data.to_csv("info.csv", index=False) 上述代码将创建一个名为info.csv的文件,并将data的数据保存到这个文件。index=False表示不保存索引信息。 除了csv文件,你还可以将数据保存到其他类型的文件,例如Excel文件、JSON文件、SQLite数据库等。具体的保存方法可以参考相关文档和教程。<span class="em">1</span><span class="em">2</span><span class="em">3</span> #### 引用[.reference_title] - *1* [python数据数据存储-五种使用python储存数据的方式](https://blog.csdn.net/weixin_37988176/article/details/109372864)[target="_blank" data-report-click={"spm":"1018.2226.3001.9630","extra":{"utm_source":"vip_chatgpt_common_search_pc_result","utm_medium":"distribute.pc_search_result.none-task-cask-2~all~insert_cask~default-1-null.142^v93^chatsearchT3_1"}}] [.reference_item style="max-width: 50%"] - *2* *3* [python保存数据方法总结](https://blog.csdn.net/qq_52764364/article/details/129527369)[target="_blank" data-report-click={"spm":"1018.2226.3001.9630","extra":{"utm_source":"vip_chatgpt_common_search_pc_result","utm_medium":"distribute.pc_search_result.none-task-cask-2~all~insert_cask~default-1-null.142^v93^chatsearchT3_1"}}] [.reference_item style="max-width: 50%"] [ .reference_list ]

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值