pandas 数据载入、存储及文件格式(文本格式数据的读写—XML 和 HTML：网络抓取)

本文链接：https://blog.csdn.net/weixin_38477351/article/details/107525200

文本格式数据的读写—XML 和 HTML：网络抓取

pandas 拥有很多可以对 HTML 和 XML 格式进行读取、写入数据的库，例如 lxml（http://lxml.de）、Beautiful Soup 和 html5lib。尽管 lxml 是相对更快的库，但其他库可以更好地处理异常的 HTML 或 XML 文件。

一、pandas.read_html 解析 HTML

pandas 的内建函数 read_html 可以使用 lxml 和 Beautiful Soup 等库将 HTML 中的表自动解析为 DataFrame 对象。

1、安装 read_html 所使用的附加库

(1) 方案一：

conda install lxml

conda install beautifulsoup4 html5lib

(2) 方案二：

pip install lxml

pip install beautifulsoup4 html5lib

2、pandas.read_html 解析美国 FDIC 政府机构的银行倒闭数据的 HTML

(1) pandas.read_html 函数有很多选项，但是默认情况下，它会搜索并尝试解析所有包含在 <table> 标签中的表格型数据，返回的结果是 DataFrame 对象的列表

(2) 对数据进行清理和分析，计算每年银行倒闭的数量

二、lxml.objectify 解析 XML

XML（eXtensible Markup Language）是另一种常用的结构化数据格式，它使用元组数据支持分层、嵌套数据。

1、纽约大都会交通局（MTA）发布的关于其公交、火车服务（http://web.mta.info/developers/download.html）的数据集

(1) XML 格式：

<INDICATOR>
  <INDICATOR_SEQ>28445</INDICATOR_SEQ>
  <PARENT_SEQ></PARENT_SEQ>
  <AGENCY_NAME>Metro-North Railroad</AGENCY_NAME>
  <INDICATOR_NAME>On-Time Performance (West of Hudson)</INDICATOR_NAME>
  <DESCRIPTION>Percent of commuter trains that arrive at their destinations within 5 minutes and 59 seconds of the scheduled time. West of Hudson services include the Pascack Valley and Port Jervis lines. Metro-North Railroad contracts with New Jersey Transit to operate service on these lines.
</DESCRIPTION>
  <PERIOD_YEAR>2008</PERIOD_YEAR>
  <PERIOD_MONTH>1</PERIOD_MONTH>
  <CATEGORY>Service Indicators</CATEGORY>
  <FREQUENCY>M</FREQUENCY>
  <DESIRED_CHANGE>U</DESIRED_CHANGE>
  <INDICATOR_UNIT>%</INDICATOR_UNIT>
  <DECIMAL_PLACES>1</DECIMAL_PLACES>
  <YTD_TARGET>95.00</YTD_TARGET>
  <YTD_ACTUAL>96.90</YTD_ACTUAL>
  <MONTHLY_TARGET>95.00</MONTHLY_TARGET>
  <MONTHLY_ACTUAL>96.90</MONTHLY_ACTUAL>
</INDICATOR>
<INDICATOR>
  <INDICATOR_SEQ>28445</INDICATOR_SEQ>
  <PARENT_SEQ></PARENT_SEQ>
  <AGENCY_NAME>Metro-North Railroad</AGENCY_NAME>
  <INDICATOR_NAME>On-Time Performance (West of Hudson)</INDICATOR_NAME>
  <DESCRIPTION>Percent of commuter trains that arrive at their destinations within 5 minutes and 59 seconds of the scheduled time. West of Hudson services include the Pascack Valley and Port Jervis lines. Metro-North Railroad contracts with New Jersey Transit to operate service on these lines.
</DESCRIPTION>
  <PERIOD_YEAR>2008</PERIOD_YEAR>
  <PERIOD_MONTH>2</PERIOD_MONTH>
  <CATEGORY>Service Indicators</CATEGORY>
  <FREQUENCY>M</FREQUENCY>
  <DESIRED_CHANGE>U</DESIRED_CHANGE>
  <INDICATOR_UNIT>%</INDICATOR_UNIT>
  <DECIMAL_PLACES>1</DECIMAL_PLACES>
  <YTD_TARGET>95.00</YTD_TARGET>
  <YTD_ACTUAL>96.00</YTD_ACTUAL>
  <MONTHLY_TARGET>95.00</MONTHLY_TARGET>
  <MONTHLY_ACTUAL>95.00</MONTHLY_ACTUAL>
</INDICATOR>

(2) JSON 格式：

[
    {
        'AGENCY_NAME': 'Metro-North Railroad',
        'INDICATOR_NAME': 'On-Time Performance (West of Hudson)',
        'DESCRIPTION': 'Percent of commuter trains that arrive at their destinations within 5 minutes and 59 seconds of the scheduled time. West of Hudson services include the Pascack Valley and Port Jervis lines. Metro-North Railroad contracts with New Jersey Transit to operate service on these lines.\n',
        'PERIOD_YEAR': 2008,
        'PERIOD_MONTH': 1,
        'CATEGORY': 'Service Indicators',
        'FREQUENCY': 'M',
        'DESIRED_CHANGE': 'U',
        'INDICATOR_UNIT': '%',
        'YTD_TARGET': 95.0,
        'YTD_ACTUAL': 96.9,
        'MONTHLY_TARGET': 95.0,
        'MONTHLY_ACTUAL': 96.9
    },
    {
        'AGENCY_NAME': 'Metro-North Railroad',
        'INDICATOR_NAME': 'On-Time Performance (West of Hudson)',
        'DESCRIPTION': 'Percent of commuter trains that arrive at their destinations within 5 minutes and 59 seconds of the scheduled time. West of Hudson services include the Pascack Valley and Port Jervis lines. Metro-North Railroad contracts with New Jersey Transit to operate service on these lines.\n',
        'PERIOD_YEAR': 2008,
        'PERIOD_MONTH': 2,
        'CATEGORY': 'Service Indicators',
        'FREQUENCY': 'M',
        'DESIRED_CHANGE': 'U',
        'INDICATOR_UNIT': '%',
        'YTD_TARGET': 95.0,
        'YTD_ACTUAL': 96.0,
        'MONTHLY_TARGET': 95.0,
        'MONTHLY_ACTUAL': 95.0
    }
]

2、使用 lxml.objectify 解析纽约大都会交通局（MTA）发布的关于其公交、火车服务的数据集文件