<code class="hljs python has-numbering" style="display: block; padding: 0px; background: transparent; color: inherit; box-sizing: border-box; font-family: "Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal;"></code>
因为最近需要用到股市的各种指数数据,刚开始想的是从同花顺等交易软件直接导出,结果发现要用滚轮滚到最开始的日期,这过于繁琐了,我对于这种重复性的劳动一向不耐烦,而且这种方法在以后每日更新的时候也很不方便。所以我把视线转向了网上的各种api。网上比较普遍的主要有两种,一种是新浪的api,一种是雅虎的api。新浪的api很方便,速度也很快,不过就网上的资料而言,似乎只能提取当天的数据。雅虎的api功能更齐全,但是连接速度比较慢,有时候一个连接请求都要10多秒甚至20秒(原谅我的渣网速-,-)这对于需要大量股票数据的我来说显然不可接受。后来发现从网易财经可以下载股票和指数的历史数据。通过在chrome的调试工具中观察请求信息,发现网址为
http://quotes.money.163.com/service/chddata.html?code=0000001&start=19901219&end=20150911&fields=TCLOSE;HIGH;LOW;TOPEN;LCLOSE;CHG;PCHG;VOTURNOVER;VATURNOVER
这里0000001指的是上证指数。注意这串数字要分0和000001两部分看。0代表sh,1代表sz。所以0000001其实是sh000001的意思。同理,0 000300也就是sh000300 沪深300的代码。后面的start和end没什么问题。fields选项中,TCLOSE,HIGH,LOW,TOPEN分别表示当日的收盘,最高,最低,开盘价;LCLOSE表示昨日收盘价。CHG,PCHG,VOTURNOVER,VAT分别表示涨跌额,涨跌幅,成交量,成交金额。如果你在浏览器中直接输入网址的话,会下载一个csv文件。如果直接用urllib.request来打开的话,会得到一长串字符串,里面包含了所有的数据,如下所示。需要自己用str.split(‘\r\n’)来截取
日期,股票代码,名称,收盘价,最高价,最低价,开盘价,前收盘,涨跌额,涨跌幅,成交量,成交金额
2015-09-11,’000001,上证指数,3200.234,3223.762,3163.449,3189.479,3197.893,2.341,0.0732,224557822,2.52769467178e+11
2015-09-10,’000001,上证指数,3197.893,3243.281,3178.904,3190.553,3243.089,-45.196,-1.3936,273261759,2.99581090523e+11
2015-09-09,’000001,上证指数,3243.089,3256.743,3165.696,3182.552,3170.452,72.637,2.2911,375327978,4.12991428267e+11
2015-09-08,’000001,上证指数,3170.452,3174.709,3011.117,3054.444,3080.42,90.032,2.9227,255415465,2.63910382052e+11
2015-09-07,’000001,上证指数,3080.42,3217.579,3066.304,3149.38,3160.167,-79.747,-2.5235,296468114,3.02689722336e+11
有几点需要指出,这里的指数代码前会加个 ’ 比如上面是’000001要把那个 ’ 去掉。还有,有些时候涨跌额和涨跌幅的值是None,简直莫名其妙,只能根据当日收盘价和前一天的收盘价自己补全;还有成交金额有几天的值也是None,这个似乎没法自己补,处理的时候需要特别注意
下面是代码。
<code class="hljs python has-numbering" style="display: block; padding: 0px; background: transparent; color: inherit; box-sizing: border-box; font-family: "Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> urllib,time, <span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">def</span> <span class="hljs-title" style="box-sizing: border-box;">get_page</span><span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">(url)</span>:</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#获取页面数据</span> req=urllib.request.Request(url,headers={ <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'Connection'</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'Keep-Alive'</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'Accept'</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'text/html, application/xhtml+xml, */*'</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'Accept-Language'</span>:<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'zh-CN,zh;q=0.8'</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'User-Agent'</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko'</span> }) opener=urllib.request.urlopen(req) page=opener.read() <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> page <span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">def</span> <span class="hljs-title" style="box-sizing: border-box;">get_index_history_byNetease</span><span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">(index_temp)</span>:</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">""" :param index_temp: for example, 'sh000001' 上证指数 :return: """</span> index_type=index_temp[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>] index_id=index_temp[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>:] <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> index_type==<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'sh'</span>: index_id=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'0'</span>+index_id <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> index_type==<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"sz"</span>: index_id=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'1'</span>+index_id url=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'http://quotes.money.163.com/service/chddata.html?code=%s&start=19900101&end=%s&fields=TCLOSE;HIGH;LOW;TOPEN;LCLOSE;CHG;PCHG;VOTURNOVER;VATURNOVER'</span>%(index_id,time.strftime(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"%Y%m%d"</span>)) page=get_page(url).decode(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'gb2312'</span>) <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#该段获取原始数据</span> page=page.split(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'\r\n'</span>) col_info=page[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>].split(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">','</span>) <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#各列的含义</span> index_data=page[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>:] <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#真正的数据</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#为了与现有的数据库对应,这里我还修改了列名,大家不改也没关系</span> col_info[col_info.index(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'日期'</span>)]=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'交易日期'</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#该段更改列名称</span> col_info[col_info.index(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'股票代码'</span>)]=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'指数代码'</span> col_info[col_info.index(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'名称'</span>)]=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'指数名称'</span> col_info[col_info.index(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'成交金额'</span>)]=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'成交额'</span> index_data=[x.replace(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"'"</span>,<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">''</span>) <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> x <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">in</span> index_data] <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#去掉指数编号前的“'”</span> index_data=[x.split(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">','</span>) <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> x <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">in</span> index_data] index_data=index_data[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>:index_data.__len__()-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>] <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#最后一行为空,需要去掉</span> pos1=col_info.index(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'涨跌幅'</span>) pos2=col_info.index(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'涨跌额'</span>) posclose=col_info.index(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'收盘价'</span>) index_data[index_data.__len__()-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>][pos1]=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#最下面行涨跌额和涨跌幅为None改为0</span> index_data[index_data.__len__()-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>][pos2]=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> i <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">in</span> range(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>,index_data.__len__()-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>): <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#这两列中有些值莫名其妙为None 现在补全</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> index_data[i][pos2]==<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'None'</span>: index_data[i][pos2]=float(index_data[i][posclose])-float(index_data[i+<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>][posclose]) <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> index_data[i][pos1]==<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'None'</span>: index_data[i][pos1]=(float(index_data[i][posclose])-float(index_data[i+<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>][posclose]))/float(index_data[i+<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>][posclose]) <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># print(col_info)</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> [index_data,col_info]</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right: 1px solid rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right: 1px solid rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li></ul>
运行get_index_history_byNetease(‘sh000001’) 就能够得到上证指数所有的历史数据