一、Django虚拟环境搭建
1. pip install virtualenv
新建文件夹,CMD进入此文件夹,创建虚拟环境:virtualenv .
文件夹下会生成3个文件:
2. 进入虚拟环境 .\Scripts\activate
命令行前会有文件夹名称显示,即当前在该虚拟环境中:
3. 安装Django
进入虚拟环境,pip install django
pip freeze 查看当前环境的安装
4. 后续爬虫需要,安装lxml,requests
pip install lxml
pip install requests
二、创建项目
1. 创建Django项目
Pycharm打开此项目,运行,报下错误:
安装migrate,去除报警
2. 在同级目录下,新建app
在mysite的settings.py中的INSTALLED_APPS添加‘kilter'
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'kilter',
]
三、配置urls
1. mysite\urls.py
from django.contrib import admin
from django.urls import path, include
urlpatterns = [
path('admin/', admin.site.urls),
path('', include('kilter.urls')), # 请求跳转到kilter中的urls
]
2. kilter\urls.py
from django.urls import path
from . import views
urlpatterns = [
path('', views.homepage, name="homepage"),
path('allNovels/', views.allNovels, name="allNovels")
]
四、创建Html模板
1. 新建模板文件夹templates,添加显示的html文件
2. 使用bootstrap框架,为html添加元素
base.html(基页)
使用bootstrap框架中最基础的模板,添加了导航条(navbar)、巨幕(jumbotron)、主页内容由homepage.html显示
<!doctype html>
<html lang="en">
<head>
<!-- Required meta tags -->
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<!-- Bootstrap CSS -->
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bootstrap@4.4.1/dist/css/bootstrap.min.css" integrity="sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh" crossorigin="anonymous">
<title>Kilter小说网</title>
</head>
<body>
<nav class="navbar navbar-expand-lg navbar-dark bg-dark">
<a class="navbar-brand" href="">主页</a>
<button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#navbarSupportedContent"
aria-controls="navbarSupportedContent" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
<div class="collapse navbar-collapse" id="navbarSupportedContent">
<ul class="navbar-nav mr-auto">
<li class="nav-item">
<a class="nav-link" href="allNovels/">全部小说</a>
</li>
</ul>
<form name="form" class="form-inline my-2 my-lg-0" action="http://www.xbiquge.la/modules/article/waps.php" method="post">
<input type="text" class="form-control mr-sm-2" value="请填入书名和作者"
name="searchkey" class="search" id="wd" baidusug="2" autocomplete="off">
<button id="sss" class="btn btn-outline-success my-2 my-sm-0" type="submit"> 搜 索</button>
</form>
</div>
</nav>
<br/>
<div class="jumbotron jumbotron-fluid">
<div class="container">
<h1 class="display-4">Kilter小说网</h1>
<p class="lead">Django搭建简单的小说网,仅学习使用</p>
</div>
</div>
<div class="container">
{% block content %}
{% endblock %}
</div>
<!-- Optional JavaScript -->
<!-- jQuery first, then Popper.js, then Bootstrap JS -->
<script src="https://cdn.jsdelivr.net/npm/jquery@3.4.1/dist/jquery.slim.min.js"
integrity="sha384-J6qa4849blE2+poT4WnyKhv5vZF5SrPo0iEjwBvKU7imGFAV0wwj1yYfoRSJoZ+n"
crossorigin="anonymous"></script>
<script src="https://cdn.jsdelivr.net/npm/popper.js@1.16.0/dist/umd/popper.min.js"
integrity="sha384-Q6E9RHvbIyZFJoft+2mJbHaEWldlvI9IOYy5n3zV9zzTtmI3UksdQRVvoxMfooAo"
crossorigin="anonymous"></script>
<script src="https://cdn.jsdelivr.net/npm/bootstrap@4.4.1/dist/js/bootstrap.min.js"
integrity="sha384-wfSDF2E50Y2D1uUdj0O3uMBJnjuUD4Ih7YwaYd1iqfktj0Uod8GCExl3Og8ifwB6"
crossorigin="anonymous"></script>
</body>
</html>
homepage.html 使用卡片组件(Card)展示信息,同时实现数据交互,使用js处理文本内容过长部分显示“......”
{% extends 'base.html' %}
{% block content %}
<div class="container">
<div class="row">
{% for img,name,describe,url in content %}
<div class="col-sm">
<div class="card" style="width:230px; height:500px;">
<img src="{{img}}" style="width:230px; height:300px;" class="card-img-top" alt="Card image cap">
<div class="card-body">
<h5 class="card-title">{{name}}</h5>
<script>
var text="{{describe}}";
if (text.length>30)
{
text=text.substring(0,30)+"......";
}
document.write(text);
document.getElementById("p1").innerHTML=text;
</script>
<p id="p1" class="card-text"></p>
<a href="{{url}}" class="btn btn-primary">访问</a>
</div>
</div>
<br>
</div>
{% endfor %}
</div>
</div>
<p>{{imgs}}</p>
{% endblock %}
allNovels.html 使用画面轮播(Carousel)、表格(tables)
<!DOCTYPE html>
<html lang="en">
<head>
<!-- Required meta tags -->
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<!-- Bootstrap CSS -->
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bootstrap@4.4.1/dist/css/bootstrap.min.css" integrity="sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh" crossorigin="anonymous">
<title>全部小说</title>
</head>
<body>
<div class="container">
<div id="carouselExampleIndicators" class="carousel slide" data-ride="carousel">
<ol class="carousel-indicators">
<li data-target="#carouselExampleIndicators" data-slide-to="0" class="active"></li>
<li data-target="#carouselExampleIndicators" data-slide-to="1"></li>
<li data-target="#carouselExampleIndicators" data-slide-to="2"></li>
<li data-target="#carouselExampleIndicators" data-slide-to="3"></li>
</ol>
<div class="carousel-inner">
<div class="carousel-item active">
<img src="http://www.xbiquge.la/files/article/image/10/10489/10489s.jpg"
style="height: 300px;width: 1200px">
<div class="carousel-caption">
<h3>三寸人间</h3>
<p>描述文字!</p>
</div>
</div>
<div class="carousel-item">
<img src="http://www.xbiquge.la/files/article/image/0/951/951s.jpg"
style="height: 300px;width: 1200px">
<div class="carousel-caption">
<h3>伏天氏</h3>
<p>描述文字!</p>
</div>
</div>
<div class="carousel-item">
<img src="http://www.xbiquge.la/files/article/image/5/5541/5541s.jpg"
style="height: 300px;width: 1200px">
<div class="carousel-caption">
<h3>异世邪君</h3>
<p>描述文字!</p>
</div>
</div>
<div class="carousel-item">
<img src="http://www.xbiquge.la/files/article/image/5/5395/5395s.jpg"
style="height: 300px;width: 1200px">
<div class="carousel-caption">
<h3>凡人修仙传</h3>
<p>描述文字!</p>
</div>
</div>
</div>
<a class="carousel-control-prev" href="#carouselExampleIndicators" role="button" data-slide="prev">
<span class="carousel-control-prev-icon" aria-hidden="true"></span>
<span class="sr-only">Previous</span>
</a>
<a class="carousel-control-next" href="#carouselExampleIndicators" role="button" data-slide="next">
<span class="carousel-control-next-icon" aria-hidden="true"></span>
<span class="sr-only">Next</span>
</a>
</div>
<br/>
<h>
全部小说
</h>
<table class="table">
<thead class="thead-dark">
<tr>
<th scope="col">#</th>
<th scope="col">
<a href="http://www.xbiquge.la/xuanhuanxiaoshuo/">玄幻小说</a>
</th>
<th scope="col">
<a href="http://www.xbiquge.la/xiuzhenxiaoshuo/">修真小说</a>
</th>
<th scope="col">
<a href="http://www.xbiquge.la/dushixiaoshuo/">都市小说</a>
</th>
</tr>
</thead>
<tbody>
<tr>
<th scope="row">1</th>
<td>
<a href="http://www.xbiquge.la/15/15409/">牧神记</a>
</td>
<td>
<a href="http://www.xbiquge.la/2/2210/">全职法师</a>
</td>
<td>
<a href="http://www.xbiquge.la/14/14930/">元尊</a>
</td>
</tr>
<tr>
<th scope="row">2</th>
<td>
<a href="http://www.xbiquge.la/0/656/">莽荒纪</a>
</td>
<td>
<a href="http://www.xbiquge.la/0/205/">执魔</a>
</td>
<td>
<a href="http://www.xbiquge.la/0/215/">魔天记</a>
</td>
</tr>
<tr>
<th scope="row">3</th>
<td>
<a href="http://www.xbiquge.la/20/20948/">最佳女婿</a>
</td>
<td>
<a href="http://www.xbiquge.la/36/36060/">豪婿</a>
</td>
<td>
<a href="http://www.xbiquge.la/3/3120/">极品全能学生</a>
</td>
</tr>
</tbody>
</table>
<br/>
<nav aria-label="Page navigation example">
<ul class="pagination">
<li class="page-item">
<a class="page-link" href="#" aria-label="Previous">
<span aria-hidden="true">«</span>
</a>
</li>
<li class="page-item"><a class="page-link" href="http://www.xbiquge.la/fenlei/3_1.html">1</a></li>
<li class="page-item"><a class="page-link" href="http://www.xbiquge.la/fenlei/3_2.html">2</a></li>
<li class="page-item"><a class="page-link" href="http://www.xbiquge.la/fenlei/3_3.html">3</a></li>
<li class="page-item">
<a class="page-link" href="#" aria-label="Next">
<span aria-hidden="true">»</span>
</a>
</li>
</ul>
</nav>
</div>
<!-- Optional JavaScript -->
<!-- jQuery first, then Popper.js, then Bootstrap JS -->
<script src="https://cdn.jsdelivr.net/npm/jquery@3.4.1/dist/jquery.slim.min.js"
integrity="sha384-J6qa4849blE2+poT4WnyKhv5vZF5SrPo0iEjwBvKU7imGFAV0wwj1yYfoRSJoZ+n"
crossorigin="anonymous"></script>
<script src="https://cdn.jsdelivr.net/npm/popper.js@1.16.0/dist/umd/popper.min.js"
integrity="sha384-Q6E9RHvbIyZFJoft+2mJbHaEWldlvI9IOYy5n3zV9zzTtmI3UksdQRVvoxMfooAo"
crossorigin="anonymous"></script>
<script src="https://cdn.jsdelivr.net/npm/bootstrap@4.4.1/dist/js/bootstrap.min.js"
integrity="sha384-wfSDF2E50Y2D1uUdj0O3uMBJnjuUD4Ih7YwaYd1iqfktj0Uod8GCExl3Og8ifwB6"
crossorigin="anonymous"></script>
</body>
</html>
五、爬取数据
views.py
from django.shortcuts import render
import requests
import json
from lxml import html
# Create your views here.
def homepage(request):
headers = {
'Accept': 'text / html, application / xhtml + xml, application / xml;q = 0.9, image / webp, image / apng, * / *;q = 0.8',
'Cookie': '_abcde_qweasd=0; Hm_lvt_169609146ffe5972484b0957bd1b46d6=1587264547; BAIDU_SSP_lcr=https://www.baidu.com/link?url=uLcdKPYEycqTY_ZNILg7FpWtP29NV3xcIQWRAsDtcrO&wd=&eqid=d1b646a70000536f000000055e9bbc22; _abcde_qweasd=0; bdshare_firstime=1587264547833; Hm_lpvt_169609146ffe5972484b0957bd1b46d6=1587268535',
'Host': 'www.xbiquge.la',
'Referer': 'http: // www.xbiquge.la /',
'User - Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36'
}
# 获取小说url
response = requests.get("http://www.xbiquge.la/paihangbang/", headers=headers, timeout=(3, 7))
etree = html.etree
xml = etree.HTML(response.text)
urls = xml.xpath('//*[@id="main"]/div[2]/ul[1]//li/a/@href')
# 获取小说对应图像
imgs_url = []
# 名字
names = []
# 简介
describes=[]
for url in urls:
response_img = requests.get(url, headers=headers, timeout=(3, 7))
response_img.encoding = "utf-8"
xml_img = etree.HTML(response_img.text)
imgs_url.append(xml_img.xpath('//div[@id="fmimg"]/img/@src')[0])
names.append(xml_img.xpath('//*[@id="info"]/h1/text()')[0])
describes.append(xml_img.xpath('//div[@id="intro"]/p[2]/text()'))
content = zip(imgs_url, names, describes, urls)
context = {}
context["content"] = content
return render(request, 'homepage.html', context)
def allNovels(request):
return render(request,'allNovels.html',{})
六、网页展示
小说链接到新笔趣阁网站小说,仅做学习使用。
参考:https://www.bilibili.com/video/BV1KJ41117HL
问题:
1. cmd运行ok python manage.py runserve ,
pycharm运行报错“ModuleNotFoundError: No module named 'django',
ImportError: Couldn't import Django. Are you sure it's installed and available on your PYTHONPATH environment variable? Did you forget to activate a virtual environment?”
解决:
a. File->settings->project interpreter ->show all
添加虚拟环境中的python路径
apply
b. run->Edit configurations->parameters中添加“runserver”
参考:https://blog.csdn.net/xty00/article/details/95354518
2. 报错:TimeoutError: [WinError 10060] 由于连接方在一段时间后没有正确答复或连接的主机没有反应,连接尝试失败
解决:
requests.get(url, headers=headers, proxies=proxies, timeout=5)
如上,在request.get()函数中加个timeout参数就行了。
timeout 是用作设置响应时间(单位:秒)的,可以设置一个float或者一个tuple参数,响应时间分为连接时间和读取时间,timeout(3,7)表示的连接时间是3秒,响应时间是7秒。
3. from lxml import etree 报错,无法导入etree
解决:
from lxml import html
etree = html.etree
xml = etree.HTML(response.text)
4. Django变量在前端js中不起作用
解决:
js是具备作用范围的,由一对{}来决定。而django的内置标签和变量也都是由大括号括起来的,这样就会起冲突。
使用时需要将django的内置标签和变量用引号括起来
eg: 变量:var a = "{{name}}"
静态资源:src = "{% static path %}"