思路
首先需要获得访问者的ip所在城市，之后通过该城市获得当前天气信息，而天气数据通过定时任务的形式定时爬取到本地数据库。
实现过程
获取访问者ip对应城市信息
通过太平洋网络提供的API接口以在页面script访问的形式获得json
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta name="referrer" content="no-referrer"/>
<title></title>
</head>
<body>
</body>
<script>
function ipJson(obj) {
console.log(obj)
//获得的json,注意在head里面添加<meta name="referrer" content="no-referrer"/>
}
</script>
<script src="http://whois.pconline.com.cn/ipJson.jsp?callback=ipJson"></script>
</html>
获得的json数据大致是这样的，接下来我用到的只是citycode这个参数，通过这个参数去请求后台接口获取当地天气
{
"ip":"210.51.167.169",
"pro":"北京市",
"proCode":"110000",
"city":"北京市",
"cityCode":"110000",
"region":"大兴区",
"regionCode":"110115",
"addr":"北京市大兴区 亦庄联通通泰IDC机房(亦庄经济技术开发区北环东路一号联通国际大厦)",
"regionNames":"",
"err":""
}
数据库
通过各种百度寻找，找到了这个行政编码数据库，然后根据这个天气预报插件定制中各个城市对应的值，增加了几个所需要的字段，这里我只填写了市级对应的weatherid
- WeatherId：天气插件的城市id
- HighTemperature：最高气温
- LowTemperature：最低气温
- CityShow：显示的城市名称
- WeatherText：天气状况
- Icon：天气状况图标
这是数据库地址
天气抓取
通过对这个天气插件的爬取获得到对应地区的天气信息，然后存储到数据库，传递参数py=城市id就可以获得对应城市天气了，该部分运用了jsoup这个html解析器，通过分析天气插件的html，可以将我们所需要的部分从中取出。
这里由于需要通过代理连接网络，所以写了连接代理的部分，这样就获得到了对应城市的天气信息。
public class GetWeather {
public Document getDocument (String url){
System.setProperty("http.proxySet", "true");
System.getProperties().put("http.proxyHost", "代理地址");
System.getProperties().put("http.proxyPort", "代理端口");
setDefaultAuthentication();
String agent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)"
+ " Chrome/56.0.2924.87 Safari/537.36" ;
try {
return Jsoup.connect(url).ignoreContentType(true)
.userAgent(agent)
.ignoreHttpErrors(true)
.timeout(10000).get();
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
public static void setDefaultAuthentication() {
BasicAuthenticator auth = new BasicAuthenticator("代理账号", "密码");
Authenticator.setDefault(auth);
}
public static class BasicAuthenticator extends Authenticator {
String userName;
String password;
public BasicAuthenticator(String userName, String password) {
this.userName = userName;
this.password = password;
}
@Override
protected PasswordAuthentication getPasswordAuthentication() {
return new PasswordAuthentication(userName, password.toCharArray());
}
}
public static MyWeather getWeather (String py,int time){
GetWeather t = new GetWeather();
String url = "http://i.tianqi.com/index.php?c=code&a=getcode&id=35&py=" + py;
Document doc = t.getDocument(url);
Boolean flag = false;
// 判断是否访问成功
try {
flag = new String(doc.select("title").text().getBytes(),"utf-8").indexOf(new String("天气预报".getBytes(),"utf-8")) != -1 && doc != null;
} catch (IOException e) {
e.printStackTrace();
}
MyWeather myWeather = new MyWeather();
// 获取目标HTML代码
if(flag) {
Elements elements1 = doc.select("li");
// 高的温度
Elements elements2 = elements1.select("p");
String highTemperature = elements2.get(0).text() + "°C";
// 低的温度
String lowTemperature1 = elements2.get(1).text().substring(1);
String lowTemperature = lowTemperature1.substring(0, lowTemperature1.length() - 1) + "°C";
// 城市
String cityShow = elements2.get(2).text();
// 天气及风向
String setWeatherText = elements1.get(1).text();
Elements elements3 = doc.select(".pngtqico");
String icon = elements3.attr("src").substring(elements3.attr("src").lastIndexOf("/")+1);
myWeather.setCityShow(cityShow);
myWeather.setHighTemperature(highTemperature);
myWeather.setLowTemperature(lowTemperature);
myWeather.setWeatherText(setWeatherText);
myWeather.setIcon(icon);
try {
Thread.sleep(time);
} catch (InterruptedException e) {
e.printStackTrace();
}
return myWeather;
} else {
return null;
}
}
}
定时任务及天气获取
定时任务这里运用了Spring Task，做了一个简单的每隔一个小时进行一次的抓取
AlarmTask
@Component
@EnableScheduling
public class AlarmTask {
@Resource
private WeatherService weatherService;
/**
* 默认是fixedDelay 上一次执行完毕时间后执行下一轮
*/
@Scheduled(cron = "0 0 * * * ?")
public void run() throws InterruptedException {
weatherService.start();
}
}
WeatherService
@Service
public class WeatherService {
@Resource
MyWeatherMapper mapper;
public MyWeather getWeather(int Id){
MyWeather myWeather = mapper.getWeather(Id);
//所在城市对应城市id是否为空，如果为空就找他的父级城市
if(myWeather != null && ("null".equals(String.valueOf(myWeather.getCityShow())) || "".equals(myWeather.getCityShow()+""))){
myWeather = getWeather(myWeather.getParentId());
}
return myWeather;
}
public void start(){
SimpleDateFormat df = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");//设置日期格式
long start = System.currentTimeMillis();
System.out.println("=====================开始执行天气抓取："+ df.format(start) +"========================");
List<MyWeather> weatherList = mapper.getAllWeatherId();//获得数据库中所有的城市id
ExecutorService exec = Executors.newFixedThreadPool(5);//5个线程执行
for (MyWeather weather:weatherList){
try {
exec.execute(() -> {
MyWeather getWeatherList = GetWeather.getWeather(weather.getWeatherId(),1000);//遍历爬取，延时1000毫秒
getWeatherList.setWeatherId(weather.getWeatherId());
if (getWeatherList != null){
mapper.updateWeather(getWeatherList);//将获取到的天气信息存储到数据库
System.out.println(getWeatherList.getCityShow());
}
});
}catch (Exception e){
e.printStackTrace();
}
}
exec.shutdown();
while (true){
if (exec.isTerminated()){
break;
}
}
long end = System.currentTimeMillis();
System.out.println("=====================本次执行结束："+ df.format(end) +"========================");
System.out.println("用时："+((float)(end-start)/1000)+"秒");
}
}
MyWeatherMapper
@Mapper
public interface MyWeatherMapper {
List<MyWeather> getAllWeatherId();
void updateWeather(MyWeather getWeatherList);
MyWeather getWeather(int Id);
}
mapper.xml
<select id="getAllWeatherId" resultMap="MyWeather">
select distinct WeatherId from weather
<where>
WeatherId IS NOT NULL and WeatherId != ''
</where>
</select>
<select id="getWeather" resultMap="MyWeather">
select * from weather
<where>
Id=#{Id}
</where>
</select>
<update id="updateWeather" parameterType="com.tanzhi.weather.entity.MyWeather">
update weather
<set>
<if test="CityShow != null">CityShow=#{CityShow},</if>
<if test="HighTemperature != null">HighTemperature=#{HighTemperature},</if>
<if test="LowTemperature != null">LowTemperature=#{LowTemperature},</if>
<if test="WeatherText != null">WeatherText=#{WeatherText},</if>
<if test="Icon != null">Icon=#{Icon}</if>
</set>
where WeatherId=#{WeatherId}
</update>
485
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
