使用工具charles,主要是用来获取访问的API数据的,为啥说初级呢,因为并没有解决mas加密,这样的话只能刷到50条新数据,然后mas就失效了
之前有篇文章 抖音API分析 大概梳理了视频地址获取方式,懒得继续写,正好最近又有兴趣,继续续上
现在既然都分析清楚了,下面就是模拟客户端获取数据下载了
private static String url = "https://aweme.snssdk.com/aweme/v1/feed/?iid=32142611788&ac=4G&os_api=18&app_name=aweme&channel=App%20Store&idfa=67642C64-6404-403A-8B0D-31A059C3A2BD&device_platform=iphone&build_number=17909&vid=9D61EDED-6680-471A-A134-D1C96399BB83&openudid=9a661cd28951ab44f0870508f7af64dfb9b5dc36&device_type=iPhone8,2&app_version=1.7.9&device_id=50862505508&version_code=1.7.9&os_version=10.2.1&screen_width=1125&aid=1128&count=6&feed_style=0&max_cursor=0&min_cursor=0&pull_type=0&type=0&user_id=96840867747&volume=0.00&mas=000171d64eb699219ac45f410bcc83d1accd3ee629e6ec51f8ceb1&as=a1859114c0ed4b50731900&ts=1531121872";
public static void main(String[] args) throws Exception{
CloseableHttpClient httpClient = org.apache.http.impl.client.HttpClients.createDefault();
RequestConfig requestConfig = RequestConfig.custom()
.setSocketTimeout(15000)
.setConnectTimeout(15000)
.build();
HttpGet get = new HttpGet(url);
get.setConfig(requestConfig);
get.setHeader("Accept","*/*");
get.setHeader("User-Agent","Aweme/1.7.9 (iPhone; iOS 10.2.1; Scale/3.00)");
get.setHeader("Cookie","这里填写个人自己的cookie");
CloseableHttpResponse response = httpClient.execute(get);
HttpEntity entity = response.getEntity();
String content = EntityUtils.toString(entity, "gbk");
JSONObject jsonObject = JSON.parseObject(content);
if(jsonObject.getInteger("status_code") == 0){
JSONArray jsonArray = jsonObject.getJSONArray("aweme_list");
for(int i=0;i<5;i++){
JSONObject detail = jsonArray.getJSONObject(i);
String url = detail.getJSONObject("video").getJSONObject("play_addr_lowbr").getJSONArray("url_list").get(0).toString();
System.out.println(url);
}
}else{
System.out.println("is not 0");
System.exit(0);
}
}
跑出来的就是
然后怎么用呢,看下对应的api
其实相当于根据视频的一些唯一标识去获取对应的视频真实地址,本来想用jsoup,想想这么简单整个正则得了
String str = "<a href=\"http://v3-dy-x.ixigua.com/1fc1320a2829de5164add4678abf2192/5b431eef/video/m/220785bb6e0882346e8aff9b7613756f4e71158d06e000055b44e3f1db4/\">Found</a>.";
String regEx = "href=\"(.*?)\">";
Pattern pattern = Pattern.compile(regEx);
Matcher matcher = pattern.matcher(str);
if(matcher.find()){
System.out.println(matcher.group(1));
}
小插曲,使用httpclient访问的时候,接受到302后自动跳转了,我说怎么一个简单的代码跑的时间感觉有点长呢,禁用302跳转就好了
private static String filePath = "/Users/xingzhe/douyin";
private static String url = "https://aweme.snssdk.com/aweme/v1/play/?video_id=v0200f660000bctojepcgf31ghmrsmdg&line=0&ratio=720p&media_type=4&vr_type=0&test_cdn=None&improve_bitrate=0";
public static void main(String[] args) throws Exception{
CloseableHttpClient httpClient = org.apache.http.impl.client.HttpClients.createDefault();
RequestConfig requestConfig = RequestConfig.custom()
.setSocketTimeout(15000)
.setConnectTimeout(15000)
.setRedirectsEnabled(false)
.build();
HttpGet get = new HttpGet(url);
get.setConfig(requestConfig);
get.setHeader("Accept","*/*");
get.setHeader("User-Agent","Aweme/1.7.9 (iPhone; iOS 10.2.1; Scale/3.00)");
CloseableHttpResponse response = httpClient.execute(get);
System.out.println(response.getStatusLine().getStatusCode());
if(response.getStatusLine().getStatusCode() != 302){
System.exit(0);
}
HttpEntity entity = response.getEntity();
String content = EntityUtils.toString(entity, "gbk");
String detail = getVideo(content);
download(detail);
}
private static String getVideo(String str){
String regEx = "href=\"(.*?)\">";
// 编译正则表达式
Pattern pattern = Pattern.compile(regEx);
// 忽略大小写的写法
// Pattern pat = Pattern.compile(regEx, Pattern.CASE_INSENSITIVE);
Matcher matcher = pattern.matcher(str);
if(matcher.find()){
return matcher.group(1);
}
return null;
}
private static void download(String videoUrl) throws Exception{
// 构造URL
URL url = new URL(videoUrl);
// 打开连接
URLConnection con = url.openConnection();
//设置请求超时为5s
con.setConnectTimeout(5*1000);
// 输入流
InputStream is = con.getInputStream();
// 1K的数据缓冲
byte[] bs = new byte[1024];
// 读取到的数据长度
int len;
// 输出的文件流
File sf=new File(filePath);
long time = System.currentTimeMillis()/1000;
OutputStream os = new FileOutputStream(sf.getPath()+"/"+time+".mp4");
// 开始读取
while ((len = is.read(bs)) != -1) {
os.write(bs, 0, len);
}
// 完毕,关闭所有链接
os.close();
is.close();
}
用上面这个代码就能下载到视频了,写的很随意主要就是完成功能,视频的文件名都直接用的时间戳,批量的话也简单,for循环一下就行,随便搞了下
记录一个小问题,最开始下载使用的 url.openConnection() 这种最原始的方式,下了一会发现403了,又转回使用httpclient下载,get方法加上header,主要是
User-Agent Aweme/1.7.9 (iPhone; iOS 10.2.1; Scale/3.00)
没再出现403
代码所有需要访问url的地方都直接重新new一个httpclient,其他的对象也没有复用也没有排重,下载也没有多线程,娱乐之作,爬抖音最重要的还是在as和mas的生成,因为我测试发现大概下载60或者多少的时候就会返回"status_code": 2151,需要重新抓包搞下mas和as,显然对效率来说是不可接受的,以后再有时间研究下as的生成方式,就这样了