目的:爬取成员信息,并导出
此处用的是hutool爬取+easy-poi导出,jsoup同理
相关jia包
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.12</version>
</dependency>
<dependency>
<groupId>cn.hutool</groupId>
<artifactId>hutool-all</artifactId>
<version>5.8.2</version>
</dependency>
<!--easypoi-->
<dependency>
<groupId>cn.afterturn</groupId>
<artifactId>easypoi-base</artifactId>
<version>3.2.0</version>
</dependency>
<dependency>
<groupId>cn.afterturn</groupId>
<artifactId>easypoi-web</artifactId>
<version>3.2.0</version>
</dependency>
<dependency>
<groupId>cn.afterturn</groupId>
<artifactId>easypoi-annotation</artifactId>
<version>3.2.0</version>
</dependency>
群成员相关接口:
https://qun.qq.com/member.html#gid=(qq群号码,直接写群号码)
进入F12看到相关信息
然后把url和相关参数放到postman测试一下:
注意cookie不要忘记了
可以得到json数据:
直接上代码:
@Test
void testGetQQInfoHutool() throws IOException {
/**
* 按照F12得出结论 群员数量66 以及相关 st end的取值
* st 0 21 42 63
* end 20 41 62 66
*/
List<QQMember> listEnd = new ArrayList<>();
//通过循环改变分页查询条件,获取所有的群成员信息
//i的值是 群成员数量/分页数
for (int i = 0; i < 4; i++) {
int st = (20 * i) + i;
if (i == 0) {
st = 0;
}
int end = st + 20;
// String url = "https://qun.qq.com/cgi-bin/qun_mgr/search_group_members?gc=438672685&sort=0&bkn=47110324&end=20&st=0";
String url = "https://qun.qq.com/cgi-bin/qun_mgr/search_group_members?gc=438672685&sort=0&bkn=1414405685&" + "end=" + end + "&st=" + st;
String cookie = "pgv_pvid=7740519374; RK=3asJQf311j; ptcz=ce3c723b8b1ea9b5f6284b1b4a79766a7905de9ff4c04cdbd7221ddc9b57cecb; luin=o2517865739; lskey=0001000097cfa5444e34636bee5b837ee6d899da77a7b37b0c9359cba6b099f2cb0001c15020791e01ffa7bc; uin=o2517865739; p_uin=o2517865739; traceid=0b047ebf48; pgv_info=ssid=s7328931644; fqm_pvqid=757f1925-5961-4478-8ba3-47167d3ff801; fqm_sessionid=47fd6f8c-9d34-41fe-a87f-7a2fe0f6f599; skey=@GoPVTRbdH; pt4_token=paLWXr2lT6J*VVAKnFzmPwbpV40mSCyFRt*k4U2WWlg_; p_skey=8saip9YaONClsvQQVQgeJAhWFg-9XzUnZi49CToqEas_";
String accept = HttpRequest.post(url)
.header("accept", "*/*")
.header("accept-encoding", "gzip, deflate, br")
.header("accept-language", "zh-CN,zh;q=0.9")
.header("content-length", "45")
.header("content-type", "application/x-www-form-urlencoded; charset=UTF-8")
.header("user-agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36")
.header("cookie", cookie)
.timeout(10000)
.execute()
.body();
//将数据转换成list
JSONObject jsonObject = JSONUtil.parseObj(accept);
JSONArray mems = jsonObject.getJSONArray("mems");
List<QQMember> list1 = mems.toList(QQMember.class);
//通过接口知道时间是: 时间戳少了3位,我默认加上了3个0,测试是正常时间,转换yyyy-MM-dd 格式输出打印 具体打印成什么样,自定义
List<QQMember> list = list1.stream().map(t -> {
t.setJoinTime(new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(Long.valueOf(t.getJoinTime() + "000")));
t.setLastSpeakTime(new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(Long.valueOf(t.getLastSpeakTime() + "000")));
return t;
}).collect(Collectors.toList());
listEnd.addAll(list);
}
//控制台打印语句 无用
listEnd.stream().forEach(t -> System.out.println("QQ号是:" + t.getUin() +
" 昵称是:" + t.getCard() +
" 入群时间:" + t.getJoinTime() + "最后一次说话时间:" + t.getLastSpeakTime()));
//这边可以不用写,我为了预防重复数据,用于去重的
ArrayList<QQMember> listEnd1 = listEnd.stream().collect(
Collectors.collectingAndThen(Collectors.toCollection(() -> new TreeSet<>(Comparator.comparing(tc -> tc.getUin()))), ArrayList::new)
);
Workbook workbook =
ExcelExportUtil.exportExcel(new ExportParams("内 卷群", "群员信息"), QQMember.class, listEnd1);
FileOutputStream os = new FileOutputStream("src/main/resources/别卷了4.xlsx");
workbook.write(os);
os.close();
}
最终,查看Excel数据
完美,成功获取群成员信息了