java jsoup解析url,Java JSoup错误提取URL

I'm creating an application which will enable me to fetch values from a specific website to the console. The value is from a element and I'm using JSoup.

My challenge has to do with this error:

Error fetching URL

Here is my Java code:

public class TestSl {

public static void main(String[] args) throws IOException {

Document doc = Jsoup.connect("https://stackoverflow.com/questions/11970938/java-html-parser-to-extract-specific-data").get();

Elements spans = doc.select("span[class=hidden-text]");

for (Element span: spans) {

System.out.println(span.text());

}

}

}

And here is the error on Console:

Exception in thread "main" org.jsoup.HttpStatusException: HTTP error fetching URL. Status=403, URL=Java Html parser to extract specific data?

at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:590)

at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:540)

at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:227)

at org.jsoup.helper.HttpConnection.get(HttpConnection.java:216)

at TestSl.main(TestSl.java:19)

What am I doing wrong and how can I resolve it?

解决方案

Set the user-agent header:

.userAgent("Mozilla")

Example:

Document document = Jsoup.connect("https://stackoverflow.com/questions/11970938/java-html-parser-to-extract-specific-data").userAgent("Mozilla").get();

Elements elements = document.select("span.hidden-text");

for (Element element : elements) {

System.out.println(element.text());

}

Stack Exchange

Inbox

Reputation and Badges