理解http request headers中Referer||User-Agent||Cookie...的意义

• 缘起

  用Python写爬虫，对于AJAX内容，需要在request.get(url, headers)中传入headers,需要了解headers中各项的意义。

• Request header 请求头

  HTTP header fields are components of the header section of request and response messages in the Hypertext Transfer Protocol (HTTP). They define the operating parameters of an HTTP transaction.

  Standard request fields and Standard response fields

  Referer : This is the address of the previous web page from which a link to the currently requested page was followed.

  User-Agent : The user agent string of the user agent. It is the fields which User agent identification is transmitted. When a software agent operates in a network protocol, it often identifies itself, its application type, operating system, software vendor, or software revision, by submitting a characteristic identification string to its operating peer.

  Cookie : An HTTP cookie previously sent by the server with Set-Cookies.

• HTTP cookie

  An HTTP cookie(also called web cookie, Internet cookie, brower cookie, or simply cookie) is a small piece of data stored on the user’s computer by the web browser while browsing a website.

  Cookies were designed to be a reliable mechanism for websites to remember stateful information (such as items added in the shopping cart in an online store) or to record the user’s browsing activity(including clicking particular buttons, logging in, or recording which pages were visited in the past). They can also be used to remember pieces of information that user previously entered into form fields, such as names, addresses, passwords, and payment card numbers.