[Clojure] Data Collection and Data Analysis on the music of www.xiami.com - Part 1

Source code

https://github.com/bluesilence/Lisp/tree/master/clojure/projects/xiami-crawler

Background

Once upon a time...I was starving at new music that I might like at xiami.com.

www.xiami.com is the only website I used to listening to music.

• Why I love xiami:

1. Tremendous amount of songs/albums/artists
2. User-friendly interface
3. You don't have to log on to play a song

• Why I wrote this xiami crawler:

1. The recommendation system is based on songs you have listened. It doesn't solve my problem of discovering songs that I haven't listened but I might like.
2. Furthermore, xiami doesn't support searching by the ranking on the rating of songs, which make it difficult to find the best music.
3. The list of representative songs provided by xiami(http://www.xiami.com/genre?spm=a1z1s.3057857.1110925385.5.8zjSZT) haven't been ranked, or are ranked by some mysterious algorithm. I want to know the best rating songs of each genre.
4. Fortunately, I was learning Clojure, and there was a piece of code of a crawler made by Clojure in the book "Clojure Programming".

So why not try to write a crawler that collects music infos from xiami and do the  DIY data analysis to find my favorite music?

Urls

The urls on xiami is quite simple and organized:

Songs

http://www.xiami.com/song/$song-id

Eg. http://www.xiami.com/song/1

Albums

http://www.xiami.com/song/$album-id

Eg. http://www.xiami.com/album/1

Artists

http://www.xiami.com/song/$artist-id

Eg. http://www.xiami.com/artist/1

So the crawler doesn't have to crawl the page for new urls. It just need to increment the id to get the url of the next page.

Data Schema

Here are the schema of the data needed to analyse:

Songs

1. song-id
2. song-name
3. song-hot
4. album-id

Albums

1. album-id
2. album-name
3. album-category
4. album-genre
5. artist-id
6. album-value

Artists

1. artist-id
2. artist-name