python 相关性分析切点寻找_利用Python进行数据分析分析-Ted Talks Data Analysis

Ted Talks

环境：Python2.7 Anaconda Jupyter Notebook

数据集: https://www.kaggle.com/rounakbanik/ted-talks

导入相应的库

%matplotlib inline

import pandas as pd

import numpy as np

from scipy import stats

import matplotlib.pyplot as plt

import seaborn as sns #matplotlib的默认作图风格就会被覆盖成seaborn的格式

import json

from pandas.io.json import json_normalize

from wordcloud import WordCloud, STOPWORDS #词云

df = pd.read_csv('ted_main.csv')

df.colums #数据集的首行表头

Index([u'comments', u'description', u'duration', u'event', u'film_date',

u'languages', u'main_speaker', u'name', u'num_speaker',

u'published_date', u'ratings', u'related_talks', u'speaker_occupation',

u'tags', u'title', u'url', u'views'],

dtype='object')

#调整表头顺序

df = df[['name', 'title', 'description', 'main_speaker', 'speaker_occupation', 'num_speaker', 'duration', 'event', 'film_date', 'published_date', 'comments', 'tags', 'languages', 'ratings', 'related_talks', 'url', 'views']]

Features Available

name: The official name of the TED Talk. Includes the title and the speaker.

title: The title of the talk

description: A blurb of what the talk is about.

main_speaker: The first named speaker of the talk.

speaker_occupation: The occupation of the main speaker.

num_speaker: The number of speakers in the talk.

duration: The duration of the talk in seconds.

event: The TED/TEDx event where the talk took place.

film_date: The Unix timestamp of the filming.

published_date: The Unix timestamp for the publication of the talk on TED.com

comments: The number of first level comments made on the talk.

tags: The themes associated with the talk.

languages: The number of languages in which the talk is available.

ratings: A stringified dictionary of the various ratings given to the talk (inspiring, fascinating, jaw dropping, etc.)

related_talks: A list of dictionaries of recommended talks to watch next.

url: The URL of the talk.

views: The number of views on the talk.

image.png

convert the Unix timestamps into a human readable format

import datetime

df['film_date'] = df['film_date'].apply(lambda x: datetime.datetime.fromtimestamp( int(x)).strftime('%d-%m-%Y'))

df['published_date'] = df['published_date'].apply(lambda x: datetime.datetime.fromtimestamp( int(x)).strftime('%d-%m-%Y'))

image.png

image.png

#根据views量排序前15行数据

pop_talks = df[['title','main_speaker','views','film_date']].sort_values('views',ascending=False)[:15]

pop_talks

image.png

<> 点击量最高4722w的观看记录

#切分main_speaker的前三个字母,新增一列abbr数据

pop_talks["abbr"] = pop_talks['main_speaker'].apply(lambda x: x[:3])

pop_talks.head()

image.png

sns.set_style("whitegrid")

plt.figure(figsize=(10,6))

sns.barplot(x='abbr',y='views',data=pop_talks)

image.png

ken 位居views点击量榜首

sns.distplot(df['views'])

image.png

sns.distplot(df[df['views'] < 0.4e7]['views'])

image.png

sns.distplot(df[(df['views'] > 0.5e4)&(df['views'] < 0.4e7)]['views']) #多条件布尔索引

image.png

df['views'].describe()

image.png

TED Talks的views平均数为160万。中位数是112万。这表明TED Talks的普及程度非常高。我们也注意到大部分Talks的views不到400万。我们将在后面的章节中将这个框图作为框图的切点。

sns.distplot(df['comments'])

image.png

sns.distplot(df[df['comments'] < 500]['comments'])

image.png

sns.jointplot(x='views', y='comments', data=df)

image.png

df['comments'].describe()

image.png

平均每次Talks都有191.5条评论。假设评论是建设性的批评，我们可以得出结论，TED社区高度参与讨论循环谈判。与评论相关的标准偏差很大。事实上，它甚至比意味着这些措施可能对异常值敏感的意思更大。我们将绘制这个图来检查分布的性质。谈话的最小评论数是2，最大数是6404.该范围是6402 ..尽管如此，最少的数字可能是最近发表的谈话的结果。

df[['views', 'comments']].corr() #相关系数函数

image.png

如散点图和相关矩阵所示，相关系数略大于0.5。这表明两个数量之间的中等到强相关性。如上所述，这个结果是相当期待的。现在让我们来检查一下有史以来最受关注的10个会谈的观点和评论数量。

df[['title', 'main_speaker','views', 'comments']].sort_values('comments', ascending=False).head(10)

image.png

df['dis_quo'] = df['comments']/df['views'] #新增一列‘dis_quo’

#评论数/点击量之比 前10行

df[['title', 'main_speaker','views', 'comments', 'dis_quo', 'film_date']].sort_values('dis_quo', ascending=False).head(10)

image.png

month_order = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']

day_order = ['Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat', 'Sun']

df['month'] = df['film_date'].apply(lambda x: month_order[int(x.split('-')[1]) - 1])

df['month'].head()

month_df = pd.DataFrame(df['month'].value_counts()).reset_index()

month_df.columns = ['month', 'talks']

sns.barplot(x='month',y='talks',data=month_df,order=month_order)

image.png

二月显然是最受欢迎的会议，而八月和一月是最不受欢迎的。二月份的人气很大程度上是由于二月份举行的官方会议。让我们只检查TEDX talks的分配。

#使用datetime的strtime方法获取一个日期是周几

def getday(x):

day, month, year = (int(i) for i in x.split('-'))

answer = datetime.date(year, month, day).strftime("%A")

return answer[:3]

#使用datetime的weekday方法获取一个日期是一周里的第几天，用这个当索引在day_order里取相应的value值

def getday2(x):

day, month, year = (int(i) for i in x.split('-'))

answer = datetime.date(year, month, day).weekday()

return day_order[answer]

image.png

df['day'] = df['film_date'].apply(getday) #新增一列day列

day_df = pd.DataFrame(df['day'].value_counts()).reset_index()

day_df.columns = ['day', 'talks']

sns.barplot(x='day', y='talks', data=day_df, order=day_order)

image.png

day的分布几乎是一个钟声曲线，星期三和星期四是最受欢迎的日子，周日是最不受欢迎的。这是非常有趣的，因为我认为大部分的会议都会在周末的某个时候发生。

df['year'] = df['film_date'].apply(lambda x: x.split('-')[2])

year_df = pd.DataFrame(df['year'].value_counts().reset_index())

year_df.columns = ['year', 'talks']

plt.figure(figsize=(18,5))

sns.pointplot(x='year', y='talks', data=year_df)

image.png

如预期的那样，自1984年成立以来，TED Talks的数量逐年增加。如果在2009年进行，数量会急剧增加。了解2009年成为TED Talks数量增加两倍以上的转折点的原因可能是有趣的。自2009年以来，TED Talks数量几乎不变。

months = {'Jan': 1, 'Feb': 2, 'Mar': 3, 'Apr': 4, 'May': 5, 'Jun': 6, 'Jul': 7, 'Aug': 8, 'Sep': 9, 'Oct': 10, 'Nov': 11, 'Dec': 12}

hmap_df = df.copy()

hmap_df['film_date'] = hmap_df['film_date'].apply(lambda x: month_order[int(x.split('-')[1]) - 1] + " " + str(x.split('-')[2]))

hmap_df = pd.pivot_table(hmap_df[['film_date', 'title']], index='film_date', aggfunc='count').reset_index()

hmap_df['month_num'] = hmap_df['film_date'].apply(lambda x: months[x.split()[0]])

hmap_df['year'] = hmap_df['film_date'].apply(lambda x: x.split()[1])

hmap_df = hmap_df.sort_values(['year', 'month_num'])

hmap_df = hmap_df[['month_num', 'year', 'title']]

hmap_df = hmap_df.pivot('month_num', 'year', 'title')

hmap_df = hmap_df.fillna(0)

image.png