1.数据可视化
kaggle中数据解释：https://www.kaggle.com/c/titanic/data
数据形式：
titanic_data
读取数据，并显示数据信息
data_train = pd.read_csv("./data/train.csv")
print(data_train.info())
数据结果如下：
RangeIndex: 891 entries, 0 to 890
Data columns (total 12 columns):
PassengerId 891 non-null int64
Survived 891 non-null int64
Pclass 891 non-null int64
Name 891 non-null object
Sex 891 non-null object
Age 714 non-null float64
SibSp 891 non-null int64
Parch 891 non-null int64
Ticket 891 non-null object
Fare 891 non-null float64
Cabin 204 non-null object
Embarked 889 non-null object
数据解释：
复制代码
PassengerId => 乘客ID
Survive => 乘客是否生还（仅在训练集中有，测试集中没有）
Pclass => 乘客等级(1/2/3等舱位)
Name => 乘客姓名
Sex => 性别
Age => 年龄
SibSp => 堂兄弟/妹个数
Parch => 父母与小孩个数
Ticket => 船票信息
Fare => 票价
Cabin => 客舱
Embarked => 登船港口
复制代码
1.1 生存/死亡人数统计
P1
复制代码
# # 统计 存活/死亡 人数
def sur_die_analysis(data_train):
fig = plt.figure()
fig.set(alpha=0.2) # 设定图表颜色alpha参数
data_train.Survived.value_counts().plot(kind='bar')# 柱状图
plt.title(u"获救情况 (1为获救)") # 标题
plt.ylabel(u"人数")
plt.show()
复制代码
1.2 PClass
Pclass
复制代码
# PClass
def pclass_analysis(data_train):
fig = plt.figure()
fig.set(alpha=0.2) # 设定图表颜色alpha参数
sur_data = data_train.Pclass[data_train.Survived == 1].value_counts()
die_data = data_train.Pclass[data_train.Survived == 0].value_counts()
pd.DataFrame({'Survived':sur_data,'Died':die_data}).plot(kind='bar')
plt.ylabel(u"人数")
plt.title(u"乘客等级分布")
plt.show()
复制代码
通过数据分布可以很明显的看出 Pclass 为 1/2 的乘客存活率比 3 的高很多
1.3 Sex
Psex
复制代码
#Sex
def sex_analysis(data_train):
no_survived_g = data_train.Sex[data_train.Survived == 0].value_counts()
no_survived_g.to_csv("no_survived_g.csv")
survived_g = data_train.Sex[data_train.Survived == 1].value_counts()
df_g = pd.DataFrame({'Survived': survived_g, 'Died': no_survived_g})
df_g.plot(kind='bar', stacked=True)
plt.title('性别存活率分析')
plt.xlabel('People')
plt.ylabel('Survive')
plt.show()
复制代码
女性的存活率比男性高
1.4 Age
Page
复制代码
# age : 将年龄分成十段，分别统计 存活人数和死亡人数
def age_analysis(data_train):
data_series = pd.DataFrame(columns=['Survived', 'dies'])
cloms = []
for num in range(0, 10):
clo = "" + str(num * 10) + "-" + str((num + 1) * 10)
cloms.append(clo)
sur_df = data_train.Age[(10 * (num + 1) > data_train.Age) & (10 * num < data_train.Age) & (data_train.Survived == 1)].shape[0]
die_df = data_train.Age[(10 * (num + 1) > data_train.Age) & (10 * num < data_train.Age) & (data_train.Survived == 0)].shape[0]
data_series.loc[num] = [sur_df,die_df]
data_series.index = cloms
data_series.plot(kind='bar', stacked=True)
plt.ylabel(u"存活率") # 设定纵坐标名称
plt.grid(b=True, which='major', axis='y')
plt.title(u"按年龄看获救分布")
plt.show()
复制代码
低年龄段的获救的百分比明显占的比例较多
1.5 Family ： SibSp + Parch
定义Family项，代表家庭成员数量，并离散分类为三个等级：
0： 代表没有任何成员
1： 1-4
2： > 4
PFamliy
复制代码
# Family: Sibsp + Parch 家庭成员人数
def family_analysis(data_train):
data_train['Family'] = data_train['SibSp'] + data_train['Parch']
data_train.loc[(data_train.Family == 0), 'Family'] = 0
data_train.loc[((data_train.Family > 0) & (data_train.Family < 4)), 'Family'] = 1
data_train.loc[((data_train.Family >= 4)), 'Family'] = 2
no_survived_g = data_train.Family[data_train.Survived == 0].value_counts()
survived_g = data_train.Family[data_train.Survived == 1].value_counts()
df_g = pd.DataFrame({'Survived': survived_g, 'Died': no_survived_g})
df_g.plot(kind='bar', stacked=True)
plt.title('家庭成员分析')
plt.xlabel('等级：0-无 1-（1~4） 2-（>4）')
plt.ylabel('存活情况')
plt.show()
复制代码
由于数据分布很不均衡，sibsp 是否和存活率的关系，可以将所有列都除以该列总人数。这里不再赘述。
1.6 Fare
费用统计：
PClassB
当费用升高到一定时，存活人数已经超过了死亡人数
PFare
复制代码
# Fare
def fare_analysis(data_train):
# data_train.Fare[data_train.Survived == 1].plot(kind='kde')
# data_train.Fare[data_train.Survived == 0].plot(kind='kde')
# data_train["Fare"].plot(kind='kde')
# plt.legend(('survived', 'died','all'), loc='best')
# plt.show()
data_train['NewFare'] = data_train['Fare']
data_train.loc[(data_train.Fare < 50), 'NewFare'] = 0
data_train.loc[((data_train.Fare>=50) & (data_train.Fare<100)), 'NewFare'] = 1
data_train.loc[((data_train.Fare >= 100) & (data_train.Fare < 150)), 'NewFare'] = 2
data_train.loc[((data_train.Fare >= 150) & (data_train.Fare < 200)), 'NewFare'] = 3
data_train.loc[(data_train.Fare >= 200), 'NewFare'] = 4
no_survived_g = data_train.NewFare[data_train.Survived == 0].value_counts()
survived_g = data_train.NewFare[data_train.Survived == 1].value_counts()
df_g = pd.DataFrame({'Survived': survived_g, 'Died': no_survived_g})
df_g.plot(kind='bar', stacked=True)
plt.title('费用-生存分析')
plt.xlabel('费用等级')
plt.ylabel('存活情况')
plt.show()
复制代码
很明显可以看出 费用等级较高的人存活率会高很多。
优化：
上述只是任意的选取了五个费用段，作为五类，但是具体是多少类才能最好的拟合数据？
这里可以通过聚类的方法查找最佳的分类个数，再将每个费用数据映射为其中一类：
复制代码
def fare_kmeans(data_train):
for i in range(2,10):
clusters = KMeans(n_clusters=i)
clusters.fit(data_train['Fare'].values.reshape(-1,1))
# intertia_ 参数是衡量聚类的效果，越大则表明效果越差
print("" + str(i) + "" + str(clusters.inertia_))
复制代码
打印结果：
复制代码
2 846932.9762272763
3 399906.26606199215
4 195618.50643749788
5 104945.73652631264
6 52749.474696547695
7 35141.316334118805
8 26030.553497795216
9 19501.242236941747
复制代码
由此可以看出看出当 类别数为 5 时分类的效果最好。所以这里将所有的费用映射到为这五类。
复制代码
#将费用进行聚类，发现 类别数为 5 时聚合的效果最好
def fare_kmeans(data_train):
clusters = KMeans(n_clusters=5)
clusters.fit(data_train['Fare'].values.reshape(-1, 1))
predict = clusters.predict(data_train['Fare'].values.reshape(-1, 1))
print(predict)
data_train['NewFare'] = predict
print(data_train[['NewFare','Survived']].groupby(['NewFare'],as_index=False).mean())
print("" + str(clusters.inertia_))
复制代码
等级映射后每个等级的存活率如下：(效果明显比上面随便分类的好)
复制代码
NewFare Survived
0 0 0.319832
1 1 0.647059
2 2 0.606557
3 3 1.000000
4 4 0.757576
复制代码
1.7 Embarked
PEmbark
复制代码
#Embarked 上船港口情况
def embarked_analysis(data_train):
no_survived_g = data_train.Embarked[data_train.Survived == 0].value_counts()
survived_g = data_train.Embarked[data_train.Survived == 1].value_counts()
df_g = pd.DataFrame({'Survived': survived_g, 'Died': no_survived_g})
df_g.plot(kind='bar', stacked=True)
plt.title('登陆港口-存活情况分析')
plt.xlabel('Embarked')
plt.ylabel('Survive')
plt.show()
复制代码
至于就登陆港口而言，三个港口并看不出明显的差距，C港生还率略高于S港与Q港。
2. 数据预处理
由开头部分数据信息可以看出，有几栏的数据是部分缺失的： Age / Cabin / Embarked
对于缺失数据这里选择简单填充的方式进行处理：（可以以中值/均值/众数等方式填充）
同时对费用进行分类
复制代码
def dataPreprocess(df):
df.loc[df['Sex'] == 'male', 'Sex'] = 0
df.loc[df['Sex'] == 'female', 'Sex'] = 1
# 由于 Embarked中有两个数据未填充，需要先将数据填满
df['Embarked'] = df['Embarked'].fillna('S')
# 部分年龄数据未空， 填充为 均值
df['Age'] = df['Age'].fillna(df['Age'].median())
df.loc[df['Embarked']=='S', 'Embarked'] = 0
df.loc[df['Embarked'] == 'C', 'Embarked'] = 1
df.loc[df['Embarked'] == 'Q', 'Embarked'] = 2
df['FamilySize'] = df['SibSp'] + df['Parch']
df['IsAlone'] = 0
df.loc[df['FamilySize']==0,'IsAlone'] = 1
df.drop('FamilySize',axis = 1)
df.drop('Parch',axis=1)
df.drop('SibSp',axis=1)
return fare_kmeans(df)
def fare_kmeans(data_train):
clusters = KMeans(n_clusters=5)
clusters.fit(data_train['Fare'].values.reshape(-1, 1))
predict = clusters.predict(data_train['Fare'].values.reshape(-1, 1))
data_train['NewFare'] = predict
data_train.drop('Fare')
# print(data_train[['NewFare','Survived']].groupby(['NewFare'],as_index=False).mean())
# print(" " + str(clusters.inertia_))
return data_train
复制代码
这里对与分类特征通过了普通的编码方式进行实现，也可以通过onehot编码使每种分类之间的间隔相等。
3. 特征选择
上述感性的认识了各个特征与存活率之间的关系，其实sklearn库中提供了对每个特征打分的函数，可以很方便的看出各个特征的重要性
复制代码
predictors = ["Pclass", "Sex", "Age", "NewFare", "Embarked",'IsAlone']
# Perform feature selection
selector = SelectKBest(f_classif, k=5)
selector.fit(data_train[predictors], data_train["Survived"])
# Plot the raw p-values for each feature,and transform from p-values into scores
scores = -np.log10(selector.pvalues_)
# Plot the scores. See how "Pclass","Sex","Title",and "Fare" are the best?
plt.bar(range(len(predictors)),scores)
plt.xticks(range(len(predictors)),predictors, rotation='vertical')
plt.show()
复制代码
skselectors
上图可以看到输入的6个特征中那些特征比较重要
4. 线性回归建模
复制代码
def linearRegression(df):
predictors = ['Pclass', 'Sex', 'Age', 'IsAlone', 'NewFare', 'Embarked']
#predictors = ['Pclass', 'Sex', 'Age', 'IsAlone', 'NewFare', 'EmbarkedS','EmbarkedC','EmbarkedQ']
alg = LinearRegression()
X = df[predictors]
Y = df['Survived']
X_train,X_test,Y_train,Y_test = train_test_split(X,Y,test_size=0.2)
# 打印 训练集 测试集 样本数量
print (X_train.shape)
print (Y_train.shape)
print (X_test.shape)
print (Y_test.shape)
# 进行拟合
alg.fit(X_train, Y_train)
print (alg.intercept_)
print (alg.coef_)
Y_predict = alg.predict(X_test)
Y_predict[Y_predict >= 0.5 ] = 1
Y_predict[Y_predict < 0.5] = 0
acc = sum(Y_predict==Y_test) / len(Y_predict)
return acc
复制代码
测试模型预测准确率： 0.79
5. 随机森林建模
选取最有价值的5个特征进行模型训练，并验证模型的效果：
复制代码
def randomForest(data_train):
# Pick only the four best features.
predictors = ["Pclass", "Sex", "NewFare", "Embarked", 'IsAlone']
X_train, X_test, Y_train, Y_test = train_test_split(data_train[predictors], data_train['Survived'], test_size=0.2)
alg = RandomForestClassifier(random_state=1, n_estimators=50, min_samples_split=8, min_samples_leaf=4)
alg.fit(X_train, Y_train)
Y_predict = alg.predict(X_test)
acc = sum(Y_predict == Y_test) / len(Y_predict)
return acc
复制代码
经过测试该模型的准确率为 0.811
初步原因分析： 选取的5个特征中没有Age，Age可能因为缺失很大部分数据对预测的准确率有一定的影响。
代码已经提交git： https://github.com/lsfzlj/kaggle
欢迎指正交流
参考：
https://blog.csdn.net/han_xiaoyang/article/details/49797143
https://blog.csdn.net/CSDN_Black/article/details/80309542
https://www.kaggle.com/sinakhorami/titanic-best-working-classifier
梦想不是浮躁,而是沉淀和积累https://www.cnblogs.com/NeilZhang/p/10092681.html
