Theoretical learning

Understanding ROC

This week's game evaluation mentioned the concept of ROC, which is actually very interesting. If a model is evaluated according to the accuracy rate, for example, there are 90 negatives and 10 positives. The first model thinks that everyone is negative, the second model recognizes 5 positives and the others are negative, the accuracy rate of the two models is only 5%, but the actual effect is very different. In addition, the cost of not identifying positive and not identifying negative in many problems is also different (such as cancer detection), Therefore, the concept of ROC is introduced from the medical field.

True positive TP is correctly predicted to be positive, false positive FP is incorrectly predicted to be positive (actually negative), FN is incorrectly predicted to be negative (actually positive), TN is correctly predicted to be negative.
ROC has two coordinate axes. The vertical axis is TPR, which is the ratio of true positive to all positive, the horizontal axis is FPR, and the ratio of false positive to all negative.

It passes through (0,0) (1,1). The larger the area of y=x, the better the evaluation. In other words, it is better to be an isosceles triangle, which can be understood as the false positive rate and the true positive rate is 1 (that is, the problem needs to find out every positive). Changing the upper and lower limits of the model can correspond to each point on the curve, which can be understood as the rise of the false positive rate (more strict), The true Yang rate is also rising. The area enclosed by the curve and y=x indicates the effect of the model. The larger the better.

Decision tree and random forest

Because I haven't repaired data mining before, I'd better supplement the basic knowledge.
First, X has n independent random events, and its information is,

log2 is used in this place because the information in the general computer is expressed to the power of 2, which is easier to calculate.
Information entropy H (X) is expressed as, and p is the probability of occurrence

Why can this formula express information entropy? Entropy refers to the degree of confusion. Here, it refers to the degree of uncertainty of an event. If an event is certain to happen and is certain not to happen, its entropy should be small, while only the events with uncertain occurrence, that is, the events with probability not at both ends (0 or 1) should have greater entropy.
If log is negative, then the curve is [0,1] (infinite, 0) (p is a tree greater than 0 and less than 1), then the entropy with occurrence probability of 0 and 1 is 0, and the entropy of those events in the middle is large.
Decision tree is a way of thinking commonly used in our daily life, as shown in the figure below.

So we can know that there are many decisions in the decision tree. After these decisions, we classify the data. Then which decision to choose as the root node needs entropy.


Random forest can be vividly understood as democratic voting by many decision trees. In order to make their output inconsistent, it is necessary to use put back sampling.

practice

In the process of officially participating in the introduction competition this week, I realized the importance of EDA. It can be said that if it is done well, it will not be so difficult to put the model behind. It is the key to determine success or failure. Several key points are to complete the default value, discretize continuous data, correct outliers and regularize. I also learned a very effective method is to divide boxes. A special point is the mapping of time information.
We have preliminarily practiced xgboost and catboost, which will be further studied next week.

Submit results

import pandas as pd
import os
import gc
import lightgbm as lgb
import xgboost as xgb
from catboost import CatBoostRegressor
from sklearn.linear_model import SGDRegressor, LinearRegression, Ridge
from sklearn.preprocessing import MinMaxScaler
import math
import numpy as np
from tqdm import tqdm
from sklearn.model_selection import StratifiedKFold, KFold
from sklearn.metrics import accuracy_score, f1_score, roc_auc_score, log_loss
import matplotlib.pyplot as plt
import time
import warnings
warnings.filterwarnings('ignore')

train = pd.read_csv('train.csv')
testA = pd.read_csv('testA.csv')

data = pd.concat([train, testA], axis=0, ignore_index=True)

data['employmentLength'].replace(to_replace='10+ years', value='10 years', inplace=True)
data['employmentLength'].replace('< 1 year', '0 years', inplace=True)


def employmentLength_to_int(s):
    if pd.isnull(s):
        return s
    else:
        return np.int8(s.split()[0])


data['employmentLength'] = data['employmentLength'].apply(employmentLength_to_int)
#print(data['employmentLength'].value_counts(dropna=False).sort_index())

data['earliesCreditLine'] = data['earliesCreditLine'].apply(lambda s: int(s[-4:]))
#print(data['earliesCreditLine'].describe())

cate_features = ['grade', 'subGrade', 'employmentTitle', 'homeOwnership', 'verificationStatus', 'purpose', 'postCode', 'regionCode', \
                 'applicationType', 'initialListStatus', 'title', 'policyCode']
# for f in cate_features:
#     print(f, 'number of types:', data[f].nunique())

data = pd.get_dummies(data, columns=['grade', 'subGrade', 'homeOwnership', 'verificationStatus', 'purpose', 'regionCode'], drop_first=True)

for f in ['employmentTitle', 'postCode', 'title']:
    data[f+'_cnts'] = data.groupby([f])['id'].transform('count')
    data[f+'_rank'] = data.groupby([f])['id'].rank(ascending=False).astype(int)
    del data[f]

features = [f for f in data.columns if f not in ['id','issueDate','isDefault']]

train = data[data.isDefault.notnull()].reset_index(drop=True)
test = data[data.isDefault.isnull()].reset_index(drop=True)

x_train = train[features]
x_test = test[features]

y_train = train['isDefault']


def cv_model(clf, train_x, train_y, test_x, clf_name):
    folds = 5
    seed = 2020
    kf = KFold(n_splits=folds, shuffle=True, random_state=seed)

    train = np.zeros(train_x.shape[0])
    test = np.zeros(test_x.shape[0])

    cv_scores = []

    for i, (train_index, valid_index) in enumerate(kf.split(train_x, train_y)):
        print('************************************ {} ************************************'.format(str(i + 1)))
        trn_x, trn_y, val_x, val_y = train_x.iloc[train_index], train_y[train_index], train_x.iloc[valid_index], \
                                     train_y[valid_index]

        if clf_name == "lgb":
            train_matrix = clf.Dataset(trn_x, label=trn_y)
            valid_matrix = clf.Dataset(val_x, label=val_y)

            params = {
                'boosting_type': 'gbdt',
                'objective': 'binary',
                'metric': 'auc',
                'min_child_weight': 5,
                'num_leaves': 2 ** 5,
                'lambda_l2': 10,
                'feature_fraction': 0.8,
                'bagging_fraction': 0.8,
                'bagging_freq': 4,
                'learning_rate': 0.1,
                'seed': 2020,
                'nthread': 28,
                'n_jobs': 24,
                'silent': True,
                'verbose': -1,
            }

            model = clf.train(params, train_matrix, 50000, valid_sets=[train_matrix, valid_matrix], verbose_eval=200,
                              early_stopping_rounds=200)
            val_pred = model.predict(val_x, num_iteration=model.best_iteration)
            test_pred = model.predict(test_x, num_iteration=model.best_iteration)

            # print(list(sorted(zip(features, model.feature_importance("gain")), key=lambda x: x[1], reverse=True))[:20])

        if clf_name == "xgb":
            train_matrix = clf.DMatrix(trn_x, label=trn_y)
            valid_matrix = clf.DMatrix(val_x, label=val_y)
            test_matrix = clf.DMatrix(test_x)

            params = {'booster': 'gbtree',
                      'objective': 'binary:logistic',
                      'eval_metric': 'auc',
                      'gamma': 1,
                      'min_child_weight': 1.5,
                      'max_depth': 5,
                      'lambda': 10,
                      'subsample': 0.7,
                      'colsample_bytree': 0.7,
                      'colsample_bylevel': 0.7,
                      'eta': 0.04,
                      'tree_method': 'exact',
                      'seed': 2020,
                      'nthread': 36,
                      "silent": True,
                      }

            watchlist = [(train_matrix, 'train'), (valid_matrix, 'eval')]

            model = clf.train(params, train_matrix, num_boost_round=50000, evals=watchlist, verbose_eval=200,
                              early_stopping_rounds=200)
            val_pred = model.predict(valid_matrix, ntree_limit=model.best_ntree_limit)
            test_pred = model.predict(test_matrix, ntree_limit=model.best_ntree_limit)

        if clf_name == "cat":
            params = {'learning_rate': 0.05, 'depth': 5, 'l2_leaf_reg': 10, 'bootstrap_type': 'Bernoulli',
                      'od_type': 'Iter', 'od_wait': 50, 'random_seed': 11, 'allow_writing_files': False}

            model = clf(iterations=20000, **params)
            model.fit(trn_x, trn_y, eval_set=(val_x, val_y),
                      cat_features=[], use_best_model=True, verbose=500)

            val_pred = model.predict(val_x)
            test_pred = model.predict(test_x)

        train[valid_index] = val_pred
        test = test_pred / kf.n_splits
        cv_scores.append(roc_auc_score(val_y, val_pred))

        print(cv_scores)

    print("%s_scotrainre_list:" % clf_name, cv_scores)
    print("%s_score_mean:" % clf_name, np.mean(cv_scores))
    print("%s_score_std:" % clf_name, np.std(cv_scores))
    return train, test

def lgb_model(x_train, y_train, x_test):
    lgb_train, lgb_test = cv_model(lgb, x_train, y_train, x_test, "lgb")
    return lgb_train, lgb_test

def xgb_model(x_train, y_train, x_test):
    xgb_train, xgb_test = cv_model(xgb, x_train, y_train, x_test, "xgb")
    return xgb_train, xgb_test

def cat_model(x_train, y_train, x_test):
    cat_train, cat_test = cv_model(CatBoostRegressor, x_train, y_train, x_test, "cat")
    return cat_train, cat_test

lgb_train, lgb_test = lgb_model(x_train, y_train, x_test)
xgb_train, xgb_test = xgb_model(x_train, y_train, x_test)
cat_train, cat_test = cat_model(x_train, y_train, x_test)

rh_test = lgb_test*0.5 + xgb_test*0.5

testA['isDefault'] = rh_test
testA[['id','isDefault']].to_csv('test_sub.csv', index=False)