catboost-training-2-step.ipynb

{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":59093,"databundleVersionId":7469972,"sourceType":"competition"},{"sourceId":7392733,"sourceType":"datasetVersion","datasetId":4297749},{"sourceId":7392775,"sourceType":"datasetVersion","datasetId":4297782},{"sourceId":7447509,"sourceType":"datasetVersion","datasetId":4334995}],"dockerImageVersionId":30636,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"This notebook is the Modified version by us of a starter notebook by Chris Deotte\nhttps://www.kaggle.com/code/cdeotte/catboost-starter-lb-0-60","metadata":{}},{"cell_type":"markdown","source":"# CatBoost \nWe use both spectrogram and spectrogram_made_from_eeg features.     \nTo make model learn better we incresed the frequency of the high quality data(total voter >=10) two times.      \nWe also used a differnt loss function that is 'MultiCrossEntropy' because it takes the Probability of all classes as labels.      \nIn this notebook, we also compare five CV scores. Kaggle's sample submission uses equal predictions of 1/6 for all targets and achieves CV 0.69, LB 0.51. \n","metadata":{}},{"cell_type":"markdown","source":"# Load Libraries","metadata":{}},{"cell_type":"code","source":"import os, gc\nos.environ[\"CUDA_VISIBLE_DEVICES\"]=\"0,1\"\nimport pandas as pd, numpy as np\nimport matplotlib.pyplot as plt\n\nVER = 3","metadata":{"execution":{"iopub.status.busy":"2024-04-06T17:44:40.475928Z","iopub.execute_input":"2024-04-06T17:44:40.476226Z","iopub.status.idle":"2024-04-06T17:44:41.412524Z","shell.execute_reply.started":"2024-04-06T17:44:40.476200Z","shell.execute_reply":"2024-04-06T17:44:41.411691Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load Train Data","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/train.csv')\nTARGETS = df.columns[-6:]\nprint('Train shape:', df.shape )\nprint('Targets', list(TARGETS))\ndf.head()","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-04-06T17:44:41.413902Z","iopub.execute_input":"2024-04-06T17:44:41.414288Z","iopub.status.idle":"2024-04-06T17:44:41.726383Z","shell.execute_reply.started":"2024-04-06T17:44:41.414261Z","shell.execute_reply":"2024-04-06T17:44:41.725397Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Create Non-Overlapping Eeg Id Train Data\nThe competition data description says that test data does not have multiple crops from the same `eeg_id`. Therefore we will train and validate using only 1 crop per `eeg_id`. There is a discussion about this [here][1].\n\n[1]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/467021","metadata":{}},{"cell_type":"code","source":"df['total_evaluators'] = df[['seizure_vote', 'lpd_vote', 'gpd_vote', 'lrda_vote', 'grda_vote', 'other_vote']].sum(axis=1)","metadata":{"execution":{"iopub.status.busy":"2024-04-06T17:44:41.727868Z","iopub.execute_input":"2024-04-06T17:44:41.728307Z","iopub.status.idle":"2024-04-06T17:44:41.765130Z","shell.execute_reply.started":"2024-04-06T17:44:41.728268Z","shell.execute_reply":"2024-04-06T17:44:41.764389Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = df.groupby('eeg_id')[['spectrogram_id','spectrogram_label_offset_seconds']].agg(\n    {'spectrogram_id':'first','spectrogram_label_offset_seconds':'min'})\ntrain.columns = ['spec_id','min']\n\ntmp = df.groupby('eeg_id')[['spectrogram_id','spectrogram_label_offset_seconds']].agg(\n    {'spectrogram_label_offset_seconds':'max'})\ntrain['max'] = tmp\n\ntmp = df.groupby('eeg_id')[['patient_id']].agg('first')\ntrain['patient_id'] = tmp\n\ntmp = df.groupby('eeg_id')[['total_evaluators']].agg('mean')\ntrain['total_evaluators'] = tmp\n\ntmp = df.groupby('eeg_id')[TARGETS].agg('sum')\nfor t in TARGETS:\n    train[t] = tmp[t].values\n    \ny_data = train[TARGETS].values\ny_data = y_data / y_data.sum(axis=1,keepdims=True)\ntrain[TARGETS] = y_data\n\ntmp = df.groupby('eeg_id')[['expert_consensus']].agg('first')\ntrain['target'] = tmp\n\ntrain = train.reset_index()\nprint('Train non-overlapp eeg_id shape:', train.shape )\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2024-04-06T17:44:41.767759Z","iopub.execute_input":"2024-04-06T17:44:41.768217Z","iopub.status.idle":"2024-04-06T17:44:41.873252Z","shell.execute_reply.started":"2024-04-06T17:44:41.768180Z","shell.execute_reply":"2024-04-06T17:44:41.872329Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature Engineer\nIn this section, we create features for our CatBoost model. \n\nFirst we need to read in all 11k train spectrogram files. Reading thousands of files takes 11 minutes with Pandas. Instead, we can read 1 file from Chris's [Kaggle dataset here][1] which contains all the 11k spectrograms in less than 1 minute! To use Chris's [Kaggle dataset][1], set variable `READ_SPEC_FILES = False`.\n\nwe also loaded EEG spectrograms from  Chris's dataset ( https://www.kaggle.com/datasets/cdeotte/brain-eeg-spectrograms\n).\n    \nNext we need to engineer features for our CatBoost model. We took the mean and min (over time) of both Kaggle spectrograms and EEG spectrograms.\n \n\n","metadata":{}},{"cell_type":"code","source":"READ_SPEC_FILES = False\nREAD_EEG_SPEC_FILES = False","metadata":{"execution":{"iopub.status.busy":"2024-04-06T17:44:41.874357Z","iopub.execute_input":"2024-04-06T17:44:41.874703Z","iopub.status.idle":"2024-04-06T17:44:41.879797Z","shell.execute_reply.started":"2024-04-06T17:44:41.874675Z","shell.execute_reply":"2024-04-06T17:44:41.878733Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# READ ALL SPECTROGRAMS\nPATH = '/kaggle/input/hms-harmful-brain-activity-classification/train_spectrograms/'\nfiles = os.listdir(PATH)\nprint(f'There are {len(files)} spectrogram parquets')\n\nif READ_SPEC_FILES:    \n    spectrograms = {}\n    for i,f in enumerate(files):\n        if i%100==0: print(i,', ',end='')\n        tmp = pd.read_parquet(f'{PATH}{f}')\n        name = int(f.split('.')[0])\n        spectrograms[name] = tmp.iloc[:,1:].values\nelse:\n    spectrograms = np.load('/kaggle/input/brain-spectrograms/specs.npy',allow_pickle=True).item()","metadata":{"execution":{"iopub.status.busy":"2024-04-06T17:44:41.881196Z","iopub.execute_input":"2024-04-06T17:44:41.881702Z","iopub.status.idle":"2024-04-06T17:45:37.677978Z","shell.execute_reply.started":"2024-04-06T17:44:41.881672Z","shell.execute_reply":"2024-04-06T17:45:37.676971Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# READ ALL EEG SPECTROGRAMS\nif READ_EEG_SPEC_FILES:\n    all_eegs = {}\n    for i,e in enumerate(train.eeg_id.values):\n        if i%100==0: print(i,', ',end='')\n        x = np.load(f'/kaggle/input/brain-eeg-spectrograms/EEG_Spectrograms/{e}.npy')\n        all_eegs[e] = x\nelse:\n    all_eegs = np.load('/kaggle/input/brain-eeg-spectrograms/eeg_specs.npy',allow_pickle=True).item()","metadata":{"execution":{"iopub.status.busy":"2024-04-06T17:45:37.679408Z","iopub.execute_input":"2024-04-06T17:45:37.680212Z","iopub.status.idle":"2024-04-06T17:46:43.971249Z","shell.execute_reply.started":"2024-04-06T17:45:37.680170Z","shell.execute_reply":"2024-04-06T17:46:43.970063Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%time\n# ENGINEER FEATURES\nimport warnings\nwarnings.filterwarnings('ignore')\n\n# FEATURE NAMES\nSPEC_COLS = pd.read_parquet(f'{PATH}1000086677.parquet').columns[1:]\nFEATURES = [f'{c}_mean_10m' for c in SPEC_COLS]\nFEATURES += [f'{c}_min_10m' for c in SPEC_COLS]\nFEATURES += [f'{c}_mean_20s' for c in SPEC_COLS]\nFEATURES += [f'{c}_min_20s' for c in SPEC_COLS]\nFEATURES += [f'eeg_mean_f{x}_10s' for x in range(512)]\nFEATURES += [f'eeg_min_f{x}_10s' for x in range(512)]\nFEATURES += [f'eeg_max_f{x}_10s' for x in range(512)]\nFEATURES += [f'eeg_std_f{x}_10s' for x in range(512)]\nprint(f'We are creating {len(FEATURES)} features for {len(train)} rows... ',end='')\n\ndata = np.zeros((len(train),len(FEATURES)))\nfor k in range(len(train)):\n    if k%100==0: print(k,', ',end='')\n    row = train.iloc[k]\n    r = int( (row['min'] + row['max'])//4 ) \n\n    # 10 MINUTE WINDOW FEATURES (MEANS and MINS)\n    x = np.nanmean(spectrograms[row.spec_id][r:r+300,:],axis=0)\n    data[k,:400] = x\n    x = np.nanmin(spectrograms[row.spec_id][r:r+300,:],axis=0)\n    data[k,400:800] = x\n\n    # 20 SECOND WINDOW FEATURES (MEANS and MINS)\n    x = np.nanmean(spectrograms[row.spec_id][r+145:r+155,:],axis=0)\n    data[k,800:1200] = x\n    x = np.nanmin(spectrograms[row.spec_id][r+145:r+155,:],axis=0)\n    data[k,1200:1600] = x\n\n    # RESHAPE EEG SPECTROGRAMS 128x256x4 => 512x256\n    eeg_spec = np.zeros((512,256),dtype='float32')\n    xx = all_eegs[row.eeg_id]\n    for j in range(4): eeg_spec[128*j:128*(j+1),] = xx[:,:,j]\n\n    # 10 SECOND WINDOW FROM EEG SPECTROGRAMS \n    x = np.nanmean(eeg_spec.T[100:-100,:],axis=0)\n    data[k,1600:2112] = x\n    x = np.nanmin(eeg_spec.T[100:-100,:],axis=0)\n    data[k,2112:2624] = x\n    x = np.nanmax(eeg_spec.T[100:-100,:],axis=0)\n    data[k,2624:3136] = x\n    x = np.nanstd(eeg_spec.T[100:-100,:],axis=0)\n    data[k,3136:3648] = x\n\ntrain[FEATURES] = data\nprint(); print('New train shape:',train.shape)","metadata":{"execution":{"iopub.status.busy":"2024-04-06T17:46:43.972433Z","iopub.execute_input":"2024-04-06T17:46:43.972694Z","iopub.status.idle":"2024-04-06T17:47:38.492587Z","shell.execute_reply.started":"2024-04-06T17:46:43.972670Z","shell.execute_reply":"2024-04-06T17:47:38.491356Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# FREE MEMORY\ndel all_eegs, spectrograms, data\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2024-04-06T17:47:38.493889Z","iopub.execute_input":"2024-04-06T17:47:38.494237Z","iopub.status.idle":"2024-04-06T17:47:38.687433Z","shell.execute_reply.started":"2024-04-06T17:47:38.494207Z","shell.execute_reply":"2024-04-06T17:47:38.686121Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train CatBoost\nWe use the default settings for CatBoost which are pretty good. We can tune CatBoost manually to improve CV and LB score. Note that CatBoost will automatically use both Kaggle T4 GPUs (when we add parameter `task_type='GPU'`)  for super fast training!","metadata":{}},{"cell_type":"code","source":"import catboost as cat\nfrom catboost import CatBoostClassifier, Pool\nprint('CatBoost version',cat.__version__)","metadata":{"execution":{"iopub.status.busy":"2024-04-06T17:47:50.391413Z","iopub.execute_input":"2024-04-06T17:47:50.391723Z","iopub.status.idle":"2024-04-06T17:47:51.388514Z","shell.execute_reply.started":"2024-04-06T17:47:50.391685Z","shell.execute_reply":"2024-04-06T17:47:51.387500Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import KFold, GroupKFold\n\nall_oof = []\nall_true = []\nall_oof2 = []\nall_true2 = []\nTARS = {'Seizure':0, 'LPD':1, 'GPD':2, 'LRDA':3, 'GRDA':4, 'Other':5}\n\ngkf = GroupKFold(n_splits=5)\nfor i, (train_index, valid_index) in enumerate(gkf.split(train, train.target, train.patient_id)):   \n    \n    print('#'*25)\n    print(f'### Fold {i+1}')\n    print(f'### train size {len(np.concatenate((train.loc[train_index,FEATURES],train.loc[train_index,FEATURES][train.iloc[train_index][\"total_evaluators\"]>=10]),axis=0))}, valid size {len(np.concatenate((train.loc[valid_index,FEATURES],train.loc[valid_index,FEATURES][train.iloc[valid_index][\"total_evaluators\"]>=10]),axis=0))}')\n    print('#'*25)\n    \n    model = CatBoostClassifier(task_type='GPU',\n                               loss_function='MultiCrossEntropy')\n   # train_pool all dataset training data \n    train_pool = Pool(\n        data = train.loc[train_index,FEATURES],\n        label = np.array(train[train.columns[6:12]])[train_index],\n    )\n    \n    valid_pool = Pool(\n        data = train.loc[valid_index,FEATURES],\n        label =np.array(train[train.columns[6:12]])[valid_index],\n    )\n    # train_pool 3 all dataset where the amount of high quality data('total voters >=10') is doubled \n    train_pool3 = Pool(\n        data = np.concatenate((train.loc[train_index,FEATURES],train.loc[train_index,FEATURES][train.iloc[train_index][\"total_evaluators\"]>=10]),axis=0),\n        label = np.concatenate((np.array(train[train.columns[6:12]])[train_index],np.array(train[train.columns[6:12]])[train_index][train.iloc[train_index][\"total_evaluators\"]>=10]),axis=0),\n    )\n    \n    valid_pool3 = Pool(\n        data = np.concatenate((train.loc[valid_index,FEATURES],train.loc[valid_index,FEATURES][train.iloc[valid_index][\"total_evaluators\"]>=10]),axis=0),\n        label = np.concatenate((np.array(train[train.columns[6:12]])[valid_index],np.array(train[train.columns[6:12]])[valid_index][train.iloc[valid_index][\"total_evaluators\"]>=10]),axis=0),\n    )\n    \n    \n    model.fit(train_pool3,\n             verbose=100,\n             eval_set=valid_pool3,\n             )\n    model.save_model(f'CAT_v{VER}_f{i}.cat')\n    \n    # train_pool 2 high quality data dataset\n    train_pool2 = Pool(\n        data = train.loc[train_index,FEATURES][train.iloc[train_index][\"total_evaluators\"]>=10],\n        label = np.array(train[train.columns[6:12]])[train_index][train.iloc[train_index][\"total_evaluators\"]>=10],\n    )\n    \n    valid_pool2 = Pool(\n        data = train.loc[valid_index,FEATURES][train.iloc[valid_index][\"total_evaluators\"]>=10],\n        label =np.array(train[train.columns[6:12]])[valid_index][train.iloc[valid_index][\"total_evaluators\"]>=10],\n    )\n    \n    oof = model.predict_proba(valid_pool)\n    all_oof.append(oof)\n    all_true.append(train.loc[valid_index, TARGETS].values)\n    oof2 = model.predict_proba(valid_pool2)\n    all_oof2.append(oof2)\n    all_true2.append(np.array(train[train.columns[6:12]])[valid_index][train.iloc[valid_index][\"total_evaluators\"]>=10])\n    \n    del train_pool, valid_pool,oof2, oof,train_pool2,valid_pool2,train_pool3,valid_pool3 #model\n    gc.collect()\n    \n    #break\n    \nall_oof = np.concatenate(all_oof)\nall_true = np.concatenate(all_true)\nall_oof2 = np.concatenate(all_oof2)\nall_true2 = np.concatenate(all_true2)","metadata":{"execution":{"iopub.status.busy":"2024-04-06T18:00:19.239089Z","iopub.execute_input":"2024-04-06T18:00:19.239865Z","iopub.status.idle":"2024-04-06T18:11:07.420481Z","shell.execute_reply.started":"2024-04-06T18:00:19.239832Z","shell.execute_reply":"2024-04-06T18:11:07.419325Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature Importance\nBelow we display the CatBoost top 25 feature importance for the last fold we trained.","metadata":{}},{"cell_type":"code","source":"TOP = 25\n\nfeature_importance = model.feature_importances_\nsorted_idx = np.argsort(feature_importance)\nfig = plt.figure(figsize=(10, 8))\nplt.barh(np.arange(len(sorted_idx))[-TOP:], feature_importance[sorted_idx][-TOP:], align='center')\nplt.yticks(np.arange(len(sorted_idx))[-TOP:], np.array(FEATURES)[sorted_idx][-TOP:])\nplt.title(f'Feature Importance - Top {TOP}')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-04-06T18:11:07.423400Z","iopub.execute_input":"2024-04-06T18:11:07.423702Z","iopub.status.idle":"2024-04-06T18:11:07.926177Z","shell.execute_reply.started":"2024-04-06T18:11:07.423676Z","shell.execute_reply":"2024-04-06T18:11:07.925204Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# CV Score for CatBoost\nThis is CV score for our CatBoost model.","metadata":{}},{"cell_type":"code","source":"import sys\nsys.path.append('/kaggle/input/kaggle-kl-div')\nfrom kaggle_kl_div import score\n\n\noof = pd.DataFrame(all_oof/np.sum(all_oof,axis=1).reshape(-1,1).copy())\noof['id'] = np.arange(len(oof))\n\ntrue = pd.DataFrame(all_true.copy())\ntrue['id'] = np.arange(len(true))\n\ncv = score(solution=true, submission=oof, row_id_column_name='id')\nprint('CV Score KL-Div for CatBoost =',cv)\n\noof2 = pd.DataFrame(all_oof2/np.sum(all_oof2,axis=1).reshape(-1,1).copy())\noof2['id'] = np.arange(len(oof2))\n\ntrue2 = pd.DataFrame(all_true2.copy())\ntrue2['id'] = np.arange(len(true2))\n\ncv2 = score(solution=true2, submission=oof2, row_id_column_name='id')\nprint('CV Score KL-Div for CatBoost for high quality data =',cv2)","metadata":{"execution":{"iopub.status.busy":"2024-04-06T18:11:07.927363Z","iopub.execute_input":"2024-04-06T18:11:07.927637Z","iopub.status.idle":"2024-04-06T18:11:08.024961Z","shell.execute_reply.started":"2024-04-06T18:11:07.927613Z","shell.execute_reply":"2024-04-06T18:11:08.024076Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}