Multiclass Multilabel Prediction For StackOverflow Questions Data set : https://www.kaggle.com/therajeshreddy/stackoverflow Objective : Given text for Questions from StackoverFlow posts, predict tags associated with them. This is a scaled down version of predecting only top 10 most occurring tags Programming Language : Python using nltk & Keras Model Architecture : Deep Learning using Recurrent Neural Network (RNN) About Data Set Dataset has text of questions, answers and thier corresponding tags from the Stack Overflow programming Q&A website. This is organized as three files: Questions contains the title, body, creation date, closed date (if applicable), score, and owner ID for all non-deleted Stack Overflow questions. Tags contains the tags on each of these questions. Answers contains the body, creation date, score, and owner ID for each of the answers to these questions. The ParentId column links back to the Questions table. We don't use this file as we want to predict Tags given a question Data Pre-Processing Questions File Code : Stackoverflow Clean Questions.ipynb Read Questions File Drop All columns except Id,Title and Body Now the text in the Body column seem to have many html tags in the text. We use Regular Expressions and Clean the Body column text by removing the html tags import re def rem_html_tags(body): regex = re.compile('<.*?>') return re.sub(regex, '', body) ques['Body'] = ques['Body'].apply(rem_html_tags) Save the questions file for later use ques.to_csv('question_clean.csv',index=False) Tags File Code : Stackoverflow Tags Map & Model.ipynb Read Tags File Identify top 10 Tags by count tagCount = collections.Counter(list(df_tags['Tag'])).most_common(10) print(tagCount) [('javascript', 124155), ('java', 115212), ('c#', 101186), ('php', 98808), ('android', 90659), ('jquery', 78542), ('python', 64601), ('html', 58976), ('c++', 47591), ('ios', 47009)] Manipulate the tags dataframe so that all the Tags for an ID are as a list in a row (grouped by Question ID) def add_tags(question_id): return tag_top10[tag_top10['Id'] == question_id['Id']].Tag.values top10 = tag_top10.apply(add_tags, axis=1) Combine the Questions and Tags Code : Stackoverflow Tags Map & Model.ipynb Merge the Questions and Tags data frame by ID total=pd.merge(ques, top10_tags, on='Id') Our Dataset would now have only Id, Title, Body & Tags Text Preprocessing Code : Stackoverflow Tags Map & Model.ipynb We will use nltk, preprocessing from Keras and sklearn to process the text data Tags preprocesing Use MultiLabelBinarizer from sklearn on the Class labels(Tags) from sklearn.preprocessing import MultiLabelBinarizer multilabel_binarizer = MultiLabelBinarizer() multilabel_binarizer.fit(total.Tags) print(multilabel_binarizer.classes_) array(['android', 'c#', 'c++', 'html', 'ios', 'java', 'javascript','jquery', 'php', 'python'], dtype=object) Title & Body Preprocessing Tokenize the words Convert the tokenized words to sequences Model Building Implemented a Hybrid model in TensorFlow using Keras as high level api. Architecture used is RNN. In this model first we train a model using the Title data, then train a model using the Body data. Outputs of both are concatenated and passed thorugh the dense layers before connecting to the output layer RNN Model : The model first uses GRU for the sequence data training with 2 GRU layers one for Title and other for Body. RNN for Title has 1 Embedding Layer has input of Title vocabulary length(68969) + 1(for 0 padding) and out put of 2000 embeddings (for better results use full vocabulary length+1) 1 Gated recurrent unit (GRU) layer 1 dense output layer of shape 10(No of classes(tags) we are trying to predict) # Title Only title_input = Input(name='title_input',shape=[max_len_t]) title_Embed = Embedding(vocab_len_t+1,2000,input_length=max_len_t,mask_zero=True,name='title_Embed')(title_input) gru_out_t = GRU(300)(title_Embed) # auxiliary output to tune GRU weights smoothly auxiliary_output = Dense(10, activation='sigmoid', name='aux_output')(gru_out_t) RNN for Body has 1 Embedding Layer has input of Title vocabulary length(1292018) + 1(for 0 padding) and out put of 170 embeddings (for better results use full vocabulary length+1) 1 Gated recurrent unit (GRU) layer # Body Only body_input = Input(name='body_input',shape=[max_len_b]) body_Embed = Embedding(vocab_len_b+1,170,input_length=max_len_b,mask_zero=True,name='body_Embed')(body_input) gru_out_b = GRU(200)(body_Embed) Combine the 2 GRU outputs com = concatenate([gru_out_t, gru_out_b]) The fully connected network has 2 Dense Layers 1 Dropout layer 1 BatchNormalization layer 1 Dense Output layer # now the combined data is being fed to dense layers dense1 = Dense(400,activation='relu')(com) dp1 = Dropout(0.5)(dense1) bn = BatchNormalization()(dp1) dense2 = Dense(150,activation='relu')(bn) main_output = Dense(10, activation='sigmoid', name='main_output')(dense2) Model Compilattion with optimizer='adam', loss='categorical_crossentropy', metrics='accuracy') Model Performance Review Classification Report to check Precision, Recall and F1 Score The Model seem to performing good enough with score of 84%. Increase in the Embedding, GRU and dense layers would help in getting better results Random Validation on Test Data Save the Model & Weights Saving the model for transfer learning or model execution later model.save('./stackoverflow_tags.h5')