MBTI Personality Test

class main.components.mbti.MBTI

The MBTI object contains functions used for MBTI Personality Test tab

Methods:

clean_text(text[, lemma])

Process text

get_bar_plot(predictions)

Get figure for plot

get_feature_importance([max_num_features])

Print top feature importance for each model

get_num_words(input_text)

Get number of input words in vocabulary

get_personality_details(personality)

Get personality details (summary and details)

get_train_test(X, y[, test_size, random_state])

Splits data into training and testing data

load_and_save_data()

Reads in data, performs preprocessing and saves data If saved data is present, directly read in the saved data

load_model(path_model)

Load and return saved best model after grid search with stratified cross validation

load_model_tf(path_model)

Load and return saved tensorflow model

load_tokenizer()

Load and return saved tokenizer

load_vectorizer()

Load and return saved vectorizer

predict_model(model, vector_test)

Perform prediction on test set

predict_model_tf(model, vector_test)

Perform prediction on test set

save_model(vector_train, y_train_series, ...)

Train, save and return best model after grid search with stratified cross validation

save_model_tf(vector_train, y_train_series, ...)

Train, save and return tensorflow model

save_tokenizer(corpus[, params])

Fit, save and return tokenizer

save_vectorizer(corpus[, params])

Fit, save and return vectorizer

test_pipeline(input_text)

Testing pipeline for new input text

tokenize_new_input(input_text)

Load saved tokenizer and transform input text

train_pipeline([train_vect, train_model])

Training pipeline for loading, preprocessing and model training

transform_tokenizer(tokenizer, corpus)

Transform corpus with tokenizer

transform_vectorizer(vect, corpus)

Transform corpus with vectorizer

vectorize_new_input(input_text)

Load saved vectorizer and transform input text
static clean_text(text: str, lemma=<WordNetLemmatizer>)

Process text

  1. Split different sentences

  2. Make words lowercase

  3. Remove URLs (i.e. http) and usernames (i.e. @username)

  4. Remove digits and punctuations

  5. Remove any mention of MBTI types

  6. Tokenize words (i.e. split the words into list)

  7. Lemmatize words (i.e. reduce words to singular form)

  8. Join text into string

Parameters:

  • text – input text

  • lemma – Lemmatizer (defaults to nltk WordNetLemmatizer)

Returns:

processed text

Return type:

(str)

static get_bar_plot(predictions: List[Any]) → Dict[str, Any]

Get figure for plot

Adds plotly.graph_objects charts for bar plot

Parameters:

predictions – list of model prediction probabilities

get_feature_importance(max_num_features: int = 10)

Print top feature importance for each model

Parameters:

max_num_features – number of top feature importance

get_num_words(input_text: str) → int

Get number of input words in vocabulary

Parameters:

input_text – input text

static get_personality_details(personality: str) → List[P]

Get personality details (summary and details)

Parameters:

personality – MBTI personality results, to retrieve detailed results

static get_train_test(X: DataFrame, y: DataFrame, test_size: float = 0.2, random_state: int = 0)

Splits data into training and testing data

Parameters:

  • X – processed input data

  • y – processed output data

  • test_size – proportion of test data, defaults to 0.2

  • random_state – fixed seed, allows reproducible result, defaults to 0

Returns:

4-element tuple

  • X_train (pd.DataFrame): training input

  • X_test (pd.DataFrame): testing input

  • y_train (pd.DataFrame): training output

  • y_test (pd.DataFrame): testing output

load_and_save_data()

Reads in data, performs preprocessing and saves data If saved data is present, directly read in the saved data

If saved data does not exist

  1. Reads in data

  2. Insert new columns as indicator for each mbti category

  3. Process text column

  4. Save data

If saved data exist

  1. Reads in saved data

Returns:

processed data

Return type:

(pd.DataFrame)

static load_model(path_model: str)

Load and return saved best model after grid search with stratified cross validation

Parameters:

path_model – location and file name of saved model

Returns:

model

static load_model_tf(path_model: str)

Load and return saved tensorflow model

Parameters:

path_model) – location and file name of saved model

Returns:

model

load_tokenizer()

Load and return saved tokenizer

Returns:

keras Tokenizer

load_vectorizer()

Load and return saved vectorizer

Returns:

(sklearn.CountVectorizer)

static predict_model(model, vector_test) → ndarray

Perform prediction on test set

Parameters:

  • model (model) – model to be used for prediction

  • vector_test (scipy.csr_matrix) – vectorized training input

Returns:

y_pred

predict_model_tf(model, vector_test: ndarray) → ndarray

Perform prediction on test set

Parameters:

  • model (model) – model to be used for prediction

  • vector_test – vectorized training input

Returns:

y_pred

static save_model(vector_train, y_train_series: Series, path_model: str)

Train, save and return best model after grid search with stratified cross validation

Parameters:

  • vector_train (scipy.csr_matrix) – vectorized training input

  • y_train_series – training output, one-column subset of y_train

  • path_model – location and file name of saved model

Returns:

model

save_model_tf(vector_train, y_train_series, path_model)

Train, save and return tensorflow model

Parameters:

  • vector_train (np.ndarray) – vectorized training input

  • y_train_series (pd.Series) – training output, one-column subset of y_train

  • path_model (str) – location and file name of saved model

Returns:

model

save_tokenizer(corpus: Series, params: Dict[str, Any] | None = None)

Fit, save and return tokenizer

Parameters:

  • corpus – input text corpus (training input)

  • params – specifies parameters for tokenizer, defaults to None

Returns:

(keras.Tokenizer)

save_vectorizer(corpus: Series, params: Dict[str, Any] | None = None)

Fit, save and return vectorizer

Parameters:

  • corpus – input text corpus (training input)

  • params – specifies parameters for vectorizer, defaults to None

Returns:

(sklearn.CountVectorizer)

test_pipeline(input_text: str)

Testing pipeline for new input text

Parameters:

input_text – input text

Returns:

2-element tuple

  • personality (str): MBTI personality results, to be shown in title of bar plot

  • predictions (list): list of tuple of model prediction probabilities

tokenize_new_input(input_text: str) → ndarray

Load saved tokenizer and transform input text

Parameters:

input_text – input text

Returns:

tokenized input_text

train_pipeline(train_vect: bool = False, train_model: bool = False)

Training pipeline for loading, preprocessing and model training

Parameters:

  • train_vect – indicates whether to retrain vectorizer, defaults to False

  • train_model – indicates whether to retrain models, defaults to False

Returns:

NA

transform_tokenizer(tokenizer, corpus: Series)

Transform corpus with tokenizer

Parameters:

  • tokenizer (keras Tokenizer) – tokenizer to be used to transform text corpus

  • corpus – input text corpus

Returns:

tokenized text corpus

Return type:

vector_corpus (np.ndarray)

transform_vectorizer(vect, corpus: Series)

Transform corpus with vectorizer

Parameters:

  • vect (sklearn.CountVectorizer) – vectorizer to be used to transform text corpus

  • corpus – input text corpus

Returns:

vectorized text corpus

Return type:

vector_corpus (scipy.csr_matrix)

vectorize_new_input(input_text: str)

Load saved vectorizer and transform input text

Parameters:

input_text – input text

Returns:

(scipy.csr_matrix) vectorized input_text