Analyze social media / SNS texts by measuring the likeliness of potential inflammatory / offensive language.
Contains the semantics and instructions for interacting with the project.
Regarding the datasets, all texts have a binary label of either 0 or 1.
0= not inflammatory / not offensive1= inflammatory / offensive
The models make predictions as a probability in range [0, 1]—the model won't explicity predict 0 or 1.
Testing & Working Python Versions:
- 3.9.6
- 3.11.5
- (Most likely versions after 3.8)
| Package | Import Name | Version |
|---|---|---|
| scikit-learn | sklearn | 1.3.2 |
| Jupyter Notebook | notebook | 7.0.6 |
| Matplotlib | matplotlib | 3.8.1 |
| TensorFlow | tensorflow | 2.14.0 |
| Transformers | transformers | 4.35.0 |
Notice: Transformers may install data into your cache to get the pre-trained BERT models to function. This is normal if you've never downloaded the pre-trained data.
- Open
main.pyand run it, no arguments required. - Follow instructions in the terminal for selecting the model and inputting text.
You can modify the main.py to experiment with other models.
All the Logistic Regression models can be loaded using pickle. In the line that loads the Logistic Regression, you may change the path to the model. Both plain and tuned models can be loaded directly without further procedure.
In the line that loads the neural network, you may simply change the path to a different model that contains a label with nn, this will not work for BERT models since they have their own loading function.
Both plain and tuned models can be loaded directly.
To load a plain TFBert model, you must create an instance using the __create_bert_model function then load its weights.
You may comment/uncomment the code and/or comment the prior code that loads the tuned model as well.
The example for main.py is written below.
bert = __create_bert_model()
bert.load_weights('./models/sns_bert.weights.h5')
models['bert']: TFBertModel = bmLoading a tuned TFBert model with its hyperparameters has another procedure which requires you to create an instance using the __create_tuned_bert_model function and pass its hyperparameters.
You may comment/uncomment the code and/or comment the other loading procedures regarding the BERT model.
The example for main.py is written below.
with open('./models/sns_tuned_bert_hp_config.json', 'r') as f:
hp_config = json.loads(f.read())
f.close()
hp = kt.HyperParameters().from_config(hp_config)
tuned_bert = __create_tuned_bert_model(hp)
tuned_bert.load_weights('./models/sns_tuned_bert.weights.h5')
models['bert']: TFBertModel = tuned_bertThis model was the project's baseline model to compare against more complex models such as feedforward neural network and BERT. The LR was manually tuned several times then automatically tuned after performing a grid search.
The regular LR had a decent accuracy but its results were generally unsure about many inflammatory texts unless it was outright explicit.
It was manually tuned.
The tuned LR had a slightly better accuracy since the hyperparameters were close to the manually tuned counterpart.
Additionally, the accuracy had a more consistent positive trend.
Performance was a bit better predicting input sentences.
An evaluated model to predict inflammatory speech. The FFNN was generally more accurate than the Logistic Regression and also performed better than the BERT model. To account for potential over-fitting in both models, there was a learning rate scheduler to decay the learning rate.
The FFNN had an acceptable trend where the losses were decreasing and the accuracies were increasing. They eventually started to reach a consistent "curve" which meant the model was generally performing well. The tuned model was more confident with inflammatory texts but sometimes had drawbacks with texts that were more implied. Additionally, the neural networks, handled longer texts better than all the other models.
The tuned FFNN performed somewhat better than the regular FFNN. The validation accuracy remained relatively consistent but the accuracy increasing in a positive trend. The loss had a negative trend but the validation loss almost went into a positive trend so 10 epochs seemed to be the limit.
The BERT model was the newest model and was also already pretrained on data. Similar to LR, it was also unsure about its results. However, it seemed better at predicting non-inflammatory content opposed to inflammatory content. When it was tuned, it did better with identifying explicit inflammatory text overall, but only did slightly better with long text. However, the tuned BERT model still performed slightly better than all other models.
The BERT model had a similar trend to the FFNNs, but was more inconsistent and less accurate. By the time it was done training, a desired trend happened but was still a bit unsure.
The tuned BERT model had a similar trend to its non-tuned counterpart, only that the trend is much more inconsistent and unsure. The accuracies were about the same when training concluded.
- Of all regular models, the feedforward neural network performed best.
- Of all tuned models, the BERT performed best (slightly).
- In general, the regular feedforward neural network performed the best.