This project is designed to analyze and classify real network traffic data to differentiate between malicious and benign traffic records. By comparing and fine-tuning several Machine Learning algorithms, it aims to achieve the highest accuracy with the lowest false positive and negative rates.

The dataset utilized in this project is CTU-IoT-Malware-Capture-34-1, a part of the Aposemat IoT-23 dataset. This labeled dataset includes malicious and benign IoT network traffic and was created through the Avast AIC laboratory with support from Avast Software.

The project consists of four phases, each represented by a corresponding notebook within the notebooks directory. Intermediate data files are stored in the data directory, while trained models are kept in the models directory.

Notebook: initial-data-cleaning.ipynb

This phase involves the initial exploration and cleaning of the dataset:

1. Load the raw dataset into a pandas DataFrame.
2. Review dataset summary and statistics.
3. Fix combined columns.
4. Remove irrelevant columns.
5. Correct unset values and validate data types.
6. Inspect the cleaned dataset.
7. Save the cleaned dataset to a CSV file.

Notebook: data-preprocessing.ipynb

In this phase, the focus is on processing and transforming the data:

1. Load the dataset into a pandas DataFrame.
2. Review dataset summary and statistics.
3. Analyze the target attribute.
4. Encode the target attribute using LabelEncoder.
5. Handle outliers with the Inter-quartile Range (IQR).
6. Encode IP addresses.
7. Manage missing values:
   • Impute missing categorical features using KNeighborsClassifier.
   • Impute missing numerical features using KNNImputer.
8. Scale numerical attributes with MinMaxScaler.
9. Handle categorical features: manage rare values and apply One-Hot Encoding.
10. Verify the processed dataset and save it to a CSV file.

Notebook: [model-training.ipynb])

This phase includes training and evaluating various classification models:

1. Naive Bayes: ComplementNB
2. Decision Tree: DecisionTreeClassifier
3. Logistic Regression: LogisticRegression
4. Random Forest: RandomForestClassifier
5. Support Vector Classifier: SVC
6. K-Nearest Neighbors: KNeighborsClassifier
7. XGBoost: XGBClassifier

Evaluation Method:

• Cross-Validation: Stratified K-Folds Cross-Validator
• Number of folds: 5
• Shuffling: Enabled

The results of each model are analyzed and compared.

Notebook: model-tuning.ipynb

This phase focuses on fine-tuning the best-performing model:

• Model: Support Vector Classifier (SVC)
• Tuning Method: GridSearchCV

The performance of the model is analyzed both before and after tuning.

By following these steps, the project aims to effectively classify network traffic and detect malicious activities with high accuracy.