Malware is any type of software created to harm or exploit another piece of software or hardware. Short for “malicious software,” malware is a collective term used to describe viruses, ransomware, spyware, Trojans, and any other type of code or software built with malicious intent.

In the past few years, the malware industry has grown very rapidly that, the syndicates invest heavily in technologies to evade traditional protection, forcing the anti-malware groups/communities to build more robust softwares to detect and terminate these attacks. The major part of protecting a computer system from a malware attack is to identify whether a given piece of file/software is a malware.

Microsoft has been very active in building anti-malware products over the years and it runs it’s anti-malware utilities over 150 million computers around the world. This generates tens of millions of daily data points to be analyzed as potential malware. In order to be effective in analyzing and classifying such large amounts of data, we need to be able to group them into groups and identify their respective families.

This dataset provided by Microsoft contains about 9 classes of malware.

[Source: Kaggle-Microsoft Malware Classification Challenge: https://www.kaggle.com/c/malware-classification]

• Minimize multi-class error.
• Multi-class probability estimates.
• Malware detection should not take hours and block the user's computer. It should fininsh in a few seconds or a minute.

1. Data Source: https://www.kaggle.com/c/malware-classification/data
2. For every malware, we have two files
   • .asm file (read more: https://www.reviversoft.com/file-extensions/asm)
   • .bytes file (the raw data contains the hexadecimal representation of the file's binary content, without the PE header)
3. Total train dataset consist of 200GB data out of which 50Gb of data is .bytes files and 150GB of data is .asm files:
4. Lots of Data for a single-box/computer.
5. There are total 10,868 .bytes files and 10,868 asm files total 21,736 files
6. There are 9 types of malwares (9 classes) in our give data
7. Types of Malware:

There are nine different classes of malware that we need to classify a given a data point => Multi class classification problem

• Multi Class Log Loss
• Confusion Matrix

Objective: Predict the probability of each data-point belonging to each of the nine classes.

Constraints:

• Class probabilities are needed.
• Penalize the errors in class probabilites => Metric is Log-loss.
• Some Latency constraints.

Split the dataset randomly into three parts train, cross validation and test with 64%,16%, 20% of data respectively.

1. http://blog.kaggle.com/2015/05/26/microsoft-malware-winners-interview-1st-place-no-to-overfitting/
2. https://arxiv.org/pdf/1511.04317.pdf
3. First place solution in Kaggle competition: https://www.youtube.com/watch?v=VLQTRlLGz5Y
4. https://github.com/dchad/malware-detection
5. http://vizsec.org/files/2011/Nataraj.pdf

1. Extract the .asm and .bytes file and copy them to seperate folders.
2. Now we have to create the features out of these .asm and .byte files.
3. .byte file:
   • Create unigram features count from each byte file.
   • Get the size of each .byte file.
   • Top 2000 Bi-Gram of Byte files
4. .asm file:
   • Create unigram features from each asm file.
   • Size of ASM Files
   • Top 500 Bigram of Opcodes of ASM Files
   • Top 800 Trigram of Opcodes of ASM Files
   • Top 800 ASM Image Features
5. Combining the different features from .byte and .asm files into a single dataframe.
6. Splitting the data into Train, Test and Cross Validation sets.
7. Applied different classification ML algorithms such as Logistic Regression, KNN, Dicision Tree, Random Forest, XGBoost.
8. Based on results obtained, we came to kn ow that XGBoost performing well.
9. Hyperparameter Tuning is done using sklearns RandomSearchCV.
10. Using the best parameters obtained from RandomSearchCV, final XGBoost model has run and logloss is calculated.

For the final dataset, i have used different classification estimators such as, Logostic Regression, KNN, Random Forest and XG Boost models. Random Forest Classifier is performing well. The results are tabulated as below.