# A method to solve the Higgs boson challenge using Least Squares – Novae

This project is the Project 1 of EPFL CS-433 Machine Learning. The project is the same as the Higgs Boson Machine Learning Challenge posted on Kaggle. The dataset and the detailed description can also be found in the GitHub repository of the course.

Team name: Novae

Team members: Giacomo Orsi, Vittorio Rossi, Chun-Tso Tsai

The task of this project is to train a model based on the provided `train.csv` to have the best prediction on the data given in `test.csv` or any other general case.

We built our model for the problem using regularized linear regression after applying some data cleaning and features engineering techniques. A report describing our approach and our results can be found in the file `report.pdf`. In the end, we obtained an accuracy of 0.836 and an F1 score of 0.751 on the `test.csv` dataset.

## Instructions

• The project runs under `Python 3.8` and requires `NumPy=1.19`.
• Please make sure to place `train.csv` and `test.csv` inside the `data` folder. Those files can be downloaded here.
• Go to the `script/` folder and execute `run.py`. A model will be trained with the given hyper-parameters and predictions for the test dataset will be outputed in the file `out.csv`.

## Modules

### `implementations.py`

Contains the implementations of different learning algorithms. Including

• Least squares linear regression
• `least_squares`: Direct computation from linear equations.
• `least_squares_GD`: Gradient descent.
• `least_squares_SGD`: Stochastic gradient descent.
• `ridge_regression`: Regularized linear regression from direct computation.
• Logistic regression
• `logistic_regression`: Gradient descent
• `reg_logistic_regression`: Gradient descent with regularization.

There are also some helper functions in this file to facilitate the above functions.

### `data_processing.py`

Calls the following files to process the data.

• `data_cleaning.py`: Contains functions used to
1. Categorize data into subgroups.
2. Replace missing values with the median.
3. Standardize the features.
• `feature_engineering.py`: Contains functions used to generate our interpretable features.

### `run.py`

Generates the submission `.csv` file based on the data of `test.csv` stored in the folder `data/`. Our optimized model is also defined in this file.

### Some helper Functions

• `models.py`: Create the models for predicting the labels for new data points without true labels.
• `expansions.py`: Contains a function to apply polynomial expansion to our features to add extra degrees of freedom for our models.
• `proj1_helpers.py`: Contains functions which loads the `.csv` files as training or testing data, and create the `.csv` file for submission.
• `cross_validation.py`: Contains a function to build the index for k-fold cross_validation.
• `disk_helper.py`: Save/load the NumPy array to disk for further usage. Useful for saving hyper-parameters when trying a long training process.

## Notebook

It is possible to use the Jupyter notebook `project_notebook.ipynb` located in the `scripts` folder to train the best hyper-parameters for the model. In the notebook it is possible to cross-validate a logistic and a least square regression model over given lambdas and degrees.

View Github