Neural network are meant to solve problems in the same way a human brain would. I always wanted to make a neural network. To replicate God’s creation.
Implementing a neural network
Using the same data set as yesterday, I followed KAI XUAN WEI’s A simple 2-layer neural network model to implement a 2-layer full-connected neural network from scratch.
The Jupyter Notebook for this little project is found here.
From what I understood, the steps are as follows:
- Get your data
- Initialize global network edge weights
- Train the neural network
- Make the predictions
Data sets for different purposes
In this exercise, we have 3 types of data sets: one for training, one for validation, and one for testing. The training set is for training the model, the validation set is for testing how well the model performed, and the test set is for making predictions. In both the training set and validation set, the values to be predicted is known; whereas, the values for the test set is not known (this is what we want to predict).
Loss function
Loss functions, otherwise known as cost functions, are used to optimize model parameters. In this exercise, a softmax loss function was implemented.
In mathematics, the softmax function, or normalized exponential function, is a generalization of the logistic function that “squashes” a K-dimensional vector z of arbitrary real values to a K-dimensional vector σ(z) of real values in the range (0, 1) that add up to 1. –wikipedia
This means that for a given outcome with e.g. 3 possibilities (A, B, or C), the probability of an input being classified as A + B + C will be 1. And in terms of optimization, we want the true outcome to have the highest probability.
Evaluation
Since the neural network was trained on Kaggle’s titanic data, I wanted to see how accurate the predictions were. Submitting the predictions to Kaggle returned a score of 0.59330.
Interesting. This performed worst than yesterday’s logistic regression.
Admittedly, there are different types of neural networks with different number of hidden layers, nodes, filters, and ways to train and connect the network. Furthermore, this shows that neural networks are not always the best machine learning approach, and that multiple approaches should instead be tried.