Neural Network

What is Neural Network?
1. Dive into the neuron
2. How does a neural network simulate an arbitrary function
3. Why do we need neural networks
How to construct a neural network
1. Fully connected neural network
2. Use graphical tool to design neural network
3. The "activation function" of the output layer
How to train a neural network
1. Learning algorithm and principle
2. Build and train neural networks from scratch
3. Rewrite the code using PyTorch
4. Use graphical tool to train neural network
Some important problems of neural network
1. Network structure
2. Overfitting
3. Underfitting
4. Overfitting vs underfitting
5. Initialization
6. Vanishing gradient and exploding gradient
Convolutional Neural Network(CNN)
1. 1D-convolution
2. 1D-convolution experiments
3. 1D-pooling
4. 1D-CNN experiments
5. 2D-CNN
6. 2D-CNN experiments
Recurrent Neural Network(RNN)
1. Vanilla RNN
2. Seq2seq, Autoencoder, Encoder-Decoder
3. Advanced RNN
4. RNN classification experiment
Natural language processing
1. Embedding: Convert symbols to values
2. Text Classification 1
3. Text Classification 2
4. TextCNN
5. Entity recognition
6. Word segmentation, POS tagging and chunking
7. Sequence tagging in action
8. Bidirectional RNN
9. BI-LSTM-CRF
10. Attention
Language Models
1. n-gram Model: Unigram
2. n-gram Model: Bigram
3. n-gram Model: Trigram
4. RNN Language Model
5. Transformer Language Model
Linear Algebra
1. Vector
2. Matrix
3. Dive in matrix multiplication
4. Tensor

How does a neural network simulate an arbitrary function

Overview

The reason why the neural network is powerful lies in its powerful simulation ability. In theory, it can simulate arbitrary functions with infinitely small errors.

In other words, we can use neural networks to construct arbitrary functions and obtain arbitrary algorithms.

We use some visual examples here to help you gain some intuitive understanding.

Simulation of unary function

Straight line

This is the simplest case, we can simulate it by using a neuron without activation function.

f(x) = wx + b


 $w$ 1
 $b$ 0

By adjusting the $w, b$ parameters, any straight line can be simulated.

Step function

We use a neuron with Sigmoidactivation function to simulate it.


 $w$ 30
 $b$ 0

As the $w$ parameter continues to increase, the neural network will gradually approach the function.

Rectangular pulse function

We divide it into several steps:

Use a single neuron to simulate the left half of the function.

f_1(x) = \text{sigmoid}(w_1x+b_1)


 $w_1$ 20
 $b_1$ 20

Use a single neurons to simulate the right half of the function (upside down).

f_2(x) = \text{sigmoid}(w_2x+b_2)


 $w_2$ 20
 $b_2$ -20

Use another neuron to synthesize the images of the first 2 steps

f_3(x, y) = \text{sigmoid}(w_{31}x + w_{32}y + b_3)


 $w_{31}$ 10
 $w_{32}$ -10
 $b_3$ -5

The result obtained is a good approximation of the objective function.

Other unary functions

Using the rectangular impulse function, we can easily approximate other arbitrary functions, just like the principle of integration.


 $n$ 10

Experiment

Complete Broken Line mission and observe the function corresponding to each neuron.

This is the simplest case, we can simulate it by using a neuron without activation function.

f(x, y) = w_1x + w_2y + b


 $w_1$ 0
 $w_2$ 1
 $b$ 0

By adjusting the parameters of $w_1, w_2, b$ , any plane can be simulated.

Binary Step Function

We use a neuron with Sigmoidactivation function to simulate it.

f(x) = \text{sigmoid}(w_1x + w_2y + b)


 $w_1$ 0
 $w_2$ 30
 $b$ 0

Binary rectangular impulse function

Similar to the case of unary functions, we implement it step by step:

Use a single neuron to simulate an edge of the function

f_1(x, y) = \text{sigmoid}(w_{11}x + w_{12}y + b_1)


 $w_{11}$ 0
 $w_{12}$ 20
 $b_1$ 20

Then we can get the following function:


 $w_i$ 10
 $w_j$ -10
 $b_i$ -5

Finally, the following functions can be synthesized


 $w_{51}$ 10
 $w_{52}$ -10
 $w_{53}$ 10
 $w_{54}$ -10
 $b_5$ -15

The final neural network structure is shown in the figure below:

Other binary functions

Using the binary rectangular impulse function, we can easily approximate any other binary function, just like the principle of integration.

Experiment

Complete the Circle mission and observe the function corresponding to each neuron.

The principle is the same, imagine for yourself! 😥

Question

We already have digital circuits and software program algorithms, why do we need neural networks?

Software programs built on digital circuits can also simulate arbitrary functions, so why invent artificial neural networks?