Initializing weights

We also have the option to initialize the neurons on each layer with specific weights. This is not a prerequisite, as they will be automatically initialized with small random numbers if not specified otherwise. Weight initialization practices are actually a whole separate sub-field of study in neural networks. It is prominently noted that the careful initialization of the network can significantly speed up the learning process.

You can use the kernel_initializer and bais_initializer parameters to set both the weights and biases of each layer, respectively. Remember that these very weights will represent the knowledge that's acquired by our network, which is why ideal initializations can significantly boost its learning:

A comprehensive review of the different parameter values is beyond the scope of this chapter. We may encounter some use cases where tweaking these parameters is beneficial later on (refer to chapter optimization). Some values for the kernel_initializer parameter include the following:

• glorot_uniform: The weights are drawn from samples of uniform distributions between -limit and limit. Here, the term limit is defined as sqrt(6 / (fan_in + fan_out)). The term fan_in simply denotes the number of input units in the weight tensor, while fan_out is the number of output units in the weight tensor.
• random_uniform: The weights are randomly initialized with small uniform values ranging between -0.05 and 0.05.
• random_normal: The weights are initialized for obeying a Gaussian distribution[1], with a mean of 0 and a standard deviation of 0.05.
• zero: The layer weights are initialized at zero.