Neurons in Deep Learning

Neuron is nothing but the central processing unit of the deep learning, In Neuron you might have multiple hidden layers which perform operations to give output to the next neuron or to the user.

Neuron is the basic building block of the neural networks, let me showcase some neuron in our body

Actual Group of Neurons in our Brains


Single Neuron structure


Simple shape of the Neuron


In above Image, 1-9 are inputs to the neuron, 10-15 are output to the other neurons or to the user, 99(circle) is the central processing unit also know as neuron. In Neuron all the calculations occurs.

In Human, inputs are taking by eyes, ears, nose, skin... outputs are like hand legs, mouth so on..

I can understand you saying "enough with your biology bullshit", now jump to deep learning.

Neurons in Deep Learning :

Neurons in Deep learning works same as neuron in biology, it takes inputs from the other neuron or from the user. Most of the time these inputs will be pre-processed, and all the calculations occurs in Neron as shown in below image.

I hope you can understand why everyone writes the Deep learning in below image format, In below image the Neuron can contain hidden layers of neurons deep-learning-neuron Neuron also sometimes called the node and has input and output signals, input signals also called as synapses

Every input value is independent of other inputs, the input variable are for one single observation for example, a row in a excel sheet or in data base.

Every output is related to one observation(one row in database), the output can of below types:

  • Continuous (0, 1,2,3,4,5 or 1.2, 1.4, 1.5)
  • Binary(0 or 1 otherwise true or false)
  • Categorical (some string values from the given categories, if you receive categorical value then you might have more than one output values)

Input Layer :

Input layer accepts input features, It provides information from the data storage(xl, spreadsheet, database etc) to the network, no computation is performed at this layer, nodes here just pass on the information(features) to the hidden layer. In above image 1-9 are input layer also known as dentries

Neuron (Hidden Layer) :

This is Neuron, Nodes of this layer are not exposed to the outer world, they are the part of the abstraction provided by any neural network. Hidden layer performs all sort of computation on the features entered through the input layer and transfer the result to the output layer.

Output Layer :

Output layer shows up the information learned by the network to the outer world.

Artificial Neuron Properties :

  • A neuron is a mathematical function modeled created based on the working of biological neurons
  • It is an basic unit in an artificial neural network
  • One or more inputs are separately weighted (basically calculated)
  • Inputs are summed and passed through a nonlinear function to produce output
  • Every neuron has an internal state called activation function
  • Each connection link carries information about the input signal process from activation/nonlinear function
  • Every neuron is connected to another(one or more) neuron via connection link

Activation Function

Activation function decides, whether a neuron should be activated or not by calculating weighted sum and further adding bias with it. The purpose of the activation function is to introduce non-linearity into the output of a neuron.

The output signal/synapses signal can be used as a input in the next layer of Neurons, or to the output.

A Neural Network without Activation function would simply be a Linear regression Model, which has limited power and does not performs good most of the times.

So most of the time, we would like out model to learn more than Linear Regression, so it is important to have activation function in Neural networks.

Also without activation function our Neural network would not be able to learn and model other complicated kinds of data such as images, videos , audio , speech etc.

Activation functions are important for a Artificial Neural Network to learn and understand the complex patterns.

Types of Activation Functions :

  • Threshold function
  • Sigmoid function
  • Rectifier
  • Hyperbolic tangent

Threshold/Step Activation function

The simplest form of activation function is binary — the neuron is either firing or not. The output y of this activation function is binary, depending on whether the input meets a specified threshold, θ.

The "signal" is sent, i.e. the output is set to one if the activation meets the threshold. step-function-neural-network A step function is a function like that used by the original Perceptron. The output is a certain value, A1, if the input sum is above a certain threshold and A0 if the input sum is below a certain threshold. The values used by the Perceptron were A1 = 1 and A0 = 0.

These kinds of step activation functions are useful for binary classification schemes. In other words, when we want to classify an input pattern into one of two groups, we can use a binary classifier with a step activation function. Another use for this would be to create a set of small feature identifiers.

Each identifier would be a small network that would output a 1 if a particular input feature is present, and a 0 otherwise. Combining multiple feature detectors into a single network would allow a very complicated clustering or classification problem to be solved.

Sigmoid Activation Function

A sigmoid function produces a curve with an "S" shape having upper limit as 1.0 and lower limit as 0.0

The sigmoid function takes any range real number and returns the output value which falls in the range of 0 to 1. sigmoid-formula-deep-learning

  • Y = output between 0 and 1 (probability estimate)
  • X = input to the function (your algorithm’s prediction e.g. mx + b)
  • e = base of natural log

when ten value X is between -2 to 2 the curve of the Sigmoid becomes more steep
It's often used in logistic regression for a binary classification and also as activation function in neural networks.

The output of the sigmoid function can be interpreted as a probability. Sigmoid function can be used for many tasks in real life: classify junk mails, classify banks transactions, etc.

The output of the sigmoid function not only used to classify YES/NO(True/False), but also be used to determine the probability of YES/NO(True/False).

Rectifier Activation Function

The Rectified Linear Unit is the most commonly used activation function in deep learning models. The function returns 0 if it receives any negative input, but for any positive value x it returns that value back.

So it can be written as f(x)=max(0,x)

Graphically it looks like this rectifier-linear-unit-function

It's surprising that such a simple function (and one composed of two linear pieces) can allow your model to account for non-linearities and interactions so well. But the ReLU function works great in most applications, and it is very widely used as a result.

Hyperbolic tangent (tanh) activation function

The Hyperbolic tangent (Tanh) function is an activation function which re-scales the values between -1 and 1 by applying a threshold just like a sigmoid function.

tanh is also one of the derivative from the sigmoid function, the values of a tanh is zero centered which helps the next neuron during propagating.

tanh-activation-function-artificial-neural-network When we apply the weighted sum of the inputs in the tanh(x), it re scales the values between -1 and 1. The large negative numbers are scaled towards -1 and large positive numbers are scaled towards 1.

If you zoom in the above images, you can understand that as the x value increase(in positive and negative side), the curve tends to become vertical.

Compare activation function


About Author

Article is written by Pavan (a) KarthiQ. Well, I am serving notice period in an MNC, Bangalore. I thought to enrich every person knowledge a little, I always have a feeling, when we teach something, we will learn more than what you know. Knowledge is the only thing that doubles when you spend it.

I have also created the reporter for Protractor Jasmine. Use for your projects without any hesitation

Comment / Suggestion Section
Point our Mistakes and Post Your Suggestions