Artificial Neural Network ANN design is a complex task because its performance depends on the architecture, the selected transfer function, and the learning algorithm used to train the set of synaptic weights. The aim of these algorithms is to evolve, at the same time, the three principal components of an ANN: the set of synaptic weights, the connections or architecture, and the transfer functions for each neuron.

Eight different fitness functions were proposed to evaluate the fitness of each solution and find the best design.

These functions are based on the mean square error MSE and the classification error CER and implement a strategy to avoid overtraining and to reduce the number of connections in the ANN. In addition, the ANN designed with the proposed methodology is compared with those designed manually using the well-known Back-Propagation and Levenberg-Marquardt Learning Algorithms.

Finally, the accuracy of the method is tested with different nonlinear pattern classification problems. Artificial Neural Networks ANNs are system composed of neurons organized in input, output, and hidden layers.

The neurons are connected to each other by a set of synaptic weights. An ANN is a powerful tool that has been applied in a broad range of problems such as pattern recognition, forecasting, and regression. During the learning process, the ANN continuously changes their synaptic values until the acquired knowledge is sufficient until a specific number of iterations is reached or until a goal error value is achieved.

When the learning process or the training stage has finished, it is mandatory to evaluate the generalization capabilities of the ANN using samples of the problem, different to those used during the training stage.

Finally, it is expected that the ANN can classify with an acceptable accuracy the patterns from a particular problem during the training and testing stage. Several classic algorithms to train an ANN have been proposed and developed in the last years. However, many of them can stay trapped in nondesirable solutions; that is, they will be far from the optimum or the best solution.

Moreover, most of these algorithms cannot explore multimodal and noncontinuous surfaces. BIAs have a good acceptance by the Artificial Intelligence community because they are powerful optimization tools and can solve very complex optimization problems. For a given problem, BIAs can explore big multimodal and noncontinuous search spaces and can find the best solution, near the optimum value.

This concept is defined in [ 1 ] as a property of systems composed of unintelligent agents with limited individual capabilities but with an intelligent collective behavior. There are several works that use evolutionary and bioinspired algorithms to train ANN as another fundamental form of learning [ 2 ]. Metaheuristic methods for training neural networks are based on local search, population methods, and others such as cooperative coevolutionary models [ 3 ].

An excellent work where the authors show an extensive literature review of evolutionary algorithms that are used to evolve ANN is [ 2 ]. Moreover, the researches do not involve the evolution of transfer functions, which are an important element of an ANN that determines the output of each neuron. In [ 7 ], the authors use Evolutionary Programming to get the architecture and the set of weights with the aim to solve classification and prediction problems. Another example is [ 8 ] where Genetic Programming is used to obtain graphs that represent different topologies.

In [ 10 ], the authors use a PSO algorithm to adjust the synaptic weights to model the daily rainfall-runoff relationship in Malaysia. In [ 11 ], the authors compare the back-propagation method versus basic PSO to adjust only the synaptic weights of an ANN for solving classification problems. In other works like [ 13 ], the three principle elements of an ANN are evolved at the same time: architecture, transfer functions, and synaptic weights.

This research has significant contributions in comparison with these last three works. First of all, eight fitness functions are proposed to deal with three common problems that emerge during the design of the ANN: accuracy, overfitting, and reduction of the ANN. In that sense, to handle better the problems that emerge during the design of the ANN, the fitness functions take into account the classification error, mean square error, validation error, reduction of architectures, and a combination of them.

Furthermore, this research explores the behavior of three bioinspired algorithms using different values for their parameters. In addition, the best configuration is used to generated a set of statistically valid experiments for each selected classification problem.

Another contribution of this research is related to a new metric that allows comparing efficiently the results provided by an ANN generated with the proposed methodology.

This metric takes into account the recognition rate obtained during training and testing stages where testing accuracy is more weighted in comparison to training accuracy. Finally, the results achieved by the three bioinspired algorithms are compared against those achieved with two classic learning algorithms.

The selection of the three bioinspired algorithms was done because NMPSO is a relatively new algorithm proposed in which is based on the metaphor of basic PSO technique so it is important to compare its performance with others inspired in the same phenomenon.A Chinese version is also available. Introduction Particle swarm optimization PSO is a population based stochastic optimization technique developed by Dr.

Eberhart and Dr. The system is initialized with a population of random solutions and searches for optima by updating generations.

In PSO, the potential solutions, called particles, fly through the problem space by following the current optimum particles. The detailed information will be given in following sections. PSO has been successfully applied in many areas: function optimization, artificial neural network training, fuzzy system control, and other areas where GA can be applied.

The remaining of the report includes six sections: Background: artificial life. Background: Artificial life The term "Artificial Life" ALife is used to describe research into human-made systems that possess some of the essential properties of life. ALife studies how computational techniques can help when studying biological phenomena 2. ALife studies how biological techniques can help out with computational problems The focus of this report is on the second topic.

Actually, there are already lots of computational techniques inspired by biological systems. For example, artificial neural network is a simplified model of human brain; genetic algorithm is inspired by the human evolution. Here we discuss another type of biological system - social system, more specifically, the collective behaviors of simple individuals interacting with their environment and each other.

Someone called it as swarm intelligence. All of the simulations utilized local processes, such as those modeled by cellular automata, and might underlie the unpredictable group dynamics of social behavior.

Some popular examples are floys and boids.

Both of the simulations were created to interpret the movement of organisms in a bird flock or fish school. These simulations are normally used in computer animation or computer aided design. There are two popular swarm inspired methods in computational intelligence areas: Ant colony optimization ACO and particle swarm optimization PSO. ACO was inspired by the behaviors of ants and has many successful applications in discrete optimization problems.

The original intent was to graphically simulate the choreography of bird of a bird block or fish school. However, it was found that particle swarm model can be used as an optimizer. The algorithm As stated before, PSO simulates the behaviors of bird flocking. Suppose the following scenario: a group of birds are randomly searching food in an area.

There is only one piece of food in the area being searched.By using our site, you acknowledge that you have read and understand our Cookie PolicyPrivacy Policyand our Terms of Service.

The dark mode beta is finally here. Change your preferences any time. Stack Overflow for Teams is a private, secure spot for you and your coworkers to find and share information. I want to train a neural network using Particle Swarm Optimization algorithm, but matlab toolbox doesn't have any function for train network with this algorithm, I've searched and founded some PSO toolboxes but they didn't work.

Can anybody help me please? Learn more. Training neural network using particle swarm optimization in matlab Ask Question. Asked 6 years, 2 months ago.

Active 5 years, 11 months ago. Viewed 3k times. Active Oldest Votes. Sign up or log in Sign up using Google. Sign up using Facebook.

**Neural Net Optimized with Particle Swarm Optimization: NOT Gate**

Sign up using Email and Password. Post as a guest Name. Email Required, but never shown. The Overflow Blog. Podcast Programming tutorials can be a real drag. Featured on Meta. Community and Moderator guidelines for escalating issues via new responseā¦. Feedback on Q2 Community Roadmap. Technical site integration observational experiment live on Stack Overflow.

Dark Mode Beta - help us root out low-contrast and un-converted bits. Linked 0. Related Hot Network Questions. Question feed. Stack Overflow works best with JavaScript enabled.By using our site, you acknowledge that you have read and understand our Cookie PolicyPrivacy Policyand our Terms of Service. The dark mode beta is finally here. Change your preferences any time. Stack Overflow for Teams is a private, secure spot for you and your coworkers to find and share information.

I have a problem to understand the concept of the Particle Swarm Algorithm. How it could help the Neural Network? Finally I have found the answer. For doing so reaching the place is not important at all, just we need to check the output and base on it decide which particle is the best.

Learn more. Asked 6 years, 2 months ago. Active 6 years ago. Viewed times. I would appreciate if you give me any information that help me to understand. Thanks in Advance. Mohammad Yousefi Mohammad Yousefi 2 2 gold badges 7 7 silver badges 20 20 bronze badges.

Active Oldest Votes. Hope I explain it properly. Is there any library in python that can train a neural network using PSO. Sign up or log in Sign up using Google. Sign up using Facebook. Sign up using Email and Password. Post as a guest Name. Email Required, but never shown.

The Overflow Blog. Podcast Programming tutorials can be a real drag. Featured on Meta. Community and Moderator guidelines for escalating issues via new responseā¦. Feedback on Q2 Community Roadmap. Technical site integration observational experiment live on Stack Overflow. Dark Mode Beta - help us root out low-contrast and un-converted bits. Related Hot Network Questions. Question feed. Stack Overflow works best with JavaScript enabled.Developed in by Eberhart and Kennedy, PSO is a biologically inspired optimization routine designed to mimic birds flocking or fish schooling.

PSO is not guaranteed to find the global minimum, but it does a solid job in challenging, high dimensional, non-convex, non-continuous environments. Below, are the only two equations that make up a bare bones PSO algorithm. Randomly initialize particle velocities. If stopping condition is satisfied, go to C.

### Artificial Neural Networks Optimization using Genetic Algorithm with Python

Update all particle velocities. Update all particle positions. Increment k. The main concept behind PSO, which is evident from the particle velocity equation above, is that there is a constant balance between three distinct forces pulling on each particle:. Keep in mind, that particle inertia is a double-edged-sword. If you're dealing with a noisy, highly multimodal cost function, too little inertia could result in the particles getting trapped at a local minimum, unable to climb out.

In vector form, these three forces can be seen below vector magnitude represents the weight value of that specific force :. Figure 1: a high energy particle that will keep exploring the search space. We can see in the above example that the weighting of the particles inertia and individual best overpower the swarms influence. In this scenario, the particle will continue exploring the search space rather than converge on the swarm. As another example below:.

Figure 2: a lazy particle that follows the herd. This time, the weighting assigned to the swarms influence overpowers the individual forces of the particle forcing it towards the swarm. This will result in a faster convergence, at the expense of not fully exploring the search space and potentially finding a better solution.

The implementation of a simple PSO routine in python is fairly straightforward. We are going to utilize some object-oriented programming and create a swarm of particles using a particle class.

These particles will be monitored by a main optimization class. Below is the entire code:. I hope this was helpful! If you want, you can download the entire code from my GitHub here. Check back later for my post on a more advanced particle swarm optimization routine.

Nathan Rooy About Contact Home. In vector form, these three forces can be seen below vector magnitude represents the weight value of that specific force : Figure 1: a high energy particle that will keep exploring the search space We can see in the above example that the weighting of the particles inertia and individual best overpower the swarms influence.

As another example below: Figure 2: a lazy particle that follows the herd This time, the weighting assigned to the swarms influence overpowers the individual forces of the particle forcing it towards the swarm. Built with Hugo. Powered by GitHub.Summary: I learn best with toy code that I can play with.

## Python Code Examples

This tutorial teaches backpropagation via a very simple toy example, a short python implementation. Edit: Some folks have asked about a followup article, and I'm planning to write one.

I'll tweet it out when it's complete at iamtrask. Feel free to follow if you'd be interested in reading it and thanks for all the feedback! Consider trying to predict the output column given the three input columns. We could solve this problem by simply measuring statistics between the input values and the output values.

If we did so, we would see that the leftmost input column is perfectly correlated with the output. Backpropagation, in its simplest form, measures statistics like this to make a model. Let's jump right in and use it to do this. As you can see in the "Output After Training", it works!!! Before I describe processes, I recommend playing around with the code to get an intuitive feel for how it works. This is what gives us a probability as output. Most of the secret sauce is here.

Everything in the network prepares for this operation. Let's walk through the code line by line. Recommendation: open this blog in two screens so you can see the code while you read it. That's kinda what I did while I wrote it.

Line This imports numpy, which is a linear algebra library.

### PSO Tutorial

This is our only dependency. Line This is our "nonlinearity". While it can be several kinds of functions, this nonlinearity maps a function called a "sigmoid". A sigmoid function maps any value to a value between 0 and 1. We use it to convert numbers to probabilities. It also has several other desirable properties for training neural networks. One of the desirable properties of a sigmoid function is that its output can be used to create its derivative.

This is very efficient.

If you're unfamililar with derivatives, just think about it as the slope of the sigmoid function at a given point as you can see above, different points have different slopes. For more on derivatives, check out this derivatives tutorial from Khan Academy.The source code used in this tutorial is available in my GitHub page. This tutorial is also available at TowardsDataScience here.

A quick summary of this tutorial is extracting the feature vector bins hue channel histogram and reducing it to just element by using a filter-based technique using the standard deviation. The ANN was not completely created as just the forward pass was made ready but there is no backward pass for updating the network weights. The solution to this problem is using an optimization technique for updating the network weights.

This tutorial uses the genetic algorithm GA for optimizing the network weights. The book is available at Springer at this link. You can find all details within this book. The source code used in this tutorial is available in my GitHub page here.

Before starting this tutorial, I recommended reading about how the genetic algorithm works and its implementation in Python using NumPy from scratch based on my previous tutorials found at the links listed in the Resources section at the end of the tutorial.

After understanding how GA works based on numerical examples in addition to implementation using Python, we can start using GA to optimize the ANN by updating its weights parameters. GA creates multiple solutions to a given problem and evolves them through a number of generations.

Each solution holds all parameters that might help to enhance the results. For ANN, weights in all layers help achieve high accuracy. According to the network structure discussed in the previous tutorial and given in the figure below, the ANN has 4 layers 1 input, 2 hidden, and 1 output. Any weight in any layer will be part of the same solution. Looking at the above figure, the parameters of the network are in matrix form because this makes calculations of ANN much easier.

For each layer, there is an associated weights matrix. Just multiply the inputs matrix by the parameters matrix of a given layer to return the outputs in such layer. Chromosomes in GA are 1D vectors and thus we have to convert the weights matrices into 1D vectors. This makes us need to convert the matrix to vector and vice versa. This figure is referred to as the main figure. Each solution in the population will have two representations.

Because a solution in GA is represented as a single 1D vector, such 3 individual 1D vectors will be concatenated into a single 1D vector. Each solution will be represented as a vector of length 24, The function accepts an argument representing the population of all solutions in order to loop through them and return their vector representation.

For each solution in matrix form, there is an inner loop that loops through its three matrices. For each matrix, it is converted into a vector using the numpy.

Note that we used the numpy. The reason is that numpy. In other words, calling this function for two lists returns a new single list with numbers from both lists. This is suitable in order to create just a 1D chromosome for each solution. But numpy. Calling it for two lists, it returns a new list which is split into two sub-lists. This is not our objective.

## Responses to

Pso ann python code