How learning works

This short article is made to explan steps needed to create recognition recipe.

For AI(or neural networks) these steps are more or less the same for any segmentation task.

After all required images are received first thisg to do is to annotate them. Usually this is done by student.

At least 15 images should be manually annotated.

Annotation process is consisting of two phases:

1. selecting contour of the objects
2. assigning a class(fancy name for candy type) to each contour

After the job is done i'm inspecting the images.

Just 15 images is not enough fo AI to understand all concepts of what candy is. So i'm using quite standard trick to extend the data set. To do that i'm addig small rotation in both directions. Together with “empty” images(with empty blister and truned over candies) this will generate a dataset of about 300 images(black images are masks with candy position and class, windows can't decode them propperly):

After dataset is prepared the learning process starts. This takes about 6 hours. All this time i need to monitor the results. If AI learning not correctly i need to interrupt it, correct some parameters and arestart learning process.

Inavoidable part of learning process. As AI learns it tries to find relation between pixels on screen and masks drawn by student. If concept is understood incorrectly - most likely there is an error at annotation, dataset balance or model needs to be tweeked. If this happens - learning process must be restarted anfter problem is eliminated.

You can think of recipe as of some program that can work with only one type of blister. Recipes are done in separate programs for multiple reasons:

• stability. if there is a critical error in recipe this will not brake other recipes
• experiments. recipes can be made differently. this allows to experiment quicly with different models and approaches
• modularity. recipes can be added or deleted without modification of program itself

Recipe is created from some sort of template. And in initial stage it's quite 'stupid' and can't even find a blister on image.

The goal of this step is to establish relation between blister position/angle and candy positions. After playing around with thansformation matrices i get result similar to this:

These records are describing where candy is located, what type it is and what its size. If this 'table' is built correctly i will get something like this:

When two candies are close to each other or touchng AI thinks that this is one single candy. This creates false-positives. To avoid this Voronoi segmentation is used. It puts virtual borders between candies ensuring that touching candies are separate.

Each candy is having different amount of different colors. This makes possible to generate a color combination pattern for each candy type for each candy position. But since each candy type may have variations - we are creating a database which contains several patterns for each candy type.

During analysis each candy is compared against database of patterns expected at specified position. If the distance is close enough - the candy is good.

Now it's time to test the recipe on good images that it's never seen. They should be all green:

The last step before it's done. On this step i'm checking how confident AI about it's decisions. This allows to discover if there is possible problems with recognition on the line.