Welcome to our GitHub page! You will find here a review of our work.

Our team is composed by:

Mattia LECCI Aude RAZAFIMBELO Paolo TESTOLINA

The goal of our work is to classify dogs images according to their breeds. To do so, we tried different architectures, techniques and optimizations as presented in the following sections.

Architectures

In order to start working directly on the Neural Network specific code, we used already existing code from Kaggle (MNIST starter code). This allowed us to learn more quickly without wasting time on image loading, encodings and language specific problems that may arise for python newbies.

MNIST Architecture

The CNN is composed by the following layers:

During the optimization processes dropout was inserted between some layers, optimizing on the number of dropout layers and the dropout probability. With this model we were able to obtain acc=15%, val_acc=5%.

After using an Adam optimizer and adding 2 dropouts in the middle of the network we were able to obtain up to acc=22%, val_acc=12%. (NewModel_best_run)

Binary Classification Architecture

We looked into a CNN that classifies cats and dogs with great accuracy (over 95%). We tried to adapt it to multiclass classification but its performance was poor, probably due to the far more complicated problem. We also tried to look into a direct extension of many binary classifiers into a single multiclass one, but what we found out was far too complicated to be implemented in this project (Erin L. Allwein, Robert E. Schapire, Yoram Singer. “Reducing Multiclass to Binary: A Unifying Approach for Margin Classifiers.”. Journal of Machine Learning Research.).

Transfer learning: VGG19

As a second baseline, we used a Transfer Learning technique based on Keras VGG19 starter. Again, to avoid wasting time on language specific problems, we just copied most of the code modifying only the interesting parts.

The main modifications were:

We then otpimize the architecture in many different ways:

We saw that in general VGG19 tended to overfit a lot, hence it required some regularization techniques. At the end, though, it improved over the results from the blank MNIST example architecture but not much. It was able to obtain up to val_acc=17%. It is possible that to avoid the overfitting problem, the regularization needed was enough to reduce the model’s performance.

We report here an example of clearly overfitted run:

Transfer learning: VGG16

We’ve replicated what we’ve done with VGG19 with other pretrained models. The first, obvious, choice was its little brother VGG16. It’s basically the same model but with 3 missing layers. We hoped that a simpler architecture would yield less overfitting problems.

This was in fact the case! Not only that, but with less overfitting control techniques we avoided crippling the real performance of the model obtaining far better results: up to acc=28%, val_acc=24%!

Transfer learning: Xception

One last model that we tried was Xception. This is a tensorflow based model using a technique called Depthwise Separable Convolution.

The particular thing about this model is that, althought it’s a very deep network (much deeper than VGG16/19), the file containing its weight weighs only a fifth of those from the VGG family. This is due to its unique architecture.

Anyway, we optimized this model less than the other since its first run yielded quite unsatisfactory results: val_acc=15% with no overfitting. Since it’s only slightly better than the MNIST architecture and worse than both VGG19 and VGG16, we decided to abandon it quite soon.

What we learned from the project

Since we all come from a telecommunication background, we are not expert programmers nor have much experience with Machine Learning-related topics.

Hence, we learned quite a lot from this project, and not only about Deep Learning!

Some transversal knowledge and skills that we learned are:

More Deep Learning related knowledge, instead, was:

As we were beginners in Neural Network, we can now assume that we have gained some basics (and something more!).

More information

Here are the Slides of our presentation, that provide further information.

Notes

Due to the many connectin problems that we had with Google Drive (as mentioned in the slides), a lot of the notebooks contain errors at the end. They are usually the notebooks that we let run overnight for many hours and when trying to save or evaluating the performance of the best model they crashed. The good thing, though, is that the training/optimization verbose outuputs were correctly saved.