from IPython.display import HTML

HTML('''<script>
code_show=true; 
function code_toggle() {
 if (code_show){
 $('div.input').hide();
 } else {
 $('div.input').show();
 }
 code_show = !code_show
} 
$( document ).ready(code_toggle);
</script>
<form action="javascript:code_toggle()"><input type="submit" value="Click here to toggle on/off the raw code."></form>''')

Training and Applying a CIFAR classifierÂ¶

In this project we are going to develop a CIFAR classifier.

To do so we will use Python and Keras, a high-level neural network API, which is written in Python and capable of running on top of either TensortFlow, Theano or CNTK.

Hence, first of all we will load all the required libraries:

from __future__ import print_function
import keras
from keras.datasets import cifar10
from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, Flatten, Input, Convolution2D, MaxPooling2D, AveragePooling2D, ZeroPadding2D, merge, Reshape
from keras.layers import Conv2D, MaxPooling2D
from keras.optimizers import SGD
from sklearn.metrics import log_loss

Using TensorFlow backend.

After having loaded all the modules needed to implement the classifier, we will proceed to define some values, which will be used as parameters for our model.

Hence we will define the batch size, i.e. the number of that that are going to be propagated through the network, to 32. Also we will also specify the number of epochs, i.e. we will train our model with 'epochs' full training cycles.

Furthermore, we will define wheter we are willing to perform data augmentation. And we will define that our data will count with 10 classes (number of classes in the dataset), this will be used to define our data as categorical.

All what we said, can be said defined in the code below:

batch_size = 32
num_classes = 10
epochs = 3
data_augmentation = False

Once we have already defined the parameters that we are going to use, we will proceed to divide our data on test and training sets.

In order to do so, we will shuffle and split the data, this is done in the code below:

# The data, shuffled and split between train and test sets:
# Load Cifar10 data. Please implement your own load_data() module for your own dataset
(x_train, y_train), (x_test, y_test) = cifar10.load_data()
print('x_train shape:', x_train.shape)
print(x_train.shape[0], 'train samples')
print(x_test.shape[0], 'test samples')

x_train shape: (50000, 32, 32, 3)
50000 train samples
10000 test samples

And we will proceed to define our data as categorical, specifying that our data will have 10 classes, as we previously defined.

# Convert class vectors to binary class matrices.
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)

Now, that we have already divided our data on train and test set, we will proceed to generate our model.

To do so we will proceed to generate a model with 2 2D Convolutional layers, in which we will define the dimensionality of the outpit space as 32 and the kernel size as 3 x 3. Additionally we will define a ReLu activation, since it produces sparsity and reduced likelihood of vanishing gradient.

Furthermore, between those 2 layers we will perform regularization by performing a Dropout with a rate equal to 0.2, to prevent complex co-adaptations on the training data.

And after the second layer we will perform max pooling, in order to reduce the dimensionality of the output of this second layer followed by a flatten process to reduce our input to a 15-d vector to then use a Dense layer with 512 units, a fully connected neural network layer where each input node is connected to each output node.

Finally we apply reguralization again via Dropout, this time with a rate equal to 0.5, and then we apply again a dense layer with 10 units and softmax activation.

All of this is done in the code below:

def simple_model(x_train, num_classes):
    model = Sequential()

    model.add(Conv2D(32, (3, 3), padding='same',
                     input_shape=x_train.shape[1:]))
    model.add(Activation('relu'))
    model.add(Conv2D(32, (3, 3)))
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Dropout(0.25))

    model.add(Conv2D(64, (3, 3), padding='same'))
    model.add(Activation('relu'))
    model.add(Conv2D(64, (3, 3)))
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Dropout(0.25))

    model.add(Flatten())
    model.add(Dense(512))
    model.add(Activation('relu'))
    model.add(Dropout(0.5))
    model.add(Dense(num_classes))
    model.add(Activation('softmax'))
    return model

from keras.applications import vgg16
from keras.applications.inception_v3 import InceptionV3
from keras.preprocessing import image
from keras.models import Model
from keras.layers import Dense, GlobalAveragePooling2D
from keras import backend as K
def vgg16_model(img_rows, img_cols, channel=1, num_classes=None):


    # create the base pre-trained model
    # base_model = InceptionV3(weights='imagenet', include_top=False)
    base_model = vgg16.VGG16(weights='imagenet', include_top=False)

    # add a global spatial average pooling layer
    x = base_model.output
    x = GlobalAveragePooling2D()(x)
    # let's add a fully-connected layer
    x = Dense(128, activation='relu')(x)
    # and a logistic layer -- let's say we have 200 classes
    predictions = Dense(num_classes, activation='softmax')(x)

    # this is the model we will train
    model = Model(inputs=base_model.input, outputs=predictions)

    # first: train only the top layers (which were randomly initialized)
    # i.e. freeze all convolutional InceptionV3 layers
    for layer in base_model.layers:
        layer.trainable = False

    # Learning rate is changed to 0.001
    sgd = SGD(lr=1e-3, decay=1e-6, momentum=0.9, nesterov=True)
    model.compile(optimizer=sgd, loss='categorical_crossentropy', metrics=['accuracy'])


    return model, base_model

    # Create model
    #model = Model(img_input, x)

model, base_model = vgg16_model(x_train.shape[1], x_train.shape[2], 3, num_classes)
# model = simple_model(x_train, num_classes)

After having specified the layers on our model, we are going to select a gradient descent optimization algorithm to find an approapiate local optimum.

The selected optimization algorithms in our case will be a RMSprop with a learning rate equal to 0.0001 and a decay equal to 1e-6 and SGD with a learning rate 0.01, a momentum equal to 0.9, a decay equal to the divison between its learning rate and the number of epochs that we defined previously.

# initiate RMSprop optimizer
opt = keras.optimizers.rmsprop(lr=0.0001, decay=1e-6)

Since now we have divided our data into train and test sets, and also we have specified the layers of our model and defined two optmization algorithms RMSprop and SGD. Now we will proceed to train our model. To do so, we will use the categorical crossentropy as our loss function and the sgd optimizer, that we previously defined. Furthermore, we will take accuracy as our main metric.

# Let's train the model using RMSprop
model.compile(loss='categorical_crossentropy',
              optimizer=opt,
              metrics=['accuracy'])

x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_test /= 255

if not data_augmentation:
    print('Not using data augmentation.')
    # Start training (or finetuning)
    hist = model.fit(x_train, y_train,
              batch_size=batch_size,
              epochs=epochs,
              validation_data=(x_test, y_test),
              shuffle=True)
else:
    print('Using real-time data augmentation.')
    # This will do preprocessing and realtime data augmentation:
    datagen = ImageDataGenerator(
        featurewise_center=False,  # set input mean to 0 over the dataset
        samplewise_center=False,  # set each sample mean to 0
        featurewise_std_normalization=False,  # divide inputs by std of the dataset
        samplewise_std_normalization=False,  # divide each input by its std
        zca_whitening=False,  # apply ZCA whitening
        rotation_range=0,  # randomly rotate images in the range (degrees, 0 to 180)
        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
        horizontal_flip=True,  # randomly flip images
        vertical_flip=False)  # randomly flip images

    # Compute quantities required for feature-wise normalization
    # (std, mean, and principal components if ZCA whitening is applied).
    datagen.fit(x_train)

    # Fit the model on the batches generated by datagen.flow().
    model.fit_generator(datagen.flow(x_train, y_train,
                                     batch_size=batch_size),
                        steps_per_epoch=x_train.shape[0] // batch_size,
                        epochs=epochs,
                        validation_data=(x_test, y_test))

Not using data augmentation.
Train on 50000 samples, validate on 10000 samples
Epoch 1/3
50000/50000 [==============================] - 197s - loss: 1.7313 - acc: 0.4160 - val_loss: 1.5055 - val_acc: 0.4879
Epoch 2/3
50000/50000 [==============================] - 195s - loss: 1.4174 - acc: 0.5153 - val_loss: 1.3844 - val_acc: 0.5250
Epoch 3/3
50000/50000 [==============================] - 194s - loss: 1.3320 - acc: 0.5426 - val_loss: 1.3339 - val_acc: 0.5375

# Make predictions
predictions_valid = model.predict(x_test, batch_size=batch_size, verbose=1)

# Cross-entropy loss score
score = log_loss(y_test, predictions_valid)
print(score)

10000/10000 [==============================] - 29s    
1.24423578936

# at this point, the top layers are well trained and we can start fine-tuning
# convolutional layers from inception V3. We will freeze the bottom N layers
# and train the remaining top layers.

# let's visualize layer names and layer indices to see how many layers
# we should freeze:
for i, layer in enumerate(base_model.layers):
   print(i, layer.name)

0 input_8
1 block1_conv1
2 block1_conv2
3 block1_pool
4 block2_conv1
5 block2_conv2
6 block2_pool
7 block3_conv1
8 block3_conv2
9 block3_conv3
10 block3_pool
11 block4_conv1
12 block4_conv2
13 block4_conv3
14 block4_pool
15 block5_conv1
16 block5_conv2
17 block5_conv3
18 block5_pool

# we chose to train the top 2 inception blocks, i.e. we will freeze
# the first 249 layers and unfreeze the rest:
for layer in model.layers[:16]:
   layer.trainable = False
for layer in model.layers[16:]:
   layer.trainable = True

# we need to recompile the model for these modifications to take effect
# we use SGD with a low learning rate
from keras.optimizers import SGD
model.compile(optimizer=SGD(lr=0.0001, momentum=0.9), loss='categorical_crossentropy', metrics=['accuracy'])

# we train our model again (this time fine-tuning the top 2 inception blocks
# alongside the top Dense layers

# And now train again
print('Not using data augmentation.')
# Start training (or finetuning)
hist = model.fit(x_train, y_train,
          batch_size=batch_size,
          epochs=epochs,
          validation_data=(x_test, y_test),
          shuffle=True)

Not using data augmentation.
Train on 50000 samples, validate on 10000 samples
Epoch 1/1
50000/50000 [==============================] - 196s - loss: 0.9625 - acc: 0.6644 - val_loss: 0.9712 - val_acc: 0.6589

from IPython.display import SVG
from keras.utils.vis_utils import model_to_dot

SVG(model_to_dot(model).create(prog='dot', format='svg'))

%matplotlib inline
import numpy as np
import matplotlib

import matplotlib.pyplot as plt
# list all data in history
#print(hist.history)
#print(hist.history.keys())
#  "Accuracy"
plt.plot([1,2, 3, 4])
plt.plot(hist.history['acc'])
plt.plot(hist.history['val_acc'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()
# "Loss"
plt.plot(hist.history['loss'])
plt.plot(hist.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()

t-sne visualization

'''Visualization of the filters of VGG16, via gradient ascent in input space.
This script can run on CPU in a few minutes (with the TensorFlow backend).
Results example: http://i.imgur.com/4nj4KjN.jpg
'''
%matplotlib inline
import numpy as np
import matplotlib

import matplotlib.pyplot as plt
from __future__ import print_function

from scipy.misc import imsave
import numpy as np
import time
from keras.applications import vgg16
from keras import backend as K

# dimensions of the generated pictures for each filter.
img_width = 128
img_height = 128

# the name of the layer we want to visualize
# (see model definition at keras/applications/vgg16.py)
layer_name = 'block5_conv1'

# util function to convert a tensor into a valid image


def deprocess_image(x):
    # normalize tensor: center on 0., ensure std is 0.1
    x -= x.mean()
    x /= (x.std() + 1e-5)
    x *= 0.1

    # clip to [0, 1]
    x += 0.5
    x = np.clip(x, 0, 1)

    # convert to RGB array
    x *= 255
    if K.image_data_format() == 'channels_first':
        x = x.transpose((1, 2, 0))
    x = np.clip(x, 0, 255).astype('uint8')
    return x

# build the VGG16 network with ImageNet weights
# model = vgg16.VGG16(weights='imagenet', include_top=False)
# print('Model loaded.')

model.summary()

# this is the placeholder for the input images
input_img = model.input

# get the symbolic outputs of each "key" layer (we gave them unique names).
layer_dict = dict([(layer.name, layer) for layer in model.layers[1:]])


def normalize(x):
    # utility function to normalize a tensor by its L2 norm
    return x / (K.sqrt(K.mean(K.square(x))) + 1e-5)


kept_filters = []
for filter_index in range(0, 150):
    # we only scan through the first 200 filters,
    # but there are actually 512 of them
    print('Processing filter %d' % filter_index)
    start_time = time.time()

    # we build a loss function that maximizes the activation
    # of the nth filter of the layer considered
    layer_output = layer_dict[layer_name].output
    if K.image_data_format() == 'channels_first':
        loss = K.mean(layer_output[:, filter_index, :, :])
    else:
        loss = K.mean(layer_output[:, :, :, filter_index])

    # we compute the gradient of the input picture wrt this loss
    grads = K.gradients(loss, input_img)[0]

    # normalization trick: we normalize the gradient
    grads = normalize(grads)

    # this function returns the loss and grads given the input picture
    iterate = K.function([input_img], [loss, grads])

    # step size for gradient ascent
    step = 1.

    # we start from a gray image with some random noise
    if K.image_data_format() == 'channels_first':
        input_img_data = np.random.random((1, 3, img_width, img_height))
    else:
        input_img_data = np.random.random((1, img_width, img_height, 3))
    input_img_data = (input_img_data - 0.5) * 20 + 128

    # we run gradient ascent for 20 steps
    for i in range(20):
        loss_value, grads_value = iterate([input_img_data])
        input_img_data += grads_value * step

        #print('Current loss value:', loss_value)
        if loss_value <= 0.:
            # some filters get stuck to 0, we can skip them
            break

    # decode the resulting input image
    if loss_value > 0:
        img = deprocess_image(input_img_data[0])
        kept_filters.append((img, loss_value))
    end_time = time.time()
    print('Filter %d processed in %ds' % (filter_index, end_time - start_time))

# we will stich the best 64 filters on a 8 x 8 grid.
n = 6

# the filters that have the highest loss are assumed to be better-looking.
# we will only keep the top 64 filters.
kept_filters.sort(key=lambda x: x[1], reverse=True)
kept_filters = kept_filters[:n * n]

# build a black picture with enough space for
# our 8 x 8 filters of size 128 x 128, with a 5px margin in between
margin = 5
width = n * img_width + (n - 1) * margin
height = n * img_height + (n - 1) * margin
stitched_filters = np.zeros((width, height, 3))

# fill the picture with our saved filters
for i in range(n):
    for j in range(n):
        img, loss = kept_filters[i * n + j]
        stitched_filters[(img_width + margin) * i: (img_width + margin) * i + img_width,
                         (img_height + margin) * j: (img_height + margin) * j + img_height, :] = img

# save the result to disk
fig =plt.figure(figsize=(15, 15), dpi= 80, facecolor='w', edgecolor='k')
img = plt.imshow(stitched_filters)
img.set_cmap('hot')
plt.axis('off')
plt.show()

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         (None, None, None, 3)     0         
_________________________________________________________________
block1_conv1 (Conv2D)        (None, None, None, 64)    1792      
_________________________________________________________________
block1_conv2 (Conv2D)        (None, None, None, 64)    36928     
_________________________________________________________________
block1_pool (MaxPooling2D)   (None, None, None, 64)    0         
_________________________________________________________________
block2_conv1 (Conv2D)        (None, None, None, 128)   73856     
_________________________________________________________________
block2_conv2 (Conv2D)        (None, None, None, 128)   147584    
_________________________________________________________________
block2_pool (MaxPooling2D)   (None, None, None, 128)   0         
_________________________________________________________________
block3_conv1 (Conv2D)        (None, None, None, 256)   295168    
_________________________________________________________________
block3_conv2 (Conv2D)        (None, None, None, 256)   590080    
_________________________________________________________________
block3_conv3 (Conv2D)        (None, None, None, 256)   590080    
_________________________________________________________________
block3_pool (MaxPooling2D)   (None, None, None, 256)   0         
_________________________________________________________________
block4_conv1 (Conv2D)        (None, None, None, 512)   1180160   
_________________________________________________________________
block4_conv2 (Conv2D)        (None, None, None, 512)   2359808   
_________________________________________________________________
block4_conv3 (Conv2D)        (None, None, None, 512)   2359808   
_________________________________________________________________
block4_pool (MaxPooling2D)   (None, None, None, 512)   0         
_________________________________________________________________
block5_conv1 (Conv2D)        (None, None, None, 512)   2359808   
_________________________________________________________________
block5_conv2 (Conv2D)        (None, None, None, 512)   2359808   
_________________________________________________________________
block5_conv3 (Conv2D)        (None, None, None, 512)   2359808   
_________________________________________________________________
block5_pool (MaxPooling2D)   (None, None, None, 512)   0         
_________________________________________________________________
global_average_pooling2d_1 ( (None, 512)               0         
_________________________________________________________________
dense_1 (Dense)              (None, 128)               65664     
_________________________________________________________________
dense_2 (Dense)              (None, 10)                1290      
=================================================================
Total params: 14,781,642
Trainable params: 4,786,570
Non-trainable params: 9,995,072
_________________________________________________________________
Processing filter 0
Filter 0 processed in 1s
Processing filter 1
Filter 1 processed in 1s
Processing filter 2
Filter 2 processed in 1s
Processing filter 3
Filter 3 processed in 1s
Processing filter 4
Filter 4 processed in 1s
Processing filter 5
Filter 5 processed in 1s
Processing filter 6
Filter 6 processed in 1s
Processing filter 7
Filter 7 processed in 1s
Processing filter 8
Filter 8 processed in 1s
Processing filter 9
Filter 9 processed in 1s
Processing filter 10
Filter 10 processed in 1s
Processing filter 11
Filter 11 processed in 1s
Processing filter 12
Filter 12 processed in 0s
Processing filter 13
Filter 13 processed in 1s
Processing filter 14
Filter 14 processed in 1s
Processing filter 15
Filter 15 processed in 1s
Processing filter 16
Filter 16 processed in 0s
Processing filter 17
Filter 17 processed in 1s
Processing filter 18
Filter 18 processed in 1s
Processing filter 19
Filter 19 processed in 0s
Processing filter 20
Filter 20 processed in 1s
Processing filter 21
Filter 21 processed in 0s
Processing filter 22
Filter 22 processed in 0s
Processing filter 23
Filter 23 processed in 1s
Processing filter 24
Filter 24 processed in 1s
Processing filter 25
Filter 25 processed in 1s
Processing filter 26
Filter 26 processed in 0s
Processing filter 27
Filter 27 processed in 1s
Processing filter 28
Filter 28 processed in 1s
Processing filter 29
Filter 29 processed in 1s
Processing filter 30
Filter 30 processed in 1s
Processing filter 31
Filter 31 processed in 1s
Processing filter 32
Filter 32 processed in 1s
Processing filter 33
Filter 33 processed in 0s
Processing filter 34
Filter 34 processed in 1s
Processing filter 35
Filter 35 processed in 0s
Processing filter 36
Filter 36 processed in 1s
Processing filter 37
Filter 37 processed in 1s
Processing filter 38
Filter 38 processed in 1s
Processing filter 39
Filter 39 processed in 1s
Processing filter 40
Filter 40 processed in 1s
Processing filter 41
Filter 41 processed in 0s
Processing filter 42
Filter 42 processed in 1s
Processing filter 43
Filter 43 processed in 1s
Processing filter 44
Filter 44 processed in 1s
Processing filter 45
Filter 45 processed in 1s
Processing filter 46
Filter 46 processed in 1s
Processing filter 47
Filter 47 processed in 1s
Processing filter 48
Filter 48 processed in 1s
Processing filter 49
Filter 49 processed in 1s
Processing filter 50
Filter 50 processed in 1s
Processing filter 51
Filter 51 processed in 1s
Processing filter 52
Filter 52 processed in 1s
Processing filter 53
Filter 53 processed in 1s
Processing filter 54
Filter 54 processed in 0s
Processing filter 55
Filter 55 processed in 1s
Processing filter 56
Filter 56 processed in 1s
Processing filter 57
Filter 57 processed in 0s
Processing filter 58
Filter 58 processed in 1s
Processing filter 59
Filter 59 processed in 1s
Processing filter 60
Filter 60 processed in 0s
Processing filter 61
Filter 61 processed in 1s
Processing filter 62
Filter 62 processed in 1s
Processing filter 63
Filter 63 processed in 0s
Processing filter 64
Filter 64 processed in 1s
Processing filter 65
Filter 65 processed in 1s
Processing filter 66
Filter 66 processed in 1s
Processing filter 67
Filter 67 processed in 0s
Processing filter 68
Filter 68 processed in 0s
Processing filter 69
Filter 69 processed in 0s
Processing filter 70
Filter 70 processed in 1s
Processing filter 71
Filter 71 processed in 0s
Processing filter 72
Filter 72 processed in 0s
Processing filter 73
Filter 73 processed in 1s
Processing filter 74
Filter 74 processed in 1s
Processing filter 75
Filter 75 processed in 1s
Processing filter 76
Filter 76 processed in 1s
Processing filter 77
Filter 77 processed in 1s
Processing filter 78
Filter 78 processed in 1s
Processing filter 79
Filter 79 processed in 1s
Processing filter 80
Filter 80 processed in 1s
Processing filter 81
Filter 81 processed in 1s
Processing filter 82
Filter 82 processed in 1s
Processing filter 83
Filter 83 processed in 1s
Processing filter 84
Filter 84 processed in 1s
Processing filter 85
Filter 85 processed in 1s
Processing filter 86
Filter 86 processed in 1s
Processing filter 87
Filter 87 processed in 1s
Processing filter 88
Filter 88 processed in 1s
Processing filter 89
Filter 89 processed in 1s
Processing filter 90
Filter 90 processed in 1s
Processing filter 91
Filter 91 processed in 1s
Processing filter 92
Filter 92 processed in 1s
Processing filter 93
Filter 93 processed in 1s
Processing filter 94
Filter 94 processed in 1s
Processing filter 95
Filter 95 processed in 1s
Processing filter 96
Filter 96 processed in 1s
Processing filter 97
Filter 97 processed in 1s
Processing filter 98
Filter 98 processed in 1s
Processing filter 99
Filter 99 processed in 1s
Processing filter 100
Filter 100 processed in 1s
Processing filter 101
Filter 101 processed in 1s
Processing filter 102
Filter 102 processed in 1s
Processing filter 103
Filter 103 processed in 1s
Processing filter 104
Filter 104 processed in 1s
Processing filter 105
Filter 105 processed in 1s
Processing filter 106
Filter 106 processed in 1s
Processing filter 107
Filter 107 processed in 1s
Processing filter 108
Filter 108 processed in 1s
Processing filter 109
Filter 109 processed in 1s
Processing filter 110
Filter 110 processed in 1s
Processing filter 111
Filter 111 processed in 1s
Processing filter 112
Filter 112 processed in 1s
Processing filter 113
Filter 113 processed in 1s
Processing filter 114
Filter 114 processed in 1s
Processing filter 115
Filter 115 processed in 1s
Processing filter 116
Filter 116 processed in 1s
Processing filter 117
Filter 117 processed in 1s
Processing filter 118
Filter 118 processed in 1s
Processing filter 119
Filter 119 processed in 1s
Processing filter 120
Filter 120 processed in 1s
Processing filter 121
Filter 121 processed in 1s
Processing filter 122
Filter 122 processed in 1s
Processing filter 123
Filter 123 processed in 1s
Processing filter 124
Filter 124 processed in 1s
Processing filter 125
Filter 125 processed in 1s
Processing filter 126
Filter 126 processed in 1s
Processing filter 127
Filter 127 processed in 1s
Processing filter 128
Filter 128 processed in 1s
Processing filter 129
Filter 129 processed in 1s
Processing filter 130
Filter 130 processed in 1s
Processing filter 131
Filter 131 processed in 1s
Processing filter 132
Filter 132 processed in 1s
Processing filter 133
Filter 133 processed in 1s
Processing filter 134
Filter 134 processed in 1s
Processing filter 135
Filter 135 processed in 1s
Processing filter 136
Filter 136 processed in 1s
Processing filter 137
Filter 137 processed in 1s
Processing filter 138
Filter 138 processed in 3s
Processing filter 139
Filter 139 processed in 1s
Processing filter 140
Filter 140 processed in 1s
Processing filter 141
Filter 141 processed in 1s
Processing filter 142
Filter 142 processed in 1s
Processing filter 143
Filter 143 processed in 1s
Processing filter 144
Filter 144 processed in 1s
Processing filter 145
Filter 145 processed in 1s
Processing filter 146
Filter 146 processed in 1s
Processing filter 147
Filter 147 processed in 1s
Processing filter 148
Filter 148 processed in 1s
Processing filter 149
Filter 149 processed in 1s

Task 5: Image Classification using Convolutional Neural NetworksÂ¶

In this project our aim was to classify images using Convolutional Neural Networks, to do so we used the Inception-V3 Network, which is trained for the ImageNet Large Visual Recognition Challenge using the data from 2012. Furthermore this model is able to achieve 76.88% Top-1 Accuracy and 93.344% Top-5 Accuracy on ILSVRC2012-Validation Set.

To do so, we used R and MXNet, generating an interactive user interface with Shiny where the user can upload images in oder to classify them. Hence, first of all we had to load all the required packages (libraries):

required_packages <- c("shiny", "mxnet", "imager", "scales", "jpeg", "ggplot2", "readr", "png", "ggthemes")
sapply(required_packages, require, character.only = TRUE)

Loading required package: shiny
Loading required package: mxnet
Loading required package: imager
Loading required package: plyr
Loading required package: magrittr

Attaching package: ‘imager’

The following object is masked from ‘package:magrittr’:

    add

The following object is masked from ‘package:plyr’:

    liply

The following objects are masked from ‘package:stats’:

    convolve, spectrum

The following object is masked from ‘package:graphics’:

    frame

The following object is masked from ‘package:base’:

    save.image

Loading required package: scales
Loading required package: jpeg
Loading required package: ggplot2
Loading required package: readr

Attaching package: ‘readr’

The following object is masked from ‘package:scales’:

    col_factor

Loading required package: png
Loading required package: ggthemes

The Inception-V3 Network has some constraints regarding the image input dimensions which need to be single crop on 299 x 299 image, to do so we defined a preprocessing function, which preprocesses the images before inputting them in the Network model, below we can find the code for the preprocessing (an adaptation of the code provided in the inception v3 model but translated to R):

preproc.image <- function(image) {
  

  shape <- dim(image)
  short.edge <- min(shape[1:2])
  xx <- floor((shape[1] - short.edge) / 2)
  yy <- floor((shape[2] - short.edge) / 2) 
  croped <- crop.borders(image, xx, yy)
  resized <- resize(croped, 299, 299)
  array <- as.array(resized) * 256
  dim(array) <- c(299, 299, 3)
  preprocessedImage <- array - 128
  preprocessedImage <- array/128
  
  dim(preprocessedImage) <- c(299, 299, 3, 1)
  return(preprocessedImage)
}

Once we defined the preprocessing function, we can proceed to download the model and define the model and the classes used in this model. To do so, we will check if the file "model", which is the name of the folder where Inception V3 model is contained is found in the computer. If it is not found, we will proceed to download and untar the model, and then we will load the model to our system and also the classes (which are in txt format).

if (!file.exists("model")) {
    download.file("http://data.dmlc.ml/mxnet/models/imagenet/inception-v3.tar.gz", destfile = "inception-v3.tar.gz")
    untar("inception-v3.tar.gz")
  }
  
  ImageNet_Model <- mx.model.load("model/Inception-7", iteration = 1)
  
  synsets <- read_lines("model/synset.txt")

After defining the preprocessing function we also defined a function to pick an image from some location (in the web or in the local computer):

picksource <- function(input){
  
  ImageFromUrl <- eventReactive(input$enter, {
    print(input$urlImage)
   
      image <- tempfile()
      download.file(input$urlImage, destfile = image, method = "auto")
      return(image)
    
  })
  
  
  src = if (input$tabs == "Upload Image") {
    if (is.null(input$Imagefile)) {
      if (input$goButton == 0 || is.null(ImageFromUrl())) {
        'image.jpg'
      } else {
        ImageFromUrl()
      }
    } else {
      input$Imagefile$datapath
    }
  } else {
    if (input$goButton == 0 || is.null(ImageFromUrl())) {
      if (is.null(input$Imagefile)) {
        'image.jpg'
      } else {
        input$Imagefile$datapath
      }
    } else {
      ImageFromUrl()
    }
  }
}

Furthermore, we will define a function to appropiately format the results, formatting the final results in a nice and visual way.

getCommonResults <- function(result){
  StringResult <- ""
  for (i in 1:5) {
    ResultInJ <- strsplit(result[i], " ")[[1]]
    for (j in 2:length(ResultInJ)) {
      StringResult <- paste(StringResult, ResultInJ[j])
    }
    StringResult <- paste(StringResult, "\n")
  }
  StringResult

}

Now that we have defined all the previous functions, we will define our shiny user interface, in this user inferface we defined a side by side layout, where we defined in one side a panel to upload an image from the computer or from an url. And in the other side we defined a panel where the results will be displayed, displaying the inputted image, the predicted classes and its probabilities.

# Define UI for application that draws a histogram
ui <- fluidPage(
  
  includeCSS('alike.css'),
  # Application title
  titlePanel("Image Classification using Convolutional Neural Networks"),
  
  # Sidebar with a slider input for number of bins 
  sidebarLayout(
    sidebarPanel(
      tabsetPanel(
        id = "tabs",
        tabPanel("Upload Image",
                 fileInput('Imagefile', '\n Upload an Image:')),
        tabPanel(
          "Use the URL",
          textInput("urlImage", "Enter an Image URL:", ""),
          actionButton("enter", "OK")
        )
      )
    ),
    
    # Show the image, the results and the probabilities
    mainPanel(
      h3("Image to classify"),
      tags$hr(),
      imageOutput("imageloaded", height = "auto"),
      tags$hr(),
      h3("What does the image contains?"),
      tags$hr(),
      verbatimTextOutput("res"),
      plotOutput("ProbPlot")
    )
  )
)

Furthermore, we had to define the actions behind the user interface, i.e. define what each interface does. To do so we generated a server, where we defined the logic required to pick an image submitted by the user and predict its class.

server <- function(input, output) {
  
  image <- NULL
  
  
  
  
  output$imageloaded = renderImage(list(src = picksource(input)), deleteFile = FALSE)
  
  
  
  output$res <- renderText({
    src <- picksource(input)
    
    im <- load.image(src)
    preprocessedImage <- preproc.image(im)
    prob <- predict(ImageNet_Model, X = preprocessedImage)
    max.idx <- order(prob[,1], decreasing = TRUE)[1:5]
    result <- synsets[max.idx]
    StringResult <- ""
    for (i in 1:5) {
      ResultInJ <- strsplit(result[i], " ")[[1]]
      for (j in 2:length(ResultInJ)) {
        StringResult <- paste(StringResult, ResultInJ[j])
      }
      StringResult <- paste(StringResult, "\n")
    }
    StringResult
  })
  
  
  
  output$res <- renderText({
    src <- picksource(input)
    
    im <- load.image(src)
    preprocessedImage <- preproc.image(im)
    prob <- predict(ImageNet_Model, X = preprocessedImage)
    max.idx <- order(prob[,1], decreasing = TRUE)[1:5]
    result <- synsets[max.idx]
    StringResult <- StringResult <- getCommonResults(result)
  })
  
  output$ProbPlot <- renderPlot({
   src <- picksource(input)
    
    im <- load.image(src)
    preprocessedImage <- preproc.image(im)
    prob <- predict(ImageNet_Model, X = preprocessedImage)
    max.idx <- order(prob[,1], decreasing = TRUE)[1:5]
    result <- synsets[max.idx]
    StringResult <- getCommonResults(result)
    
    
    StringResult1 <- unlist(strsplit(StringResult, split="\n"))
    StringResult1 <- data.frame(names = StringResult1, probability = sort(prob[,1], decreasing = TRUE)[1:5])
    ggplot(StringResult1) + geom_bar(aes(reorder(names, probability ), probability * 100), stat = "identity", fill = "cyan", alpha = 1/3) + coord_flip() +
      labs(x = "Names", y = "Probability (%)") + theme_solarized_2(light = FALSE) + theme_hc(bgcolor = "darkunica") +
      scale_colour_hc("darkunica")
  })
  
  
}

After defining the user and server, we can run the application by running the following chunck:

shinyApp(ui = ui, server = server)

Listening on http://127.0.0.1:5759

Note: In order to access to the application click on the url generated after running the previous chunk.

ReferencesÂ¶

[1] Szegedy, Christian, et al. "Rethinking the Inception Architecture for Computer Vision." arXiv preprint arXiv:1512.00567 (2015).

Task 5: Image Classification using Convolutional Neural Networks¶

In this project our aim was to classify images using Convolutional Neural Networks, to do so we used the Inception-V3 Network, which is trained for the ImageNet Large Visual Recognition Challenge using the data from 2012. Furthermore this model is able to achieve 76.88% Top-1 Accuracy and 93.344% Top-5 Accuracy on ILSVRC2012-Validation Set.

To do so, we used R and MXNet, generating an interactive user interface with Shiny where the user can upload images in oder to classify them. Hence, first of all we had to load all the required packages (libraries):

required_packages <- c("shiny", "mxnet", "imager", "scales", "jpeg", "ggplot2", "readr", "png", "ggthemes")
sapply(required_packages, require, character.only = TRUE)

Loading required package: shiny
Loading required package: mxnet
Warning message:
“Installed Rcpp (0.12.11) different from Rcpp used to build dplyr (0.12.7).
Please reinstall dplyr to avoid random crashes or undefined behavior.”Loading required package: imager
Loading required package: plyr
Loading required package: magrittr

Attaching package: ‘imager’

The following object is masked from ‘package:magrittr’:

    add

The following object is masked from ‘package:plyr’:

    liply

The following objects are masked from ‘package:stats’:

    convolve, spectrum

The following object is masked from ‘package:graphics’:

    frame

The following object is masked from ‘package:base’:

    save.image

Loading required package: scales
Loading required package: jpeg
Loading required package: ggplot2
Loading required package: readr

Attaching package: ‘readr’

The following object is masked from ‘package:scales’:

    col_factor

Loading required package: png
Loading required package: ggthemes

The Inception-V3 Network has some constraints regarding the image input dimensions which need to be single crop on 299 x 299 image, to do so we defined a preprocessing function, which preprocesses the images before inputting them in the Network model, below we can find the code for the preprocessing (an adaptation of the code provided in the inception v3 model but translated to R):

preproc.image <- function(image) {
  

  shape <- dim(image)
  short.edge <- min(shape[1:2])
  xx <- floor((shape[1] - short.edge) / 2)
  yy <- floor((shape[2] - short.edge) / 2) 
  croped <- crop.borders(image, xx, yy)
  resized <- resize(croped, 299, 299)
  array <- as.array(resized) * 256
  dim(array) <- c(299, 299, 3)
  preprocessedImage <- array - 128
  preprocessedImage <- array/128
  
  dim(preprocessedImage) <- c(299, 299, 3, 1)
  return(preprocessedImage)
}

Once we defined the preprocessing function, we can proceed to download the model and define the model and the classes used in this model. To do so, we will check if the file "model", which is the name of the folder where Inception V3 model is contained is found in the computer. If it is not found, we will proceed to download and untar the model, and then we will load the model to our system and also the classes (which are in txt format).

if (!file.exists("model")) {
    download.file("http://data.dmlc.ml/mxnet/models/imagenet/inception-v3.tar.gz", destfile = "inception-v3.tar.gz")
    untar("inception-v3.tar.gz")
  }
  
  ImageNet_Model <- mx.model.load("model/Inception-7", iteration = 1)
  
  synsets <- read_lines("model/synset.txt")

After defining the preprocessing function we also defined a function to pick an image from some location (in the web or in the local computer):

picksource <- function(input){
  
  ImageFromUrl <- eventReactive(input$enter, {
    print(input$urlImage)
   
      image <- tempfile()
      download.file(input$urlImage, destfile = image, method = "auto")
      return(image)
    
  })
  
  
  src = if (input$tabs == "Upload Image") {
    if (is.null(input$Imagefile)) {
      if (input$goButton == 0 || is.null(ImageFromUrl())) {
        'image.jpg'
      } else {
        ImageFromUrl()
      }
    } else {
      input$Imagefile$datapath
    }
  } else {
    if (input$goButton == 0 || is.null(ImageFromUrl())) {
      if (is.null(input$Imagefile)) {
        'image.jpg'
      } else {
        input$Imagefile$datapath
      }
    } else {
      ImageFromUrl()
    }
  }
}

Furthermore, we will define a function to appropiately format the results, formatting the final results in a nice and visual way.

getCommonResults <- function(result){
  StringResult <- ""
  for (i in 1:5) {
    ResultInJ <- strsplit(result[i], " ")[[1]]
    for (j in 2:length(ResultInJ)) {
      StringResult <- paste(StringResult, ResultInJ[j])
    }
    StringResult <- paste(StringResult, "\n")
  }
  StringResult

}

Now that we have defined all the previous functions, we will define our shiny user interface, in this user inferface we defined a side by side layout, where we defined in one side a panel to upload an image from the computer or from an url. And in the other side we defined a panel where the results will be displayed, displaying the inputted image, the predicted classes and its probabilities.

# Define UI for application that draws a histogram
ui <- fluidPage(
  
  includeCSS('alike.css'),
  # Application title
  titlePanel("Image Classification using Convolutional Neural Networks"),
  
  # Sidebar with a slider input for number of bins 
  sidebarLayout(
    sidebarPanel(
      tabsetPanel(
        id = "tabs",
        tabPanel("Upload Image",
                 fileInput('Imagefile', '\n Upload an Image:')),
        tabPanel(
          "Use the URL",
          textInput("urlImage", "Enter an Image URL:", ""),
          actionButton("enter", "OK")
        )
      )
    ),
    
    # Show the image, the results and the probabilities
    mainPanel(
      h3("Image to classify"),
      tags$hr(),
      imageOutput("imageloaded", height = "auto"),
      tags$hr(),
      h3("What does the image contains?"),
      tags$hr(),
      verbatimTextOutput("res"),
      plotOutput("ProbPlot")
    )
  )
)

Furthermore, we had to define the actions behind the user interface, i.e. define what each interface does. To do so we generated a server, where we defined the logic required to pick an image submitted by the user and predict its class.

server <- function(input, output) {
  
  image <- NULL
  
  
  
  
  output$imageloaded = renderImage(list(src = picksource(input)), deleteFile = FALSE)
  
  
  
  output$res <- renderText({
    src <- picksource(input)
    
    im <- load.image(src)
    preprocessedImage <- preproc.image(im)
    prob <- predict(ImageNet_Model, X = preprocessedImage)
    max.idx <- order(prob[,1], decreasing = TRUE)[1:5]
    result <- synsets[max.idx]
    StringResult <- ""
    for (i in 1:5) {
      ResultInJ <- strsplit(result[i], " ")[[1]]
      for (j in 2:length(ResultInJ)) {
        StringResult <- paste(StringResult, ResultInJ[j])
      }
      StringResult <- paste(StringResult, "\n")
    }
    StringResult
  })
  
  
  
  output$res <- renderText({
    src <- picksource(input)
    
    im <- load.image(src)
    preprocessedImage <- preproc.image(im)
    prob <- predict(ImageNet_Model, X = preprocessedImage)
    max.idx <- order(prob[,1], decreasing = TRUE)[1:5]
    result <- synsets[max.idx]
    StringResult <- StringResult <- getCommonResults(result)
  })
  
  output$ProbPlot <- renderPlot({
   src <- picksource(input)
    
    im <- load.image(src)
    preprocessedImage <- preproc.image(im)
    prob <- predict(ImageNet_Model, X = preprocessedImage)
    max.idx <- order(prob[,1], decreasing = TRUE)[1:5]
    result <- synsets[max.idx]
    StringResult <- getCommonResults(result)
    
    
    StringResult1 <- unlist(strsplit(StringResult, split="\n"))
    StringResult1 <- data.frame(names = StringResult1, probability = sort(prob[,1], decreasing = TRUE)[1:5])
    ggplot(StringResult1) + geom_bar(aes(reorder(names, probability ), probability * 100), stat = "identity", fill = "cyan", alpha = 1/3) + coord_flip() +
      labs(x = "Names", y = "Probability (%)") + theme_solarized_2(light = FALSE) + theme_hc(bgcolor = "darkunica") +
      scale_colour_hc("darkunica")
  })
  
  
}

After defining the user and server, we can run the application by running the following chunck:

shinyApp(ui = ui, server = server)

Listening on http://127.0.0.1:6978

Note: In order to access to the application click on the url generated after running the previous chunk.

References¶

[1] Szegedy, Christian, et al. "Rethinking the Inception Architecture for Computer Vision." arXiv preprint arXiv:1512.00567 (2015).

Deep Learning for Computer Vision Project

Group 1

Group Members

Task 1

Task 2

Task 3

Task 4

Task 5

Training and Applying a CIFAR classifierÂ¶

Task 5: Image Classification using Convolutional Neural NetworksÂ¶

ReferencesÂ¶

Task 5: Image Classification using Convolutional Neural Networks¶

References¶