Monday, 27 November 2017

Implementing Keras model in Tensorflow

Deep learning is the latest trend in the field of artificial intelligence. Deep learning based models have produced incredible results in the fields such as computer vision, natural language processing, robotics etc. Tensorflow and Keras are two popular frameworks used for building the deep neural networks used in deep learning. Tensorflow is one of the advanced software libraries used for implementing deep learning models whereas Keras is a high level abstract library that runs on top of tensorflow.

In this blog, we will see a simple implementation of using a Keras model with Tensorflow backend in a Tensorflow serving framework. The Jupyter notebook implementation of the code can be found in github.

First of all import the libraries required.
## Imports
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
from keras.layers import Dense, Dropout, Flatten, MaxPooling2D, Conv2D, Input
from keras.models import Model
import numpy as np
import pylab as plt
import os

Next define values for the variables used in the program.
## Defining variables

Then load the MNIST dataset.
mnist = input_data.read_data_sets('MNIST_data', one_hot=True)

Now the model has to be defined in Keras. For that, first the placeholder variables are defined in tensorflow. The data placeholder variable will be used for serving the input data and the target placeholder variable will be used for feeding the expected target values.

## Defining placeholders in tf
data = tf.placeholder(tf.float32, [None, height,width,channel]) #step_size=No: of frames in video sequence
target = tf.placeholder(tf.float32, [None, nb_classes])

Next the model has to be defined in Keras. A simple CNN is used as an example in this program. The model is defined using Keras functional API.

## Defining the model in keras using functional layers
input_layer = Input(shape=input_shape)
layer1= Conv2D(32, kernel_size=(3, 3),activation='relu')(input_layer)
layer2=Conv2D(64, (3, 3), activation='relu')(layer1)
layer3=MaxPooling2D(pool_size=(2, 2))(layer2)
layer5=Dense(128, activation='relu')(layer4)
output_layer=Dense(nb_classes, activation='linear')(layer5)

After that the remaining part is defined as normal in tensorflow. This includes defining the cross entropy loss function, Adam optimiser for optimisation of the network and defining accuracy.

## Making optimisation method, loss function and calculating accuracy
predictions= tf.nn.softmax(out)
optimizer = tf.train.AdamOptimizer(lr_rate)
minimize = optimizer.minimize(cross_entropy)
mistakes = tf.equal(tf.argmax(target, 1), tf.argmax(predictions, 1))
accuracy = tf.reduce_mean(tf.cast(mistakes, tf.float32))

Once the model and the related operations are defined, the model has to be trained. The training is done using the tensorflow serving mechanism. First a session is started, the variables are initialised and then the network is trained by feeding the the input and target data batch by batch.

## The keras model is trained as a tensorlfow graph
init_op = tf.global_variables_initializer()
sess = tf.InteractiveSession()

for i in range(iterations):
    batch_x, batch_y = mnist.train.next_batch(batch_size)
    batch_x = batch_x.reshape(batch_x.shape[0], height, width, channel)[minimize],{data: batch_x, target: batch_y})
    print('Iteration {}'.format(i))

And finally the trained model is tested and session is closed.
mnist_test = mnist.test.images.reshape(mnist.test.images.shape[0], height, width, channel), feed_dict={data:mnist_test ,target: mnist.test.labels})
print('accuracy on test set: {}'.format(acc))

There are also other ways of using Keras alongside tensorflow. Refer to this blog for further reading.

Monday, 12 September 2016

Mango Images In the Wild (MIW) Database

Mango Images In the Wild (MIW) Database:

Lack of databases of mango images is a major hurdle faced by researchers while developing image processing algorithms for  mango related projects. This database was collected by us for developing computer vision algorithms for sorting and grading of mangoes.
The database consists of images of mangoes taken under unconstrained real world conditions using a consumer grade digital camera. Since the images are captured under real world conditions, they are subject to variations in lightning, blur, image quality, clarity, orientation etc. The database consists of 52 images of mangoes each of size 3648 x 2736 pixels captured against a white back ground. This database could be used for developing and testing Computer vision/ Image processing algorithms in mango related projects.
Sample mango images from MIW Database
 How to get it:
The images could be downloaded from here : Link
 All are free to use this database for their projects and research works without any restrictions.

But if you use this dataset please cite the following paper in your work:
Leo Pauly and Deepa Sankar. "A new method for sorting and grading of mangos based on computer vision system." In Advance Computing Conference (IACC), 2015 IEEE International, pp. 1191-1195. IEEE, 2015.

Contact if any question. 

Computer Vision System for Drowsiness Detection Using Web camera

This post briefly explains about my project "Computer vision system for Drowsiness Detection using Web cameras". The aim of the project was to develop a computer vision system for drowsiness detection. The developed system uses a ordinary web camera and computer vision algorithms to detect whether the subject is drowsy or active. The main advantage of the system is that it is non-intrusive and hence it can be implemented in real world conditions.

The project was completed in 3 stages :
Stage 1 : Identifying an eye blink detection method with high rate of accuracy
Stage 2 : Developing a eye tracking and blink detection system
Stage 3 : Using the eye tracking and blink detection system for drowsiness detection

Stage 1
In this stage an analysis of five non-intrusive methods for eye blink detection from low resolution eye images using different features like mean intensity, Fisher faces and Histogram of Oriented Gradients (HOG) and classifiers like Support Vector Machines (SVM) and Artificial neural network (ANN) was conducted. A comparative study was performed by varying the number of training images and in uncontrolled lighting conditions with low resolution eye images. The results showed that HOG features combined with SVM classifier outperforms all other methods with an accuracy of 85.62% even when tested on images taken from a totally unknown dataset.
Paper based on stage 1 :  (to be updated soon)

Stage 2
In this stage a novel method for eye tracking and blink detection in the video frames obtained from low resolution consumer grade web cameras was developed. It uses a method involving Haar based cascade classifier for eye tracking and a combination of HOG features with SVM classifier for eye blink detection. The eye tracking method had an accuracy of 92.3% and the blink detection method had an accuracy of 92.5% when tested using standard databases and a combined accuracy of 86% when tested under real world conditions of a normal room.
Paper based on stage 2: here

                                            Demo of Eye Tracking and Blink Detection method

Stage 3
In this stage an accurate method for drowsiness detection from the images obtained using low resolution consumer grade web cameras under normal lighting conditions was developed. The drowsiness detection method uses Haar based cascade classifier for eye tracking and combination of Histogram of oriented gradient (HOG) features combined with Support Vector Machine (SVM) classifier for blink detection developed in the second stage. Once the eye blinks are detected then the PERCLOS is calculated from it. The PERCLOS is a parametric measurement of how long the eyes of the subject has been closed in one minute. If the PERCLOS value is greater than 6 seconds then the person is said to be drowsy. The method was validated by comparing the prediction of the system with that of a human rater. The system matched with the human observer with 91.6 % accuracy.
Paper based on stage 3: here

The project was completed under the guidance of Dr. Deepa Sankar, Associate professor, Division of Electronics Engineering, Cochin University of Science and Technology.

Saturday, 13 September 2014

LabBot1.0 : Labview Controlled Mobile Robot

Ever since i saw a mobile robot named Accbot  made by my friend praveen Sridhar, which  used an arduino board programmed with javascript, i began exploring for other ways of programming an arduino without using the standard c++ code.

It was during that time i came across the new firmware LIFA by National Instruments which provided an interface to integrate arduino with LabView.More about LIFA can be found here.

Since I was so passionate about robots i decide to build a mobile robot controlled by labview.
And that was how LabBot1.0 , my labview controlled mobile robot was created.

The LabBot had the following interfaces:

1.Control Panel: A control panel for controlling and guiding the movement of the robot.The front control panel had 4 buttons for each direction and 4 indicator lights to indicate the direction of movement.

2.Sensor Interface :  The front panel also included a sensor interface for  providing an interface between sensors used in the mobile robot and the user.Here for this robot i used only a IR sensor and a corresponding level indicator was included in the front panel.

The hardware of the LabBot consisted of the following items:
1.Arduino Microcontroller:
2.IR analog sensor
3.L293A motor driver
4.two DC motors

It took me some time to integrate the arduino with labview ,since my laptop didnt had  a few drivers.But once i got the drivers installed in my system it took me just few hours to get my robot running.Here is the video of my LabBot1.0.

Tuesday, 19 August 2014

ROS Tutorials 1 : Introduction to ROS , the Robotic Operating system

This is my very first blog on ROS aka the robotic operating system.In this blog i would like to give you a overview of what it is and why do we need it.

Why ROS ?

ROS is an dedicated software framework created for running on robotic systems.The main objective of ROS is code reusability.Before the origin of ROS there was no dedicated software framework for implementing  robotic software architectures.As a result different software frameworks were being used by researchers , industrialists ,academicians and hobbyists all around the globe.There was no common platform.Every Time you created a robot you had to reinvent the wheel.Every new robot had to be build from the scratch.This led to the wastage of time and resources.So in rorder to tackle this issue ROS was created.It provided a common platform for creating and sharing codes and programs for robots all around the world.Once a code is written in ROS it can be used again by any other user anywhere in the world.

You can have a look a the history of ROS here.
Also the original research paper on ROS can be downloaded from here .

Basics of ROS :

In this section i would like to describe the basic working concept of ROS.We all know that a robot is an assembly of different modules.It is an integration of hardware components like cameras , sensors , microcontrollers, laptops etc with software modules for vision,AI,speech recognition , navigation etc.It is really a difficult task to integrate all of it together.

ROS has the solution to this problems.It implements a nodal architecture.That is each of the modules let it be hardware or software is implemented as a node.These nodes are then made to communicate with each other.A general nodal structure is shown in the following picture.

Nodal Structure of ROS framework

Consider the speech recognition and navigation nodes given in the above figure.Here the speech recognition node publishes the ros messages ,according to the words spoken.A node can publish their messages under different topics.Here the speech node publishes under the topic "Speech".Any other node which need to receive this data can subscribe to this topic.If the navigation node has to listen to the words spoken by the user it can subscribe to this rostopic "Speech".Similarly any node can publish its data to a topic.Any other node let it be hardware or software can listen to  that published values by subscribing to the corresponding topics.

Considering one more instance from the above pictorial representation , we have a temperature sensor connected to a microcontroller and a text to speech node.The sensor connected to the microcontroller unit is initiated as a rosserial node.It publishes the sensor values to the Rostopic "sensor_values".The text to speech node which is responsible for announcing the sensor values will subscribe to this rostopic "Sensor_values".This received values will then be converted to to speech and announced to the user by the text to speech node.

This is the working of ROS in a nutshell.
For further reading and references to get a better overview of ROS visit :

1) ROS official Page
2) Intro to Robotic operating system
3) What is ROS ?

In the next tutorial we will learn how to instal ros in your computer and get started.Until then stay tuned.

Thursday, 31 July 2014


Robotics is an exciting field of technology. It is also one of the most rapidly evolving field. It is predicated that robots are going to be a common thing like smart phones by the end of next decade. But in INDIA the robotics scenario is entirely different. We don't progress at a rate as other countries do. So in this blog I would like to outline some of my observations on why robotics research is not flourishing in INDIA.

1.Lack of a good environment:

For any research to flourish what is essentially needed is a good research environment .But we in INDIA lack this environment in two ways:

a) Labs

A personal robotics Lab

Labs are an essential part of  any research activity. But here we don't have many labs dedicated exclusively for robotics research. Robotics is a very expensive field of interest and it involves a lot of costly hardware tools. This emphasise  the need of well equipped robotics labs in INDIA like Personal Robotics lab(MIT) , BRL (Bristol) ,Stanford AI laboratory (Stanford university) ,Social Robotics lab (NUS, Singapore)...ect


Robotics is a collaboration of many fields of science and technology including vision (image processing), speech(speech recognition), AI(Artificial neural networks  & Machine learning), simulation (vrep,rviz..ect) mechanics(kinematics, dynamics, statics ) ,electronics (microcontrollers), language processing (Natural language  processing) and many more. We need faculties who are capable of looking at the big picture of a robotics project. But unfortunately we don't have many faculties who have got such an insight. We have got great faculties who are specialised in one or two fields mentioned above. But a person who understands NLP may not understand mechanics or vice versa and so on. This has affected a lot of robotics projects as a whole.    

Knowledge about robotics in India is below average generally. Even in reputed institutes microcontroller programming and basic mechanical kinematics of robotic arms is taught as robotics. Texts like “Fundamentals of Robotics: analysis and control’ by Robert J Schilling (1990)”, “Robotics: Control Sensing vision and intelligence by K.S Fu (1987)” written in 1990’s are still considered to be the standard textbooks in robotics. The technology has evolved to unimaginable heights in the first decade of 21st century .Robotics has changed in all aspects after that. The availability of powerful computers, hardware platforms, and latest software tools like ROS (robotic operating system) has made rapid changes in the field of robotics. But in India it seems like we are totally ignorant about all this and hold on to 1990s.
      3.Miss consumption & wrong trends:

Robotics is a technological field in INDIA which has been under constant exploitation. Here whatever is taught as robotics is just either microcontroller programming or an extension of embedded system. The many  so called ‘robotics companies’ introduces workshops called ‘Robotics workshops’ and teach students few lines of microcontroller programming. Students go back home thinking that they have learned robotics without actually knowing what actual robotics is all about. Personally I was also a victim of this trend for major part of my under graduation.

 4.Absence of a good hardware research platform :

PR2 : Robot
One of the major reasons for lack of research in the field of robotics is the absence of a very good robotic research platform. If you want to be a good workshop mechanic you need to work on a real hardware piece and not in just a computer simulated car engine. Same is applicable to robotics too. The robots can be simulated in a computer and coded, but that won't create real robots. We need efficient and real robotics hardware research platforms like PR2 (willow garage ) , rollin justin (DLR ), UBR-1 or at least something like a Nao.But in INDIA we don’t anything like that available at a affordable rate.

 5.Attitude :

Now this about the attitude of students towards projects and is applicable not just to robotics. 90 percent of students do projects only to score marks. Marks is the only factor that drives the students. Only a very few feels that they has to do something different and innovative and among that only a few dare to take up really challenging projects. Student has to understand one thing : only those who dare to challenge the system has become great , let it be Steve jobs, Bill Gates , Larry page or Elon musk. I personally believe that before any student start his/her project they have to read this :
“Here’s to the crazy ones....The misfits....The rebels....The troublemakers....The round pegs in the square holes.

The ones who see things differently. They’re not fond of rules. And they have no respect for the status quo. You can quote them, disagree with them, glorify or vilify them.

But the only thing you can’t do is ignore them. Because they change things. They push the human race forward. And while some may see them as the crazy ones, we see genius. Because the people who are crazy enough to think they can change the world, are the ones who do “

-From Apple’s famous ad ‘Think Different’ : Link to video

PS: Everything written above are my personal observations.Anyone can disagree to my viewpoints.All your suggestions are welcome.

Tuesday, 1 July 2014

Identifying Coloured Objects Using OpenCv

Hey everyone , in this blog we will be learning how to create a mask for a particular colour  and extract the particular coloured object from an image.

Algorithm used:

1) Read the BGR image
2) Convert the image into HSV color space
3) Create a color mask by setting the upper and lower thresholds from the hsv image
4) Filter the original BGR  image using the color mask

Sample Code :

import cv2
import numpy as np

img = cv2.imread("tomato.jpg",-1)

cv2.imshow("blurred image",blur)

cv2.imshow("masked image",out)



The sample code above separates all the red coloured objects from your image.

import cv2
import numpy as np

img = cv2.imread("tomato.jpg",-1)

The above lines are for reading the original image with the name tomato.jpg stored in the working directory.In case the image is stored in some other directory you have to give the complete path.

cv2.imshow("blurred image",blur)

The first two lines removes the noise in the image using a gaussian filter and displays the blurred image. Next line converts the image into into hsv color space.For more details of the conversion visit my blog.


The lines #1,#2 sets the lower and upper threshold for the mask.The hue value is selected to be in between 0-20 since we want to get the red objects.If we want to select some other colour you can do it by just changing the hue value here.The value for saturation and value are set to be 100-255 to include all tones and shades of red.The line #3 creates a mask from the hsv image.Line #4 displays the mask.

cv2.imshow("masked image",out)

The line 1 use a bitwise 'and' operation to filter the original BGR image using the mask and return a masked image in which only the red objects will be shown.

Original Image
Blurred Image
Masked Image

 Hoping the blog was helpful.Take care & Continue Hacking ;-)