tangible Essay

Abstract

1) Group no. : 48

2) Title of the Project : A tangible product to enhance the real time expe-

rience of learning by character detection.

3) Subject area : Image processing and Embedded system.

4) Nature of the Project : Hardware + Software.

5) Internal Guide : Prof.Padma Hirave.

6) Sr.No. Names Exam Seat No.

1. Chapalgaonkar Amulya Shirish B150203022

2. Kulkarni Dhanshri Laxmikant B150203091

3. Late Aditi Lalit B150203098

7) Academic year : 2018-2019

i) Objective :-

As the digital applications are evolving day by day platforms are made

available to make routine tasks more engrossing and interactive.

These tech-

nologies provide us a chance to de

ne new ways to make our day to day life

much easier by providing man-to-machine interface.

Being an electronics engineer we have considered the newer application

emerging in market and tried to collaborate them with real time use. In

this project we are trying to give an interactive and engrossing experience of

learning and teaching to the user by using image processing and embedded

application. The user will get output in audio and visual format according

to the input received after image processing.

A tangible product will give users a hands-on experience which is helpful

for toddlers considering their high grasping power. Character detection can

be new way to make students familiar alphabets. Also, the extent of this

product can increased further by introducing number detection and mathe-

matical operation.

ii) Features of your Project :-

1) user-friendly

2) compact and portable

3) interactive and harmless to toddlers

iv

Contents

Acknowledgement iii

Abstract iv

1 Introduction 1

1.1 Aim . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.2 Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.3 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2 Literature Survey 3

2.1 Currently available products . . . . . . . . . . . . . . . . . . . 3

2.2 Related work done in past . . . . . . . . . . . . . . . . . . . . 3

3 Speci

cations 4

4 Methodology 5

4.1 Technical Block Diagram . . . . . . . . . . . . . . . . . . . . . 5

4.2 Modes of operations . . . . . . . . . . . . . . . . . . . . . . . 6

4.2.1 Character recognition . . . . . . . . . . . . . . . . . . 6

4.2.2 Mathematical operation . . . . . . . . . . . . . . . . . 6

4.2.3 Instrumental tone playing . . . . . . . . . . . . . . . . 7

4.3 Flow chart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

4.4 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

4.4.1 Otsu’s Method . . . . . . . . . . . . . . . . . . . . . . 8

4.4.2 Segmentation . . . . . . . . . . . . . . . . . . . . . . . 8

4.4.3 Thresholding . . . . . . . . . . . . . . . . . . . . . . . 8

4.4.4 KNN classi

er . . . . . . . . . . . . . . . . . . . . . . 9

4.4.5 HOG feature extraction . . . . . . . . . . . . . . . . . 9

v

5 Detail Design 10

5.1 Hardware Design . . . . . . . . . . . . . . . . . . . . . . . . . 10

5.1.1 Real-time picture . . . . . . . . . . . . . . . . . . . . . 10

5.2 Software Design . . . . . . . . . . . . . . . . . . . . . . . . . . 12

5.2.1 Selection of region of interest . . . . . . . . . . . . . . 12

5.2.2 HOG feature extraction . . . . . . . . . . . . . . . . . 13

6 Results 14

6.1 Sample Images . . . . . . . . . . . . . . . . . . . . . . . . . . 14

6.1.1 Sample Images A . . . . . . . . . . . . . . . . . . . . . 14

6.1.2 Sample Images R . . . . . . . . . . . . . . . . . . . . . 15

6.1.3 Sample Images Y . . . . . . . . . . . . . . . . . . . . . 15

6.1.4 Sample Images V . . . . . . . . . . . . . . . . . . . . . 16

7 Conclusion 17

7.1 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

7.2 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

7.3 Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

7.4 Future scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Bibliography 19

Appendix 20

vi

List of Figures

4.1 Technical Block Diagram . . . . . . . . . . . . . . . . . . . . . 5

4.2 Flow chart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

5.1 Real-time picture of table . . . . . . . . . . . . . . . . . . . . 10

5.2 Pictorial View . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

5.3 Selection of region of interest . . . . . . . . . . . . . . . . . . . 12

5.4 HOG feature extraction . . . . . . . . . . . . . . . . . . . . . 13

6.1 Sample Images A . . . . . . . . . . . . . . . . . . . . . . . . . 14

6.2 Sample Images R . . . . . . . . . . . . . . . . . . . . . . . . . 15

6.3 Sample Images Y . . . . . . . . . . . . . . . . . . . . . . . . . 15

6.4 Sample Images V . . . . . . . . . . . . . . . . . . . . . . . . . 16

vii

List of Tables

7.1 Work Plan Table . . . . . . . . . . . . . . . . . . . . . . . . . 20

viii

Chapter 1

Introduction

Conventional educational systems use books, blackboards and var-

ious such stationary materials. These methods cause a lot of paper wastage

and also provide just 2D experience. According to researches done previously

teaching tasks collaborated with some fun activities can give better results.

Some new technologies emerging in the market can make use of conventional

tools along with some techniques which will make the whole task of teaching

and learning more engrossing.

In proposed system we have considered both pros and cons of

generally used tools and tried to overcome some of their disadvantages by

adding new functionality.

1.1 Aim

To provide an interactive and engrossing experience of learning and

teaching to the user by using image processing and embedded application.

1.2 Objective

i)To design an interactive system which can used to educational as

well as brainstorming purpose.

ii)To create an user-friendly environment to make the process com-

fortable and interesting for toddlers.

1

A tangible product to enhance the real time user experience of learning by character detection

1.3 Overview

The proposed system comprises for shape detection of alphanumeric

blocks using image processing as

rst step .This output is given to Raspberry

Pi unit for processing. Using the image processing output and the prede

ned

database stored the required information is retrieved specifying the alphabet

and number .Signals are then supplied to speakers which will play audio clip

giving sounds of alphabet pronunciation and a spelling with corresponding

alphabet as initials . Video signal provided to display unit will display the

picture of object having corresponding alphabet as initials. Same is applica-

ble for the number blocks.

Department of E&TC, MKSSS’S Cummins College of Engineering for Women, 2018-19 2

Chapter 2

Literature Survey

2.1 Currently available products

Smart educational tabs available in market provide same purpose of

educating toddlers in various versions with portable and compact structures.

But along with these come the cons as well :

i) Unnecessary distractions

ii) Harm to eye sights

iii) No tangible experience

2.2 Related work done in past

1) Paper by Bhawna Agarwal : CaptuRing { A tangible tool for brainstorm-

ing :

CaptuRing is tool used by designers to augment the brainstorming pro-

cess during project development. It provides capturing of area of interest

containing the information using the intuitive interaction.

2) Ityam Vasal, Suyog Deshpande, Kumar Mayank,”A tangible product to

enhance real time user experience of enjoying music” ,USD foundation :

The paper elaborates the development of a group interactive table top

device for youth, along with the background research. The design requires

precise and dynamic positioning of multiple objects in order to enable real-

time multi-user interactions with media applications.

3

Chapter 3

Speci

cations

1. Height of table 37cm

2. Perimeter of Table top 94.2cm

3. Vertical range of object detection 25cm

4. Response time 10 sec

5. Weight of table 2.5 Kg

6. Object dimension 5-15 Cm

7. Object weight 2-3 Kg

8. Supply voltage 230V/50Hz

4

Chapter 4

Methodology

4.1 Technical Block Diagram

Figure 4.1: Technical Block Diagram

5

A tangible product to enhance the real time user experience of learning by character detection

4.2 Modes of operations

In this project is we are trying to design a system which will be

interactive and harmless to kids and can be used for educating purpose.

4.2.1 Character recognition

The

rst step will be shape detection of alphanumeric blocks .The

user will place the block within prede

ned area on the tangible table.

The camera attached at bottom of table will be continuously cap-

turing the video. Using this video frame extraction will be done consisting

required shapes.This frame will undergo RGB to Gray conversion process un-

der image processing followed by segmentation .For segmentation threshold

can be used .This segmented output will be given for morphological opera-

tion which may consist erosion and dilation method .Morphological output

will undergo feature extraction.

These features need to be classi

ed according to their similarities

and di erences which will fall under machine learning concept .The classi

ed

information and previously stored data are matched for actual character or

number recognition.

After recognizing character or number placed the processor will gen-

erate the output accordingly .The output will consist of audio format dictat-

ing pronunciation of alphabets, numbers and their corresponding spellings.

4.2.2 Mathematical operation

In this mode of various basic binary mathematical operations can

be taught like addition, subtraction, multiplication and division.For selecting

required operation we can use push buttons assigned with speci

c operation.

The tangible table is divided into two equal parts. The two operands

are placed in these two parts respectively .The number of operands can be

calculated using contour application in image processing and operation is

selected using push button.The output after mathematical operation will be

given to aux-speakers and display unit.

Department of E&TC, MKSSS’S Cummins College of Engineering for Women, 2018-19 6

A tangible product to enhance the real time user experience of learning by character detection

4.2.3 Instrumental tone playing

Di erent instrumental tones can be played in this mode of oper-

ation for brainstorming purpose. Speci

c shapes of blocks can be assigned

to instruments whose speci

c tones are stored in prede

ned database.The

volume intensity of tone can varied with reference of distance from centre of

the table.

4.3 Flow chart

Figure 4.2: Flow chart

Department of E&TC, MKSSS’S Cummins College of Engineering for Women, 2018-19 7

A tangible product to enhance the real time user experience of learning by character detection

4.4 Algorithm

1. Start

2. Turn on camera module ,audio speaker module.

3. Create remote network for raspberry pi.

4. Capture video with camera.

5. Frame extraction.

6. Preprocessing on image captured (RGB to gray conversion).

7. Thresholding using Otsu’s method(Black for background and white for

foreground).

8. Finding contours and maximum contours.

9. Select region of interest.

10. Features extraction using HOG method.

11. Database creation in Excel sheet.

12. Classi

cation using KNN classi

er.

13. Compare input image with classi

er output.

14. Final output.

15. Stop.

4.4.1 Otsu’s Method

Otsu’s method chooses a threshold that minimizes the intraclass

variance of the thresholded black and white pixels. The global threshold can

be used with ` imbinarize ‘ to convert a grey-scale image to a binary image.

In simple words, it automatically calculates a threshold value from image

histogram for a bi-modal image (an image whose histogram has two peaks).

4.4.2 Segmentation

Image segmentation is the process of partitioning a digital image

into multiple segments.

4.4.3 Thresholding

Image thresholding is a simple, yet e ective, way of partitioning an

image into a foreground and background that isolates objects by converting

Department of E&TC, MKSSS’S Cummins College of Engineering for Women, 2018-19 8

A tangible product to enhance the real time user experience of learning by character detection

grey-scale images into binary images. It is most e ective in images with high

levels of contrast.

4.4.4 KNN classi

er

The K-nearest neighbours (KNN) algorithm is a type of supervised

machine learning algorithms. KNN is extremely easy to implement in its

most basic form, and yet performs quite complex classi

cation tasks. It is a

lazy learning algorithm since it does not have a specialized training phase.

4.4.5 HOG feature extraction

A feature descriptor is a representation of an image or an image

patch that simpli

es the image by extracting useful information and throwing

away extraneous information.

The histogram of oriented gradients (HOG) is a feature descriptor

used in computer vision and image processing for the purpose of object de-

tection. The technique counts occurrences of gradient orientation in localized

portions of an image.

Department of E&TC, MKSSS’S Cummins College of Engineering for Women, 2018-19 9

Chapter 5

Detail Design

5.1 Hardware Design

Design consideration of mechanical enclosure : We have designed

a complete black coloured enclosure for the system to avoid any noise or

unnecessary image pixels that might be captured by the camera. Also, this

provides uniform lighting conditions for image capturing.

5.1.1 Real-time picture

Figure 5.1: Real-time picture of table

10

A tangible product to enhance the real time user experience of learning by character detection

Figure 5.2: Pictorial View

Department of E&TC, MKSSS’S Cummins College of Engineering for Women, 2018-19 11

A tangible product to enhance the real time user experience of learning by character detection

5.2 Software Design

5.2.1 Selection of region of interest

Figure 5.3: Selection of region of interest

Department of E&TC, MKSSS’S Cummins College of Engineering for Women, 2018-19 12

A tangible product to enhance the real time user experience of learning by character detection

5.2.2 HOG feature extraction

Figure 5.4: HOG feature extraction

Department of E&TC, MKSSS’S Cummins College of Engineering for Women, 2018-19 13

Chapter 6

Results

1) The output of the system will be in audio and video format.

2) Audio of pronunciation of the particular character block and spelling of a

word initiating with the character will be played.

3) Video displaying the spelling will be provided

4) In second mode of operation results of mathematical operation will be

given in audio and video format.

5) Finally,in last mode instrumental tone associated with particular shape

of block will be played.

6.1 Sample Images

6.1.1 Sample Images A

Figure 6.1: Sample Images A

14

A tangible product to enhance the real time user experience of learning by character detection

6.1.2 Sample Images R

Figure 6.2: Sample Images R

6.1.3 Sample Images Y

Figure 6.3: Sample Images Y

Department of E&TC, MKSSS’S Cummins College of Engineering for Women, 2018-19 15

A tangible product to enhance the real time user experience of learning by character detection

6.1.4 Sample Images V

Figure 6.4: Sample Images V

Department of E&TC, MKSSS’S Cummins College of Engineering for Women, 2018-19 16

Chapter 7

Conclusion

7.1 Conclusions

The system has collaborated educating tasks with some fun

activities resulting into engrossing experiences. It also overcomes

disadvantages of conventional tool like paper wastage and single tone

experience.

7.2 Limitations

1. Similar characters (ex :- `o’ and `0′,’i’and `1′) may produce ambiguous

outputs.

2. Misplaced character blocks may not be detected properly.

3. Noise, irregular lighting condition detected in background image may

produce error.

7.3 Features

i)Aplication independent

ii)Portable

iii)Interactive and user friendly

17

A tangible product to enhance the real time user experience of learning by character detection

7.4 Future scope

i)Brainstorming purpose

ii)Entertainment

iii)Various educational uses

Department of E&TC, MKSSS’S Cummins College of Engineering for Women, 2018-19 18

Bibliography

[1] Bhawna Agarwal, CaptuRing -A tangible tool for brainstorming”

[2] Ityam Vasal, Suyog Deshpande, Kumar Mayank, A tangible product

toenhance real time user experience of enjoying music”,USD foundation

[3] Rondney Berry, Mao Makino, Naoto Hikawa, Masumi Suzuki,The Aug-

mented Composer Project: The Music Table”, ATR Media Information

Science Laboratories

19

A tangible product to enhance the real time user experience of learning by character detection

Appendix

Table 7.1: Work Plan Table

Task Month Week no. date

Project approved July 3rd week 14/07/18

Concept Development August 4th week 23/08/18

Search paper discussion August 5th week 29/08/18

Rough literature survey August 5th week 31/08/18

Final literature survey September 1st week 07/09/18

Physical Layout of system September 3rd week 18/09/18

Seminar Report(Draft) October 2nd week 12/10/18

Seminar Report October 2nd week 15/10/18

Internal Seminar October 2nd week 16/10/18

Seminar Report(Fair) December 3rd week 21/12/18

Project Report(Draft) January 2nd week 17/01/19

Discussion on algorithm February 2nd week 12/02/19

External Review March 1st week 01/03/19

Final Report(Draft) March 2nd week 15/03/19

Project competition March 3rd week 27/03/19

Soft copy of Report(Draft) March 3rd week 28/03/19

Department of E&TC, MKSSS’S Cummins College of Engineering for Women, 2018-19 20