Monthly Archives: October 2018

Research – Detection & Tracking

Posted on by

When I started doing the research for this project I was to a degree thinking of the terms ‘detection and ‘tracking’ in an interchangeable way that was not helpful in developing the program we are going to need fulfill the brief.

This was drawn into sharp focus when I was exploring HOG (Histogram of Oriented Gradients) and SVM (Support Vector Machine) as a possible combined approach to the project.

Image courtesy of software.intel.com

A Histogram of Oriented Gradients (HOG) is a feature descriptor that works by counting the occurences of a gradient orientation within an image (detection). As gradients tend to represent a sharp change in intensity value there is a high likelihood that a gradient could be an edge. HOG trys to #

1
2
#Histogram of Oriented Gradients
hog = cv2.HOGDescriptor()

There is also a an SVM Training function

1
2
# Support Vector Machine
SVM = cv2.SVM.train

 

REFERENCES:

HOG Descriptor Struct Reference (accessed 31.10.2018)
https://docs.opencv.org/3.4.1/d5/d33/structcv_1_1HOGDescriptor.html#details

Histogram of Oriented Gradients (HOG) Descriptor (accessed 31.10.2018)
https://software.intel.com/en-us/ipp-dev-reference-histogram-of-oriented-gradients-hog-descriptor

Dalal N. Triggs B. Histograms of oriented gradients for human detection. 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05).
https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=1467360

Juliano E. C. Cruza, Elcio H. Shiguemorib, Lamartine N. F. Guimaraes. A comparison of Haar-like, LBP and HOG approaches to concrete and asphalt
runway detection in high resolution imagery
http://epacis.net/jcis/PDF_JCIS/JCIS11-art.0101.pdf

Histograms of Oriented Gradients (accessed October 31st 2018)
https://www2.cs.duke.edu/courses/fall17/compsci527/notes/hog.pdf

Research – Tracking Texture & Tracking Motion

Posted on by

Part 1 – Tracking Texture

Our project has decided to explore four different approaches as described in an earlier post in this blog. My project partner is focusing on Shape Based & Colour Based tracking and I am going to focus my efforts on tracking which focuses on motion and texture.

Kalman Filtering

The first thing I notice while researching this topic is that Kalman filtering seems to be a popular algorithm when it comes to this approach.

Kalman filtering has a variety of applications and like all filters it allows certain things to pass through and other things not. The Kalman filter’s aim is to filter imperfect information, sort out the useful parts of interest and reduce the uncertainty and noise.

Apparently an early application of Kalman filtering was used for guided missiles and was also used as part of the onboard navigation system aboard the Apollo 11 Lunar module.

Particle Filtering

Unlike the Kalman Filter, particle filtering takes a non-linear approach. Particle filtering in a nutshell involves representing a posterior function by a set of random samples (particles).

Initial research into the area of texture tracking seems to suggest that it’s effectiveness is dependent on the image in question possessing strong textures (unsurprisingly) and texture tracking techniques tend to get used in conjunction with other means of detection and tracking.

The series of images we are working with in this project is neither rich in texture or indeed detail as the image is quite limited in terms of resolution.

So with that in mind, I am going to park this avenue of research here for now and may revisit it later should I decide to use this technique in combination with another approach

OpenCV offers its own Kalman filter function, the constructor of which I have included below.

1
2
# Kalman Filter
kalman = cv2.KalmanFilter(dynamParams, measureParams, controlParams)

 

REFERENCES:

Pressigout M, Marchand E. Real-Time Hybrid Tracking using Edge and Texture Information. The International Journal of Robotics Research Vol 26, No 7, July 2007
http://www.irisa.fr/lagadic/pdf/2007_ijrr_pressigout.pdf

Particle filter (accessed 25.10 2018)
https://en.wikipedia.org/wiki/Particle_filter

Motion Analysis and Object Tracking (accessed 25.10 2018)
https://docs.opencv.org/3.0-beta/modules/video/doc/motion_analysis_and_object_tracking.html

Tracking – Tracking by Background Subtraction (accessed 25.10 2018)
https://www.hdm-stuttgart.de/~maucher/Python/ComputerVision/html/Tracking.html

Vacchetti L. Lepetit V. Fua P. Combining Edge and Texture Information
for Real-Time Accurate 3D Camera Tracking. CVlab. Swiss Federal Institute of Technology, 1015 Lausanne, Switzerland
https://www.labri.fr/perso/vlepetit/pubs/vacchetti_ismar04.pdf

Research – Haar Cascades

Posted on by

A Haar Cascade is an object detection algorithm.

image courtesy of: https://docs.opencv.org

It uses a classifier which is trained to detect an object and does this by superimposing a positive image over a set of negative images over a number of stages. Quite a large number of stages.

This machine learning approach is computationally heavy and reliant upon possession of a training set. However there seem to be a number of them available online. Each with their own focus of object detection.

 

Haar-cascade Detection in OpenCV

OpenCV has it’s own trainer and detector which can be used to train your own classifier, which in our case would need to track the left eye of a person over a series of three images.

By applying some demo code kindly provided by OpenCV to one of our project images, we can see that this could well be a viable approach to take with our project and worthy of further research.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
import numpy as np
import cv2
 
face_cascade = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')
eye_cascade = cv2.CascadeClassifier('haarcascade_eye.xml')
 
img = cv2.imread('Ilovecats1.bmp')
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
 
faces = face_cascade.detectMultiScale(gray, 1.3, 5)
for (x,y,w,h) in faces:
    cv2.rectangle(img,(x,y),(x+w,y+h),(255,0,0),2)
    roi_gray = gray[y:y+h, x:x+w]
    roi_color = img[y:y+h, x:x+w]
    eyes = eye_cascade.detectMultiScale(roi_gray)
    for (ex,ey,ew,eh) in eyes:
        cv2.rectangle(roi_color,(ex,ey),(ex+ew,ey+eh),(0,255,0),2)
 
cv2.imshow('img',img)
cv2.waitKey(0)
# cv2.destroyAllWindows()

Similar to HAAR Cascades are LBP (Local Binary Pattern) Cascades which seem to operate similarly in the respect that they both use training sets. However, LBP Cascades can provide quicker results with slightly less accuracy.

Local Binary Patterns could be thought of as a computationally cheaper version of the Haar Cascade. LBP works by thresholding with the use of a 3×3 grid. The value of the centre pixel in the grid is used as a threshold value and the surrounding pixels in the grid are thresholded and summed together to form the descriptor used. LBP yields results that would be similar to Haar Cascades while offering a much simpler process.

References:

Face Detection using Haar Cascades (accessed 25.10 2018)
https://docs.opencv.org/3.0-beta/doc/py_tutorials/py_objdetect/py_face_detection/py_face_detection.html#face-detection

Viola P. Jones MJ. Rapid Object Detection using a Boosted Cascade of Simple Features. IEEE Computer Society Conference on Computer Vision and Pattern Recognition 1:I-511- I-518 vol.1
https://www.researchgate.net/publication/3940582_Rapid_Object_Detection_using_a_Boosted_Cascade_of_Simple_Features

Deep Learning Haar Cascaded Explained (accessed 25.10 2018)
http://www.willberger.org/cascade-haar-explained/

Pietikäinen M. Local Binary Patterns. Machine Vision Group, Department of Electrical and Information Engineering, University of Oulu, Finland.
http://www.scholarpedia.org/article/Local_Binary_Patterns

Huang D. Shan C. Ardebilian M. Wang Y. Chen L. Local Binary Patterns and Its Application to Facial Image Analysis: A Survey
https://liris.cnrs.fr/Documents/Liris-5004.pdf

Week 6 – Lab Work

Posted on by

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# 1. Open image Googly.jpg;
 
# 2. Use thresholding to separate Googly and his friend from the background;
 
# 3. Use statistical analysis to choose a threshold for better results;
 
# 4. Try multiple thresholds on different channels.
 
import numpy as np
import cv2
from matplotlib import pyplot as plt
from matplotlib import image as image
import easygui
 
I = cv2.imread("googly.jpg", cv2.IMREAD_GRAYSCALE)
 
# I = cv2.imread("image.jpg")
 
# RGB = cv2.cvtColor(I, cv2.COLOR_BGR2RGB)
 
T = 100
T, B = cv2.threshold(I, thresh = T, maxval = 255,
type = cv2.THRESH_BINARY)
 
cv2.imshow("Googly", B)
key = cv2.waitKey(0)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# 1. Open image sudoku.jpg
 
# 2. Use adaptive thresholding to create a black & white image;
 
# 3. Modify the region size and constant for better results.
 
import numpy as np
import cv2
from matplotlib import pyplot as plt
from matplotlib import image as image
import easygui
 
I = cv2.imread("sudoku.jpg", cv2.IMREAD_GRAYSCALE)
 
J = cv2.adaptiveThreshold(I, maxValue = 255,
  adaptiveMethod = cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
  thresholdType = cv2.THRESH_BINARY,
  blockSize = 5, C = 15)
  
cv2.imshow("Sudoku", J)
key = cv2.waitKey(0)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
# 1. Open image Orange.png;
 
# 2. Using thresholding, create a mask with the orange as ROI;
 
# 3. Use this mask to extract the orange from the image;
 
# 4. Open Water.jpg;
 
# 5. Use the inverse of the orange mask to cut an orange-shaped hole in
# the water picture;
 
# 6. Combine the orange and water masked images to create a
# composite image.
 
import numpy as np
import cv2
from matplotlib import pyplot as plt
from matplotlib import image as image
import easygui
 
I = cv2.imread("orange.png")
Z = cv2.imread("water.jpg")
G = cv2.cvtColor(I, cv2.COLOR_BGR2GRAY)
 
J = cv2.bitwise_not(G)
 
T = 20
T, B = cv2.threshold(J, thresh = T, maxval = 255,
type = cv2.THRESH_BINARY)
 
K = cv2.bitwise_not(B)
 
ROI = cv2.bitwise_and(I,I,mask=B)
 
ROI2 = cv2.bitwise_and(Z,Z,mask=K)
 
ROI3 = cv2.bitwise_or(ROI,ROI2)
 
cv2.imshow("Composite", ROI3)
key = cv2.waitKey(0)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
# 1. Open Trump.jpg;
 
# 2. Use an appropriate colour space and range to segment out areas of
# flesh (you will need to research this);
 
# 3. Use morphological processes to clean up the ROI.
 
import numpy as np
import cv2
from matplotlib import pyplot as plt
from matplotlib import image as image
import easygui
 
I = cv2.imread("trumpcopy.jpg")
 
HSV = cv2.cvtColor(I, cv2.COLOR_BGR2HSV)
 
# A Numpy array is created of upper and lower fleshtone values.
lowerFleshValue = np.array([3, 48, 80])
upperFleshValue = np.array([20, 255, 255])
# Mask is created of all fleshtone values.
fleshMask = cv2.inRange(HSV, lowerFleshValue, upperFleshValue)
 
# Structuring element
shape = cv2.getStructuringElement(cv2.MORPH_RECT,(6,6))
 
# Closing
# newMask = cv2.dilate(fleshMask,shape)
# newerMask = cv2.erode(newMask,shape)
 
# Opening
# newestMask = cv2.erode(newerMask,shape)
# mostNewestMask = cv2.dilate(newestMask,shape)
 
newMask = cv2.morphologyEx(fleshMask,cv2.MORPH_CLOSE,shape)
newerMask = cv2.morphologyEx(newMask,cv2.MORPH_OPEN,shape)
boundary = cv2.morphologyEx(newerMask,cv2.MORPH_GRADIENT,shape)
 
C = cv2.bitwise_and(I,I,mask=newerMask)
 
cv2.imshow("Original", I)
cv2.imshow("HSV", HSV)
cv2.imshow("Mask", fleshMask)
cv2.imshow("Closing", newMask)
cv2.imshow("Opening", newerMask)
cv2.imshow("Boundary", boundary)
cv2.imshow("Combined", C)
key = cv2.waitKey(0)

 

Assignement 1 – Lighting

Posted on by

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
# To accomplish this task I did the following:
 
# Stage 1. I converted the image to grayscale.
 
# Stage 2. I then applied Contrast Limited Adaptive Histogram  
# Equalization to try and correct the uneven lighting on the page.
 
# Stage 3. I then applied an Adaptive Threshold which turned
# the information on the page black or white and therefore
# getting rid of a lot of detail making the text unreadable.
# So I ended up using the threshold just to assist canny find
# the edges of the ROI.
 
# Stage 4. I used the opencv'canny' function to find the edges of my
# ROI (Region of Interest)
 
# Stage 5. I used the numpy 'argwhere' function to crop the image
# using the edges located by the 'canny' function.
 
# Final stage. I output a copy of the corrected image and display
# the corrected image on screen.
 
# Summary: I was hoping that my result would be a crisp white
# background with clear black text in the foreground, but as the
# priority was to make the text easier to read I had to make a
# call with that in mind, so in terms of contrast, I think my
# program has provided results somewhere in the middle, but
# the text is definitely more readable.
 
import numpy as np
import cv2
from matplotlib import pyplot as plt
from matplotlib import image as image
import easygui
 
stage1 = cv2.imread("text.jpg", cv2.IMREAD_GRAYSCALE)
 
stage2a = cv2.createCLAHE(clipLimit = 2.0, tileGridSize = (8, 8))
stage2b = stage2a.apply(stage1)
 
stage3 = cv2.adaptiveThreshold(stage2b, maxValue = 255,
  adaptiveMethod = cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
  thresholdType = cv2.THRESH_BINARY,
  blockSize = 5, C = 15)
 
height = np.size(stage3, 0)
width = np.size(stage3, 1)
 
stage4 = cv2.Canny(stage3, height, width)
 
stage5a = stage4 > 0
 
stage5b = np.argwhere(stage5a)
 
horiz0, vertic0 = stage5b.min(axis = 0)
horiz1, vertic1 = stage5b.max(axis = 0) + 1
 
stage5d = stage2b[horiz0: horiz1, vertic0: vertic1]
 
cv2.imwrite('Corrected_image.jpg', stage5d)
 
cv2.imshow("Corrected Image", stage5d)
key = cv2.waitKey(0)

 

Week 5 – Lab Work

Posted on by

Today we explored kernels in more depth.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# 1. Open any image;
 
# 2. Build a kernel for sharpening an image
# (you will need to research this);
 
# 3. Apply this filter to the image.
 
import numpy as np
import cv2
from matplotlib import pyplot as plt
from matplotlib import image as image
import easygui
 
I = cv2.imread("Lenna.png")
 
G = cv2.cvtColor(I, cv2.COLOR_BGR2GRAY)
 
k = np.array([[0,-1,0], [-1,5,-1], [0,-1,0]],
dtype=float)
 
F = cv2.filter2D(G,ddepth=-1,kernel=k)
 
cv2.imshow("image", F)
key = cv2.waitKey(0)

The second part of this lab was used to finish off our first modules .assignment.

Project Research – Preliminary Stage

Posted on by

To succeed in the aims of our project to track an item within an image across a series of similar images we are going to need to explore the technology within four potential ways of tackling this challenge before deciding on any particular approach.

Tracking motion – Looking for things that are different between images, if it doesn’t move and the pixel value hasn’t changed we are not interested, we are only interested in what has changed.

Tracking texture – The gradient orientation that occurs in the localized portions of an image is counted in texture based technique. It is then computed using overlapping local contrast normalization on a dense grid of uniformly spaced cells for improving accuracy.

Shape based – Simple pattern matching can be applied with appropriate templates.

Colour based – Creates a gaussian mixture model to describe the colour disrtibution within the sequence of images and to segment the images into background and objects.

References:

Klimek RB. Wright TW. Sielken RS. Color Image Processing and Object
Tracking System. Lewis Research Center, Cleveland, Ohio. Trinity University, San Antonio, Texas .
https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/19960016954.pdf

Yazdi M. Bouwmans T. New Trends on Moving Object Detection in Video
Images Captured by a moving Camera: A Survey. HAL Archives
https://hal.archives-ouvertes.fr/hal-01724322/document

Object tracking & Motion detection in video sequences (accessed 16.10.2018)
http://sccg.sk/~cernekova/

Balaji S RDr. S. Karthikeyan. A survey on moving object tracking using image processing. Conference: 2017 11 th International Conference on Intelligent Systems and Control (ISCO)
https://www.researchgate.net/publication/315711962_A_survey_on_moving_object_tracking_using_image_processing

Pramod Patil SP. Techniques and Methods for Detection and Tracking of Moving Object in a Video . International Journal of Innovative Research in Computer and Communication Engineering. Vol. 4, Issue 5, May 2016
http://www.rroij.com/open-access/techniques-and-methods-for-detection-and-tracking-f-moving-object-in-a-video-.pdf

The First Computer I used

Posted on by

Today in class I was asked, what was the first computer I used and I had to think waaaaay back. It also took a while.

Was it the Aston 3 I though! A primitive beast that looked like it had been made in somebody’s shed and generated TV captions. It stored everything on a fearsome looking 7″ floppy disc that could be erased by a change of wind direction.

 

Or was it the Abekas A72, another Caption Generator with a sleek futuristic keyboard and half a rack of hardware to support its amazing ant-aliasing powers.

 

Or could it have been the CMX Edit control system, driven by Unix command prompt with GPI’s for external FX units such as Vision Mixers and DVE’s.

 

Or maybe it the Quantel Paintbox, which was the forerunner to Adobe Photoshop but was definitely not as portable.

 

No it was in fact the Amiga 500.

This was a popular gaming machine in its day but we used it for generating info trivia boxes for the 1980’s Channel 4/ITV music Show ‘The Chart Show’. An illegible example of this can be seen at around 2 minutes on this clip from the show.

Week 4 – Lab Material

Posted on by

Transformation along with more image adjustments such as cropping, rotating, scaling. We also covered image Math such as adding, dividing, subtraction and multiplication. Finally we had in introduction to Kernels, yikes…

Code for the lab tasks:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# For finding the size of an image
 
import numpy as np
import cv2
from matplotlib import pyplot as plt
from matplotlib import image as image
import easygui
 
# I = cv2.imread("orange.png")
J = cv2.imread("water.jpg")
 
# Getting the size of the image:
size = np.shape(J)
# size = np.shape(J)
 
print size
 
# orange.png 360L, 630L, 3L
# water.jpg 360L, 630L, 3L

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
1. Open image "Wartime.jpg";
 
2. View its histogram alongside the image;
 
3. Use histogram equalisation to improve its contrast;
 
4. View the new histogram alongside the new image.
 
import numpy as np
import cv2
from matplotlib import pyplot as plt
from matplotlib import image as image
import easygui
 
I = cv2.imread("wartime.jpg")
G = cv2.cvtColor(I, cv2.COLOR_BGR2GRAY)
 
plt.figure()
plt.subplot(2,2,1)
 
plt.imshow(I)
plt.subplot (2,2,3)
H =cv2.equalizeHist(G)
plt.imshow(H)
 
 
Values = G.ravel()
plt.subplot(2,2,2)
plt.hist(Values,bins =256,range=[0,256]);
Values = H.ravel()
plt.subplot(2,2,4)
plt.hist(Values,bins =256,range=[0,256]);
plt.show()

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# 1. Open any image;
 
# 2. Crop out the fourth quadrant of the image;
 
# 3. Rotate this cropped section by 45 degrees
 
import numpy as np
import cv2
from matplotlib import pyplot as plt
from matplotlib import image as image
import easygui
 
I = cv2.imread("colors.jpg")
 
C = I[90:180,135:270]
 
M = cv2.getRotationMatrix2D(center=(150,5),
angle=45, scale=1)
R = cv2.warpAffine(C, M = M, dsize=(270,180))
 
cv2.imshow("image", R)
 
key = cv2.waitKey(0)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
# 1. Open images Orange and Water;
# 2. Scale each image to 50 % by multiplying each by 0.5;
# 3. Add the two scaled images to get a composite image;
# 4. Adjust the scaling to give a nicer output.
 
import numpy as np
import cv2
from matplotlib import pyplot as plt
from matplotlib import image as image
import easygui
 
I = cv2.imread("orange.png")
J = cv2.imread("water.jpg")
 
# doesn't accept float so couldn't mulitple by 0.5
h, w, d = I.shape
S = cv2.resize(I, dsize=(2*w, 2*h))
 
# doesn't accept float so couldn't mulitple by 0.5
h, w, d = J.shape
T = cv2.resize(J, dsize=(2*w, 2*h))
 
M = cv2.add(S,T)
 
cv2.imshow("M", M)
cv2.imshow("S", S)
cv2.imshow("T", T)
key = cv2.waitKey(0)