Skip to main content

Face Detection with python using opencv

So I have been continuing on with the live edge detection I looked out a few weeks ago... I have made that code alot more object oriented and hopefully re-useable. I am now using both pygame and opencv built from svn instead of the ubuntu repositories. I wanted independence in the image rendering, the webcam capturing and the image processing. So I needed to convert between a numpy array (which pygame and any scipy processing uses) and cvMat which is opencv's data type. This was not immediately obvious as the opencv.adaptors module was full of routines for converting via the Python Image Library (PIL). These images were annoyingly being rotated by the functions when going from numpy to cvMat, then rotated back to the correct way going back from cvMat to numpy.
First up is the VideoCapturePlayer class, it can be used to simply display a video feed. It uses pygame camera, stores the images as a pygame.surface and shows the video with pygame. The latest pygame has an option to force pygame.camera to use opencv. Also in this class is an optional process function, something that takes and returns a surface... So in here one could put in the edge detection functionality that I looked at in the last post.
import pygame
import pygame.camera
from pygame.locals import *
import numpy
class VideoCapturePlayer(object):
"""A VideoCapturePlayer object is an encapsulation of
the display of a video stream. A process can be
given (as a function) that is done on every frame
For example a filter function that takes and returns a
surface can be given. This player will take the webcam image,
pass it through the filter then display the result.
If the function takes significant computation time (>1second)
The VideoCapturePlayer takes 3 images between each, this ensures
an updated picture is used in the next computation.
"""
size = width,height = 640, 480
def __init__(self, processFunction = None, forceOpenCv = False, **argd):
self.__dict__.update(**argd)
super(VideoCapturePlayer, self).__init__(**argd)
pygame.init()
pygame.camera.init()
if forceOpenCv:
import os
os.environ["PYGAME_CAMERA"] = "opencv"
self.processFunction = processFunction
# create a display surface. standard pygame stuff
self.display = pygame.display.set_mode( self.size, 0 )
# gets a list of available cameras.
self.clist = pygame.camera.list_cameras()
if not self.clist:
raise ValueError("Sorry, no cameras detected.")
# creates the camera of the specified size and in RGB colorspace
self.camera = pygame.camera.Camera(self.clist[0], self.size, "RGB")
# starts the camera
self.camera.start()
self.waitForCam()
self.clock = pygame.time.Clock()
self.processClock = pygame.time.Clock()
# create a surface to capture to. for performance purposes, you want the
# bit depth to be the same as that of the display surface.
self.snapshot = pygame.surface.Surface(self.size, 0, self.display)
def get_and_flip(self):
"""We will take a snapshot, do some arbitrary process (eg in numpy/scipy)
then display it.
"""
# capture an image
self.snapshot = self.camera.get_image(self.snapshot).convert()
if self.processFunction:
self.processClock.tick()
if self.processClock.get_fps() < 2:
print "Running your resource intensive process at %f fps" % self.processClock.get_fps()
# flush the camera buffer to get a new image...
# we have the time since the process is so damn slow...
for i in range(3):
self.waitForCam()
self.snapshot = self.camera.get_image(self.snapshot).convert()
self.snapshot = self.processFunction(self.snapshot)
# blit it to the display surface. simple!
self.display.blit(self.snapshot, (0,0))
pygame.display.flip()
def waitForCam(self):
# Wait until camera is ready to take image
while not self.camera.query_image():
pass
def main(self):
print "Video Capture & Display Started... press Escape to quit"
going = True
fpslist = []
while going:
events = pygame.event.get()
for e in events:
if e.type == QUIT or (e.type == KEYDOWN and e.key == K_ESCAPE):
going = False
# if you don't want to tie the framerate to the camera, you can check and
# see if the camera has an image ready. note that while this works
# on most cameras, some will never return true.
# note seems to work on my camera at hitlab - Brian
if self.camera.query_image():
self.get_and_flip()
self.clock.tick()
if self.clock.get_fps():
fpslist.append(self.clock.get_fps())
print fpslist[-1]
print "Video Capture & Display complete."
print "Average Frames Per Second "
avg = numpy.average(fpslist)
print avg
if __name__ == "__main__":
vcp = VideoCapturePlayer(processFunction=None,forceOpenCv=True)
vcp.main()
pygame.quit()
Standalone running the module (from pydev in eclipse)
If you see behind me I have it printing the frames per second... I am sadly limited somewhat on my ibook in this respect. It cannot keep up with the 15fps that my machine at hitlab could do just displaying video.
Well that is all good and fun, say to add in the edge detection filter from last post I would write this:
(As a random aside the indentation was lost importing this into google docs.... so you will have to work out how the code was indented to run it.... Stupid problem I know! Actually on second thought I might remove most of the code from here - if any one wants it just message me. Eventually I will put it into an svn repo...

def edgeDetectionProcess(surf):
    if useScipy:
        imageArray1 = numpy.mean(surfarray.pixels3d(surf),2) # converting here to one col
        if scipySpline:
            imageArray2 = edgeDetect2(imageArray1)
        else:
            imageArray2 = edgeDetect1(imageArray1)
        surf = surfarray.make_surface(imageArray2)
    else:
        # use pygame transform
        surf = transform.laplacian(surf)
    return surf

def main():
    vcp = VideoCapturePlayer(processFunction=edgeDetectionProcess)
    vcp.main()
    pygame.quit()

Musing on the coolness of it all...
I also changed the edge detection to work on an average of the red, green and blue values instead of just one. Once I had got everything working I compared the performance of forcing pygame.camera to use opencv internally for the capturing.
opencv edgeDetection scipy spline result
true false N/A N/A 66ms
false false n/A n/a 66ms
true true false N/A 209ms // opencv capture, pygame edge detection
false true false n/a 211ms
true true true false 553ms
false true true false 551ms
true true true true 790ms
false true true true 795
Might as well just throw it out there, next I thought Face detection would be a good idea. Cue a lot of research into haarcascades and object recognition. So eventually found some working code in the opencv samples and hacked away at it to come up:

def detect(img):
    gray = cvCreateImage( cvSize(img.width,img.height), 8, 1 )
    small_img = cvCreateImage( cvSize( cvRound (img.width/image_scale),
    cvRound (img.height/image_scale)), 8, 1 )
    cvCvtColor( img, gray, CV_BGR2GRAY )
    cvResize( gray, small_img, CV_INTER_LINEAR )
    cvEqualizeHist( small_img, small_img )
    cvClearMemStorage( storage )
    if( cascade ):
        t = cvGetTickCount()
        faces = cvHaarDetectObjects( small_img, cascade, storage,
        haar_scale, min_neighbors, haar_flags, min_size )
        t = cvGetTickCount() - t
        print "%i faces found, detection time = %gms" % (faces.total,t/(cvGetTickFrequency()*1000.))
        return faces
    else:
        print "no cascade"

def detect_and_draw( img ):
"""
draw a box with opencv on the image around the detected faces.
"""
faces = detect(img)
if faces:
    for r in faces:
        print "Face found at (x,y) = (%i,%i)" % (r.x*image_scale,r.y*image_scale)
        pt1 = cvPoint( int(r.x*image_scale), int(r.y*image_scale))
        pt2 = cvPoint( int((r.x+r.width)*image_scale), int((r.y+r.height)*image_scale) )
        cvRectangle( img, pt1, pt2, CV_RGB(255,0,0), 3, 8, 0 )
        cvShowImage( "result", img ) # TODO is this reqd if pygame renders?

Now the main function in the above file (now edited out) is using opencv for the capture, analysis and the rendering. I also made the next script to interface with it using the VideoCapturePlayer class from above and to use pygame for the rendering.

def drawFacesOnSurface(surf,faces):
"""draw rectangles around detected cvObjects with pygame
"""
...Snip....
if __name__ == "__main__":
    vcp = VideoCapturePlayer(processFunction=locateFaces)
    vcp.main()
    pygame.quit()
I suppose showing it working would be a good idea :-P

Yes it still works close up!
And further back...

And it works on more than one face at once!
Now if you see the blue box at the top of my screen... yeah thats my CPU usage... This is a rather intese process and seems to take about a second for the loop that does a capture, a conversion, analysis, reconversion and rendering.
Well that should just about do it for today! Actually no, maybe I'll do a quick mash of the edgedetection + face detection :-P
#!/usr/bin/python
from VideoCapturePlayer import *
import pygameFaceDetect
import edgeDetect
def process(surf):
faces = pygameFaceDetect.getFaces(surf)
surf = edgeDetect.edgeDetectionProcess(surf)
if faces:
pygameFaceDetect.drawFacesOnSurface(surf, faces)
return surf
if __name__ == "__main__":
vcp = VideoCapturePlayer(processFunction=process)
vcp.main()
pygame.quit()
Yes that is a book I am holding up...
Next maybe eye detection within a face...? Who knows! Then face recognition? Oh the possibilities!

Popular posts from this blog

Driveby contribution to Python Cryptography

While at PyConAU 2016 I attended the Monday sprints and spent some time looking at a proposed feature I hoped would soon be part of cryptography. As most readers of this blog will know, cryptography is a very respected project within the Python ecosystem and it was an interesting experience to see how such a prominent open source project handles contributions and reviews.

The feature in question is the Diffie-Hellman Key Exchange algorithm used in many cryptography applications. Diffie-Helman Key Exchange is a way of generating a shared secret between two parties where the secret can't be determined by an eavesdropper observing the communication. DHE is extremely common - it is one of the primary methods used to provide "perfect forward secrecy" every time you initiate a TLS connection to an HTTPS website. Mathematically it is extremely elegant and the inventors were the recipients of the 2015 Turing award.

I wanted to write about this particular contribution because man…

My setup for downloading & streaming movies and tv

I recently signed up for Netflix and am retiring my headless home media pc. This blog will have to serve as its obituary. The box spent about half of its life running FreeNAS, and half running Archlinux. I’ll briefly talk about my experience with FreeNAS, the migration, and then I’ll get to the robust setup I ended up with.

The machine itself cost around $1000 in 2014. Powered by an AMD A4-7300 3.8GHz cpu with 8GB of memory. A SilverStone DS380 case is both functional, quiet and looks great. The hard drives have been updated over the last two years until it had a full compliment of 6 WD Green 4TiB drives - all spinning bits of metal though.

Initially I had the BSD based FreeNAS operating system installed. I had a single hard drive in its own ZFS pool for TV and Movies, and a second ZFS pool comprised of 5 hard drives for documents and photos.

FreeNAS is straight forward to use and setup, provided you only want to do things supported out of the box or by plugins. Each plugin is install…

Markdown Editor Component for Angular2

Thought I'd share a component I've been hacking on for angular2: a syntax highlighted markdown editor with rendered preview.

The code including a basic example is available on github. Because Angular2 hasn't yet been released this is really just me kicking the tyres.



This component relies on two libraries:

- marked for rendering markdown as html
- and ace editor for editing markdown
Basic Usage Example Add to your html template:
<markdown-editor (save)="updatedText($event)" [initial-text]="markdownContent"></markdown-editor> Remember to include the Markdowndirective in your @Component annotation:
@Component({ selector:'about', directives: [CORE_DIRECTIVES, Markdown] }) Another Example You can also control the component with external ui:
<button (click)="md.editMode = true">Custom Edit Button</button><markdown-editor [initial-text]="myMarkdownText" [show-edit-but…