Introduction
The world cup season is here and off to an interesting start. Who ever thought the reining champions Germany would be eliminated in the group stage :(
For the data scientist within you lets use this opportunity to do some analysis on soccer clips. With the use of deep learning and opencv we can extract interesting insights from video clips. See example gif below of the game b/w Australia and Peru played where we can identify all the players + referees, the soccer ball and also predict which team the player is based on the color of their jersey. And all of this can be done real time.
You can find the code I used on my Github repo.
Overview of the steps
Tensorflow Object Detection API is a very powerful source for quickly building object detection models. If you are not familiar with this API, please see the following blogs from me that introduce the API and teach you how to build a custom model using the API.
Introduction to Tensorflow Object Detection API
Building a custom model using Tensorflow Object Detection API
The API provides pre-trained object detection models that have been trained on the COCO dataset. COCO dataset is a set of 90 commonly found objects. See image below of objects that are part of COCO dataset.
coco object categories
In this case we care about classes — persons and soccer ball which are both part of COCO dataset.
Small subset of models supported by the API
The models have a trade off between speed and accuracy. Since I was interested in real time analysis, I chose SSDLite mobilenet v2.
Once we identify the players using the object detection API, to predict which team they are in we can use OpenCV which is powerful library for image processing. If you are new to OpenCV please see the tutorial below:
OpenCV allows us to identify masks of specific colours and we can use that to identify red players and yellow players. See example below of how OpenCV masking works to detect red colour in the image.
Prediction of red areas in an image
Deep Dive into the main steps
Now lets go into the code in detail.
If you are using the Tensorflow Object Detection API for the first time, please download the GitHub from this link. And install all the dependencies using these instructions.
If you don't have OpenCV setup, please build it from source using this tutorial.
The main steps I followed are (please follow along in the jupyter notebook on my Github):
· Load the SSDLite mobilenet model into a graph and load the list of classes that are part of COCO dataset
· Open the video using cv2.VideoCapture(filename) and read each frame one by one
· For each frame perform object detection using the loaded graph
· The result that comes back from the SSDLite is each identified class along with its confidence score and bounding box prediction. So now identify all persons identified with confidence > 0.6 and crop them out.
· Now you have each player extracted out. We need to read the color of their jersey to predict if they are an Australian player or a Peru player. This is done by the code block detect team. We first define the color ranges for red and blue colors. Then we use cv2.inRange and cv2.bitwise to create a mask of that color. To detect team, I count how many red pixels and yellow pixels were detected and what is the percent of that compared to total num of pixels in the cropped image.
· Finally all the code pieces are combined to run everything at the same time and display results using cv2.imshow
Conclusion and References
Awesome. So now you can see how simple combination of deep learning and OpenCV can produce interesting results. Now that you have this data, there are many ways to draw additional insights from it:
1. With the camera angle at the Australian goal area you can calculate how many Peru players are in the zone vs Australian players
2. You can draw a heat map of each teams' footprint — Example what are the areas which saw high occupancy from Peru team
3. You can draw out the path of the goalkeeper
The Object Detection API also offers other models that are more accurate but slower. You can try those as well.
Other writings: http://deeplearninganalytics.org/blog
0 件のコメント:
コメントを投稿