Tutorial: Detection / recognition of multiple rectangles and extracting with OpenCV

Categories Computer Vision, Uncategorized

 This tutorial will be focused on being able to take a picture and extract the rectangles in the image that are above a certain size:

I am using OpenCV 2.4.2 on Microsoft Visual Express 2008 but it should work with other version as well.

Thanks to: opencv-code.com for their helpful guides

Step 1: Clean up

So once again, we’ll use my favourite snippet for cleaning up an image:
Apply a Gaussian blur and using an adaptive threshold for binarzing the image
//Apply blur to smooth edges and use adapative thresholding  
 cv::Size size(3,3);  
 adaptiveThreshold(img, img,255,CV_ADAPTIVE_THRESH_MEAN_C, CV_THRESH_BINARY,75,10);  
 cv::bitwise_not(img, img);  

Step 2: Hough Line detection

Use a probabilistic Hough line detection to figure out where the lines are. This algorithm works by going through every point in the image and checking every angle. 
 vector<Vec4i> lines;  
 HoughLinesP(img, lines, 1, CV_PI/180, 80, 100, 10);  
And here we have the results of the algorithm:

Step 3: Use connected components to determine what they shapes are

This is the most complex part of the algorithm (general pseudocode):
First, initialize every line to be in an undefined group
For every line compute the intersection of the two line segments (if they do not intersect ignore the point)
      If both lines are undefined, make a new group out of them
      If only one line is defined in a group, add the other line into the group. 
      If both lines are defined than add all the lines from one group into the other group
      If both lines are in the same group, do nothing
cv::Point2f computeIntersect(cv::Vec4i a, cv::Vec4i b)  
   int x1 = a[0], y1 = a[1], x2 = a[2], y2 = a[3];  
   int x3 = b[0], y3 = b[1], x4 = b[2], y4 = b[3];  
   if (float d = ((float)(x1-x2) * (y3-y4)) - ((y1-y2) * (x3-x4)))  
     cv::Point2f pt;  
     pt.x = ((x1*y2 - y1*x2) * (x3-x4) - (x1-x2) * (x3*y4 - y3*x4)) / d;  
     pt.y = ((x1*y2 - y1*x2) * (y3-y4) - (y1-y2) * (x3*y4 - y3*x4)) / d;  
           //-10 is a threshold, the POI can be off by at most 10 pixels
                return Point2f(-1,-1);  
                return Point2f(-1,-1);  
     return pt;  
     return cv::Point2f(-1, -1);  
Connected components
int* poly = new int[lines.size()];  
  for(int i=0;i<lines.size();i++)poly[i] = - 1;  
  int curPoly = 0;  
       vector<vector<cv::Point2f> > corners;  
      for (int i = 0; i < lines.size(); i++)  
           for (int j = i+1; j < lines.size(); j++)  
                cv::Point2f pt = computeIntersect(lines[i], lines[j]);  
                if (pt.x >= 0 && pt.y >= 0&&pt.x<img2.size().width&&pt.y<img2.size().height){  
                     if(poly[i]==-1&&poly[j] == -1){  
                          vector<Point2f> v;  
                          poly[i] = curPoly;  
                          poly[j] = curPoly;  
                          poly[i] = poly[j];  
                          poly[j] = poly[i];  
                          for(int k=0;k<corners[poly[j]].size();k++){  
                          poly[j] = poly[i];  
The circles represent the points of intersection and the colours represent the different shapes. 

Step 4: Find corners of the polygon

Now we need to find corners of the polygons to get the polygon formed from the point of intersections.
For each group of points:
       Compute mass center (average of points)
        For each point that is above the mass center, add to top list
        For each point that is below the mass center, add to bottom list
        Sort top list and bottom list by x val
       first element of top list is  left most (top left point)
        last element of top list is right most (top right point) 
       first element of bottom list is  left most  (bottom left point)
       last element of bottom list is right most  (bottom right point) 


 bool comparator(Point2f a,Point2f b){  
           return a.x<b.x;  
 void sortCorners(std::vector<cv::Point2f>& corners, cv::Point2f center)  
   std::vector<cv::Point2f> top, bot;  
   for (int i = 0; i < corners.size(); i++)  
     if (corners[i].y < center.y)  
   cv::Point2f tl = top[0];
   cv::Point2f tr = top[top.size()-1];
   cv::Point2f bl = bot[0];
   cv::Point2f br = bot[bot.size()-1];  
for(int i=0;i<corners.size();i++){  
           cv::Point2f center(0,0);  
           for(int j=0;j<corners[i].size();j++){  
                center += corners[i][j];  
           center *= (1. / corners[i].size());  
           sortCorners(corners[i], center);  

Step 5: Extraction

The final step is extract each rectangle from the image. We can do this quite easily with the perspective transform from OpenCV. To get an estimate of the dimensions of the rectangle we can use a bounding rectangle of the corners. If the dimensions of that rectangle are under our wanted area, we ignore the polygon. If the polygon also has less than 4 points we can ignore it as well. 
for(int i=0;i<corners.size();i++){  
           Rect r = boundingRect(corners[i]);  
           // Define the destination image  
           cv::Mat quad = cv::Mat::zeros(r.height, r.width, CV_8UC3);  
           // Corners of the destination image  
           std::vector<cv::Point2f> quad_pts;  
           quad_pts.push_back(cv::Point2f(0, 0));  
           quad_pts.push_back(cv::Point2f(quad.cols, 0));  
           quad_pts.push_back(cv::Point2f(quad.cols, quad.rows));  
           quad_pts.push_back(cv::Point2f(0, quad.rows));  
           // Get transformation matrix  
           cv::Mat transmtx = cv::getPerspectiveTransform(corners[i], quad_pts);  
           // Apply perspective transformation  
           cv::warpPerspective(img3, quad, transmtx, quad.size());  
           stringstream ss;  
           imshow(ss.str(), quad);  


  • xao
    May 7, 2013

    Any chance you can convert this to C# using emgu cv (http://www.emgu.com)?

  • Foo
    May 20, 2013

    what's the img2 & img3 means

    • ayoungprogrammer
      May 21, 2013

      Mat img2;
      cvtColor(img,img2, CV_GRAY2RGB);
      img2 is just a Mat in RGB format for the graphic display of the lines.
      just use imshow("rects",img2) to see the lines

      img3 is also just a Mat in RGB format for the perspective transform.
      Mat img3;
      cvtColor(img,img3, CV_GRAY2RGB);

  • Ahsin Choudhry
    June 7, 2013

    Hi Mate,

    I want to measure the width in pixels of a shape. Kindly explain me how can I do it. I'm thinking if we find the corners of the shape, then we can do it.
    Please give me your suggestions.
    Email: [email protected]

  • Kelvin ZHU
    June 26, 2013

    Hi Mate,
    Could you please send the code to me?
    Thank you very much!

    Email: [email protected]

  • xi sizhe
    September 2, 2013

    Hi ,Michael:
    for some reason, the HoughLinesP doesn't work for me. It turns out to be that the resulted image after smoothing has aliasing on the edge, I suppose something parameters wrong with the adaptiveThreshold methods.

    I am still figuring out how to do, do you have any idea?
    Best rgds

  • Nurzhan Mussabekov
    May 11, 2014

    Hi, Michael.
    Could you send me the code?
    my email: [email protected]
    Thank you!!!

  • Can
    September 10, 2014

    How can i merge outputs. I dont need separate.

    • ayoungprogrammer
      September 10, 2014

      Sorry, I don't understand

  • Dav Car
    November 4, 2014

    Hi Michael! Your project is interesting!
    Could you please send the code to me or create a github to cooperate?
    email: [email protected]

    • ayoungprogrammer
      November 4, 2014

      The code is on a different machine, but when I have time, I will put the code up

    • Dav Car
      November 5, 2014

      i'm trying to rewrite your code for iOS 🙂

  • Isakki Raja
    June 15, 2015

    Could you send me the code?
    my email: [email protected]
    Thank you!!!

  • Unknown
    November 22, 2015

    What format do you load your image with ?
    image = imread(filename, CV_8UC1); ???

  • Abbie Liu
    February 18, 2016

    Hi, Michael
    Could you send me the code?
    my email: [email protected]
    Thank you!!!

  • ilario
    May 31, 2016

    hi Michael, I also would have the need to see the code . I would be grateful if I send it to my address [email protected]

  • kishore
    August 1, 2016

    Hi i’m kishore kumar… Please could you share me the opencv c++ code to detect “Windows in the apartments/building” please do needful..

  • sam
    September 9, 2016

    Hi..Thanks for putting this tutorial..

    Can you please send me the code on [email protected] ?

  • diyan
    November 22, 2016

    thanks for your post, very good for me learn about opencv..
    please share your code to my email [email protected]
    thank you..

  • Max
    November 22, 2016

    Thank for this . I’m interesting about computer vision.
    Can you send me the code pls ?
    [email protected]

  • James
    November 29, 2016

    Any chance this code is transferable to GoLang?

    • ayoungprogrammer
      December 8, 2016

      If GoLang has OpenCV bindings it should be transferable.

  • SB
    December 5, 2016

    Thanks for this tutorial. Can you send me the code please ([email protected])

  • Charaf
    February 4, 2017

    Thanks for this tutorial.
    Do you have python version of it by any chance ?

    Also can you send me the code please ([email protected])


  • Rafii
    March 4, 2017

    Thanks for this tutorial.
    Do you have python version of it by any chance ? If you dont have, no problem, I will try to make own, but I need little reference from you.. thanks before for the method knowledge

    Also can you send me the code please ([email protected])

    • ayoungprogrammer
      March 29, 2017

      Sorry, there is no current Python version of this code

  • idan
    April 19, 2017

    Thanks for this tutorial.
    Can you send me the code?
    [email protected]

  • chayma
    April 26, 2017

    can you send me the code please ([email protected])

  • chayma
    April 27, 2017

    can you send me the code please ([email protected]) please

  • fatih
    May 18, 2017

    heeyy guys I guess captain left the building 🙂 I find same thing with different algorithm if you want to have code you can send a email to me [email protected] 🙂

  • Amr
    January 18, 2018

    Many Thanks for your descriptive sample, I tried to build it up but there are missing some libraries, would you please share your code please ([email protected])

  • JoelW
    August 10, 2018

    Great tutorial and application of Hough lines! Just wanted to say something encouraging, not like all those too-lazy commenters who can’t copy/paste code into their own programs.

  • Dev
    October 14, 2018

    Hey, it is very helpful. But we dont know how to get the intermediate result. Can you help with that or share your code([email protected])?

  • Harsh
    August 27, 2019

    Hey I’m trying write a code for only recognizing rectangles but there are some issue
    can you send me the code for it…to – [email protected]

Leave a Reply

Your email address will not be published. Required fields are marked *