# Tutorial: Detection / recognition of multiple rectangles and extracting with OpenCV

Categories Computer Vision, Uncategorized

This tutorial will be focused on being able to take a picture and extract the rectangles in the image that are above a certain size:

I am using OpenCV 2.4.2 on Microsoft Visual Express 2008 but it should work with other version as well.

Thanks to: opencv-code.com for their helpful guides

### Step 1: Clean up

So once again, we’ll use my favourite snippet for cleaning up an image:
Apply a Gaussian blur and using an adaptive threshold for binarzing the image
```//Apply blur to smooth edges and use adapative thresholding
cv::Size size(3,3);
cv::GaussianBlur(img,img,size,0);
cv::bitwise_not(img, img);  ```

### Step 2: Hough Line detection

Use a probabilistic Hough line detection to figure out where the lines are. This algorithm works by going through every point in the image and checking every angle.
``` vector<Vec4i> lines;
HoughLinesP(img, lines, 1, CV_PI/180, 80, 100, 10);  ```
And here we have the results of the algorithm:

### Step 3: Use connected components to determine what they shapes are

This is the most complex part of the algorithm (general pseudocode):
First, initialize every line to be in an undefined group
For every line compute the intersection of the two line segments (if they do not intersect ignore the point)
If both lines are undefined, make a new group out of them
If only one line is defined in a group, add the other line into the group.
If both lines are defined than add all the lines from one group into the other group
If both lines are in the same group, do nothing
```cv::Point2f computeIntersect(cv::Vec4i a, cv::Vec4i b)
{
int x1 = a[0], y1 = a[1], x2 = a[2], y2 = a[3];
int x3 = b[0], y3 = b[1], x4 = b[2], y4 = b[3];
if (float d = ((float)(x1-x2) * (y3-y4)) - ((y1-y2) * (x3-x4)))
{
cv::Point2f pt;
pt.x = ((x1*y2 - y1*x2) * (x3-x4) - (x1-x2) * (x3*y4 - y3*x4)) / d;
pt.y = ((x1*y2 - y1*x2) * (y3-y4) - (y1-y2) * (x3*y4 - y3*x4)) / d;
//-10 is a threshold, the POI can be off by at most 10 pixels
if(pt.x<min(x1,x2)-10||pt.x>max(x1,x2)+10||pt.y<min(y1,y2)-10||pt.y>max(y1,y2)+10){
return Point2f(-1,-1);
}
if(pt.x<min(x3,x4)-10||pt.x>max(x3,x4)+10||pt.y<min(y3,y4)-10||pt.y>max(y3,y4)+10){
return Point2f(-1,-1);
}
return pt;
}
else
return cv::Point2f(-1, -1);
}  ```
Connected components
```int* poly = new int[lines.size()];
for(int i=0;i<lines.size();i++)poly[i] = - 1;
int curPoly = 0;
vector<vector<cv::Point2f> > corners;
for (int i = 0; i < lines.size(); i++)
{
for (int j = i+1; j < lines.size(); j++)
{

cv::Point2f pt = computeIntersect(lines[i], lines[j]);
if (pt.x >= 0 && pt.y >= 0&&pt.x<img2.size().width&&pt.y<img2.size().height){

if(poly[i]==-1&&poly[j] == -1){
vector<Point2f> v;
v.push_back(pt);
corners.push_back(v);
poly[i] = curPoly;
poly[j] = curPoly;
curPoly++;
continue;
}
if(poly[i]==-1&&poly[j]>=0){
corners[poly[j]].push_back(pt);
poly[i] = poly[j];
continue;
}
if(poly[i]>=0&&poly[j]==-1){
corners[poly[i]].push_back(pt);
poly[j] = poly[i];
continue;
}
if(poly[i]>=0&&poly[j]>=0){
if(poly[i]==poly[j]){
corners[poly[i]].push_back(pt);
continue;
}

for(int k=0;k<corners[poly[j]].size();k++){
corners[poly[i]].push_back(corners[poly[j]][k]);
}

corners[poly[j]].clear();
poly[j] = poly[i];
continue;
}
}
}
}  ```
The circles represent the points of intersection and the colours represent the different shapes.

### Step 4: Find corners of the polygon

Now we need to find corners of the polygons to get the polygon formed from the point of intersections.
Pseudocode:
For each group of points:
Compute mass center (average of points)
For each point that is above the mass center, add to top list
For each point that is below the mass center, add to bottom list
Sort top list and bottom list by x val
first element of top list is  left most (top left point)
last element of top list is right most (top right point)
first element of bottom list is  left most  (bottom left point)
last element of bottom list is right most  (bottom right point)

``` bool comparator(Point2f a,Point2f b){
return a.x<b.x;
}
void sortCorners(std::vector<cv::Point2f>& corners, cv::Point2f center)
{
std::vector<cv::Point2f> top, bot;
for (int i = 0; i < corners.size(); i++)
{
if (corners[i].y < center.y)
top.push_back(corners[i]);
else
bot.push_back(corners[i]);
}
sort(top.begin(),top.end(),comparator);
sort(bot.begin(),bot.end(),comparator);
cv::Point2f tl = top[0];
cv::Point2f tr = top[top.size()-1];
cv::Point2f bl = bot[0];
cv::Point2f br = bot[bot.size()-1];
corners.clear();
corners.push_back(tl);
corners.push_back(tr);
corners.push_back(br);
corners.push_back(bl);
}  ```
```for(int i=0;i<corners.size();i++){
cv::Point2f center(0,0);
if(corners[i].size()<4)continue;
for(int j=0;j<corners[i].size();j++){
center += corners[i][j];
}
center *= (1. / corners[i].size());
sortCorners(corners[i], center);
}  ```

### Step 5: Extraction

The final step is extract each rectangle from the image. We can do this quite easily with the perspective transform from OpenCV. To get an estimate of the dimensions of the rectangle we can use a bounding rectangle of the corners. If the dimensions of that rectangle are under our wanted area, we ignore the polygon. If the polygon also has less than 4 points we can ignore it as well.
```for(int i=0;i<corners.size();i++){
if(corners[i].size()<4)continue;
Rect r = boundingRect(corners[i]);
if(r.area()<50000)continue;
cout<<r.area()<<endl;
// Define the destination image
cv::Mat quad = cv::Mat::zeros(r.height, r.width, CV_8UC3);
// Corners of the destination image
// Get transformation matrix
cv::Mat transmtx = cv::getPerspectiveTransform(corners[i], quad_pts);
// Apply perspective transformation
stringstream ss;
ss<<i<<".jpg";
}  ```

• xao
May 7, 2013

Any chance you can convert this to C# using emgu cv (http://www.emgu.com)?

• Foo
May 20, 2013

what's the img2 & img3 means

• ayoungprogrammer
May 21, 2013

Mat img2;
cvtColor(img,img2, CV_GRAY2RGB);
img2 is just a Mat in RGB format for the graphic display of the lines.
just use imshow("rects",img2) to see the lines

img3 is also just a Mat in RGB format for the perspective transform.
Mat img3;
cvtColor(img,img3, CV_GRAY2RGB);

• Ahsin Choudhry
June 7, 2013

Hi Mate,

I want to measure the width in pixels of a shape. Kindly explain me how can I do it. I'm thinking if we find the corners of the shape, then we can do it.
Thanks,
Ahsin
Email: [email protected]

• Kelvin ZHU
June 26, 2013

Hi Mate,
Could you please send the code to me?
Thank you very much!

Email: [email protected]

• xi sizhe
September 2, 2013

Hi ,Michael:
for some reason, the HoughLinesP doesn't work for me. It turns out to be that the resulted image after smoothing has aliasing on the edge, I suppose something parameters wrong with the adaptiveThreshold methods.

I am still figuring out how to do, do you have any idea?
Best rgds

• Nurzhan Mussabekov
May 11, 2014

Hi, Michael.
Could you send me the code?
my email: [email protected]
Thank you!!!

• Can
September 10, 2014

How can i merge outputs. I dont need separate.

• ayoungprogrammer
September 10, 2014

Sorry, I don't understand

• Dav Car
November 4, 2014

Hi Michael! Your project is interesting!
Could you please send the code to me or create a github to cooperate?
Thanks
email: [email protected]

• ayoungprogrammer
November 4, 2014

The code is on a different machine, but when I have time, I will put the code up

• Dav Car
November 5, 2014

i'm trying to rewrite your code for iOS ðŸ™‚

• Isakki Raja
June 15, 2015

Could you send me the code?
my email: [email protected]
Thank you!!!

• Unknown
November 22, 2015

What format do you load your image with ?
image = imread(filename, CV_8UC1); ???

• Abbie Liu
February 18, 2016

Hi, Michael
Could you send me the code?
my email: [email protected]
Thank you!!!

• ilario
May 31, 2016

hi Michael, I also would have the need to see the code . I would be grateful if I send it to my address [email protected]
tank's.

• kishore
August 1, 2016

Hi i’m kishore kumar… Please could you share me the opencv c++ code to detect “Windows in the apartments/building” please do needful..

• sam
September 9, 2016

Hi..Thanks for putting this tutorial..

Can you please send me the code on [email protected] ?

• diyan
November 22, 2016

thanks for your post, very good for me learn about opencv..
please share your code to my email [email protected]
thank you..

• Max
November 22, 2016

Thank for this . I’m interesting about computer vision.
Can you send me the code pls ?
[email protected]

• James
November 29, 2016

Any chance this code is transferable to GoLang?

• ayoungprogrammer
December 8, 2016

If GoLang has OpenCV bindings it should be transferable.

• SB
December 5, 2016

Thanks for this tutorial. Can you send me the code please ([email protected])

• Charaf
February 4, 2017

Thanks for this tutorial.
Do you have python version of it by any chance ?

Also can you send me the code please ([email protected])

ðŸ™‚

• Rafii
March 4, 2017

Thanks for this tutorial.
Do you have python version of it by any chance ? If you dont have, no problem, I will try to make own, but I need little reference from you.. thanks before for the method knowledge

Also can you send me the code please ([email protected])

• ayoungprogrammer
March 29, 2017

Sorry, there is no current Python version of this code

• idan
April 19, 2017

Thanks for this tutorial.
Can you send me the code?
[email protected]

• chayma
April 26, 2017

can you send me the code please ([email protected])

• chayma
April 27, 2017

can you send me the code please ([email protected]) please

• fatih
May 18, 2017

heeyy guys I guess captain left the building ðŸ™‚ I find same thing with different algorithm if you want to have code you can send a email to me [email protected] ðŸ™‚

• Amr
January 18, 2018

Many Thanks for your descriptive sample, I tried to build it up but there are missing some libraries, would you please share your code please ([email protected])

• JoelW
August 10, 2018

Great tutorial and application of Hough lines! Just wanted to say something encouraging, not like all those too-lazy commenters who can’t copy/paste code into their own programs.

• Dev
October 14, 2018

Hey, it is very helpful. But we dont know how to get the intermediate result. Can you help with that or share your code([email protected])?

• Harsh
August 27, 2019

Hey I’m trying write a code for only recognizing rectangles but there are some issue
can you send me the code for it…to – [email protected]