3D vision with 2 NXTcams?

[linusa]

I was just wondering (since I'm currently heavy involved with 3D vision):

Did anyone ever try to build a robot with two (or even more) NXTcams? Wouldn't it be cool to have a robot that could estimate the distance of colored objects by 3D vision?

Since I've never used an NXTcam, here some questions:

• How many different blobs can it detect simultaneously?
• Given stable light conditions (indoors, "calibrated"), how many differently colored blobs could it distinguish reliably?
• Does it calculate the centroid/position of the blobs? How accurate?
• What is the resolution/total range of these blob coordinates (as in what precision do the coordinates have)?

I actually think it would be possible to estimate the 3D position of blobs with respect to a camera coordinate system -- and that in turn can be related to the robots local coordinate system.

If the NXTcam can somehow output 4 blobs, calibration would be possible (with less blobs, it might get more complicated).

If anyone was interested, we could discuss this in more detail. But overall, given the right tools, stereo vision is not that hard

[HaWe]

How many different blobs can it detect simultaneously?

8

Given stable light conditions (indoors, "calibrated"), how many differently colored blobs could it distinguish reliably?

I tested up to 4 or 5, depending on distance, size, light source (fluorescent light is worst), environmental appearing colors and shades, blob color (green appears like blue or black). In the worst cases which I tested: none.
E.g., the colors of a Rubik's cube (white, orange, blue, green, red, yellow) can not be safely distinguished at all (orange vs. yellow/red, white vs. yellow, green vs. blue), even if placed in front of a light grey cartoon. Light shine and reflections of the colored surfaces appear as different colors, if the biases are raised, the discernment of different colors is lowered.

Does it calculate the centroid/position of the blobs? How accurate?
What is the resolution/total range of these blob coordinates (as in what precision do the coordinates have)?

it' like the AVR CCD, but the blobs buzz and swirr over the NXT screen if visualized that way.

[loopymech]

I remembered seeing this had been done. See link below...

http://www.youtube.com/watch?v=o0X9PKMVxBY


cheers!
-Loopy

[linusa]

I remembered seeing this had been done. See link below...
http://www.youtube.com/watch?v=o0X9PKMVxBY

Thanks for the link. I'm not sure however if they do in fact calculate the 3D coordinates of the red ball. I don't see why this specific application needs 2 NXTcams, and I see no need for stereo vision algorithms...

[gloomyandy]

Th video explains why they use two cameras. It allows them to track distance to the object without having to know the size of the object. They show how a single cam can be used if you know the size of the object....

[linusa]

Th video explains why they use two cameras. It allows them to track distance to the object without having to know the size of the object. They show how a single cam can be used if you know the size of the object....

Ah, shouldn't have been watching it without sound -- no it's all clear.

I'm now convinced the video author used stereo vision. Even though I think the demo is a bit misleading, as you could do the ball tracking in this case with just 1 cam, no compass and no tilt sensor (by just keeping the object in the center of the cam, and moving the motors accordingly).

Anyway, using just 1 cam and known size of the object, this 3D range method is the same as the Playstation's controller PSMove does.

Given 2 NXTcams, I'd personally do intrinsic and extrinsic camera calibration using Bouguet's toolbox for MATLAB or OpenCV's 3D calib functions (which is based on Bouguet anyway). A stereo vision example can be found here (see pic at the bottom of the page).

For calibration you need at least 4 points / trackable objects on a plane. Show this plane to the two NXTcams from various positions and angles. Record the tracked marker locations. Save them to NXT memory, and export them to a PC. Then use the above mentioned calibration engines.

After that, you have calibrated stereo system with the transformation you need to get from cam1's coordinate system to cam2's (i.e. a rigid motion transformation, covering 6 degrees of freedom (3 translation + 3 rotation)). If you align your cameras precisely, there won't be any rotation and only displacement in 1 direction (the baseline). But that doesn't really matter...

Given this calbration data, you can work out the 3D coordinates in world coordinates for any given object / marker, which is recognized by both NXTcams simultaneously. Take away certain degenerate disambiguities...

[linusa]

So let me rephrase:

Anybody here with 2 or more NXTCams who would like to have 3D vision for his robot?
By "3D vision" I mean: The NXT could calculate the 3D world coordinates of some coloured objects in "real time". I'd be willing to help with the programming and act as guide for the algorithms. And I know it's possible (I don't know with what quality though).

[HaWe]

I would use 2 cams (having 2 DOF for rotation) which can triangulate a single specific unique color blob by "centering the center" by independend 2D rotations in the center of each CCD screen. Then it's automatically independend from the object size.
I dare say, it will work only in widely restricted experimental lab environments, not in real commonplace environments with changing colored backgrounds, undergrounds, and light conditions.

[linusa]

I would use 2 cams (having 2 DOF for rotation) which can triangulate a single specific unique color blob by "centering the center" by independend 2D rotations in the center of each CCD screen. Then it's automatically independend from the object size.

In my plan, I'd roughly follow the steps described by S. Hay et. al. (http://www.cl.cam.ac.uk/~sjeh3/wii/ , good short intro is the poster http://www.cl.cam.ac.uk/~sjeh3/wii/poster.pdf ). I tried this, it works well, and I feel I understood the concept. There are no object sizes, just dot-markers. So I have N sets of 2D coordinates, and I calculate N 3D coordinates from it. The marker size (i.e. the radius of the colored ball) is only useful to get the position of the marker with better sub-pixel accuracy, and it's relevant to the NXTcam's internal software (don't know what it does with it).

I dare say, it will work only in widely restricted experimental lab environments, not in real commonplace environments with changing colored backgrounds, undergrounds, and light conditions.

It's ok if it's working in a lab with controlled and constant light conditions. It's even ok if ALL MARKERS HAVE THE SAME COLOR, I'm not sure I made that clear. As long as 4 or more "blob positions" can be detected by each camera, that's fine.

A short calibration to adapt the NXTcams settings to ambient light conditions would be acceptable (this works for the light and color sensors, too, after all).

So, anybody up to it?