# Ground Truth Images from 3D model

Look at this photo of a katydid. Amazing camouflage, isn’t it?

By Jeff Kwapil – Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=50923289

Recently I have been tasked with a project related to the detection of a rather elusive object. Our aim is to use image processing algorithm to detect the object, and the fact that we cannot obtain too many samples of the real object is a problem. I mean, imagine you have to capture the images of a katydid, searching high and low, but it is so easy to overlook! Perhaps we can automate photo taking, and then from the many photos obtained, we want to use algorithm to detect which photos actually contain a katydid. We need to train the model of the image processing algorithm, but has too few samples. What are we to do? Let us assume we know the shape of this object.

Here, we use 3D modelling to generate many samples of the object seen from different perspective in a camouflage background. In our training, the images to be used for training need to be marked with, say, a red box to mark out the region where the elusive object is present, as shown above. We will be using ground truth images to help draw the red boxes. What we want to show here is the simple steps to create this ground truth image. We will be using Blender and python. The codes can be found here.

Let us suppose we want to detect the elusive “animal” from figure (A). The model of the object might not be easy to create, but let us also assume we have created a 3D model that replicates well the “real animal”, as shown in figure (B).

1. In Object mode, see yellow box in figure (B); select the object by right-clicking it. Once selected, change to Edit mode.
2. Select the entire object. You can also do this by selecting one vertex, edge or face, and press A twice.
3. Press SHIFT+D (duplicate the object), then left-click wherever the mouse cursor is now. Press P and choose Selection. Notice that a separate object has been created. The name of my object is animal, and the duplicated object generated is called Animal.001. This can be seen in the Outliner panel; see figure (C) yellow dashed rectangle.
4. Scale Animal.001 to be just very slightly larger than the actual Animal object, just so that Animal.001 will completely cover it. Now can change the material to a color very different from the background. I change it to red. To do this, use the icon marked with dashed yellow circle in figure (C).
5. Render and save the images from different point of views, as shown in figure D. You can use a python script to automate the process (see example here). The files are saved as a1.png, a2.png, a3.png and a4.png.
6. Use the following code ground.py to generate the ground truth images b1.png, b2.png, b3.png and b4.png.
import cv2

# Set the current directory to the current directory
# where the Blender file is located.
current_dir = "path\\to\\current\\directory"
for i in range(4):
count = i + 1
myimg = cv2.imread(current_dir + '\\a' + str(count) + '.png')
myimg_hsv = cv2.cvtColor(myimg, cv2.COLOR_BGR2HSV)
output = cv2.inRange(myimg_hsv, (0, 40, 50), (30, 255, 255))
cv2.imwrite(current_dir + '\\b' +str(count) + '.png', output)

In this code, the image is converted to hsv format, and we are to extract the part of image with red color, more specifically the one with hue between 0 to 30, saturation from 40 to 255 and value (brightness) from 50 to 255. This is done through inRange() with arguments (0, 40, 50) and (30, 255, 255). The desired color is marked white and the rest is black; this is our ground truth image and we are done! Of course, these images are to be further processed for image processing, though not within the scope of this post.

One final note: of course we can just set the object Animal to red color. However, when we need to extract only a part of an object, use the above method. Highlight the part, duplicate it, change its color to red and capture the ground truth image.