Composition and Machine Learning: Bounding Boxes for Photographers

Bounding boxes are a fascinating concept for photographers– and a concept that many machine learning folks use to classify images, and detect images.

Video

See on YouTube >

Bounding boxes and self-driving cars

For example, this is how self-driving cars ‘see’ other cars and objects on the road, in order not to hit them.

For example, here is the infamous TESLA self-driving car demo. Note how the car “sees” the roads with the cyan or magenta boxes or grids around the world:

An example of simple bounding boxes used in computer vision:

And even more ambitious– bounding boxes (plus) image masks (search ‘Mask R CNN’ or download the PDF paper:

Why is this a game-changer?

Anyways, the question is this:

Why is this significant for photographers or visual artists?

This is my idea of a ‘brave new world’ of photography:

We can use our phones, and automatically see the world in ‘real time’ to make better photos and videos.

Learn more at UC Berkeley Vision Course >

1. How to apply this to your photography

So the question is this–

How can we practically apply this (today) in our photography?

Basic ideas:

Start drawing bounding boxes (in red) over your own pictures.

Then we can start to abstract our vision, to create more innovative and simple and elegant compositions:

2. Composition

Another practical application — we can start to ‘see like machines‘ and start trying to think about bounding boxes, and composition lines:

Now this looks a bit crazy, but you already see yellow ‘bounding boxes’ used in the AI (artificial intelligence) of cameras such as in the Lumix cameras– with the “iA” (intelligent auto) mode (as on my Lumix G9 camera).

3. Multiple-subjects

Bounding boxes are also interesting visual tools to use to better see multiple-subjects and layers in a photo:

Also you can see– by creating more three-dimensional bounding boxes, we can better determine the position and tilt of the heads of the subjects, and also which direction they are looking:

You can also see the cyan for the faces closest to us– and the magenta for the faces a bit further away:

4. Proportions in a frame

Eric Kim photography Bauhaus Piet Mondrian

I think photography and visual art is all about proportions:

How big is your subject in respect to the background?

This is what Piet Mondrian was trying to figure out with his artwork and composition:

What are the ideal proportions of the subject versus the background?

For his entire life, Mondrian experimented with different sizes for his red, yellow, and blue boxes– in proportion to his white/black backgrounds. I sketched some of his compositions for fun on my iPad, to better understand: