{"id":113984,"date":"2018-09-24T09:35:41","date_gmt":"2018-09-24T16:35:41","guid":{"rendered":"http:\/\/erickimphotography.com\/blog\/?p=113984"},"modified":"2018-09-24T11:09:05","modified_gmt":"2018-09-24T18:09:05","slug":"brave-new-world-of-photography-and-ai","status":"publish","type":"post","link":"https:\/\/erickimphotography.com\/blog\/2018\/09\/24\/brave-new-world-of-photography-and-ai\/","title":{"rendered":"Brave New World of Photography and AI"},"content":{"rendered":"

Dear friend,<\/p>\n

We are currently undergoing a phenomenally exciting time in photography– the marriage of (human) photographers and AI (artificial intelligence):<\/p>\n

<\/p>\n

What is the future of pictures and images?<\/h2>\n

\"picasso_bulls\"<\/p>\n

“A picture used to be a sum of additions. In my case, a picture is a sum of destructions.” – Picasso<\/p><\/blockquote>\n

A lot is going on. To get caught up, I recommend reading the lecture slides from the Stanford Computer Vision Course<\/a>.<\/p>\n

\"33823288584_1d21cf0a26_k_example_output.jpg\"<\/p>\n

Some concepts which are fascinating to me:<\/p>\n

    \n
  1. Segmentation masks<\/li>\n
  2. Edge detection<\/li>\n
  3. ‘Whitened’ images<\/li>\n
  4. Color ‘jittering’<\/li>\n
  5. Semantic and Instance segmentation<\/li>\n
  6. 3D Object detection<\/li>\n
  7. 3D Bounding Boxes<\/li>\n<\/ol>\n
    \n

    How can all this AI stuff help us?<\/h2>\n
    \"picasso_bull_plate_1\" <\/a> <\/div> <\/div>
    \"picasso_bull_plate_2\" <\/a> <\/div>
    \"picasso_bull_plate_3\" <\/a> <\/div> <\/div> <\/div>
    \"picasso_bull_plate_4\" <\/a> <\/div> <\/div>
    \"picasso_bull_plate_5\" <\/a> <\/div> <\/div>
    \"picasso_bull_plate_10\" <\/a> <\/div> <\/div>
    \"picasso_bull_plate_11\" <\/a> <\/div> <\/div> <\/div> <\/div>\n

    Ultimately, I like this simple idea:<\/p>\n

    The future of humanity is that we will become ‘centaurs<\/a>‘ — half-man, half machine.<\/p><\/blockquote>\n

    So simply in photography– this is what I am interested in:<\/p>\n

    How can we use computer vision (how computers see images, objects, and the real world) to augment (increase) our own human vision– to see the world more vividly, and to ultimately make better photos?<\/p><\/blockquote>\n

    \n

    Imagine<\/h2>\n

    \"\"<\/p>\n

    This is one of my visions:<\/p>\n

    \n Imagine a world in which your phone could see the world like the image above. You can click a toggle switch, and suddenly you see the world ‘augmented’ with these little dots, lines, to outline human skeletons. This would help you better see the poses and body language of your subject– to hopefully make better pictures.\n<\/p><\/blockquote>\n

    Or imagine you are photographing your friend (or shooting video), and you can see them abstracted as a skeleton — to better see their poses:<\/p>\n

    \"\"<\/p>\n

    Now imagine when this can all be easily automated– in real-time (or perhaps afterwards when you’re looking at pictures in your phone or computer):<\/p>\n

    \"\"<\/p>\n

    A sneak peek of how AI ‘sees’ human poses (with heat maps). Learn more on the MIT Deep Learning Course<\/a>:<\/p>\n

    \"\"<\/p>\n

    The future is already here<\/h2>\n

    \"\"<\/p>\n

    Jonathan Hui did a pretty good overview of how this ‘image segmentation’ technology works<\/a> (I also reading the scholarly paper on ‘MASK R CNN [PDF]<\/a>‘.<\/p>\n

    The basic idea is this:<\/p>\n

    \n The program will automatically create ‘bounding boxes<\/a>‘ around the subjects in the frame, then create ‘segmentation masks’, and do both:\n<\/p><\/blockquote>\n

      \n
    1. Semantic segmentation (What is in the scene? Is it a person or a dog?)<\/li>\n
    2. Instance segmentation (If there are 5 people in a scene, how can we create separate segmentation masks around each person)?<\/li>\n<\/ol>\n

      \"\"<\/p>\n

      A free tool we can play around with is the ‘Pixel Annotation Tool<\/a>‘ — to create customs masks for our images:<\/p>\n

      \"\"<\/p>\n

      \"\"<\/p>\n

      \n

      Computer Vision and Video<\/h2>\n

      \"\"<\/p>\n

      When I first saw the above GIF of a man doing parkour (then computer vision abstracting him into a simple 3D-shape) my mind was blown.<\/p>\n

      Another vision I have:<\/p>\n

      \n Imagine if you see your friend breakdancing or doing something interesting, and you take out your iPhone. You turn it into ‘Augmented’ mode, hold up your camera, and then you can (in real-time) see your friend suddenly augmented into a 3D model.\n<\/p><\/blockquote>\n

      It ain’t so farfetched. Look at some of this insane technology that exists RIGHT NOW!<\/p>\n

      For example, look at this real-time ‘pose estimation’ framework<\/a>:<\/p>\n

      \"\"<\/p>\n

      You can play with the PoseNet Browser Demo Here ><\/a>. Some examples:<\/p>\n

      \"\"<\/p>\n

      \"\"<\/p>\n

      \"\"<\/p>\n

      \"\"<\/p>\n

      Also see the PoseNet Code on GitHub ><\/a><\/p>\n

      For more inspiration, check out Dan Oved– doing lots of interesting cross-pollination between eye-tracking, machine vision, and art ><\/a><\/p>\n

      Also check out TensorFlow.JS — to experiment with machine learning on your browser ><\/a><\/p>\n

      \n

      Self-driving cars, machine vision, AI, and photography<\/h2>\n

      \"\"<\/p>\n

      Andrej Karpathy<\/a> is probably the most interesting person in AI, computer vision, etc to follow (@karpathy on Twitter<\/a>). See his ‘TRAIN AI 2.0′ PDF presentation<\/a>‘ or watch his Video Presentation ><\/a><\/p>\n

      \"\"<\/p>\n

      The future is already here with autopilot and self-driving cars, as you can see in this (2016 video) of Telsa’s Autopilot in Action:<\/p>\n