When prepping images for publishing on the web, it is common to look for shortcuts that might speed things up.  And since images generally make up 40-80% of the tonnage on a website, incorrectly prepping image can have a very heavy performance cost,

A few months back, I blogged about using screen shots in production (a full 2.1% of the websites in HTTP Archive have an image named screenshot).  With so many great tools out there, surely there are some automated processes we can utilize?  Let’s take a look at some tools we can use (or easily create) to ease the image processing pipeline:

 

Automated Image Cropping

It is pretty common to want to crop an image to custom dimensions.  Let’s take my photo from dougsillars.com and crop it around my head:

In this case, I’m using Cloudinary to do the cropping:

https://res.cloudinary.com/dougsillars/image/upload/c_crop,h_1000,w_1000,q_auto,f_auto/v1558503538/doug_headshot_alps_tfzado.jpg

The parameters in the url crop the image. The blue text calls for a center crop of 1000×1000 pixels.  The red text gives the image the optimal quality and format for the browser (lowering the image size in KB without affecting quality).  We get this image:

This is awesome, but not quite ready for automated image processing, because if I utilize this image,

my autocropping algorithm fails to give a good result:

AI to the rescue

Luckily Cloudinary has several tools that allow you to find the most important objects in the image (g_auto, g_face, etc).  In this case, I use the G_auto parameter, and the AI crops the image perfectly:

http://res.cloudinary.com/dougsillars/image/upload/g_auto,c_crop,w_1000,h_1000/v1558504239/IMG_20160527_131042692_HDR_c21a3k.jpg

Machine Learning Aided Object Identification

For images with a lot of content, it might be useful to identify areas to crop around. For this we can use libraries build on machine learning to identify objects.  In the example below, I used ImageAI and followed their tutorial.  Here is Berwick upon Tweed:

The AI library found a lot of content (it may be tough to see, but there are cars and people are enclosed in boxes):

The algorithm also provides an output of what it found, and what certainty and the x,y coordinates of the box:

As you can see, it even finds the clock on the church tower!  If you were hoping to crop around an image of a bus – we now know the location, and can crop the image around the bus.

Machine Learning Limitations

The object identification libraries can only identify items it has been trained to identify. For example, in this image of me with a couple of llamas, the algorithm found both the llamas and myself:

but the model was not trained to identify llamas:

• person : 99.72
• dog : 88.67
• dog : 96.92

and it is pretty darn sure that it found two dogs.  This is clearly incorrect. In order to properly identify a new object, a new model must be trained.  In the next post in this series, I’l train a model for object detection, and use it to perform image cropping in an automated way.

 

Conclusion

There are existing AI models that will find items of interest in images to allow you to crop and modify images in an automated way.  Tools like Cloudinary work to find objects of interest, and Machine Learning libraries can identify objects that the library was trained to identify.  In the next post, I’ll build a library to identify a new object.