Code associated with this post: ColorBooked.
Motivation
This post describes a pet project that was inspired by the involvement of UC Berkeley graduate students in the scientific outreach program known as BASIS. For this program, we create simple hands-on experiments to demonstrate various scientific topics and then visit elmentary school students in the Bay Area to give the lesson plans in person. Recently, a fellow grad student had the great idea to create and distribute a coloring book publication that illustrates and describes our research in a way that is accessible to children of this age group. Since it is a very time consuming task to draw these coloring book images by hand, I wondered if there was a feasible way to keep creating these scientific color books beyond our first publication. For example, could we take cover images from scientific journals and automatically turn them into coloring book images for a monthly or bi-monthly publication?
Algorithm
After reading about various image processing and unsupervised learning techniques and some trial and error, here is a simple procedure one can use to automate the task of coloring book image generation with a few simple steps.
- Apply Gaussian blur to the intial image. This smooths out overly detailed features that otherwise result in excessively small color domains in the final coloring book image.
- Utilize the simple yet powerful unsupervised learning (clustering) algorithm known as K-means clustering to reduce the RGB color image into k principal colors that best represent the original image.
- Perform edge detection separately on each color channel in the color segmented image.
The Python program I wrote to do this can be found here.
Image processing with K-means
The important step in this workflow is the k-means color segmentation of the original image. The goal of k-means is mathematically formalized as follows:
Here are set of m observations (in the case of an RGB image
just corresponds to the pixel values). Each of the m data points are assigned to one of k different sets
, where
is just the mean of all points assigned to set i. The algorithm seeks to perform this assingment so as to minimize the loss function above. While this problem is np-hard, the k-means algorithm quickly converges to a local optimum and is sufficient for many applications. For those in which it is not sufficient there exist a variety of more complex clustering algorithms.
Here's an example of an original image...
... for which I've plotted a random sampling of 10,000 pixels by their RGB color values on the x, y, and z axes, and each is color coded by its RGB value.
After running the k-means algorithm and optimizing the cluster centroids, each pixel is assigned to the closest centroid and its RGB color set to the RGB coordinates of that centroid.
The resulting coloring book image then looks like:
The advantage of segmenting the image into domains of only k principal colors is twofold. First, it immediately provides the "already filled in" coloring book image. Secondly, a subtle but important consequence of the k-means clustering is that the edge detection step is now trivial. The edge can simply be drawn between any adjacent pixels with differing RGB values. If performing edge detection directly after Step 1, the final coloring book image would be dependent on the edge detection algorithm and its input parameters, and would not necessarily produce closed contours corresponding to one color domain. Therefore k-means gives us a robust and well-defined way of drawing the edges that are consistent with the requirements of a coloring book image:
In the future it will also be fun to play around with more complex unsupervised learning algorithms for this particular application.
Some more examples
Here are a few examples of generating coloring book images. In general, to make easier (more complex) images one should apply a a higher (lower) strength Gaussian blur in Step 1 and choose less (more) colors in Step 2.
By Alvesgaspar - Self-photographed, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=4636035
Next steps
Now that we have a simple tool for automating coloring book image generation, we can contact the major publishers in our research fields and try to obtain permission to take various cover images and convert them into coloring book images for distribution in our BASIS outreach program. For example, I converted a cover photo from the Journal of the American Chemical Society (found here) into the coloring book image below:
