The Intersection of Art and Technology
Where Art and Technology Meet
Immalla Chen
Thomas Jefferson High School for Science and Technology
As the paint washes out of my brushes, I watch the colors swirl and disappear through the sinkhole. I quickly dry my hands and scurry over to the table to grab my belongings. “Thank you, Mr. Davis,” I say as I head out the door. The long main hallway is crowded, but I zigzag through the students and arrive at my next class, Web/Mobile App Development. I sit and shuffle through ideas in my head. What should my final project be? Since I enjoy art and design, I wanted to incorporate it somehow into my project. I decided to create a photo editing web app with one simple function. You could isolate a color and make it pop on a grey-scale image. This year I am continuing this exploration between art and technology in my senior research lab and focusing on color manipulation and color transfer.
I first stumbled upon color transfer while writing my lab research proposal. I was reading through articles on color and images and came across an article titled, “Color transfer between images” by Reinhard, Ashikhmin, Gooch, & Shirley (2001, p. 34-41). Instead of editing images by changing basic features like brightness and saturation, this article described a process called color transfer where you could use one image to edit another image. Colors of a target image with the colors you want to transfer could be moved onto an original image with the colors you want to change [5]. By using this method, a picture of the ocean taken at noon could easily be changed to look like a picture of the ocean taken at sunset.
The first step of color transfer is to convert an image’s RGB color system to the Lab color space. Colors are displayed on a screen using the 2D RGB color system which stands for Red, Green, and Blue. The Lab color space is 3D and spherical. The vertical axis is luminance and the other two are the a and b axes. To transfer colors from one image to another, the mean and standard deviations from each axis of the L, a, and b color channels are measured. For each pixel from an image, the mean is subtracted from each axis and then scaled by a factor of the standard deviation of the target image over the standard deviation of the original image. Lastly, the Lab coordinates are then converted back to the RGB color system [5].
Since Reinhard et al.’s initial research, scientists have been developing new algorithms that are more accurate and efficient. One new method of color transfer introduced in a 2015 study uses pixel comparison and equalization [4]. Instead of using the mean and standard deviation like in the first method, this method sorts every pixel by color value and puts it back onto to a gray-scale image. One problem with this method is that the images become grainy. In order to combat this, Pang, Zhu, and Zhou (2015, p. 3510-3515) use an equalization method where every pixel is divided into the R, G, and B color channels, sorted by value, and put back together to create new values. This helps even out the colors and reduce graininess [4]. This method is fast and efficient, but only works well with images with few colors. For images with many colors, each color would need to be separated out and processed individually. All calculations of data and color manipulations were processed on OpenCV, a machine learning software library (P. Zhou, personal communication, January 17, 2017).
Another method is color transfer based on texture. This method works to combat the problem of spatial recognition. If you wanted to change a flower’s color on an original image to a flower’s color on a target image, but they weren’t in similar areas on both images, then the transfer would not be accurate. The actual transfer of colors in this method is based on the one proposed by Reinhard et al., but this method groups areas of an image by texture and transfers colors based on those groupings. Sky colors would transfer to the sky and grass colors would transfer to grass. Different textures are distinguished by edge-detection [1].
A third new method created by Khan et al. (2016, p. 1-12), combines the color aspect and the spatial aspect into one by bringing in multiple target images. The goal of this method is to transfer colors from multiple target images onto different areas of an original image. For example, the sky color of an image you want to manipulate could be transferred from one image, the flower color from another image, and the grass color from another image. This is unique because it allows users to have more input into what colors they want and where they want the colors to go [3].
Another method that allows for even more user interaction is manipulation by color-palette. This method takes an image and changes its colors based on a small set of chosen colors. A palette of five colors is first taken from the original image. All the colors on an image are put into bins and the bins with the most colors determine the five colors in the palette. Then, when the palette colors are changed, the colors from those bins in the image will also change accordingly. This allows for different color combinations and color themes [2].
Recently, in the news Kristen Stewart, a well-known actress, released a scientific paper on style transfer. Instead of transferring colors, her method transfers styles and strokes to mimic a target artist’s unique way of presenting their art. The paper is titled "Bringing Impressionism to Life with Neural Style Transfer in Come Swim". This Neural Style Transfer was used in a film she recently directed called Come Swim [6]. This highlights the importance of bringing art and technology together in the entertainment industry to create new and aesthetic content.
Many think of technology and art as being completely separate things on opposite sides of a spectrum, but they help each other out. As I further improve upon on my senior research project, I hope to continue to explore how these two seemingly different things work together as one in unique and efficient ways.
References
[1] Arbelot, B., Vergne, R., Hurtut, T., & Thollot, J. (2017). Local texture-based color transfer and colorization. Computers & Graphics, 62, 15-27. http://dx.doi.org/10.1016/j.cag.2016.12.005
[2] Chang, H., Fried, O., Liu, Y., DiVerdi, S., & Finkelstein, A. (2015). Palette-based photo recoloring. ACM Transactions on Graphics, 34(4), 139:1-139:11. http://dx.doi.org/10.1145/2766978
[3] Khan, A., Jiang, L., Li, W., & Liu, L. (2016, December 28). Fast color transfer from multiple images. Retrieved from https://arxiv.org/pdf/1612.08927v1.pdf
[4] Pang, G., Zhu, M., & Zhou, P. (2015). Color transfer and image enhancement by using sorting pixels comparison. Optik - International Journal for Light and Electron Optics, 216(23), 3510-3515. http://dx.doi.org/10.1016/j.ijleo.2015.08.263
[5] Reinhard, E., Ashikhmin, M., Gooch, B., & Shirley, P. (2001). Color transfer between images. IEEE Computer Graphics and Applications, 21(5), 34-41. http://dx.doi.org/10.1109/38.946629
[6] Rusu, L. (2017, January 20). Kristen Stewart co-authored a paper on artificial intelligence. Retrieved January 29, 2017, from Tech Times website: http://www.techtimes.com/articles/193654/20170120/kristen-stewart-co-authored-a-paper-on-artificial-intelligence.htm
