Progress and Impediments in Deep Learning-Driven Image Style Transfer

Yuqi Jiang

doi:10.62051/kqkdq196

Authors

Yuqi Jiang

DOI:

https://doi.org/10.62051/kqkdq196

Keywords:

Image Style Transfer, Deep Learning Perspective.

Abstract

This detailed review explores key developments and ongoing challenges in image style transfer, emphasizing the transformative role of deep learning approaches, notably Convolutional Neural Networks (CNNs) and Generative Adversarial Networks (GANs). The paper examines the fundamental principles of this field, particularly the intricate process of blending 'style' and 'content' via advanced neural network designs. It chronicles critical breakthroughs, and progresses to contemporary solutions addressing real-time execution, diversity enhancement, and stability in results. Emerging techniques such as Deep Feature Interpolation and Multi-Scale Style Transfer are also scrutinized, offering insights into potential research directions. The review not only traces the technical evolution but also considers the wider impact of image style transfer, underscoring its significance in bridging art and technology. This intersection is demonstrated through applications that span from digital art creation to innovative adaptations in medical imaging.

Downloads

Download data is not yet available.

References

Gatys, L. A., Ecker, A. S., & Bethge, M. (2016). A neural algorithm of artistic style. Journal of Vision, 16(12), 326.

Johnson, J., Alahi, A., & Fei-Fei, L. (2016). Perceptual losses for real-time style transfer and super-resolution. European Conference on Computer Vision (ECCV).

Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014). Generative adversarial nets. Advances in Neural Information Processing Systems (NIPS).

Zhu, J. Y., Park, T., Isola, P., & Efros, A. A. (2017). Unpaired image-to-image translation using cycle-consistent adversarial networks. IEEE International Conference on Computer Vision (ICCV).

Jing, Y., Yang, Y., Feng, Z., Ye, J., Yu, Y., & Song, M. (2020). Neural style transfer: A review. IEEE Transactions on Visualization and Computer Graphics, 26(11), 3365-3385.

Luan, F., Paris, S., Shechtman, E., & Bala, K. (2017). Deep photo style transfer. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

Ulyanov, D., Vedaldi, A., & Lempitsky, V. (2016). Instance normalization: The missing ingredient for fast stylization. ArXiv preprint arXiv: 1607.08022.

Isola, P., Zhu, J. Y., Zhou, T., & Efros, A. A. (2017). Image-to-image translation with conditional adversarial networks. Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR).

Li, C., & Wand, M. (2016). Combining Markov random fields and convolutional neural networks for image synthesis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

Chen, D., Yuan, L., Liao, J., Yu, N., & Hua, G. (2017). Stylebank: An explicit representation for neural image style transfer. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

Simonyan, K., & Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations (ICLR).

He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR).

Karras, T., Aila, T., Laine, S., & Lehtinen, J. (2018). Progressive growing of GANs for improved quality, stability, and variation. International Conference on Learning Representations (ICLR).

Huang, X., & Belongie, S. (2017). Arbitrary style transfer in real-time with adaptive instance normalization. Proceedings of the IEEE International Conference on Computer Vision.

Kolkin, N., Salavon, J., & Shakhnarovich, G. (2019). Style transfer by relaxed optimal transport and self-similarity. Conference on Computer Vision and Pattern Recognition (CVPR).

Li, Y., Fang, C., Yang, J., Wang, Z., Lu, X., & Yang, M. H. (2017). Universal style transfer via feature transforms. Advances in Neural Information Processing Systems (NIPS).

Dumoulin, V., Shlens, J., & Kudlur, M. (2017). A learned representation for artistic style. International Conference on Learning Representations (ICLR).

Selim, A., Elgharib, M., & Doyle, L. (2016). Painting style transfer for head portraits using convolutional neural networks. ACM Transactions on Graphics (TOG), 35(4).

Risser, E., Wilmot, P., & Barnes, C. (2017). Artistic style transfer for videos. Graphical Models, 82, 23-36.

Chen, T. Q., & Schmidt, M. (2016). Fast patch-based style transfer of arbitrary style. arXiv preprint arXiv:1612.04337.

Mechrez, R., Talmi, I., & Zelnik-Manor, L. (2018). The contextual loss for image transformation with non-aligned data. Proceedings of the European Conference on Computer Vision (ECCV).

Sanakoyeu, A., Kotovenko, D., Lang, S., & Ommer, B. (2018). A style-aware content loss for real-time HD style transfer. Proceedings of the EuropeanConference on Computer Vision (ECCV).

Park, D., Liu, S., Wang, T.-C., & Zhu, J.-Y. (2019). Semantic image synthesis with spatially-adaptive normalization. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

Sheng, L., Lin, Z., Shao, J., & Wang, X. (2018). Avatar-net: Multi-scale zero-shot style transfer by feature decoration. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

An, J., & Pellacini, F. (2018). CHAMELEON: Adaptive selection of collections of style-transfer filters. ACM Transactions on Graphics (TOG), 37(4).

Liao, J., Yao, Y., Yuan, L., Hua, G., & Kang, S. B. (2017). Visual attribute transfer through deep image analogy. ACM Transactions on Graphics (TOG), 36(4), 120.

Elgammal, A., Liu, B., Elhoseiny, M., & Mazzone, M. (2017). CAN: Creative adversarial networks, generating "art" by learning about styles and deviating from style norms. International Conference on Computational Creativity.

Gharbi, M., Chen, J., Barron, J. T., Hasinoff, S.., & Durand, F. (2017). Deep bilateral learning for eal-time image enhancement. ACM Transactions on Graphics (TOG), 36(4).

J. Johnson, & Fei-Fei, L. (2018). Image generation from scene graphs. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

Mordvintsev, A., Olah, C., & Tyka, M. (2015). Inceptionism: Going deeper into neural networks. Google Research Blog.