Overview
Timeframe
Role
Team
Tools
Aug 2021 - May 2022
Researcher, Software Engineer
Esha Saini
Google Colab, Texture & Normal Map Generator, Cinema 4D
Background
As a member of the Art & AI Vertically Integrated Project (VIP) team, our work greatly deals with finding a means to create technology that is creative in nature or aids creative minds in their roles. Currently, searching for unique, tileable, high quality textures catered to an artist’s needs is an exhausting and difficult process. This project was focused on creating a tool that generates artistic RGB and bump 3D textures, fully synthesized using GANs. Via a starting image and style transfer image, the tool would generate a new tileable texture available for artists to use in their own artistic endeavors.
How can the scope of Artificial Intelligence (AI) inspire the creative process?
Anton Gudiswitz
Michelle Lee
Diana Kim
Ramya Iyer
Ethan Yu
Process
Research
Design
Assess
1
Existing Tool Analysis
Software Modification
Google Colab Prototyping
Texture Creation
Usability Testing
Final Presentation
Prospective Improvement
Resource Compliation
Requirements Gathering
3
2
Research
The research phase of this project, consisted of understanding what current tools exist and how existing resources could be modified to reach our end goal. Each team member researched various Colab files, style transfer GANs (generative adversarial networks), and bump/normal map generation tools that could potentially give us the result we desired. This gathering phase ultimately took the longest amount of time in the overall project timeline as it was a very niche end goal that required lots of thorough, targeted research.
After understanding what requirements needed to be met with our tool, we set out to find existing bits and pieces that could aid us in creating something functional. The main moving parts of our tool were implementations of an existing Style Transfer Neural Network and a Normal/Displacement Map Generator. These parts were coded in a Google Colab file to work in a specific, step-by-step process to generate the final downloadable texture.
Our initial research goal was to research what current tools exist for artists to generate their own textures and what some potential issues or reservations regarding these tools are. Overall we discovered that the existing tools were mostly not free to use and even if they were free, there were no tools that generated unique, inorganic (not naturally occurring) textures. If an artist wanted something abstract to base their work off of, they would struggle.
Existing Tool Analysis
Resource Compliation
Materialize
Material Maker
Before Style Transfer:
After Style Transfer:
After Style Transfer:
Before Style Transfer:
Generated Bump Map:
Generated Bump Map:
Style Transfer Neural Network - [1] Golnaz Ghiasi, Honglak Lee, Manjunath Kudlur, Vincent Dumoulin, Jonathon Shlens. Exploring the structure of a real-time, arbitrary neural artistic stylization network. Proceedings of the British Machine Vision Conference (BMVC), 2017.
Normal Map Generation - adapted from https://github.com/Mehdi-Antoine/NormalMapGenerator
Resources Used
Design
For the design phase of this project, the primary focus was combining and modifying the resources we found during our research phase in order to produce viable prototypes. With some trial and error of which types of images were best suited as source and style images, we were able to form a solid foundation for our tool to use as base prototypes. The ideal end goal was to consolidate all the necessary elements into one Google Colab file that with a few clicks, could create unique, abstract, tileable textures.
Initial Prototypes
Advanced Applications
Final Tool
Assess
Once the tool was refined to an extent that produced quality textures upon every use, we deemed it ready for presentation. This tool was demoed live in front of our entire Art & AI project class and users were able to create their own textures in order to test usability. A crucial part of our assessment phase was discussing future refinements and scope for real world application. The major improvement we wanted to make to the tool was increasing the quality of the texture’s tileability. Tileability in this context is defined as the ability to have seamless transitions when the texture is duplicated and placed side by side to cover more area. After addressing tileability concerns, next steps would include using these textures in online animation and 3D-printing them onto physical models.
