An experimental framework for running ML Models inside the Grasshopper / Rhino environment

SmokingGAN is a Hackathon project for the 2020 AEC Tech Symposium organized by CORE Studio at Thornton Tomasetti

This project has been a collaborative effort between Mayur Mistry, Sergey Pigach, Charlie Portelli and Sergio Guindon

SmokingGAN borrows heavily from the following open-source projects:

Fast Style Transfer: https://github.com/lengstrom/fast-style-transfer

TF Monodepth 2: https://github.com/FangGet/tf-monodepth2

SmokingGAN is an attempt at incorporating GANs into a Rhino / Grasshopper design workflow. The two major parts of the SmokingGAN are the Flask server exposing pre-trained checkpoints (This hackathon version currently serves A Style GAN and a Depth Map generator, but can be extended to work with any other pretrained checkpoints) and a Grasshopper plugin that accesses the Flask API and also provides functionality for converting images to meshes based on the depth information.

This is the basic architectural diagram showing the main components of the API:

We recommend creating an Anaconda environment with Python 3.7 (IMPORTANT: we've ran into serious issues with Tensorflow and Python 3.8)

conda create --name your-env-name python=3.7
conda activate your-env-name

Install the necessary dependencies:

conda install flask numpy imageio marshmallow matplotlib scipy pillow pandas
pip install tensorflow==2.3.1
pip install keras==2.4.3
pip install tf_slim
pip install flask
pip install marshmallow
pip install imageio
pip install matplotlib

Download the following checkpoint files:

Style Transfer - https://drive.google.com/drive/folders/0B9jhaT37ydSyRk9UX0wwX3BpMzQ

Depth Map - https://drive.google.com/drive/folders/0B9jhaT37ydSyRk9UX0wwX3BpMzQ

Place them into Flask/Checkpoints/StyleTransfer and Flask/Checkpoints/DepthMap respectively.

Run the server:

cd GenerativeModeling/Flask
python MainFrame.py

Open and build the GenerativeModeling solution from the .NET folder

In Rhino's command line type in GrasshopperDeveloperSettings. In the dialog paste the path to the bin folder for the Visual Studio solution. Click ok and launch Grasshopper. You should see the SmokingGAN tab appear in the panel.

Have fun: