aitmsi / pipes Goto Github PK
View Code? Open in Web Editor NEWThis project forked from liebannam/pipes
This project forked from liebannam/pipes
Howdy there! This is still very much in development, so please accept my apologies if anything is confusing (or worse, wrong!). -Anna Lieb (7/3/2015) contact me with comments/suggestions/questions: [email protected] /\/\/\\/\/\/\\/\/\/\\/\/\/\\/\/\/\\/\/\/\/\\ Section (0): Introduction /\/\/\\/\/\/\\/\/\/\\/\/\/\\/\/\/\\/\/\/\/\\ *********************** There are two ways to run the code. Details in Sections (1) and (2) below. *********************** (1) from command line (just compile the C++ and you're good to go) (2) from Python (requires you to build with cython) *********************** Some things you will need: *********************** --a C++ compiler. I use gcc-4.9. --Open MP --Python (I use the Anaconda distribution) --Cython (if you are doing option (1)) --Jupyter notebook (latest version of iPython notebook) *********************** Some things you may want: *********************** --POVRAY (if you want to make fancy plots) --qt_tools or ffmpeg (if you want to make movies of your fancy plots) --EPANET (if you want to draw networks that are complicated) --a stiff drink (kidding) (sort of) *********************** Some basics about input and output data: *********************** the original code used two input files: .inp and .config to describe the network simulation you were running. --the .inp file: same structure as a .inp file generated by the software EPANET. this has the advantage that you can use the EPANET graphical user interface to draw a big network and get the connectivities right. Or, if you have .inp files lying around from simulating them in EPANET, you can use them! Caveat: this code is picky about how nodes are labeled, so if you do use an EPANET-generated file, you need to run the python script (which lives in indata/) to rewrite it with acceptable numbering. To do this, just run Python cleanupinpfiles.py nameoforiginalinpfile.inp this generates a new file nameoforiginalinpfile2.inp which you can use for this code. --the .config file: not all the things the intermittency scenario needs to know about will be described in an EPANET inp file. So I have created a .config file which contains other information, such as boundary conditions, number of grid cells, simulation time, pressure wave speed, and number of time steps. Pretty self-evident, I hope. the output data goes to a directory called output_data, which the code assumes exists. If you're starting from scratch, you'll need to make this directory. For the python interface, I have made some helper functions to avoid having to deal with .inp and .config files all the time (e.g. when changing a parameter and rerunning the code many times.). Those functions are found in the module writeit.py, whose use can be examined, in, e.g. VaryLength.ipynb. /\/\/\\/\/\/\\/\/\/\\/\/\/\\/\/\/\\/\/\/\/\\ Section (1): Calling from command line /\/\/\\/\/\/\\/\/\/\\/\/\/\\/\/\/\\/\/\/\/\\ *********************** Why would you want use the code from command line? *********************** Well, it's probably simpler and faster to compile, for one thing. It's also easier to add new features to the code and test them. Lastly, some of the features of the original code (such as writing out data to make into movies) has not been wrapped for calling from Python. *********************** Getting started *********************** --Go to the folder pipes/ In this folder, you should have a folder called "output_data". If you don't have this, your simulation will crash when it tries to write output data there, so run mkdir output_data if it's not there yet. If it existed already, clear out all the files so that you don't get old data mixed up with the stuff you're about to generate. --Go to the folder called Build/ --The compilation information is in the file called Makefile and the supporting info is in the file called Sysdep. Quite frankly, I think compiling is voodoo magic, so if it's not compiling for you, then all I have to say is "good luck, have fun". --To run a simulation, you want to use the driver routine "justrunit.cpp". so you would compile as follows: make justrunit --you can also compile the optimization stuff using make optimizeit (to do optimizaiton stuff...but that cpp file may be outdated after some cython-related source code changes. no guarantees.) --calling snytax is ./justrunit nameofinpfile.inp nameofconfigfile.config (what are .inp and .config files? go back up and read about input and output data!) I have traditionally kept the .inp and .config files in the folder /indata. So, for example, you can run ./justrunit 7deSeptiembre.inp ../indata/7deSeptiembre.config. --if your simulation is crashing, one simple debug stragegy is to crank down the number of time steps. Or! set the flag WTF=1, that prints a lot of random information about what's going on. *********************** After you run a simulation *********************** You can make really pretty figures out of the data you generated in output_data (see Section (3)). /\/\/\\/\/\/\\/\/\/\\/\/\/\\/\/\/\\/\/\/\/\\ Section (2): Calling from Python /\/\/\\/\/\/\\/\/\/\\/\/\/\\/\/\/\\/\/\/\/\\ *********************** Why would you want use the code from Python? *********************** It's easier to play with parameters and data visualization. You can make lovely figures, change parameters, dig in and look at the data in your networks to verify that it is what you think it is. *********************** Getting started *********************** You will need to use Cython to compile the C++ code into a python module. To do this, run Python setup.py build_ext -i There are a couple things that usually go wrong for me here: (i) Clang butts in and pretends to be GCC, which means that omp.h can't be found. I force the compiler to be gcc-4.9 so this doesn't happen. (ii) The Cython bit gets confused and tries to pass Clang-specific flags anyway. (Are you seeing a trend here? GO AWAY, CLANG, NO ONE LIKES YOU.) so if you get the following error gcc-4.9: error: unrecognized command line option '-Wshorten-64-to-32' then try running the following commands in your terminal: export ARCHFLAGS="" export CFLAGS = "-arch i386 -arch x86_64" ...Lots of other things can go wrong, dig in to the file setup.py to look at compilation paths and flags. I have commented out (and labeled) the lines that work on Orinoco so that it will compile on my macbook air. Just uncomment the orinco lines and comment the macbook air ones, see if it works. The file allthethings.pyx has all the actual wrapping for the C++ code. It's a bit of a mess but I did try to document all the classes you would use in a python call to do something interesting (e.g. run a simulation of a network or run an optimization.) once you compile (1) give yourself a high-five!! (it's not easy) (2) you can either use python scripts or notebooks to call the C++ code. I personally like notebooks so all the examples are in .ipynb form. But there's no reason you can't use a Python script to do the exact same thing if you like. (2.5) check to make sure compiled module allthethings actually works. Sometimes it compiles but can't load or link libraries. Or something. Which is sad and means...go back to beginning and figure out compiler issues. Sorry :( (3) there are a lot of notebooks, many correspond to examples in the paper. you can use them to see some of the calling syntax and possible uses. Partial list of them: --cythondemo.ipynb (just shows lots of stuff you can do with the wrapped code) --Convergence.ipynb (do a convergence study) --VaryLength.ipynb (vary length of one pipe and simulate new network. demonstrates using the module writeit.py to make networks without touching .config or .inp files) --mysteryBC.ipynb (optimize to fit unknown BC) --reallyoptimizeDH.ipynb (optimize dh/dx by controlling boundary inflow at one node) (4) You can run a notebook on your own machine just by starting one up: ipython noteboook (5) You can also run one remotely on Orinoco, which is really nice if you are having trouble compiling the code on your own machine (or dont' want to run a big simulation there). To do this, you start one ssh pipe as follows: ssh -N -f -L localhost:8889:localhost:8889 [email protected] Then you start a new terminal, and ssh in normally: ssh [email protected] And once you're logged in to Orinoco, you start up a notebook with no browser, which instead points to the pipe you already made: ipython notebook --no-browser --port=8889 now go to your browser (on your machine!) and enter the address http://localhost:8889 You should see the usual ipython notebook tree. You can run cells and make plots--Orinoco is doing the work. *********************** After you run a simulation *********************** You can sit all day and make pretty figures in matplotlib. With tex labels! /\/\/\\/\/\/\\/\/\/\\/\/\/\\/\/\/\\/\/\/\/\\ Section (4): Dealing with all the data! /\/\/\\/\/\/\\/\/\/\\/\/\/\\/\/\/\\/\/\/\/\\ After you run a command line simulation ./justrunit blah.inp blah.config you will have filled up the folder output_data/ with a bunch of .txt and .tga files. You should never ever have to deal with them directly! The python script smarterputittogether.py and blockputittogether.py do the work of writing povray scripts and running them to create images. Smartputitogether.py can do either a topview (with false colors for pressure) or an angleview (with pretty glass pipes!). Notes on this: (0) Make a folder called movie/ if it's not already there-- that's where these scripts write the image files to. (1) before you run either of these scripts, open them and comment out the line that says os.sys(command) this prevents you from running 100 or 300 povray renderings before you know if you like the colors, view, etc. (2) Also check that the number of writes is correct (it should correspond to the number of .tga files in the output_data folder) (3) once the os.sys(command) bit is commented out, run the python script and use povray angleview.pov or povray topview.pov to generate a test figure. You may want to tweak the view or color scheme--those properties are in topview.pov and angleview.pov. (4) when you like the test figure, uncomment the line from step (1). Run the script, go get some coffee and put on a movie...rendering is slow! Go to the folder movie/ and you can then stitch the together with qt_tools, for example: quick time movies with qt_tools qt_export --sequencerate=15 tmp_000.png --dodialog test3again.mov or with ffmpeg: ffmpeg -f image2 -i tmp_%03d.png -vcodec libx264 -b 2400k output-filename.mp4
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.