Giter Club home page Giter Club logo

spectral-workbench's Introduction

Spectral Workbench

Copyright Public Lab publiclab.org | spectralworkbench.org

Gitpod Ready-to-Code tests View performance data on Skylight

Spectral Workbench is an open-source tool to perform low-cost spectral analysis and to share those results online. It consists of a Ruby on Rails web application for publishing, archiving, discussing, and analyzing spectra online -- running at http://spectralworkbench.org

The core library for analyzing and manipulating spectral data has been spun out into its own self-contained JavaScript module, at https://github.com/publiclab/spectral-workbench.js, which is then included into this application. To use Spectral Workbench on the commandline, try that out!

Read about how to build and use your own spectrometer with this software here: http://publiclab.org/wiki/spectrometer


Simple installation with Cloud9

  1. If you have a GitHub account, visit https://c9.io and log in with the GitHub button.
  2. Fork this repository to your own GitHub account, creating a yourname/spectral-workbench project.
  3. Name your project, then (order important!) choose the Ruby template, THEN enter yourname/spectral-workbench in the "Clone from Git or Mercurial URL" field, and press Create Workspace
  4. In the command line prompt at the bottom of the page, type ./install_cloud9.sh and press enter.
  5. Enter your username when prompted, and click "Run Project" when it's done.
  6. You're done! Go to the URL shown!

Installation

The app now runs on Ruby 2.6.6 up to Ruby 2.6.x (preferred), and Rails 5.2.x, and uses Bundler for gem management and Yarn for static asset management.

Prerequisites:

Recommended; for an Ubuntu/Debian system. Varies slightly for mac/fedora/etc

Install a database, if necessary:

sudo apt-get install mysql-server

Note: You will have to update the config/database.yml file appropriately. The example file assumes usage of SQLite. For which you need not install MySQL.

RMagick dependencies are required for processing uploaded spectrum images: apt-get install imagemagick ruby-rmagick libmagickwand-dev libmagick++-dev

  • On Fedora/centOs: yum install ImageMagick-devel
  • On mac, you can use Homebrew: brew install imagemagick

Install rvm for Ruby management (http://rvm.io)

curl -L https://get.rvm.io | bash -s stable

Note: At this point during the process, you may want to log out and log back in, or open a new terminal window; RVM will then properly load in your environment.

Ubuntu users: You may need to enable Run command as a login shell in Ubuntu's Terminal, under Profile Preferences > Title and Command. Then close the terminal and reopen it.

Then, use RVM to install version 2.6.6 of Ruby:

rvm install 2.6.6

You'll also need yarn which is available through NPM. To install NPM, you can run:

sudo apt-get install npm

However, on Ubuntu, you may need to also install the nodejs-legacy package, as due to a naming collision, some versions of Ubuntu already have an unrelated package called node. To do this, run:

sudo apt-get install nodejs-legacy

Once NPM is installed, you should be able to run:

sudo npm install -g yarn

  • On mac, you can use Homebrew: brew install yarn

Installation steps:

  1. Download a copy of the source with git clone https://github.com/publiclab/spectral-workbench.git
  2. Install gems with bundle install from the rails root folder. You may need to run bundle update if you have older gems in your environment.
  3. Copy and configure database with cp config/database.yml.example config/database.yml. Similarly, configure cp config/config.yml.example config/config.yml
  4. Initialize database with bundle exec rake db:setup
  5. Install static assets (like external javascript libraries, fonts) with yarn install
  6. Start rails with bundle exec passenger start from the Rails root and open http://localhost:3000 in a web browser. (For some, just passenger start will work; adding bundle exec ensures you're using the version of passenger you just installed with Bundler.)

Sign in instructions:

  • Create a account at PublicLab.org and use that username to log in.
  • Then you will be redirected to publiclab.org to "approve" a use of the openid identity.
  • Note that this applies for development environment as well.

Bugs and support

To report bugs and request features, please use the GitHub issue tracker provided at http://github.com/publiclab/spectral-workbench/issues

For additional support, join the Public Laboratory website and mailing list at http://publiclab.org/lists or for urgent requests, email [email protected]

For questions related to the use of this software and your open source spectrometer, the same page links to the "plots-spectrometry" group.

API

Using your secret API token on your SpectralWorkbench.org profile, you can submit spectral via the API in JSON format:

POST to "https://spectralworkbench.org/spectrums.json"

The required parameters are:

{"spectrum": {
  "title": "Test spectrum",
  "data_type": "json",
  "data": [
    {"average": 64.3, "r": 69, "g": 46, "b": 78, "wavelength": 269.089 },
    {"average": 63.3, "r": 71, "g": 45, "b": 74, "wavelength": 277.718 },
    {"average": 64,   "r": 71, "g": 47, "b": 74, "wavelength": 291.524 },
    {"average": 64,   "r": 68, "g": 49, "b": 75, "wavelength": 303.604 }
  ]},
  "token": "00000000"
}

This will return a path (from the root URL by default) to the spectrum, as in: /spectrums/12345

Developers

Development is occurring at https://github.com/publiclab/spectral-workbench/; please fork and submit pull requests; for more guidelines on contributing to Public Lab projects, see http://publiclab.org/wiki/contributing-to-public-lab-software

If you're a developer, consider joining the Public Lab developer list, also at http://publiclab.org/wiki/developers

Testing

Before submitting changes, please run tests with rake test to ensure that your code passes.

Also run rake jasmine and navigate to http://localhost:8888 to check client-side tests.

New tests are also appreciated to increase coverage; Rails tests are in /test and Jasmine tests (JavaScript tests) are in /spec.


License

Spectral Workbench is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

Spectral Workbench is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with Spectral Workbench. If not, see http://www.gnu.org/licenses/.

spectral-workbench's People

Contributors

alaxalves avatar anantraghuvanshi avatar aniruddhabagal avatar cesswairimu avatar cmcgee0308 avatar dependabot-preview[bot] avatar dependabot[bot] avatar dev-suren avatar developer22-university avatar georjane avatar gr455 avatar icarito avatar jrslagle avatar jywarren avatar mbuiro avatar rhonall avatar samn avatar saraswathy-krish avatar shiv2708 avatar shuchitgandhi avatar sidharthbansal avatar sloutrel avatar sreyanth avatar stevesayantan avatar tildadares avatar tlazypanda avatar tsparksh avatar uzay-g avatar vidurangaw avatar waridrox avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

spectral-workbench's Issues

improve CSV format consistency and standards-compliance

The CSV download of a spectrum is two (unlabeled) columns. I presume this is only the spectral data.

For JSON, XML, and HTML, the full set of metadata and data is included.

One issue that needs to be addressed is that the spectral data is always returned as embedded JSON (e.g. in XML it is <data>{ json stuff }</data>).

The current format returned by the CSV is a good start towards demuxing the JSON into CSV, but we need to append/prepend the rest of the metadata as well as provide column names.


Addendum by @jywarren: include FWHM values in CSV; details in jywarren#137

design UI for wavelength range limiting

The current version of Spectralworkbench is hard-coded to display reflectance spectra between 250 and 800nm. Since the webcam is sensitive past 800nm, could you add functionality in the UI to extend this display to spectra beyond 800nm (say 850nm or so)?

Thanks! -david mccloskey

add energy scale in eV

@feranick sez:

Currently wavenumbers and wavelength are used as units. Sometimes eV are needed as well as a unit. It would be great to have that as an option for the x axis; the conversion is: eV = 1239.82/wavelength

index should not return data, but there should be equivalent call

Presently, if one calls http://spectralworkbench.org?format=xml, all the spectra (possibly truncated at some arbitrary point) are returned with metadata and spetrum data.

def index
if logged_in?
redirect_to "/dashboard"
else
@spectrums = Spectrum.find(:all,:order => "created_at DESC", :conditions => ["author != 'anonymous'"], :limit => 100)
@spectrums = @spectrums.paginate :page => params[:page], :per_page => 24
# @anon_spectrums = Spectrum.find(:all,:order => "created_at DESC", :conditions => {:author => "anonymous"})
# @anon_spectrums = @anon_spectrums.paginate :page => params[:page], :per_page => 4
@sets = SpectraSet.find(:all,:limit => 4,:order => "created_at DESC")
@comments = Comment.all :limit => 12, :order => "id DESC"
respond_to do |format|
format.html {
render :template => "spectrums/index.html.erb"
} # show.html.erb
format.xml { render :xml => @spectrums }
end
end
end

The index should contain just enough to decide which spectra to download without including the actual spectrum data. This is more friendly to front-end javascript decision making, but can also by used by offline massive processing applications.

There should be a similar call, however, which returns all the metadata and the spectral data. The concept is that the front end javascript has decided which spectra to grab, and will now make this new call to get it. So it'll also need some selectors, minimally a set of ids will suffice.

Possibly this interface could be called bulk or bulkdata.

Additionally, index and bulk should support csv and json on top of html and xml..

can't use "more tools" after saving a set

after I save a set I no longer have access to the "more tools" pallette. I'd like to create a set adn then equalize the area underneath the spectra for purposes of comparison. To do that I have to re-build the set, adding in each of the spectra again.

I wish I could just apply transforms to saved sets.

cross-section start/end powertagging

We have a terminology collision here; extraction in spectroscopy can refer to extracting something from a sample with solvents. But what I mean here is a means, like Infragrammar (http://publiclab.org/wiki/infragrammar), to specify how data should be taken from an image.

Another collision is "recipes" -- here I mean like a parseable expression, but in #64 "recipes" refers to something different; the steps in an experimental method. The feature in this issue needs renaming.

This could include start/end pixels for a cross section, whether to take the average or sum of the pixel color values, etc.

The recipe might be stored as a tag, such as start:151,355, end:454,1266 and rgb:average or rgb:sum

presumably we'd have to enforce certain rules for namespaces like rgb: or start: -- maybe most recent wins; also, when you add such tags, it should enter the changes in a log

commenting on specific wavelengths/annotating graphs

$("#placeholder").bind("plotclick", function (event, pos, item) {
  alert("You clicked at " + pos.x + ", " + pos.y);
  // axis coordinates for other axes, if present, are in pos.x2, pos.x3, ...
  // if you need global screen coordinates, they are pos.pageX, pos.pageY

  if (item) {
    highlight(item.series, item.datapoint);
    alert("You clicked a point!");
  }
});

spectra calibrated with a macro do not get tagged "calibrated"

@Fastie sez:

Spectra calibrated with a macro ($w.calibrate(...)) never show up as a choice for calibrating while uploading or from the analyze page. This means that it is not possible to use some well-calibrated spectra for calibration. The macro feature offers a workaround, but it would be nice to calibrate on upload using a spectrum carefully calibrated with a macro.

/capture doesn't sample from the middle by default

at startup in the settings menu, if no cross-section is selected and I just click "Begin Capturing" I do not see the middle cross section when I start capturing.
screen shot 2014-09-21 at 4 50 47 pm
screen shot 2014-09-21 at 4 48 25 pm
There should be a default cross section-- and I remember there being one before... seems like a new problem.

bulk image download

Presently the way to download the raw image data is one at a time using the format http://spectralworkbench.org/system/photos/<id>/original/capture.png. If the format is wrong, the filename is in the metadata for each spectrum.

If a bulk download of 60+ images is required for offline processing experiments, iterating this format will be a major strain on the server. Worse yet would be if the data wasn't downloaded, so that each experimental modification required a new download of 60+ images.

Desired is a system which will gather the raw images for a set of spectrum ids submit by the user. The backend will grab the images, compress them, and send them to the user in a single push.

For all images ever, it might be good if we back up the images to S3, DropBox, or some photo sharing site. Many external services have a bulk download button which does the process of compressing a large set of files and sending them as a single file to the user. This would certainly eliminate effort on our part.

I don't think S3 has that capability built in by default, but it will statically host files. Monthly caches could be put there, but that feels far too arbitrary.

finish upgrading to Rails 3 conventions

Deprecated session secret setting
Previously, session secret was set directly on ActionController::Base; it's now config.secret_token.
More information: http://lindsaar.net/2010/4/7/rails_3_session_secret_and_session_store

The culprits: 
    - config/initializers/session_store.rb

Deprecated test_help path
You now must require 'rails/test_help' not just 'test_help'.
More information: http://weblog.rubyonrails.org/2009/9/1/gem-packaging-best-practices

The culprits: 
    - test/test_helper.rb

Soon-to-be-deprecated ActiveRecord calls
Methods such as find(:all), find(:first), finds with conditions, and the :joins option will soon be deprecated.
More information: http://m.onkey.org/2010/1/22/active-record-query-interface

The culprits: 
    - app/models/processed_spectrum.rb
    - app/models/spectrum.rb
    - app/controllers/spectrums_controller.rb
    - app/models/processed_spectrum.rb
    - app/models/spectrum.rb
    - app/models/user.rb
    - app/controllers/spectrums_controller.rb
    - app/controllers/likes_controller.rb

Deprecated ActionMailer API
You're using the old ActionMailer API to send e-mails in a controller, model, or observer.
More information: http://lindsaar.net/2010/1/26/new-actionmailer-api-in-rails-3

The culprits: 
    - app/models/spectrum.rb
    - app/models/spectra_set.rb
    - app/models/user.rb
    - app/controllers/users_controller.rb
    - app/controllers/sets_controller.rb
    - app/controllers/comments_controller.rb

Old ActionMailer class API
You're using the old API in a mailer class.
More information: http://lindsaar.net/2010/1/26/new-actionmailer-api-in-rails-3

The culprits: 
    - app/models/user_mailer.rb
    - app/models/user_mailer.rb

subtracting baseline spectra

For the life of me, I cannot get SpectralWorkbench to subtract a baseline spectrum. Was this feature removed? I am using the latest version of FireFox on a PC.

Offline Use/RasPi Image

This gizmo is small enough that it would be useful in field camps where there is no internet connectivity. Making available a RaspberryPi image that permitted "off-line" usage with possibly a locally stored database of popular exaple spectrograms would greatly improve the usefuleness of this appliance

Support open standards for exporting spectra

Because open data and portability are good things(TM), we probably want to support open standards for this sort of data. The custom exports that currently exist certainly work, but might not be as portable or as easy to work with as open standards.

We might want to export some small amount of metadata regarding ownership and licensing for each spectra which I don't think is part of any standard.

mobile selection of cross section does not show data in live graph during capture

screenshots_2015-01-06-15-55-50
screenshots_2015-01-06-15-54-58
I have attached two screenshots from my Galaxy Note 3 smartphone. I am using the smartphone DIY instructions and the back facing camera. I found first that the online spectralworkbench utility does not work with both Chrome (back facing camera only displays 'black' with nothing covering camera!) and Opera. I can only use it properly with current Firefox. The screenshots come from the DIY spec/smartphone pointed at a perfectly white display on a 12" tablet. Why when I perform the capture does NOTHING SHOW UP! I have also tried to point it at a compact fluorescent, but same thing happens. What am I doing wrong?
Thank you,
Jason

recent calibrations not available to capture interface

  1. We should have a button that lets you choose a recent calibration, in the /capture interface. It should show an up-to-date list of recent calibrations, ajaxly in case the capture page hasn't been reloaded

  2. We should have the button turn red when a new calibration is detected, via 10-sec AJAX polling, and display an alert that a new calibration is available with a button to use it.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.