• Singular data points: cards with a plain number, like kibana
  
• Temporal data points: graph

we should probably just use travis.

http://www.chartjs.org seems to fit for this, yet to be determined

… and prevent artificial shortening where we don't need it.

We should use tables in the experiment and model listings to provide a better overview.

Possible columns:

• experiment
  • model (name, description, hyperparameter)
    • metric (name, value if singular data point *or * "max value, min value, avg value" if series)

Maybe I can incorporate this somehow … not sure yet. Would be useful to have an overview of the chosen hyperparams after the search has finished or something.

no test checks the hyperparam attribute …

If we have a number of models which share some metrics, we should highlight the best and worst performing models. The maximum value of a metric should be marked with some color, and the minimum value as well. The colors however should not be red and green, as errors ought to be minimized, and scores maximized, and we have no way to tell (other than trying to be smart and parsing some scores and errors, which isn't really a good approach). Maybe blue and orange.

• add legend above the models (which are only displayed if we have more than one model) which a re hyperlinked to the respective value (max and min value via #min-val or #max-val)
• detect max and min of a value
  • mark row with color and underline the value
  • create an anchor to allow jumping to the row from the legend
• introduce help button that displays a short explanation of the difference between error and score.

• CRUD /experiment: the experiment
  • POST /experiment: create new experiment
  • GET /experiment: get all experiments
  • GET /experiment/:id: get experiment data (all of it)
  • PUT /experiment/:id: change experiment data (only the meta data about the experiment)
  • DELETE /experiment/:id: remove experiment from database
• CRUD /experiment/:id/model: the different models that ought to be compared for the experiment
  • POST /experiment/:id/model: create a new model
  • GET /experiment/:id/model: get all models
  • GET /experiment/:id/model/:id: get model data
  • PUT /experiment/:id/model/:id: change model data (only meta data about the model)
  • DELETE /experiment/:id/model/:id: remove model
• CRUD /experiment/:id/model/:id/measurements
  • POST /experiment/:id/model/:id/measurements: create a new measurement point
  • GET /experiment/:id/model/:id/measurements: get all measurement points (sorted by name)
  • GET /experiment/:id/model/:id/measurements/forname/:name: get all metrics with a name (for example f1_micro or totalLoss)

Atm we even return __v which shouldn't be returned.

I guess tags would be cool if one runs a lot of projects/experiments.

As we have to generate canvas elements per metric with more than one data point the elements are created dynamically. Chartjs can't really find these elements, hence the chart will only be displayed after another point of measurement is added later on.

GET /api/v1 and GET /api wont work, I think. They don't call it … done(err).

add these fields to models/experiment

• it was proposed to me to include tensorboard somehow
  • either try to read the files created by SummaryWriter (not sure if and how this works)
  • or as plain uri
• jupyter notebook integration would be splendid
  • either as html directly via nbconvert
  • or as plain uri

thus, hereby I propose introducing a hypertext resource attribute of models.

new models are added to the table, though measurements and other updates are not propagated properly. after reloading the information is displayed correctly.

y/Users/x/Documents/Development/Pythia/node_modules/mongoose/lib/types/embedded.js:178
    throw new Error(msg);
    ^

Error: Unable to invalidate a subdocument that has not been added to an array.
    at EmbeddedDocument.invalidate (/Users/x/Documents/Development/Pythia/node_modules/mongoose/lib/types/embedded.js:178:11)
    at EmbeddedDocument.Document.set (/Users/x/Documents/Development/Pythia/node_modules/mongoose/lib/document.js:687:10)
    at EmbeddedDocument.Document.set (/Users/x/Documents/Development/Pythia/node_modules/mongoose/lib/document.js:540:18)
    at EmbeddedDocument.Document (/Users/x/Documents/Development/Pythia/node_modules/mongoose/lib/document.js:66:10)
    at new EmbeddedDocument (/Users/x/Documents/Development/Pythia/node_modules/mongoose/lib/types/embedded.js:30:12)
    at EmbeddedDocument (/Users/x/Documents/Development/Pythia/node_modules/mongoose/lib/schema/documentarray.js:26:17)
    at Array.create (/Users/x/Documents/Development/Pythia/node_modules/mongoose/lib/types/documentarray.js:204:12)
    at Query.<anonymous> (/Users/x/Documents/Development/Pythia/api/v1/experiment/index.js:475:54)
    at /Users/x/Documents/Development/Pythia/node_modules/mongoose/node_modules/kareem/index.js:177:19
    at /Users/x/Documents/Development/Pythia/node_modules/mongoose/node_modules/kareem/index.js:109:16
    at _combinedTickCallback (internal/process/next_tick.js:73:7)
    at process._tickCallback (internal/process/next_tick.js:104:9)
2017-03-24T08:52:03.220+0100 I -        [conn1] end connection 127.0.0.1:56966 (1 connection now open)

caused by:

m1.add_measurement('avg_smape', m)

with m:

{ name: 'avg_smape',
  value:
   [ 'Figure(720x432)',
     'Figure(720x432)',
     'Figure(720x432)',
     'Figure(720x432)',
     'Figure(720x432)',
     'Figure(720x432)',
     'Figure(720x432)',
     'Figure(720x432)',
     'Figure(720x432)',
     'Figure(720x432)',
     'Figure(720x432)',
     'Figure(720x432)' ],
  epoch: '0',
  step: '0' }

mongodb disappoints throughout when it comes to security. we should look for alternatives. I'm open for suggestions.

• show only measurements with a given metric/loss and sort them

these fields are only updated after reloading the page

• PUT /experiment/:id
• PUT /experiment/:id/model/:id

Every route in the api should have its predefined actions, where all of them should be implemented. Yet to be determined.

only after the site has been reloaded