Giter Club home page Giter Club logo

pixiedust_node's Introduction

pixiedust_node

PixieDust extension that enable a Jupyter Notebook user to invoke Node.js commands.

schematic

How it works

The pixiedust_node Python module has access to Pixiedust's display API to render charts and maps. When pixiedust_node is imported into a notebook, a Node.js sub-process is setup and the notebook is configured so that cells beginning with '%%node' may contain JavaScript code: that code is piped to the Node.js sub-process automatically. The output of the Node.js process is parsed by pixiedust_node to handle the use of functions display/print/store/html/image. The pixiedust_node module also allows npm installs to be initiated from within the notebook. This achieved with further npm sub-processes whose output appears in the notebook.

Prerequisites

To use pixiedust_node you need to be running a Jupyter notebooks with the Pixedust extension installed. Notebooks can be run locally by installing Pixiedust and its prerequisites.

You also need Node.js/npm installed. See the Node.js downloads page to find an installer for your platform.

Installation

Inside your Jupyter notebook, install pixiedust_node with

!pip install pixiedust_node

Running

Once installed, a notebook can start up pixiedust_node with:

import pixiedust_node

Using %%node

Use the %%node prefix in a notebook cell to indicate that the content that follows is JavaScript.

%%node
print(new Date());

Installing npm modules

You can install any npm module to use in your Node.js code from your notebook. To install npm modules, in a Python cell:

npm.install('silverlining')

or install multiple libraries in one go:

npm.install( ('request', 'request-promise') )

and then "require" the modules in your Node.js code.

%%node
var silverlining = require('silverlining');
var request = require('request-promise');

You may also do :

  • npm.uninstall('packagename') - to remove an npm module (or npm.remove('packagename'))
  • npm.list() - to list the installed modules

Node.js helper functions

Node.js functions are available to interact with the Notebook

  • print(x) - print out the value of variable x
  • display(x) - use Pixiedust's display function to visualise an array of data
  • store(x,'y') - turn a JavaScript array x into a Pandas data frame and store in Python variable y
  • html(x) - render HTML string x in a notebook cell
  • image(x) - render image URL x in a notebook cell
  • help() - show help

print

%%node
// connect to Cloudant using Silverlining
var url = 'https://reader.cloudant.com/cities';
var cities = silverlining(url);

// fetch number of cities per country
cities.count('country').then(print);

display

%%node

// fetch cities called York
cities.query({name: 'York'}).then(display);

store

** This function is deprecated as Node.js global variables are copied to the Python environment automatically **

%%node

// fetch the data and store in Pandas dataframe called 'x'
cities.all({limit: 2500}).then(function(data) {
  store(data, 'x');
});

The dataframe 'x' is now available to use in a Python cell:

x['population'].sum()

html

%%node
var str = 'Sales are up <b>25%</b>';
html(str);

image

%%node
var url = 'http://myserver.com/path/to/image.jpg';
image(url);

help

%%node
help();

Node.js-Python bridge

Any global variables that you create in your %%node cells will be automatically copied to equivalent variables in Python. e.g if you create some variables in a Node.js cell:

%%node
var str = "hello world";
var n1 = 4.1515;
var n2 = 42;
var tf = true;
var obj = { name:"Frank", age: 42 };
var array_of_strings = ["hello", "world"];
var array_of_objects = [{a:1,b:2}, {a:3, b:4}];

Then these variables can be used in Python:

# Python cell
print str, n1, n2, tf
print obj
print array_of_strings
print array_of_objects

Strings, numbers, booleans and arrays of such are converted to their equivalent in Python. Objects are converted into Python dictionaries and arrays of objects are automatically converted into a Pandas DataFrames.

Note that only variables declared with var are moved to Python, not constants declared with const.

If you want to move data from an asynchronous Node.js callback, remember to write it to a global variable:

%%node
var googlehomepage = '';
request.get('http://www.google.com').then(function(data) {
  googlehomepage = data;
  print('Fetched Google homepage');
});

Similarly, Python variables of type str, int, float, bool, unicode, dict or list will be moved to Node.js when a cell is executed:

# Python cell
a = 'hello'
b = 2
b = 3
c= False
d = {}
d["x"] = 1
d["y"] = 2
e = 3.142

The variables can then be used in Node.js:

%%node
console.log(a,b,c,d,e);
// hello 3 false { y: 2, x: 1 } 3.142

Managing the Node.js process

If enter some invalid syntax into a %%node cell, such as code with more opening brackets than closing brackes, then the Node.js interpreter may not think you have finished typing and you receive no output.

You can cancel execution by running the following command in a Python cell:

node.cancel()

If you need to clear your Node.js variables and restart from the beginning then issue the following command in an Python cell:

node.clear()

Help

You can view the help in a Python cell:

node.help()

pixiedust_node's People

Contributors

bradnoble avatar elainethale avatar glynnbird avatar jsnowacki avatar ptitzler avatar shnizzedy avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pixiedust_node's Issues

Erratic output behaviour

Sometimes, output from Node cells and/or node functions is cut or not displayed at all.
Short descriptive video (trying to display the result of the help() function) available here.

TypeError: Converting circular structure to JSON at JSON.stringify

I'm attempting to use the node-postgres library from pixiedust_node. The following simple setup fails:

import pixiedust_node
npm.install(('node-fetch', 'pg')) 
var { Pool } = require('pg');
var pool = new Pool({
  user: 'congress23',
  host: 'localhost',
  database: 'vol_congress23',
  password: '',
  port: 5431,
});
pool.query('SELECT NOW()', (err, res) => {
    console.log(err,res);
});

Oddly, the error occurs even when trying to console.log("it works") instead of the results, so I'm not even sure where/what is causing the circular reference error. The same code executes just fine when run directly from node. Here's the stacktrace:

TypeError: Converting circular structure to JSON
at JSON.stringify (<anonymous>)
at globalVariableChecker (/anaconda3/lib/python3.6/site-packages/pixiedust_node/pixiedustNodeRepl.js:26:22)
at REPLServer.writer (/anaconda3/lib/python3.6/site-packages/pixiedust_node/pixiedustNodeRepl.js:67:5)
at finish (repl.js:512:38)
at REPLServer.defaultEval (repl.js:279:5)
at bound (domain.js:301:14)
at REPLServer.runBound [as eval] (domain.js:314:12)
at REPLServer.onLine (repl.js:468:10)
at emitOne (events.js:116:13)
at REPLServer.emit (events.js:211:7)
er
/anaconda3/lib/python3.6/site-packages/pixiedust_node/pixiedustNodeRepl.js:26
const j = JSON.stringify(r.context[v]);
^
TypeError: Converting circular structure to JSON
at JSON.stringify (<anonymous>)
at Timeout.globalVariableChecker [as _onTimeout] (/anaconda3/lib/python3.6/site-packages/pixiedust_node/pixiedustNodeRepl.js:26:22)
at ontimeout (timers.js:482:11)
at Timer.unrefdHandle (timers.js:595:5)

weird dotted output

a node cell that does only function declaration
is outputting this weird dotted output

Screen Shot 2020-05-05 at 17 32 57

thanks

Use Nodejs vars in Python

Today, I can use store to save Nodejs vars for use in Python cells, but I can't do the opposite. That is, I can't access Python vars in %%node cells.

x.sum('field').then(console.log) triggers 'int' object has no attribute '__getitem__' error

PD 1.1.7 pd_node 0.2.3

Didn't try other aggregation operators.

%%node
var cqs = require('cloudant-quickstart');
const cities = cqs('https://56953ed8-3fba-4f7e-824e-5498c8e1d18e-bluemix.cloudant.com/cities');
cities.get('2636749').then(print).catch(console.error);
// works
cities.get('2636749').then(console.log).catch(console.error);
// works
cities.sum('population').then(print).catch(print);
2694222973
// fails
cities.sum('population').then(console.log).catch(print);

2694222973
'int' object has no attribute '__getitem__'

Workaround: use print instead of console.log or console.error

Source notebook: https://github.com/ibm-watson-data-lab/nodebook-code-pattern/blob/master/notebooks/nodebook_1.ipynb

Working Directory should match Jupyter Python environment

It seems like the working directory for the node process should match the python working directory in jupyter. I think, related to #38, this would also default the npm location to this same working directory which makes sense and mirrors the behavior of the %%script node jupyter cell magic.

I'm happy to put together a PR here if needed. My expertise is definitely more on the node side and less on the jupyter and python side so feel free to correct any misunderstandings I may have here.

Problem with mysql module

Hi!
I'm experiencing trouble trying to use node-mysql within a pixiedust-node project.

The cell setup is the following:

import pixiedust_node
npm.install('mysql')

This works as expected showing the logos of both pixiedust and pixiedust node and then the stats of the module install.
The problem comes with the nodejs part. The code works perfectly in a nodejs project, but not within pixiedust-node:

%%node
var mysql = require('mysql');
var dburl = 'database-url.com';
var con = mysql.createConnection({
  host: dburl,
  user: 'dbuser',
  password: 'dbpassword',
  database: 'dbdatabase'
});
con.connect(function(err) {
    if (err) throw err;
});
var query ='SELECT user_created_date from user LIMIT 100';
con.query(query, function (err, result, fields) {
            if (err) throw err;
            console.log(result);
});
con.end();

I write it down here as a single block, but I've tried it in different cell layouts to find more precisely where the error is located. And it is located in the mysql.createConnection() call:

... ... ... ... ... TypeError: Converting circular structure to JSON
at JSON.stringify (<anonymous>)
at globalVariableChecker (/home/javier/anaconda3/lib/python3.7/site-packages/pixiedust_node/pixiedustNodeRepl.js:26:22)
at REPLServer.writer (/home/javier/anaconda3/lib/python3.7/site-packages/pixiedust_node/pixiedustNodeRepl.js:67:5)
at finish (repl.js:683:38)
at finishExecution (repl.js:310:7)
at REPLServer.defaultEval (repl.js:396:7)
at bound (domain.js:395:14)
at REPLServer.runBound [as eval] (domain.js:408:12)
at REPLServer.onLine (repl.js:639:10)
at REPLServer.emit (events.js:182:13)
/home/javier/anaconda3/lib/python3.7/site-packages/pixiedust_node/pixiedustNodeRepl.js:26
const j = JSON.stringify(r.context[v]);
^
TypeError: Converting circular structure to JSON
at JSON.stringify (<anonymous>)
at Timeout.globalVariableChecker [as _onTimeout] (/home/javier/anaconda3/lib/python3.7/site-packages/pixiedust_node/pixiedustNodeRepl.js:26:22)
at ontimeout (timers.js:436:11)
at tryOnTimeout (timers.js:300:5)
at unrefdHandle (timers.js:520:7)
at Timer.processTimers (timers.js:222:12)

Running npm package in cell like one would do on command line

Is it possible to run a npm package within a cell after it's been installed like you would from the command line? I have installed shp2json with npm.install('shp2json') and would like to then run it with !shp2json /path/to/shapefile > output.json but receive an error saying /bin/sh: 1: shp2json: not found

Python 3.x: Typerror following the medium post examples

!pip install pixiedust
!pip install pixiedust_node
import pixiedust_node
%%node
var date = new Date();
print(date);
TypeError                                 Traceback (most recent call last)
<ipython-input-4-26ef6f3fb6e5> in <module>()
----> 1 get_ipython().run_cell_magic('node', '', 'var date = new Date();\nprint(date);')

/opt/conda/lib/python3.6/site-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
   2129             magic_arg_s = self.var_expand(line, stack_depth)
   2130             with self.builtin_trap:
-> 2131                 result = fn(magic_arg_s, cell)
   2132             return result
   2133 

<decorator-gen-126> in node(self, line, cell)

/opt/conda/lib/python3.6/site-packages/IPython/core/magic.py in <lambda>(f, *a, **k)
    185     # but it's overkill for just that one bit of state.
    186     def magic_deco(arg):
--> 187         call = lambda f, *a, **k: f(*a, **k)
    188 
    189         if callable(arg):

/opt/conda/lib/python3.6/site-packages/pixiedust_node/__init__.py in node(self, line, cell)
     48     def node(self, line, cell):
     49         # write the cell contents to the Node.js process
---> 50         self.n.write(cell)
     51 
     52 try:

/opt/conda/lib/python3.6/site-packages/pixiedust_node/node.py in write(self, s)
     53 
     54     def write(self, s):
---> 55         self.ps.stdin.write(s)
     56         self.ps.stdin.write("\r\n")
     57 

TypeError: a bytes-like object is required, not 'str'

Any ideas?

Exception thrown when calling Jupyter function

Hi,
I am trying to get the cell index in javascript and then using it as a python variable.
I tried execute this line to capture the cell index:

%%node
var cell_index=Jupyter.notebook.get_selected_index()

But I got this message:

Thrown:

This line is working in jupyter (but the problem is that it's javacript variable and not python):

%%javascript
var cell_index=Jupyter.notebook.get_selected_index()

Do you have any solution for this problem?

Will not display any graphs

This is the error I got:

def join_path(self, template, parent):
\n in template()
\nTemplateAssertionError: no filter named 'tojson'\n

Wrong working directory used when running node

The node subprocess is launched using the wrong cwd (see https://github.com/ibm-watson-data-lab/pixiedust_node/blob/master/pixiedust_node/node.py#L27)
it should match the npm command which is using the current working directory.

I think we should have both node and npm using the central working directory e.g. PIXIEDUST_HOME/node.

Note: you can get the PIXIEDUST_HOME directory using the Environment class
from pixiedust.utils.environment import Environment
Environment.pixiedustHome

/cc @glynnbird @bradnoble

Does this still work? Deprecation and Uncaught error.

When I try to import this project, I get a deprecation error, which (from what I can see) means this project doesn't work? I tried creating a variable in Python and accessing it in node/JS, and it just says Uncaught :/

image

Thanks!

Exception from popen in python 3.5

Trying to import pixiedust_node fails in Python 3.5, with the following error:

Pixiedust database opened successfully

Pixiedust version 1.1.17

Table USER_PREFERENCES created successfully
Table service_connections created successfully

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-1-4a0fd16c56d9> in <module>()
----> 1 import pixiedust_node

~/miniconda/envs/tmp/lib/python3.5/site-packages/pixiedust_node/__init__.py in <module>()
     60         # start up a Node.js sub-process running a REPL
     61         path = os.path.join(__path__[0], 'pixiedustNodeRepl.js')
---> 62         node = Node(path)
     63 
     64         # pass the node process to the Node magics

~/miniconda/envs/tmp/lib/python3.5/site-packages/pixiedust_node/node.py in __init__(self, path)
    208         # process that runs the Node.js code
    209         args = (self.node_path, path)
--> 210         self.ps = self.popen(args)
    211         #print ("Node process id", self.ps.pid)
    212 

TypeError: __init__() got an unexpected keyword argument 'encoding'

In the code, the encoding argument is passed to popen if the Python version is 3 or greater:

    if sys.version_info.major == 3:
        popen_kwargs['encoding'] = 'utf-8'

But the encoding argument was only added to popen in python 3.6 (see: https://docs.python.org/3.6/library/subprocess.html). I'm not sure if just changing the if to sys.version_info.major == 3 and sys.version_info.minor > 6 will fix the problem.

add requests to install dependencies

When importing pixiedust_node in a notebook, the following error happens:

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-9-327dd5855a1c> in <module>()
      8 from scipy.interpolate import interp1d
      9 from skimage.draw import bezier_curve
---> 10 import pixiedust_node

~/dev/readonce/venv/lib/python3.6/site-packages/pixiedust_node/__init__.py in <module>()
     19 from IPython.core.error import TryNext
     20 import warnings
---> 21 from .node import Node, Npm
     22 import os
     23 from pixiedust.utils.shellAccess import ShellAccess

~/dev/readonce/venv/lib/python3.6/site-packages/pixiedust_node/node.py in <module>()
     11 import IPython
     12 import pandas
---> 13 from pixiedust.display import display
     14 from pixiedust.utils.environment import Environment
     15 from pixiedust.utils.shellAccess import ShellAccess

~/dev/readonce/venv/lib/python3.6/site-packages/pixiedust/__init__.py in <module>()
     29 
     30     #shortcut to logging
---> 31     import pixiedust.utils.pdLogging as pdLogging
     32     logger = pdLogging.getPixiedustLogger()
     33     getLogger = pdLogging.getLogger

~/dev/readonce/venv/lib/python3.6/site-packages/pixiedust/utils/__init__.py in <module>()
     16 
     17 import os
---> 18 from . import storage
     19 import pkg_resources
     20 import binascii

~/dev/readonce/venv/lib/python3.6/site-packages/pixiedust/utils/storage.py in <module>()
     26 from pkg_resources import get_distribution
     27 from re import search
---> 28 from requests import post
     29 from os import environ as env
     30 from pixiedust.utils.printEx import *

So the requests package should probably be explicitely listed in install_requires.

Installing Node if it's not there

If pixiedust_node is installed, could it conceivably install Node.js (node/npm) on the target system? If so how?

@DTAIEB said

"yeah, I think we could do a command line download/install of node under PIXIEDUST_HOME directory when we detect it’s not there"

Currently, we assume node is in the PATH so we run

self.ps = subprocess.Popen( ('node', path), stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

This could be changed to:

home = get_ipython().home_dir
self.ps = subprocess.Popen( ('node', path), stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, cwd = home)

This would ensure node's home directory is the same as Pixiedust's.

TypeError: x.get is not a function

We saw the following error intermittently:

screen shot 2018-02-09 at 11 13 32 am

Sequence:

%%node
// connect to Cloudant using cloudant-quickstart
var cqs = require('cloudant-quickstart');
var cities = cqs('https://...-bluemix.cloudant.com/cities');

[works]

%%node
cities.get('2636749').then(print).catch(print);

[fails sometimes; see screen cap]

%%node
cities.get(['4562407','2636749','3530597']).then(print).catch(print);

Once the error was encountered, cities is rendered unusable (no method calls work anymore)

Error running npm.list

When I run npm.list from a %%node cell in a notebook I receive the following error:

CalledProcessError: Command '['npm', 'list', '-s']' returned non-zero exit status 1

Here is the out from npm list -s from the command line:

/Users/markwatson
├── UNMET PEER DEPENDENCY @angular/[email protected]
├── UNMET PEER DEPENDENCY @angular/[email protected]
├── UNMET PEER DEPENDENCY @angular/[email protected]
├── UNMET PEER DEPENDENCY @angular/[email protected]
├── UNMET PEER DEPENDENCY @angular/[email protected]
├── UNMET PEER DEPENDENCY @angular/[email protected]
├── UNMET PEER DEPENDENCY @angular/[email protected]
├── UNMET PEER DEPENDENCY @angular/[email protected]
├── [email protected]
├─┬ [email protected]
│ ├── [email protected]
│ ├─┬ [email protected]
...

print statement that is not compatible with Python 3 in last release

I see from your Python3 milestone that you are working toward Python 3 compatibility and have already merged in many fixes. However, your most recent release (v0.2.0) contains a fairly new line of code that does not work in Python 3. In my notebook I see this right away:

import pixiedust_node
Traceback (most recent call last):

  File "c:\python36\lib\site-packages\IPython\core\interactiveshell.py", line 2862, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-1-4377929b0946>", line 1, in <module>
    import pixiedust_node

  File "c:\python36\lib\site-packages\pixiedust_node\__init__.py", line 20, in <module>
    from .node import Node, Npm

  File "c:\python36\lib\site-packages\pixiedust_node\node.py", line 91
    print '!!! Warning: store is now deprecated - Node.js global variables are automatically propagated to Python !!!'
                                                                                                                     ^
SyntaxError: Missing parentheses in call to 'print'

Cannot run node commands - a byte like object is required

Could not start a node command. Got the message that byte object is required. I installed jupyter using anaconda3. Here is how I fixed it

  • Added encode when writing to stdin
  • When reading from stdout, convert from byte to string
  • Also needed to flush after writing to stdin

Let me know if you need it as a pull request

NPM Modules Installed in Local Directory

npm.install() seems to install modules in the user directory. Is there some way in pixiedust_node to refer to modules installed in the same directory as the notebook file or control the installation location of npm.install()?

How do I convert an array of values so that they are correctly understood by `display()`?

I'm new to Jupyter and pixiedust and I'm having a hardtime on my first experiments with it.

I'm looping through an array to recover values by time stamp and put them in a new array to be displayed as a line graph.

        var data = [];
        body.donors.forEach(
            function(donor) {
                var epochDate = new Date(donor.data_envio).setHours(0,0,0,0);
                var obj = {
                    date: new Date(epochDate),
                    valor: Number(donor.valor)
                };
                data.push(obj);
            }
        );
        display(data);

When I do this, the generated chart says "x must be a label or position".

Opening "Options", the date field is shown as "string".

I've tried formating the date field as ISO 8601 but it is still understood as string.

I've found no information/documentation on how to "cast" my data so that pixiedust correctly understands it.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.