oracle / graalpython Goto Github PK

A Python 3 implementation built on GraalVM

License: Other

Python 47.97% R 0.01% C 24.26% C++ 0.19% Java 27.14% Shell 0.06% Batchfile 0.01% HTML 0.24% CSS 0.01% Roff 0.03% DTrace 0.01% PLSQL 0.03% PowerShell 0.01% Jsonnet 0.01% VBScript 0.01% XSLT 0.01% Starlark 0.01% CMake 0.04% Jasmin 0.01% Objective-J 0.01%

graalpython's Introduction

GraalPy, the GraalVM Implementation of Python

GraalPy is a high-performance implementation of the Python language for the JVM built on GraalVM. GraalPy has first-class support for embedding in Java and can turn Python applications into fast, standalone binaries.

Why GraalPy?

Low-overhead integration with Java and other languages

Use Python in Java applications on GraalVM JDK, Oracle JDK, or OpenJDK
Use JVM tools like Maven, JFR, or GraalVM Native Image
Manage Python libraries' system access thanks to GraalPy's Java-based emulation of Python OS APIs

Compatible with the Python ecosystem

Install packages like NumPy, PyTorch, or Tensorflow; run Hugging Face models like Stable Diffusion or GPT
See if the packages you need work with our Python Compatibility Checker
Use almost any standard Python feature, the CPython tests run on every commit and pass ~85% ^{We run the tests of the most depended on PyPI packages every day.
For 96% of those packages a recent version can be installed on GraalPy and GraalPy passes about 50% of all tests of all packages combined.
We assume that CPython not passing 100% of all tests is due to problems in our infrastructure that may also affect GraalPy.
Packages where CPython fails all tests are marked as "not tested" for both CPython and GraalPy.}

Runs Python code faster

Pure Python code is often faster than on CPython after JIT compilation
C extension performance is near CPython, but varies depending on the specific interactions of native and Python code
GraalPy is ~4x faster than CPython on the official Python Performance Benchmark Suite ^{Benchmarks run via pip install pyperformance && pyperformance run on each of CPython and GraalPy.
Harness and benchmarks were adapted by hand for Jython due to missing Python 3 support.
Each interpreter was installed via pyenv.
Geomean speedup was calculated against CPython on the intersection of benchmarks that run on all interpreters.}

Getting Started

Embedding GraalPy in Java

GraalPy is available on Maven Central for inclusion in Java projects. Refer to our embedding documentation for more details.

Maven

<dependency>
    <groupId>org.graalvm.polyglot</groupId>
    <artifactId>polyglot</artifactId>
    <version>24.0.0</version>
</dependency>
<dependency>
    <groupId>org.graalvm.polyglot</groupId>
    <artifactId>python</artifactId>
    <version>24.0.0</version>
    <type>pom</type>
</dependency>

Gradle

implementation("org.graalvm.polyglot:polyglot:24.0.0")
implementation("org.graalvm.polyglot:python:24.0.0")

Replacing CPython with GraalPy

GraalPy should in many cases work as a drop-in replacement for CPython. You can use pip to install packages as usual. Packages with C code usually do not provide binaries for GraalPy, so they will be automatically compiled during installation. This means that build tools have to be available and installation will take longer. We provide Github actions to help you build binary packages with the correct dependencies. Thanks to our integration with GraalVM Native Image, we can deploy Python applications as standalone binary, all dependencies included.

Linux

The easiest way to install GraalPy on Linux is to use Pyenv (the Python version manager). To install version 24.0.0 using Pyenv, run the following commands:
```
pyenv install graalpy-24.0.0
```
```
pyenv shell graalpy-24.0.0
```
Alternatively, you can download a compressed GraalPy installation file from GitHub releases.
1. Find the download that matches the pattern graalpy-XX.Y.Z-linux-amd64.tar.gz or graalpy-XX.Y.Z-linux-aarch64.tar.gz (depending on your platform) and download.
2. Uncompress the file and update your PATH environment variable to include the graalpy-XX.Y.Z-linux-amd64/bin (or graalpy-XX.Y.Z-linux-aarch64/bin) directory.
macOS

The easiest way to install GraalPy on macOS is to use Pyenv (the Python version manager). To install version 24.0.0 using Pyenv, run the following commands:
```
pyenv install graalpy-24.0.0
```
```
pyenv shell graalpy-24.0.0
```
Alternatively, you can download a compressed GraalPy installation file from GitHub releases.
1. Find the download that matches the pattern graalpy-XX.Y.Z-macos-amd64.tar.gz or graalpy-XX.Y.Z-macos-aarch64.tar.gz (depending on your platform) and download.
2. Remove the quarantine attribute.
```
sudo xattr -r -d com.apple.quarantine /path/to/graalpy
```
  For example:
```
sudo xattr -r -d com.apple.quarantine ~/.pyenv/versions/graalpy-24.0.0
```
3. Uncompress the file and update your PATH environment variable to include to the graalpy-XX.Y.Z-macos-amd64/bin (or graalpy-XX.Y.Z-macos-aarch64/bin) directory.
Windows

The Windows support of GraalPy is still experimental, so not all features and packages may be available.
1. Find and download a compressed GraalPy installation file from GitHub releases that matches the pattern graalpy-XX.Y.Z-windows-amd64.tar.gz.
2. Uncompress the file and update your PATH variable to include to the graalpy-XX.Y.Z-windows-amd64/bin directory.

Using GraalPy in Github Actions

The setup-python action supports GraalPy:

    - name: Setup GraalPy
      uses: actions/setup-python@v5
      with:
        python-version: graalpy # or graalpy24.0 to pin a version

Migrating Jython Scripts to GraalPy

Most existing Jython code that uses Java integration will be based on a stable Jython release—however, these are only available in Python 2.x versions. To migrate your code from Python 2 to Python 3, follow the official guide from the Python community. GraalPy provides a special mode to facilitate migration. To run Jython scripts, you need to use a GraalPy distribution running on the JVM so you can access Java classes from Python scripts.

Linux
1. Find and download a compressed GraalPy installation file from GitHub releases that matches the pattern graalpy-jvm-XX.Y.Z-linux-amd64.tar.gz or graalpy-jvm-XX.Y.Z-linux-aarch64.tar.gz (depending on your platform) and download.
2. Uncompress the file and update your PATH environment variable to include the graalpy-jvm-XX.Y.Z-linux-amd64/bin (or graalpy-jvm-XX.Y.Z-linux-aarch64/bin) directory.
3. Run your scripts with graalpy --python.EmulateJython.
macOS
1. Find and download a compressed GraalPy installation file from GitHub releases that matches the pattern graalpy-jvm-XX.Y.Z-macos-amd64.tar.gz or graalpy-jvm-XX.Y.Z-macos-aarch64.tar.gz (depending on your platform) and download.
2. Remove the quarantine attribute.
```
sudo xattr -r -d com.apple.quarantine /path/to/graalpy
```
  For example:
```
sudo xattr -r -d com.apple.quarantine ~/.pyenv/versions/graalpy-24.0.0
```
3. Uncompress the file and update your PATH environment variable to include to the graalpy-jvm-XX.Y.Z-macos-amd64/bin (or graalpy-jvm-XX.Y.Z-macos-aarch64/bin) directory.
4. Run your scripts with graalpy --python.EmulateJython.
Windows
1. Find and download a compressed GraalPy installation file from GitHub releases that matches the pattern graalpy-jvm-XX.Y.Z-windows-amd64.tar.gz.
2. Uncompress the file and update your PATH variable to include to the graalpy-jvm-XX.Y.Z-windows-amd64/bin directory.
3. Run your scripts with graalpy --python.EmulateJython.

Examples

^{Java AWT app with Python graph library using JBang | Standalone binary of a Python game by Joey Navarro with all dependencies included.}

Documentation

GraalPy Quick Reference Sheet should help you get started. More GraalPy-specific user documentation is available in docs/user. General documentation about polyglot programming and language embedding is available on the GraalVM website.

Community

The best way to get in touch with us is to join the #graalpy channel on GraalVM Slack or tweet us.

Contributing

This project welcomes contributions from the community. Before submitting a pull request, please review our contribution guide.

If you're thinking about contributing something to this repository, you will need to sign the Oracle Contributor Agreement for us to able to merge your work. Also take a look at the code of conduct for contributors.

Security

Consult the security guide for our responsible security vulnerability disclosure process.

License

This GraalVM implementation of Python is Copyright (c) 2017, 2024 Oracle and/or its affiliates and is made available to you under the terms the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/. This implementation is in part derived from and contains additional code from 3rd parties, the copyrights and licensing of which is detailed in the LICENSE and THIRD_PARTY_LICENSE files.

graalpython's People

Contributors

Stargazers

Watchers

Forkers

timfel smarr fniephaus hbcbh1999 daniellansun numberpi kontrafiktion renesugar isaiah rowhit rsvoboda vishalbelsare qunaibit sudosalim cmueh muellren weaver-viii hpi-swa-lab bryantchhun b3838438org corona10 gamarino davidmrdavid todun dereisele mohamedrez manishchainthat pombredanne johenning jkreindl zolstein transposit sighingnow micki0271 codinghase dev-bobbie lucaswerkmeister zhutony jyukutyo xmonader tminussi cclauss lvyitian strogo manrick653 msimacek massimiliano76 rahulmutt acproject mnislam01 jeffcarpenter fangerer kariya-mitsuru sudong-beijing hartsock kingking888 tanmibts charygao octavelarose errikos tu-berlin-dima cjluzzl stefreschke alexdtorres migroskub classicvalues itasks inno157 naijun0403 edric-shen testlabor cyberflamego isabella232 doytsujin yuhala trellixvulnteam 2767mr bencleric gg-big-org apetcho empeje the-alchemist h-cusworth jiridanek reynoldsm88 bhoran jaroslavtulach miguelzamora13 msalman-abid phkeese k2662 lokmeinmatz goodstudio wangxi761 yalchinalv omidsa1 mohammedaziz02 mjschnie

graalpython's Issues

Recover from Exception, not die with segmentation fault

➜  ~ graalpython
Please note: This Python implementation is in the very early stages, and can run little more than basic benchmarks at this point.
>>> 1 / 0
[1]    3747 segmentation fault  graalpython
➜  ~ python
Python 3.6.5 (default, Mar 30 2018, 06:42:10) 
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 1 / 0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ZeroDivisionError: division by zero
>>>

Build and Usage Instructions

I was experimenting with the graalpython following the directions from https://www.graalvm.org/docs/reference-manual/graal-updater/

I noticed that none of the installed jars seemed to contain SignalModuleBuiltins, so I figured I'd try to build this project myself but I can't seem to find build instructions.

Thanks!

Graal.Python pollutes polyglot bindings

Similar to TruffleRuby (see oracle/truffleruby#1561), Graal.Python pollutes the polyglot bindings object with various entries (all prefixed with python_) to communicate with Sulong. This happens as soon as Graal.Python is initialized and should be avoided (imagine someone happens to override python_error for some reason).

gu install python not working.

I am on OSX. Not sure if that is the problem. Should this be pre installed?

$ /Library/Java/JavaVirtualMachines/graalvm-1.0.0-rc1/Contents/Home/jre/lib/installer/bin/gu install -cv org.graalvm.python
Downloading: Component catalog
Receiving 4 kB: Done.
Processing component archive: Component org.graalvm.python
Preparing to install Component org.graalvm.python, contains org.graalvm.python, version 1.0.0-rc1 (org.graalvm.python)
Checking requirements of component Graal.python (org.graalvm.python), version 1.0.0-rc1
Requires Graal Version = 1.0.0-rc1, GraalVM provides: 1.0.0-rc1
Requires Architecture = amd64, GraalVM provides: amd64
Requires Operating System = darwin, GraalVM provides: darwin
Downloading: Component org.graalvm.python
Receiving 0 kB: Error: Package for component org.graalvm.python is not accessible at https://github.com/graalvm/graalpython/releases/download/vm-1.0.0-rc1/python-installable-macos-amd64.jar

There are no releases available in the releases directory. Both R and ruby installed fine. Should I use another link for the install?

ModuleNotFoundError: No module named 'fcntl'

>>> import fcntl
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "_bootstrap.py", line 1074, in __import__
  File "_bootstrap.py", line 994, in _gcd_import
  File "_bootstrap.py", line 971, in _find_and_load
  File "_bootstrap.py", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'fcntl'

time.localtime missing, best steps to contribute?

After porting ReBench to Python 3, and having it running on CPython and PyPy, I thought I take it for a spin on GraalPython:

graalpython --jvm ~/Projects/ReBench/main.py --help
Please note: This Python implementation is in the very early stages, and can run little more than basic benchmarks at this point.
Traceback (most recent call last):
  File "_bootstrap.py", line 957, in _find_and_load_unlocked
  File "_bootstrap.py", line 662, in _load_unlocked
  File "_bootstrap_external.py", line 687, in _LoaderBasics.exec_module
  File "_bootstrap.py", line 206, in _call_with_frames_removed
  File "functions", line 106, in exec
  File "Projects/ReBench/rebench/rebench.py", line 30, in <module>
    b'import logging'
  File "_bootstrap.py", line 1066, in __import__
  File "_bootstrap.py", line 985, in _gcd_import
  File "_bootstrap.py", line 968, in _find_and_load
  File "_bootstrap.py", line 957, in _find_and_load_unlocked
  File "_bootstrap.py", line 662, in _load_unlocked
  File "_bootstrap_external.py", line 687, in _LoaderBasics.exec_module
  File "_bootstrap.py", line 206, in _call_with_frames_removed
  File "functions", line 106, in exec
  File "graalvm-1.0.0-rc1/Contents/Home/jre/languages/python/lib-python/3/logging/__init__.py", line 426, in <module>
    b'class Formatter(object):'
  File "classes", line 115, in __build_class__
  File "classes", line 47, in new_class
  File "graalvm-1.0.0-rc1/Contents/Home/jre/languages/python/lib-python/3/logging/__init__.py", line 468, in Formatter
    b'converter = time.localtime'
AttributeError: 'module' object has no attribute localtime

Assuming that this localtime property might be useful eventually, what would be the preferred approach for me to take to possibly contribute an implementation?

Thanks

Setup CI publishing the artifacts built after each commit.

GitLab CI or AppVeyor can be used for that.

Graalpython stucked after help()

To the procedure:
after which graalpython is started and help() is entered hangs GraalVM.

What did you want to do?.:
in help() you can query modules that can be used in Python,
since graalpython modules are unknown to me I wanted to find out about them.

class is not available

class A:
   def b(self):
      return __class__()

pathlib.PosixPath is completely broken

from pathlib import Path
p = Path("/usr")
str(p) # __str__ returned non-string (type NoneType)
p.absolute() # None
p / "local" # None

NumPy float32 error

Hi,

Thank you for supporting NumPy and all the hard work.

I tried to create a float32 array but I got an error.

np.array([1,2,3,4], dtype=np.float32)

error:

java.lang.IllegalStateException: Missing LLVM builtin: @llvm.x86.sse.cmp.ss

float64 is fine though.

Is there a prerequisite that I missed?

pandas.read_csv throws error

I tried to read data.csv

"a" "b" "c"
1 2 3
4 5 6
7 8 9

but

>>> import pandas as pd
>>> pd.read_csv('data.csv', sep=' ')
java.lang.ClassCastException: com.oracle.graal.python.builtins.objects.cext.PythonNativeObject cannot be cast to java.lang.Boolean

Module object (`itertools`) has no attribute `cycle`

Binary build?

Hi, is it possible to provide release binaries for Oracle/CentOS/Debian using native image so we don't have to install JVM?

Cannot create weak reference with arbitrary (non-function) callbacks

  File "~/graalvm-ce-1.0.0-rc11/jre/languages/python/lib-python/3/ensurepip/__init__.py", line 91, in _bootstrap
    with tempfile.TemporaryDirectory() as tmpdir:
  File "~/graalvm-ce-1.0.0-rc11/jre/languages/python/lib-python/3/tempfile.py", line 791, in __init__
    self._finalizer = _weakref.finalize(
  File "~/graalvm-ce-1.0.0-rc11/jre/languages/python/lib-python/3/weakref.py", line 534, in __init__
    info.weakref = ref(obj, self)
TypeError: cannot create weak reference to 'TemporaryDirectory' object

Is it presently possible to install pip and use python modules?

root@fe64b215f82d:/graalvm/jre/languages/python# curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1604k  100 1604k    0     0   415k      0  0:00:03  0:00:03 --:--:--  415k
root@fe64b215f82d:/graalvm/jre/languages/python# bin/graalpython get-pip.py 
Please note: This Python implementation is in the very early stages, and can run little more than basic benchmarks at this point.
Traceback (most recent call last):
  File "get-pip.py", line 20651, in <module>
    main()
  File "get-pip.py", line 186, in main
    tmpdir = tempfile.mkdtemp()
  File "/graalvm/jre/languages/python/lib-python/3/tempfile.py", line 358, in mkdtemp
    prefix, suffix, dir, output_type = _sanitize_params(prefix, suffix, dir)
  File "/graalvm/jre/languages/python/lib-python/3/tempfile.py", line 130, in _sanitize_params
    dir = gettempdir()
  File "/graalvm/jre/languages/python/lib-python/3/tempfile.py", line 296, in gettempdir
    tempdir = _get_default_tempdir()
  File "/graalvm/jre/languages/python/lib-python/3/tempfile.py", line 205, in _get_default_tempdir
    name = next(namer)
  File "/graalvm/jre/languages/python/lib-python/3/tempfile.py", line 161, in _RandomNameSequence.__next__
    return ''.join(letters)
TypeError: sequence item 0: expected str instance, NotImplementedType found
root@fe64b215f82d:/graalvm/jre/languages/python#

rc6 won't execute script if it's given as `/dev/stdin` (and won't report error either)

rc5

$ bash -c "graalpython /dev/stdin <<<'print(2**42)'"
Please note: This Python implementation is in the very early stages, and can run little more than basic benchmarks at this point.
4398046511104

$ echo 'print(2**42)' | graalpython /dev/stdin
Please note: This Python implementation is in the very early stages, and can run little more than basic benchmarks at this point.
4398046511104

rc6

$ bash -c "graalpython /dev/stdin <<<'print(2**42)'"
Please note: This Python implementation is in the very early stages, and can run little more than basic benchmarks at this point.
4398046511104

$ echo 'print(2**42)' | graalpython /dev/stdin
Please note: This Python implementation is in the very early stages, and can run little more than basic benchmarks at this point.

i.e. the second call didn't print anything

Repro env

For reproducing, I use isolated docker environment.

FROM debian:stable

ARG GRAAL_VERSION=1.0.0-rc6

RUN apt-get update && \
    apt-get install -y curl && \
    apt-get clean && rm -r "/var/lib/apt/lists"/*

RUN curl -fsSL https://github.com/oracle/graal/releases/download/vm-${GRAAL_VERSION}/graalvm-ce-${GRAAL_VERSION}-linux-amd64.tar.gz -o graalvm.tar.gz
RUN tar xf graalvm.tar.gz
ENV PATH=/graalvm-ce-${GRAAL_VERSION}/bin:/bin:/usr/bin
RUN gu install -c org.graalvm.python

`graalpython -S` renders quit() undefined

$ graalpython -S
Python 3.7.0 (Sat Jan 12 12:16:56 GMT 2019)
[GraalVM CE, Java 1.8.0_191] on linux
Please note: This Python implementation is in the very early stages, and can run little more than basic benchmarks at this point.
>>> quit()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'quit' is not defined

Usage of unpacking operator causes syntax error

def a(b, c, *args, d=None):
    pass

Importing classes from jar is unobvious

Here is how to import classes from a jar:

from os import pathsep as PATH_sep
import java
ClassLoader = java.type("java.lang.ClassLoader")
scl = ClassLoader.getSystemClassLoader()
sys = java.type("java.lang.System")

origClassPath = sys.getProperty("sys.class.path")

dedupClassPath=OrderedDict()
for el in origClassPath.split(PATH_sep):
	dedupClassPath[el]=True

for f in libDir.glob("*.jar"):
	el=str(f.absolute())
	dedupClassPath[el]=True

classPathz=PATH_sep.join(dedupClassPath.keys())

sys.setProperty("sys.class.path", classPathz)
Main=scl.loadClass("a.package.in.a.jar.Main")

There are lot of issues here.

java.type returns only types built into Java
classpath cannot be set up right using java object
python import statement is not used (it is possible to write a loader using the stuff above).
This way of loading java classes is not documented in GraalPython docs. I had to spend some time googling Java classes to do that.
Only java classes are clearly loaded this way, not scala ones (I can still use them as Java classes and it works fine, but it requires some perversions and type info like default arguments is missing and is not populated into __annotations__).

Probably it should look like

from java.classes.a.package.in.a.jar import Main

This can even be implemented purely in python via a custom module loader wrapping the above code.

Some keywords to make this issue more searcheable:
How to import a class from a jar into graalvm graalpython python.

Eval crash on custom filesystem

I install a custom filesystem in my code that redirects access to a custom root directory. Running any Python code in a Graal context causes this to happen.

[python] WARNING: could not determine Graal.Python's standard library path. You need to pass --python.StdLibHome if you want to use the standard library.
[python] WARNING: could not determine Graal.Python's core path - you may need to pass --python.CoreHome.
org.graalvm.polyglot.PolyglotException: java.lang.ClassCastException: com.oracle.graal.python.builtins.objects.PNone cannot be cast to com.oracle.graal.python.builtins.objects.type.PythonClass
	at com.oracle.graal.python.builtins.Python3Core.findKnownExceptionTypes(Python3Core.java:770)
	at com.oracle.graal.python.builtins.Python3Core.initialize(Python3Core.java:343)
	at com.oracle.graal.python.PythonLanguage.initializeContext(PythonLanguage.java:233)
	at com.oracle.graal.python.PythonLanguage.initializeContext(PythonLanguage.java:72)
	at com.oracle.truffle.api.TruffleLanguage$Env.postInit(TruffleLanguage.java:1875)
	at com.oracle.truffle.api.TruffleLanguage$LanguageImpl.postInitEnv(TruffleLanguage.java:2088)
	at com.oracle.truffle.api.vm.PolyglotLanguageContext.ensureInitialized(PolyglotLanguageContext.java:315)
	at com.oracle.truffle.api.vm.PolyglotLanguageContext.parseCached(PolyglotLanguageContext.java:155)
	at com.oracle.truffle.api.vm.PolyglotContextImpl.eval(PolyglotContextImpl.java:745)
	at org.graalvm.polyglot.Context.eval(Context.java:311)
	at org.graalvm.polyglot.Context.eval(Context.java:336)
	at tf.veriny.vt1000.application.polyglot.PolyglotApplication.invoke(PolyglotApplication.kt:25)
	at tf.veriny.vt1000.kernel.KernelApplication.invoke(KernelApplication.kt:41)
	at tf.veriny.vt1000.kernel.Kernel.invoke(Kernel.kt:45)
	at tf.veriny.vt1000.VT1000$create$2.invoke(VT1000.kt:54)
	at tf.veriny.vt1000.VT1000$create$2.invoke(VT1000.kt:22)
	at kotlin.concurrent.ThreadsKt$thread$thread$1.run(Thread.kt:30)
Original Internal Error: 
java.lang.ClassCastException: com.oracle.graal.python.builtins.objects.PNone cannot be cast to com.oracle.graal.python.builtins.objects.type.PythonClass
	at com.oracle.graal.python.builtins.Python3Core.findKnownExceptionTypes(Python3Core.java:770)
	at com.oracle.graal.python.builtins.Python3Core.initialize(Python3Core.java:343)
	at com.oracle.graal.python.PythonLanguage.initializeContext(PythonLanguage.java:233)
	at com.oracle.graal.python.PythonLanguage.initializeContext(PythonLanguage.java:72)
	at com.oracle.truffle.api.TruffleLanguage$Env.postInit(TruffleLanguage.java:1875)
	at com.oracle.truffle.api.TruffleLanguage$LanguageImpl.postInitEnv(TruffleLanguage.java:2088)
	at com.oracle.truffle.api.vm.PolyglotLanguageContext.ensureInitialized(PolyglotLanguageContext.java:315)
	at com.oracle.truffle.api.vm.PolyglotLanguageContext.parseCached(PolyglotLanguageContext.java:155)
	at com.oracle.truffle.api.vm.PolyglotContextImpl.eval(PolyglotContextImpl.java:745)
	at org.graalvm.polyglot.Context.eval(Context.java:311)
	at org.graalvm.polyglot.Context.eval(Context.java:336)
	at tf.veriny.vt1000.application.polyglot.PolyglotApplication.invoke(PolyglotApplication.kt:25)
	at tf.veriny.vt1000.kernel.KernelApplication.invoke(KernelApplication.kt:41)
	at tf.veriny.vt1000.kernel.Kernel.invoke(Kernel.kt:45)
	at tf.veriny.vt1000.VT1000$create$2.invoke(VT1000.kt:54)
	at tf.veriny.vt1000.VT1000$create$2.invoke(VT1000.kt:22)
	at kotlin.concurrent.ThreadsKt$thread$thread$1.run(Thread.kt:30)

Document how to contribute

Not really sure, whether you are ready to accept contributions.
If yes, please document

Basic setup
Limitations
How to handle CPython extensions using shared libraries

Type annotations and typed fields in data classes not parseable

Hi,

I know dataclasses is not yet implemented but the following unittest fails with runtime exception
while CPython 3.6 simply returns without error.

import unittest


class DataClassesTests(unittest.TestCase):
    def test_dataclasses(self):
        try:
            from dataclasses import dataclass
        except ImportError:
            return

        @dataclass
        class Point:
            x: float
            y: float

        def f(x):
            return x


        self.assertEqual(f(Point(1, 2)), Point(1, 2))

graalpython ./test_dataclass.py

java.lang.RuntimeException: not implemented
	at com.oracle.graal.python.parser.AssignmentTranslator.translate(AssignmentTranslator.java:76)
	at com.oracle.graal.python.parser.PythonTreeTranslator.visitExpr_stmt(PythonTreeTranslator.java:1313)
	at com.oracle.graal.python.parser.antlr.Python3Parser$Expr_stmtContext.accept(Python3Parser.java:2132)
	at com.oracle.graal.python.parser.PythonTreeTranslator.visitChildren(PythonTreeTranslator.java:155)
	at com.oracle.graal.python.parser.antlr.Python3BaseVisitor.visitSmall_stmt(Python3BaseVisitor.java:196)
	at com.oracle.graal.python.parser.antlr.Python3Parser$Small_stmtContext.accept(Python3Parser.java:1997)
	at com.oracle.graal.python.parser.PythonTreeTranslator.visitChildren(PythonTreeTranslator.java:162)
	at com.oracle.graal.python.parser.antlr.Python3BaseVisitor.visitSimple_stmt(Python3BaseVisitor.java:189)
	at com.oracle.graal.python.parser.antlr.Python3Parser$Simple_stmtContext.accept(Python3Parser.java:1900)
	at com.oracle.graal.python.parser.PythonTreeTranslator.visitChildren(PythonTreeTranslator.java:155)
	at com.oracle.graal.python.parser.antlr.Python3BaseVisitor.visitStmt(Python3BaseVisitor.java:182)
	at com.oracle.graal.python.parser.antlr.Python3Parser$StmtContext.accept(Python3Parser.java:1801)
	at com.oracle.graal.python.parser.PythonTreeTranslator.visitChildren(PythonTreeTranslator.java:162)
	at com.oracle.graal.python.parser.antlr.Python3BaseVisitor.visitSuite(Python3BaseVisitor.java:434)
	at com.oracle.graal.python.parser.antlr.Python3Parser$SuiteContext.accept(Python3Parser.java:4569)
	at com.oracle.graal.python.parser.PythonTreeTranslator.visitClassdef(PythonTreeTranslator.java:1764)
	at com.oracle.graal.python.parser.antlr.Python3Parser$ClassdefContext.accept(Python3Parser.java:7443)
	at com.oracle.graal.python.parser.PythonTreeTranslator.visitDecorated(PythonTreeTranslator.java:910)
	at com.oracle.graal.python.parser.antlr.Python3Parser$DecoratedContext.accept(Python3Parser.java:649)
	at com.oracle.graal.python.parser.PythonTreeTranslator.visitChildren(PythonTreeTranslator.java:155)
	at com.oracle.graal.python.parser.antlr.Python3BaseVisitor.visitCompound_stmt(Python3BaseVisitor.java:371)
	at com.oracle.graal.python.parser.antlr.Python3Parser$Compound_stmtContext.accept(Python3Parser.java:3807)
	at com.oracle.graal.python.parser.PythonTreeTranslator.visitChildren(PythonTreeTranslator.java:155)
	at com.oracle.graal.python.parser.antlr.Python3BaseVisitor.visitStmt(Python3BaseVisitor.java:182)
	at com.oracle.graal.python.parser.antlr.Python3Parser$StmtContext.accept(Python3Parser.java:1801)
	at com.oracle.graal.python.parser.PythonTreeTranslator.visitChildren(PythonTreeTranslator.java:162)
	at com.oracle.graal.python.parser.antlr.Python3BaseVisitor.visitSuite(Python3BaseVisitor.java:434)
	at com.oracle.graal.python.parser.antlr.Python3Parser$SuiteContext.accept(Python3Parser.java:4569)
	at com.oracle.graal.python.parser.PythonTreeTranslator.visitFuncdef(PythonTreeTranslator.java:1485)
	at com.oracle.graal.python.parser.antlr.Python3Parser$FuncdefContext.accept(Python3Parser.java:772)
	at com.oracle.graal.python.parser.PythonTreeTranslator.visitChildren(PythonTreeTranslator.java:155)
	at com.oracle.graal.python.parser.antlr.Python3BaseVisitor.visitCompound_stmt(Python3BaseVisitor.java:371)
	at com.oracle.graal.python.parser.antlr.Python3Parser$Compound_stmtContext.accept(Python3Parser.java:3807)
	at com.oracle.graal.python.parser.PythonTreeTranslator.visitChildren(PythonTreeTranslator.java:155)
	at com.oracle.graal.python.parser.antlr.Python3BaseVisitor.visitStmt(Python3BaseVisitor.java:182)
	at com.oracle.graal.python.parser.antlr.Python3Parser$StmtContext.accept(Python3Parser.java:1801)
	at com.oracle.graal.python.parser.PythonTreeTranslator.visitChildren(PythonTreeTranslator.java:162)
	at com.oracle.graal.python.parser.antlr.Python3BaseVisitor.visitSuite(Python3BaseVisitor.java:434)
	at com.oracle.graal.python.parser.antlr.Python3Parser$SuiteContext.accept(Python3Parser.java:4569)
	at com.oracle.graal.python.parser.PythonTreeTranslator.visitClassdef(PythonTreeTranslator.java:1764)
	at com.oracle.graal.python.parser.antlr.Python3Parser$ClassdefContext.accept(Python3Parser.java:7443)
	at com.oracle.graal.python.parser.PythonTreeTranslator.visitChildren(PythonTreeTranslator.java:155)
	at com.oracle.graal.python.parser.antlr.Python3BaseVisitor.visitCompound_stmt(Python3BaseVisitor.java:371)
	at com.oracle.graal.python.parser.antlr.Python3Parser$Compound_stmtContext.accept(Python3Parser.java:3807)
	at com.oracle.graal.python.parser.PythonTreeTranslator.visitChildren(PythonTreeTranslator.java:155)
	at com.oracle.graal.python.parser.antlr.Python3BaseVisitor.visitStmt(Python3BaseVisitor.java:182)
	at com.oracle.graal.python.parser.antlr.Python3Parser$StmtContext.accept(Python3Parser.java:1801)
	at com.oracle.graal.python.parser.PythonTreeTranslator.visitChildren(PythonTreeTranslator.java:162)
	at com.oracle.graal.python.parser.antlr.Python3BaseVisitor.visitFile_input(Python3BaseVisitor.java:56)
	at com.oracle.graal.python.parser.PythonTreeTranslator.visitFile_input(PythonTreeTranslator.java:232)
	at com.oracle.graal.python.parser.antlr.Python3Parser$File_inputContext.accept(Python3Parser.java:328)
	at com.oracle.graal.python.parser.PythonTreeTranslator.translate(PythonTreeTranslator.java:133)
	at com.oracle.graal.python.parser.PythonParserImpl.parse(PythonParserImpl.java:101)
	at com.oracle.graal.python.PythonLanguage.doParse(PythonLanguage.java:245)
	at com.oracle.graal.python.PythonLanguage.parse(PythonLanguage.java:235)
	at com.oracle.truffle.api.TruffleLanguage$ParsingRequest.parse(TruffleLanguage.java:798)
	at com.oracle.truffle.api.TruffleLanguage.parse(TruffleLanguage.java:1233)
	at com.oracle.truffle.api.TruffleLanguage$LanguageImpl.parse(TruffleLanguage.java:2413)
	at com.oracle.truffle.polyglot.PolyglotSourceCache.parseImpl(PolyglotSourceCache.java:92)
	at com.oracle.truffle.polyglot.PolyglotSourceCache.parseCached(PolyglotSourceCache.java:73)
	at com.oracle.truffle.polyglot.PolyglotLanguageContext.parseCached(PolyglotLanguageContext.java:186)
	at com.oracle.truffle.polyglot.PolyglotContextImpl.eval(PolyglotContextImpl.java:751)
	at org.graalvm.polyglot.Context.eval(Context.java:335)
	at com.oracle.graal.python.shell.GraalPythonMain.evalNonInteractive(GraalPythonMain.java:356)
	at com.oracle.graal.python.shell.GraalPythonMain.launch(GraalPythonMain.java:280)
	at org.graalvm.launcher.AbstractLanguageLauncher.launch(AbstractLanguageLauncher.java:132)
	at org.graalvm.launcher.AbstractLanguageLauncher.launch(AbstractLanguageLauncher.java:72)
	at com.oracle.graal.python.shell.GraalPythonMain.main(GraalPythonMain.java:56)
Traceback (most recent call last):
java.lang.RuntimeException: Failed in com.oracle.graal.python.parser.PythonTreeTranslator@7fac575914a8 with error java.lang.RuntimeException: not implemented

Pandas import is not working proprely

With graalvm RC12 I got following error while importing pandas in graalpython repl.

On Ubuntu 16.04 with clang-7

~/workspace/polyglot$ /usr/lib/jvm/graalvm-ce-1.0.0-rc12/bin/graalpython
Python 3.7.0 (Fri Feb 01 11:43:57 UTC 2019)
[GraalVM CE, Java 1.8.0_192] on linux
Type "help", "copyright", "credits" or "license" for more information.
Please note: This Python implementation is in the very early stages, and can run little more than basic benchmarks at this point.
>>> import pandas as pd
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap.py", line 1074, in __import__
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap.py", line 994, in _gcd_import
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap.py", line 971, in _find_and_load
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap.py", line 955, in _find_and_load_unlocked
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap.py", line 665, in _load_unlocked
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap_external.py", line 681, in exec_module
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap.py", line 219, in _call_with_frames_removed
  File "/home/etienne/.local/lib/python3.7/site-packages/pandas-0.20.3-py3.7-linux-x86_64.egg/pandas/__init__.py", line 58, in <module>
    from pandas.io.api import *
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap.py", line 1074, in __import__
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap.py", line 994, in _gcd_import
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap.py", line 971, in _find_and_load
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap.py", line 955, in _find_and_load_unlocked
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap.py", line 665, in _load_unlocked
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap_external.py", line 681, in exec_module
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap.py", line 219, in _call_with_frames_removed
  File "/home/etienne/.local/lib/python3.7/site-packages/pandas-0.20.3-py3.7-linux-x86_64.egg/pandas/io/api.py", line 18, in <module>
    from pandas.io.packers import read_msgpack, to_msgpack
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap.py", line 1074, in __import__
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap.py", line 994, in _gcd_import
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap.py", line 971, in _find_and_load
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap.py", line 955, in _find_and_load_unlocked
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap.py", line 665, in _load_unlocked
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap_external.py", line 681, in exec_module
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap.py", line 219, in _call_with_frames_removed
  File "/home/etienne/.local/lib/python3.7/site-packages/pandas-0.20.3-py3.7-linux-x86_64.egg/pandas/io/packers.py", line 68, in <module>
    from pandas.io.msgpack import Unpacker as _Unpacker, Packer as _Packer, ExtType
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap.py", line 1074, in __import__
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap.py", line 994, in _gcd_import
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap.py", line 971, in _find_and_load
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap.py", line 955, in _find_and_load_unlocked
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap.py", line 665, in _load_unlocked
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap_external.py", line 681, in exec_module
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap.py", line 219, in _call_with_frames_removed
  File "/home/etienne/.local/lib/python3.7/site-packages/pandas-0.20.3-py3.7-linux-x86_64.egg/pandas/io/msgpack/__init__.py", line 9, in <module>
    class ExtType(namedtuple('ExtType', 'code data')):
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap.py", line 1074, in __import__
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap.py", line 994, in _gcd_import
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap.py", line 971, in _find_and_load
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap.py", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'pandas.io.msgpack._packer'

If a try to import a second time I got this error :

>>> import pandas as pd
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap.py", line 1074, in __import__
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap.py", line 994, in _gcd_import
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap.py", line 971, in _find_and_load
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap.py", line 955, in _find_and_load_unlocked
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap.py", line 665, in _load_unlocked
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap_external.py", line 681, in exec_module
  File "/b/b/e/graalpython/graalpython/lib-python/3/importlib/_bootstrap.py", line 219, in _call_with_frames_removed
  File "/home/etienne/.local/lib/python3.7/site-packages/pandas-0.20.3-py3.7-linux-x86_64.egg/pandas/__init__.py", line 51, in <module>
    plot_params = pandas.plotting._style._Options(deprecated=True)
AttributeError: 'module' object has no attribute 'plotting'

I'm also wondering why it's so slow to do imports, numpy take something like 30s to be imported in the graalpython repl.

findLocalScopes(node, null) provides all local variables in the scope ignoring node's position

def myfunc(arg):
    y = 1
    x = 2

findLocalScopes(node, null) returns always [arg, y, x] for a node inside myfunc(arg)

See SLLanguage.findLocalScopes() how it should be done collecting only the local variables already defined.

`graalpython --polyglot` fails with "Could not find launcher"

Steps to reproduce:

Download GraalVM 1.0.0 RC8 EE for macOS
Install Python via gu
Launch ../graalvm-ee-1.0.0-rc8/Contents/Home/bin/graalpython --polyglot

Result:

$ ../graalvm-ee-1.0.0-rc8/Contents/Home/bin/graalpython --polyglot
ERROR: Could not find launcher 'com.oracle.graal.python.shell.GraalPythonMain'
[1]    45618 exit 1     ../bin/graalvm-ee-1.0.0-rc8/Contents/Home/bin/graalpython --polyglot

Expected result:
Interactive Python shell "with all other guest languages accessible" as suggested by --help option.

shutil.copyfile doesn't overwrite exiting file

On CPython 3.6 shutil.copyfile overwrites even if the file exits.
GraalPython doesn't.

  File "copyfile.py", line 19, in <module>
    copyfile(a,b)
  File "/u/yokohama55/people/ryutas/workspace/packages/graal/graal/vm/mxbuild/linux-amd64/GRAALVM_CMP_GU_GVM_INS_JS_LIBPOLY_POLY_POLYNATIVE_PRO_PYN_RGX_SLG_SVM_TFL_VVM/graalvm-cmp-gu-gvm-ins-js-libpoly-poly-polynative-pro-pyn-rgx-slg-svm-tfl-vvm-1.0.0-rc12-dev/jre/languages/python/lib-python/3/shutil.py", line 104, in copyfile
    raise SameFileError("{!r} and {!r} are the same file".format(src, dst))
shutil.SameFileError: <unprintable SameFileError object>

import os
from shutil import copyfile

a = os.path.join('.', 'a')
b = os.path.join('.', 'b')

if os.path.exists(a):
  os.remove(a)

if os.path.exists(b):
  os.remove(b)

open(a, 'w')

copyfile(a,b)

# should overwrite 'b' by 'a'
copyfile(a,b)

Lib random throws error

Hey there,
i cant import the libary "random" inside of java.

import random
print(random.randint(0, 9))

these lines will throw an error. org.graalvm.polyglot.PolyglotException: java.lang.ClassCastException: com.oracle.graal.python.builtins.objects.PNone cannot be cast to com.oracle.graal.python.builtins.objects.module.PythonModule

if i run it with graalpyton testfile.py it runs without errors

PythonBuiltinClass.getArity() fails with ClassCastException

In PythonBuiltinClass.getArity() returns the call to getAttribute(__NEW__) PNone, because the shape has no such field:

@44f9dee3{"__doc__":Object@2,
"__qualname__":Object@1,
"__name__":Object@0
}

Trace:

Caused by: java.lang.ClassCastException: com.oracle.graal.python.builtins.objects.PNone cannot be cast to com.oracle.graal.python.builtins.objects.function.PythonCallable
    at com.oracle.graal.python.builtins.objects.type.PythonBuiltinClass.getArity(PythonBuiltinClass.java:69)

Bytecode caching

GraalPython starts up very slowly. It may make sense to cache preprocessed bytecode.

TypeError in itertools.zip_longest

Hi,

zip_longest from itertools in lib-graalpython

from itertools import zip_longest
x = [1,2,3]
y = [4,5,6,7]
zip_longest(x,y)

results in TypeError:

...
  File "itertools.py", line 646, in __new__
    self.iterators = [iter1] + args
TypeError: can only concatenate list (not "tuple") to list

line 646: self.iterators = [iter1] + args

where

args is <class 'tuple'>

Calling python from java, polyglot error: UnicodeDecodeError: Input length = 1

Here's my python code, I'm running into problems importing any modules. Are there any suggestions?

FWIW -- I am passing as a JVM arg inside of a docker container -Dpython.home=/graalvm/jre/languages/python

import polyglot
  
log = polyglot.import_value('log')
log.info("Geocoder running")

import json
log.info("Imported json")
import reverse_geocoder as rg
log.info("Imported rg")

def main():
   arg = polyglot.import_value('arguments')
   log.info("Python args")
   log.info(arg)
   lat = arguments.get(0)
   lon = arguments.get(1)
   log.info("Running on %s, %s" % (str(lat), str(lon)))
   locations = rg.search((lat, lon))
   return locations

main()

I'm seeing this error:

2018-08-20 12:44:53.050+0000 INFO  Executing reverse-geocoder/python with arguments [35.577045, 51.462464] of type class java.util.ArrayList
2018-08-20 12:44:56.586+0000 INFO  Geocoder running
UnicodeDecodeError: Input length = 1
	at <python> __truffle_decode(Unknown)
	at <python> utf_8_decode(graalvm/jre/languages/python/lib-graalpython/_codecs.py:157:5119-5159)
	at <python> staticmethod.__call__(graalvm/jre/languages/python/lib-graalpython/staticmethod.py:46:2134-2163)
	at <python> decode(/graalvm/jre/languages/python/lib-python/3/encodings/utf_8.py:16:244-283)
	at <python> decode(graalvm/jre/languages/python/lib-graalpython/_codecs.py:112:3952-3971)
	at <python> decode(graalvm/jre/languages/python/lib-graalpython/bytes.py:53:2476-2529)
	at <python> SRE_Pattern._decode_string(graalvm/jre/languages/python/lib-graalpython/_sre.py:146:4973-4987)
	at <python> SRE_Pattern.__init__(graalvm/jre/languages/python/lib-graalpython/_sre.py:130:4421-4455)
	at <python> compile(/graalvm/jre/languages/python/lib-python/3/sre_compile.py:582-586:19292-19421)
	at <python> _compile(/graalvm/jre/languages/python/lib-python/3/re.py:301:12964-12998)
	at <python> compile(/graalvm/jre/languages/python/lib-python/3/re.py:233:10969-10992)
	at <python> <module '/graalvm/jre/languages/python/lib-python/3/json/encoder.py'>(/graalvm/jre/languages/python/lib-python/3/json/encoder.py:20:515-540)
	at <python> eval(Unknown)
	at <python> exec(graalvm/jre/languages/python/lib-graalpython/functions.py:112:3611-3666)
	at <python> _call_with_frames_removed(graalvm/jre/languages/python/lib-python/3/importlib/_bootstrap.py:206:6419-6434)
	at <python> _LoaderBasics.exec_module(graalvm/jre/languages/python/lib-python/3/importlib/_bootstrap_external.py:687:26296-26360)
	at org.graalvm.polyglot.Context.eval(Context.java:313)
	at com.neo4j.magnolia.polyglot.ExternalFn.invoke(ExternalFn.java:74)
	at com.neo4j.magnolia.polyglot.CallExternal.fn(CallExternal.java:102)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.neo4j.kernel.impl.proc.ReflectiveProcedureCompiler$ReflectiveUserFunction.apply(ReflectiveProcedureCompiler.java:811)

The java code that calls this python can be found here: https://github.com/moxious/magnolia/blob/master/src/main/java/com/neo4j/magnolia/polyglot/ExternalFn.java#L56

No module named '_bz2'

Trying to get https://github.com/Blizzard/heroprotocol running via Blizzard/heroprotocol#59

It includes a new version of mpyp (the library to read MPQ files), but it bails on No module named '_bz2'

Which is used in mpyq import bz2

I see in SciPy also bz2 is imported, any issues with that?

This is my code

package heroes

import org.graalvm.polyglot._

object Main extends App {

  val context = Context.newBuilder().allowIO(true).build
  val v = context.getPolyglotBindings()
  val _ = context.eval("python",
    """
      |from heroprotocol import protocol29406
      |from heroprotocol.mpyq import mpyq
      |
      |archive = mpyq.MPQArchive('/path/to/replay')
      |contents = archive.header['user_data_header']['content']
      |header = protocol.decode_replay_header(contents)
      |
      |# The header's baseBuild determines which protocol to use
      |baseBuild = header['m_version']['m_baseBuild']
      |try:
      |    protocol = __import__('protocol%s' % (baseBuild,))
      |except:
      |    print >> sys.stderr, 'Unsupported base build: %d' % baseBuild
      |    sys.exit(1)
      |
      |""".stripMargin)

}

Can put up a repo if you want?

graalpython pollutes polyglot bindings

Not sure what this is used for (Sulong interop?), but graalpython RC10 exports the following keys into the Polyglot bindings:

python_ArithmeticError
python_AssertionError
python_AttributeError
python_BaseException
python_BlockingIOError
python_BrokenPipeError
python_BufferError
python_builtins
python_BytesWarning
python_cext
python_ChildProcessError
python_ConnectionAbortedError
python_ConnectionError
python_ConnectionRefusedError
python_ConnectionResetError
python_DeprecationWarning
python_EOFError
python_error
python_Exception
python_FileExistsError
python_FileNotFoundError
python_FloatingPointError
python_FutureWarning
python_GeneratorExit
python_ImportError
python_ImportWarning
python_IndentationError
python_IndexError
python_InterruptedError
python_IOError
python_IsADirectoryError
python_KeyboardInterrupt
python_KeyError
python_LookupError
python_MemoryError
python_ModuleNotFoundError
python_NameError
python_NotADirectoryError
python_NotImplementedError
python_OSError
python_OverflowError
python_PendingDeprecationWarning
python_PermissionError
python_ProcessLookupError
python_RecursionError
python_ReferenceError
python_ResourceWarning
python_RuntimeError
python_RuntimeWarning
python_StopIteration
python_SyntaxError
python_SyntaxWarning
python_SystemError
python_SystemExit
python_TabError
python_TimeoutError
python_TypeError
python_UnboundLocalError
python_UnicodeDecodeError
python_UnicodeEncodeError
python_UnicodeError
python_UnicodeTranslateError
python_UnicodeWarning
python_UserWarning
python_ValueError
python_Warning
python_ZeroDivisionError
python_ZipImportError

I don't think this is a good idea. Imagine someone happens to export and effectively overrides python_error, this could cause some hard-to-debug misbehavior if it's used for Sulong interop or some other important infrastructure.

NameError: name 'PyCF_ONLY_AST' is not defined

>>> import ast
>>> line = '__version__ = \"0.0.1\"'
>>> ast.literal_eval(line.split('=', 1)[1].strip())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "ast.py", line 48, in literal_eval
    node_or_string = parse(node_or_string, mode='eval')
  File "ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
NameError: name 'PyCF_ONLY_AST' is not defined

`os` has no attribute 'readlink'

Python 3.7.0 (Sat Jan 12 12:16:56 GMT 2019)
[GraalVM CE, Java 1.8.0_191] on linux
Type "help", "copyright", "credits" or "license" for more information.
Please note: This Python implementation is in the very early stages, and can run little more than basic benchmarks at this point.
>>> import os
>>> os.readlink
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'readlink'

[Interop] UnsupportedSpecializationException when calling polyglot.export_value with wrong parameters

Repro via polyglot shell:

GraalVM MultiLanguage Shell 1.0.0-rc10
Copyright (c) 2013-2018, Oracle and/or its affiliates
  JavaScript version 1.0
  Python version 3.7.0
  R version 3.4.0
  Ruby version 2.4.4
Usage:
  Use Ctrl+L to switch language and Ctrl+D to exit.
  Enter -usage to get a list of available commands.
python> import polyglot
python> polyglot.export_value('o', object())
Internal error occured: org.graalvm.polyglot.PolyglotException: com.oracle.truffle.api.dsl.UnsupportedSpecializationException: Unexpected values provided for ExportSymbolNodeGen@29bcf51d: [o, <object object at 0x1e54a6b1>], [String,PythonObject]
Run with --verbose to see the full stack trace.

Expected result: value is correctly exported.

Implement `mmap`

Python context isolation broken?

import org.graalvm.polyglot.Context;
import org.graalvm.polyglot.Engine;
import org.graalvm.polyglot.Source;

/** Created by revin on Oct.1,2018. */
public class BugReport{
    public void trace(Object x){System.out.println(x);}
    public static void main(String...args)throws Exception{
        Engine engine=Engine.create();
        Context c0=Context.newBuilder().allowIO(true).engine(engine).build();
        Context c1=Context.newBuilder().allowIO(true).engine(engine).build();

        Source source=Source.newBuilder("python",
                "import polyglot\n"+
                "try:bug.trace(\"SHOULD NOT SEE THIS\")\nexcept:pass\n"+
                "bug=polyglot.import_value(\"bug\")\n"
                ,"bugReport.py").build();

        BugReport bug=new BugReport();
        long nano0=System.nanoTime();c0.getPolyglotBindings().putMember("bug",bug);c0.eval(source);
        long nano1=System.nanoTime();c1.getPolyglotBindings().putMember("bug",bug);c1.eval(source);
        long nano2=System.nanoTime();System.out.println((nano1-nano0)/1000_000+" ms without init, "+(nano2-nano1)/1000_000+" ms after init");
    }
}

expected output: none
actual output: "SHOULD NOT SEE THIS"

as you can see from the simplified code above:
it creates two separate contexts, run the same code in each one of them
but it seems somehow the global variable bug leaked from the first context into the second one

am I misunderstood something here? how can I achieve performant context isolation?
context initialization(even if they use the same engine instance underneath) seems very heavy

Can't import prompt_toolkit due to "unexpected values provided for InitNodeGen"

>>> import sys
>>> sys.path += ['/Users/mike/Library/Python/3.7/lib/python/site-packages']
>>> import prompt_toolkit

Traceback (most recent call last):
  File "x.py", line 3, in <module 'x.py'>
  "/Users/mike/Library/Python/3.7/lib/python/site-packages/prompt_toolkit/__init__.py", line 17, in <module '/Users/mike/Library/Python/3.7/lib/python/site-packages/prompt_toolkit/__init__.py'>
  "/Users/mike/Library/Python/3.7/lib/python/site-packages/prompt_toolkit/application/__init__.py", line 2, in <module '/Users/mike/Library/Python/3.7/lib/python/site-packages/prompt_toolkit/application/__init__.py'>
  "/Users/mike/Library/Python/3.7/lib/python/site-packages/prompt_toolkit/application/application.py", line 12, in <module '/Users/mike/Library/Python/3.7/lib/python/site-packages/prompt_toolkit/application/application.py'>
  "/Users/mike/Library/Python/3.7/lib/python/site-packages/prompt_toolkit/input/typeahead.py", line 44, in <module '/Users/mike/Library/Python/3.7/lib/python/site-packages/prompt_toolkit/input/typeahead.py'>
com.oracle.truffle.api.dsl.UnsupportedSpecializationException: Unexpected values provided for InitNodeGen@a9ed6db: [<class 'defaultdict'>, [Ljava.lang.Object;@126c12e08, [Lcom.oracle.graal.python.builtins.objects.function.PKeyword;@10e2297a0], [PythonClass,Object[],PKeyword[]]

line 44 of typeahead.py is:

from collections import defaultdict
_buffer = defaultdict(list)  # input hash -> list of key presses.

Problem with using the uuid module

Hi, when I try to make a simple test script for general uuid generating like:

import uuid
print('id ', str(uuid.uuid4()))

When I run with graalpython, I have the following exception:

Please note: This Python implementation is in the very early stages, and can run little more than basic benchmarks at this point.
Traceback (most recent call last):
  File "test.py", line 3, in <module>
    import uuid
  File "_bootstrap.py", line 1074, in __import__
  File "_bootstrap.py", line 994, in _gcd_import
  File "_bootstrap.py", line 971, in _find_and_load
  File "_bootstrap.py", line 955, in _find_and_load_unlocked
  File "_bootstrap.py", line 665, in _load_unlocked
  File "_bootstrap_external.py", line 681, in _LoaderBasics.exec_module
  File "_bootstrap.py", line 219, in _call_with_frames_removed
  File "functions.py", line 123, in exec
  File "/opt/graalvm/jre/languages/python/lib-python/3/uuid.py", line 633, in <module>
    NAMESPACE_DNS = UUID('6ba7b810-9dad-11d1-80b4-00c04fd430c8')
  File "/opt/graalvm/jre/languages/python/lib-python/3/uuid.py", line 141, in UUID.__init__
    int = int_(hex, 16)
ValueError: invalid literal for int() with base 16: 6ba7b8109dad11d180b400c04fd430c8
)

OS: Ubuntu 18.04
GraalVM version: graalvm-ce-1.0.0-rc6

The `resource` module is not implemented yet

Hello. Seems some specific modules are not implemented yet.
For example, the resource module, which should locate on path like
/usr/lib/python3.6/lib-dynload/resource.cpython-36m-x86_64-linux-gnu.so
(using Python 3.6 as example).
The official documentation for the resource module is here: https://docs.python.org/3.7/library/resource.html .

Here is my test on Arch Linux using the AUR package https://aur.archlinux.org/packages/graalpython-bin

[kaizhao@zbox ~]$ uname -a
Linux zbox 4.17.5-1-ARCH #1 SMP PREEMPT Sun Jul 8 17:27:31 UTC 2018 x86_64 GNU/Linux
[kaizhao@zbox ~]$ graalpython --version
Graal Python 3.7.0 (GraalVM CE Native 1.0.0-rc3)
[kaizhao@zbox ~]$ graalpython -c "import resource"
Please note: This Python implementation is in the very early stages, and can run little more than basic benchmarks at this point.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "_bootstrap.py", line 1066, in __import__
  File "_bootstrap.py", line 985, in _gcd_import
  File "_bootstrap.py", line 968, in _find_and_load
  File "_bootstrap.py", line 955, in _find_and_load_unlocked
ModuleNotFoundError: ModuleNotFound: 'resource'. No module named 'resource'

Although this is a specific module just for Unix like systems, some third packages/tools may depend on it. For example:
https://github.com/ipython/ipython/blob/master/IPython/utils/timing.py
https://github.com/saltstack/salt/blob/develop/salt/utils/verify.py
https://github.com/saltstack/salt/blob/develop/salt/utils/vt.py
https://github.com/lilydjwg/swapview/blob/master/bench.py
etc.
I have read the "Python Compatibility" on https://www.graalvm.org/docs/reference-manual/languages/python/#python-compatibility .
And is there any plan for such issues?
Thanks!

Inconsitency with dictonary members

the following code call a lambda which return a python dictionary.

val result = calculator.execute(columns,lines)
println("result: "+result)
println("hasMembers: "+result.hasMembers)
println("hasMember(\"Col1\"): "+result.hasMember("Col1"))
println("hasMember(\"0\"): "+result.hasMember("0"))
println("getMemberKeys: "+result.getMemberKeys)
println("getMember(\"Col1\"): "+result.getMember("Col1"))
println("getMember(\"Col1\").hasMembers: "+result.getMember("Col1").hasMembers)
println("getMember(\"Col1\").getMemberKeys: "+result.getMember("Col1").getMemberKeys)
println("getMember(\"Col1\").getMember(\"0\"): "+result.getMember("Col1").getMember("0"))

the result of the execution.

result: {'Col1': {0: 1, 1: 4, 2: 7}, 'Col2': {0: 2, 1: 5, 2: 8}, 'Col3': {0: 3, 1: 6, 2: 9}}
hasMembers: true
hasMember("Col1"): false
hasMember("0"): false
getMemberKeys: [pop, fromkeys, setdefault, keys, values, get, clear, update, popitem, copy, items]
getMember("Col1"): {0: 1, 1: 4, 2: 7}
getMember("Col1").hasMembers: true
getMember("Col1").getMemberKeys: [pop, fromkeys, setdefault, keys, values, get, clear, update, popitem, copy, items]
getMember("Col1").getMember("0"): null

hasMember("Col1") return false but getMember("Col1") return the inner object.
getMemberKeys don't contain "Col1"
getMember("Col1").getMember("0") return null because the inner key is an Int and not a String

Python can not interpret float, char, short and byte types in a Java object

I'd like to pass a Java object to a python function, however, some types can not be interpreted in type char, short, float and byte.
Here is my Java code, two classes
HelloPolyglot: get an object from a python file, and pass a Java object to one of its function
JavaObject: the Java object to be passed

import org.graalvm.polyglot.*;
import org.graalvm.polyglot.proxy.*;
import java.io.File;

public class HelloPolyglot {
    public static void main(String[] args) throws Exception {
    	JavaObject order = new JavaObject();
    	object.setStrategyId("strategyIdxx");
    	object.setCumAmount(1000d);
        object.setByteValue((byte)127);
        object.setShortValue((short)32767);
        object.setIntValue(2_147_483_647);
        object.setLongValue(9_223_372_036_854_775_807L);
        object.setFloatValue(0);
        object.setDoubleValue(2000d);
        object.setBooleanValue(true);
        object.setCharValue('a');

    	File file = new File("/home/appadmin/graalvm/script", "polyglot_example.py");
		String language = Source.findLanguage(file);

		Context context = Context.newBuilder(language)
            .allowAllAccess(false)
            .allowHostAccess(false)
            .allowHostClassLoading(false)
            .allowIO(true)
            .allowNativeAccess(false)
            .allowCreateThread(false)               
            .build();
        Value v = context.getPolyglotBindings();
		Source source = Source.newBuilder(language, file).build();
		context.eval(source);
		Value main = context.getPolyglotBindings().getMember("foo");		
		Value instance = main.execute();
		Value onNewOrderRejected = instance.getMember("onNewOrderRejected");
		onNewOrderRejected.execute(object);
		context.close();
    }
}

public class JavaObject {
    private byte byteValue;
    private short shortValue;
    private int intValue;
    private long longValue;
    private float floatValue;
    private double doubleValue;
    private boolean booleanValue;
    public char charValue;

    public byte getByteValue() {
        return byteValue;
    }

    public void setByteValue(byte byteValue) {
        this.byteValue = byteValue;
    }

    public short getShortValue() {
        return shortValue;
    }

    public void setShortValue(short shortValue) {
        this.shortValue = shortValue;
    }

    public Integer getIntValue() {
        return intValue;
    }

    public void setIntValue(int intValue) {
        this.intValue = intValue;
    }

    public long getLongValue() {
        return longValue;
    }

    public void setLongValue(long longValue) {
        this.longValue = longValue;
    }

    public float getFloatValue() {
        return floatValue;
    }

    public void setFloatValue(float floatValue) {
        this.floatValue = floatValue;
    }

    public double getDoubleValue() {
        return doubleValue;
    }

    public void setDoubleValue(double doubleValue) {
        this.doubleValue = doubleValue;
    }

    public boolean isBooleanValue() {
        return booleanValue;
    }

    public void setBooleanValue(boolean booleanValue) {
        this.booleanValue = booleanValue;
    }

    public Character getCharValue() {
        return 'a';
    }

    public void setCharValue(char charValue) {
        this.charValue = charValue;
    }
}

#!/usr/bin/python
# -*- coding: UTF-8 -*-
import polyglot
import java

@polyglot.export_value
def foo():
	return PythonStrategy()

class PythonStrategy:

	def onNewOrderRejected(self, order):
		print(order.getIntValue())
		print(order.getLongValue())
		print(order.getDoubleValue())		
		print(order.isBooleanValue())	
		# print(order.getByteValue())  # error
		# print(order.getShortValue()) # error
		# print(order.getFloatValue()) # error
		# print(order.getCharValue()) # error
		return

I got the following exception for byte, short, float and char type:
Exception in thread "main" org.graalvm.polyglot.PolyglotException: com.oracle.truffle.api.dsl.UnsupportedSpecializationException: Unexpected values provided for GetClassNodeGen@650ae78c: [234.5], [Float]
at com.oracle.graal.python.nodes.object.GetClassNodeGen.executeAndSpecialize(GetClassNodeGen.java:531)
at com.oracle.graal.python.nodes.object.GetClassNodeGen.executeGetClass(GetClassNodeGen.java:216)
at com.oracle.graal.python.nodes.object.GetClassNode.execute(GetClassNode.java:86)
at com.oracle.graal.python.nodes.attributes.LookupInheritedAttributeNode.execute(LookupInheritedAttributeNode.java:74)
at com.oracle.graal.python.nodes.call.special.LookupAndCallUnaryNode.callObject(LookupAndCallUnaryNode.java:204)
at com.oracle.graal.python.nodes.call.special.LookupAndCallUnaryNodeGen.executeAndSpecialize(LookupAndCallUnaryNodeGen.java:801)
at com.oracle.graal.python.nodes.call.special.LookupAndCallUnaryNodeGen.executeDouble(LookupAndCallUnaryNodeGen.java:416)
at com.oracle.graal.python.builtins.modules.BuiltinConstructors$FloatNode.doubleFromObject(BuiltinConstructors.java:680)
at com.oracle.graal.python.builtins.modules.BuiltinConstructorsFactory$FloatNodeFactory$FloatNodeGen.executeAndSpecialize(BuiltinConstructorsFactory.java:1774)
at com.oracle.graal.python.builtins.modules.BuiltinConstructorsFactory$FloatNodeFactory$FloatNodeGen.execute(BuiltinConstructorsFactory.java:1693)
at com.oracle.graal.python.nodes.function.BuiltinFunctionRootNode$BuiltinAnyCallNode.execute(BuiltinFunctionRootNode.java:72)
at com.oracle.graal.python.nodes.function.BuiltinFunctionRootNode.execute(BuiltinFunctionRootNode.java:266)
at float(Unknown)
at PythonStrategy.onNewOrderRejected(polyglot_example.py:36:723-750)
at org.graalvm.polyglot.Value.execute(Value.java:338)
at HelloPolyglot.main(HelloPolyglot.java:54)
Original Internal Error:
com.oracle.truffle.api.dsl.UnsupportedSpecializationException: Unexpected values provided for GetClassNodeGen@650ae78c: [234.5], [Float]
at com.oracle.graal.python.nodes.object.GetClassNodeGen.executeAndSpecialize(GetClassNodeGen.java:531)
at com.oracle.graal.python.nodes.object.GetClassNodeGen.executeGetClass(GetClassNodeGen.java:216)
at com.oracle.graal.python.nodes.object.GetClassNode.execute(GetClassNode.java:86)
at com.oracle.graal.python.nodes.attributes.LookupInheritedAttributeNode.execute(LookupInheritedAttributeNode.java:74)
at com.oracle.graal.python.nodes.call.special.LookupAndCallUnaryNode.callObject(LookupAndCallUnaryNode.java:204)
at com.oracle.graal.python.nodes.call.special.LookupAndCallUnaryNodeGen.executeAndSpecialize(LookupAndCallUnaryNodeGen.java:801)
at com.oracle.graal.python.nodes.call.special.LookupAndCallUnaryNodeGen.executeDouble(LookupAndCallUnaryNodeGen.java:416)
at com.oracle.graal.python.builtins.modules.BuiltinConstructors$FloatNode.doubleFromObject(BuiltinConstructors.java:680)
at com.oracle.graal.python.builtins.modules.BuiltinConstructorsFactory$FloatNodeFactory$FloatNodeGen.executeAndSpecialize(BuiltinConstructorsFactory.java:1774)
at com.oracle.graal.python.builtins.modules.BuiltinConstructorsFactory$FloatNodeFactory$FloatNodeGen.execute(BuiltinConstructorsFactory.java:1693)
at com.oracle.graal.python.nodes.function.BuiltinFunctionRootNode$BuiltinAnyCallNode.execute(BuiltinFunctionRootNode.java:72)
at com.oracle.graal.python.nodes.function.BuiltinFunctionRootNode.execute(BuiltinFunctionRootNode.java:266)
at org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.callProxy(OptimizedCallTarget.java:285)
at org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.callRoot(OptimizedCallTarget.java:274)
at org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.callBoundary(OptimizedCallTarget.java:264)
at org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.doInvoke(OptimizedCallTarget.java:246)
at org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.callDirect(OptimizedCallTarget.java:229)
at org.graalvm.compiler.truffle.runtime.OptimizedDirectCallNode.callProxy(OptimizedDirectCallNode.java:88)
at org.graalvm.compiler.truffle.runtime.OptimizedDirectCallNode.call(OptimizedDirectCallNode.java:72)
at com.oracle.graal.python.nodes.call.InvokeNode.doNoKeywords(InvokeNode.java:244)
at com.oracle.graal.python.nodes.call.InvokeNodeGen.executeAndSpecialize(InvokeNodeGen.java:61)
at com.oracle.graal.python.nodes.call.InvokeNodeGen.execute(InvokeNodeGen.java:48)
at com.oracle.graal.python.nodes.call.CallDispatchNode.callFunction(CallDispatchNode.java:106)
at com.oracle.graal.python.nodes.call.CallDispatchNodeGen.executeAndSpecialize(CallDispatchNodeGen.java:315)
at com.oracle.graal.python.nodes.call.CallDispatchNodeGen.executeCall(CallDispatchNodeGen.java:139)
at com.oracle.graal.python.nodes.call.CallNode.builtinFunctionCall(CallNode.java:126)
at com.oracle.graal.python.nodes.call.CallNodeGen.executeAndSpecialize(CallNodeGen.java:102)
at com.oracle.graal.python.nodes.call.CallNodeGen.execute(CallNodeGen.java:59)
at com.oracle.graal.python.nodes.call.special.CallVarargsMethodNode.call(CallVarargsMethodNode.java:126)
at com.oracle.graal.python.nodes.call.special.CallVarargsMethodNodeGen.executeAndSpecialize(CallVarargsMethodNodeGen.java:523)
at com.oracle.graal.python.nodes.call.special.CallVarargsMethodNodeGen.execute(CallVarargsMethodNodeGen.java:184)
at com.oracle.graal.python.builtins.objects.type.TypeBuiltins$CallNode.op(TypeBuiltins.java:190)
at com.oracle.graal.python.builtins.objects.type.TypeBuiltins$CallNode.doItUnboxed(TypeBuiltins.java:165)
at com.oracle.graal.python.builtins.objects.type.TypeBuiltinsFactory$CallNodeFactory$CallNodeGen.executeAndSpecialize(TypeBuiltinsFactory.java:510)
at com.oracle.graal.python.builtins.objects.type.TypeBuiltinsFactory$CallNodeFactory$CallNodeGen.execute(TypeBuiltinsFactory.java:469)
at com.oracle.graal.python.builtins.objects.type.TypeBuiltins$CallNode.varArgExecute(TypeBuiltins.java:155)
at com.oracle.graal.python.nodes.call.special.CallVarargsMethodNode.call(CallVarargsMethodNode.java:68)
at com.oracle.graal.python.nodes.call.special.CallVarargsMethodNodeGen.executeAndSpecialize(CallVarargsMethodNodeGen.java:234)
at com.oracle.graal.python.nodes.call.special.CallVarargsMethodNodeGen.execute(CallVarargsMethodNodeGen.java:184)
at com.oracle.graal.python.nodes.call.CallNode.specialCall(CallNode.java:90)
at com.oracle.graal.python.nodes.call.CallNodeGen.executeAndSpecialize(CallNodeGen.java:74)
at com.oracle.graal.python.nodes.call.CallNodeGen.execute(CallNodeGen.java:59)
at com.oracle.graal.python.nodes.call.PythonCallNode.call(PythonCallNode.java:261)
at com.oracle.graal.python.nodes.call.PythonCallNodeGen.executeAndSpecialize(PythonCallNodeGen.java:82)
at com.oracle.graal.python.nodes.call.PythonCallNodeGen.execute(PythonCallNodeGen.java:55)
at com.oracle.graal.python.nodes.argument.positional.PositionalArgumentsNode.evaluateArguments(PositionalArgumentsNode.java:78)
at com.oracle.graal.python.nodes.call.PythonCallNode.evaluateArguments(PythonCallNode.java:216)
at com.oracle.graal.python.nodes.call.PythonCallNode.call(PythonCallNode.java:259)
at com.oracle.graal.python.nodes.call.PythonCallNodeGen.executeAndSpecialize(PythonCallNodeGen.java:82)
at com.oracle.graal.python.nodes.call.PythonCallNodeGen.execute(PythonCallNodeGen.java:55)
at com.oracle.graal.python.nodes.expression.ExpressionNode$ExpressionStatementNode.executeVoid(ExpressionNode.java:117)
at com.oracle.graal.python.nodes.control.BlockNode.executeVoid(BlockNode.java:54)
at com.oracle.graal.python.nodes.control.BlockNode.executeVoid(BlockNode.java:54)
at com.oracle.graal.python.nodes.control.ReturnTargetNode.execute(ReturnTargetNode.java:58)
at com.oracle.graal.python.nodes.function.FunctionRootNode.execute(FunctionRootNode.java:142)
at org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.callProxy(OptimizedCallTarget.java:285)
at org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.callRoot(OptimizedCallTarget.java:274)
at org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.callBoundary(OptimizedCallTarget.java:264)
at org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.doInvoke(OptimizedCallTarget.java:246)
at org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.callDirect(OptimizedCallTarget.java:229)
at org.graalvm.compiler.truffle.runtime.OptimizedDirectCallNode.callProxy(OptimizedDirectCallNode.java:88)
at org.graalvm.compiler.truffle.runtime.OptimizedDirectCallNode.call(OptimizedDirectCallNode.java:72)
at com.oracle.graal.python.nodes.call.InvokeNode.doNoKeywords(InvokeNode.java:244)
at com.oracle.graal.python.nodes.call.InvokeNodeGen.executeAndSpecialize(InvokeNodeGen.java:61)
at com.oracle.graal.python.nodes.call.InvokeNodeGen.execute(InvokeNodeGen.java:48)
at com.oracle.graal.python.nodes.call.CallDispatchNode.callMethod(CallDispatchNode.java:74)
at com.oracle.graal.python.nodes.call.CallDispatchNodeGen.executeAndSpecialize(CallDispatchNodeGen.java:184)
at com.oracle.graal.python.nodes.call.CallDispatchNodeGen.executeCall(CallDispatchNodeGen.java:139)
at com.oracle.graal.python.nodes.call.CallNode.methodCall(CallNode.java:111)
at com.oracle.graal.python.nodes.call.CallNodeGen.executeAndSpecialize(CallNodeGen.java:81)
at com.oracle.graal.python.nodes.call.CallNodeGen.execute(CallNodeGen.java:59)
at com.oracle.graal.python.runtime.interop.PythonMessageResolution$ExecuteNode.doCall(PythonMessageResolution.java:303)
at com.oracle.graal.python.runtime.interop.PythonMessageResolution$ExecuteNode.execute(PythonMessageResolution.java:298)
at com.oracle.graal.python.runtime.interop.PythonMessageResolution$PForeignFunctionExecuteNode.access(PythonMessageResolution.java:430)
at com.oracle.graal.python.runtime.interop.PythonMessageResolutionForeign$PForeignFunctionExecuteSubNode.accessWithTarget(PythonMessageResolutionForeign.java:122)
at com.oracle.graal.python.runtime.interop.PythonMessageResolutionForeignFactory$PForeignFunctionExecuteSubNodeGen.executeAndSpecialize(PythonMessageResolutionForeignFactory.java:65)
at com.oracle.graal.python.runtime.interop.PythonMessageResolutionForeignFactory$PForeignFunctionExecuteSubNodeGen.executeWithTarget(PythonMessageResolutionForeignFactory.java:51)
at com.oracle.graal.python.runtime.interop.PythonMessageResolutionForeign$PForeignFunctionExecuteSubNode$EXECUTERootNode.execute(PythonMessageResolutionForeign.java:144)
at org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.callProxy(OptimizedCallTarget.java:285)
at org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.callRoot(OptimizedCallTarget.java:274)
at org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.callBoundary(OptimizedCallTarget.java:264)
at org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.doInvoke(OptimizedCallTarget.java:246)
at org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.callDirect(OptimizedCallTarget.java:229)
at org.graalvm.compiler.truffle.runtime.OptimizedDirectCallNode.callProxy(OptimizedDirectCallNode.java:88)
at org.graalvm.compiler.truffle.runtime.OptimizedDirectCallNode.call(OptimizedDirectCallNode.java:72)
at com.oracle.truffle.api.interop.InteropAccessNode.doCached(InteropAccessNode.java:202)
at com.oracle.truffle.api.interop.InteropAccessNodeGen.executeAndSpecialize(InteropAccessNodeGen.java:82)
at com.oracle.truffle.api.interop.InteropAccessNodeGen.executeImpl(InteropAccessNodeGen.java:45)
at com.oracle.truffle.api.interop.InteropAccessNode.execute(InteropAccessNode.java:83)
at com.oracle.truffle.api.interop.ForeignAccess.sendExecute(ForeignAccess.java:436)
at com.oracle.truffle.polyglot.PolyglotValue$Interop$AbstractExecuteNode.executeShared(PolyglotValue.java:2146)
at com.oracle.truffle.polyglot.PolyglotValue$Interop$ExecuteNode.executeImpl(PolyglotValue.java:2226)
at com.oracle.truffle.polyglot.PolyglotValue$PolyglotNode.execute(PolyglotValue.java:529)
at org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.callProxy(OptimizedCallTarget.java:285)
at org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.callRoot(OptimizedCallTarget.java:274)
at org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.callBoundary(OptimizedCallTarget.java:264)
at org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.doInvoke(OptimizedCallTarget.java:246)
at org.graalvm.compiler.truffle.runtime.GraalTVMCI.callProfiled(GraalTVMCI.java:86)
at com.oracle.truffle.api.impl.Accessor.callProfiled(Accessor.java:718)
at com.oracle.truffle.polyglot.VMAccessor.callProfiled(VMAccessor.java:91)
at com.oracle.truffle.polyglot.PolyglotValue$Interop.execute(PolyglotValue.java:1388)
at org.graalvm.polyglot.Value.execute(Value.java:338)
at HelloPolyglot.main(HelloPolyglot.java:54)
Caused by: Attached Guest Language Frames (4)

[Interop] Foreign objects do not understand getattribute/str/repr/...

Example:

GraalVM MultiLanguage Shell 1.0.0-rc10
Copyright (c) 2013-2018, Oracle and/or its affiliates
  JavaScript version 1.0
  Python version 3.7.0
js> Polyglot.eval('python', 'lambda x: x.__repr__()')(Object.create(null))
NameError: Unknown identifier: __repr__
        at <python> anonymous(<eval>:1:10-21)
        at <js> :program(<shell>:1:49-69)

It looks like TruffleObjectBuiltins is missing an implementation for __GETATTRIBUTE__.

Related: oracle/graaljs#101

`typing` is broken

import typing

raises

rc5 release makes lib-python files not world readable

Hi! I publish the graalpython-bin AUR package, and while updating it to rc5, I noticed that using graalpython didn’t completely work anymore:

$ graalpython 
Please note: This Python implementation is in the very early stages, and can run little more than basic benchmarks at this point.
org.graalvm.launcher.Launcher$AbortException
Caused by: java.lang.RuntimeException: error while retrieving prompt
        at java.lang.Throwable.<init>(Throwable.java:287)
        at java.lang.Exception.<init>(Exception.java:84)
        at java.lang.RuntimeException.<init>(RuntimeException.java:80)
        at com.oracle.graal.python.shell.GraalPythonMain.getPrompt(GraalPythonMain.java:605)
        at com.oracle.graal.python.shell.GraalPythonMain.readEvalPrint(GraalPythonMain.java:516)
        at com.oracle.graal.python.shell.GraalPythonMain.launch(GraalPythonMain.java:199)
        at org.graalvm.launcher.AbstractLanguageLauncher.launch(AbstractLanguageLauncher.java:107)
        at org.graalvm.launcher.AbstractLanguageLauncher.launch(AbstractLanguageLauncher.java:54)
        at com.oracle.graal.python.shell.GraalPythonMain.main(GraalPythonMain.java:56)
        at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:177)
Caused by: org.graalvm.polyglot.PolyglotException: OSError: /usr/lib/jvm/java-8-graal/jre/languages/python/lib-python/3/_pyio.py
        at <python>.open(Unknown)
        at <python>.FileIO.__init__(_io:345)
        at <python>.FileLoader.get_data(_bootstrap_external.py:841)
        at <python>.SourceLoader.get_code(_bootstrap_external.py:789)
        at <python>._LoaderBasics.exec_module(_bootstrap_external.py:682)
        at org.graalvm.polyglot.Context.eval(Context.java:313)
        at com.oracle.graal.python.shell.GraalPythonMain.getPrompt(GraalPythonMain.java:600)
        at com.oracle.graal.python.shell.GraalPythonMain.readEvalPrint(GraalPythonMain.java:516)
        at com.oracle.graal.python.shell.GraalPythonMain.launch(GraalPythonMain.java:199)
        at org.graalvm.launcher.AbstractLanguageLauncher.launch(AbstractLanguageLauncher.java:107)
        at org.graalvm.launcher.AbstractLanguageLauncher.launch(AbstractLanguageLauncher.java:54)
        at com.oracle.graal.python.shell.GraalPythonMain.main(GraalPythonMain.java:56)
        at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:177)
        at <python>._load_unlocked(_bootstrap.py:662)
        at <python>._find_and_load_unlocked(_bootstrap.py:957)
        at <python>._find_and_load(_bootstrap.py:968)
        at <python>._gcd_import(_bootstrap.py:985)
        at <python>.__import__(_bootstrap.py:1066)
        at <python>.<module '__builtins_patches__'>(__builtins_patches__:48)

$ Rscript --polyglot -e 'eval.polyglot("python", "1")'
Error in eval.polyglot("python", "1") :
  Error while parsing: ImportError: /usr/lib/jvm/java-8-graal/jre/languages/python/lib-python/3/importlib/_bootstrap_external.py

As part of the installation process, the package applies the META-INF/permissions file found in the jar, and it seems that in the rc5 jar, that file marks everything in lib-python as not readable by anyone other than root:

…
jre/languages/python/include/object.h = rw-rw-r--
jre/languages/python/include/pyctype.h = rw-rw-r--
jre/languages/python/include/pyconfig.h = rw-rw-r--
jre/languages/python/lib-python/3/sre_constants.py = rw-------
jre/languages/python/lib-python/3/symbol.py = rw-------
jre/languages/python/lib-python/3/abc.py = rw-------
jre/languages/python/lib-python/3/_sitebuiltins.py = rw-------
…

root can still run graalpython or run polyglot Python from other languages, but other users are no longer allowed to read the files with graalpython requires. In rc4 this was not a problem: there, the permissions file listed lib-python as world-readable (mode rw-rw-r--).

cannot `from socket import error`

  File "~/graalvm-ce-1.0.0-rc11/jre/languages/python/lib-python/3/site-packages/pip/_vendor/urllib3/connectionpool.py", line 7, in <module>
    from socket import error as SocketError, timeout as SocketTimeout
ImportError: cannot import name 'error'

NameError: name 'basestring' is not defined

Repro via polyglot shell:

GraalVM MultiLanguage Shell 1.0.0-rc10
Copyright (c) 2013-2018, Oracle and/or its affiliates
  JavaScript version 1.0
  Python version 3.7.0
  R version 3.4.0
  Ruby version 2.4.4
Usage:
  Use Ctrl+L to switch language and Ctrl+D to exit.
  Enter -usage to get a list of available commands.
python> isinstance('foo', basestring)
NameError: name 'basestring' is not defined

Expected result: True.

[Python-Java-Interop] default type conversions

Context context=Context.newBuilder().allowAllAccess(true).build();
Value value=context.eval("python","[\"1\",2,3]");
Object obj=value.as(Object.class);List list=value.as(List.class);
System.out.println(obj.getClass()+": "+((Map)obj).size());
System.out.println(list.getClass()+": "+list.size());

as the code above shown, ["1",2,3] is a Python <class 'list'>
however when converted to Java object, it became a PolyglotMap(with zero size)
instead of Object[] or List

is this behavior intended? if so, is there any way to customize it to return Object[] or List?
when used like this: JavaMap getMap(){...}; Pythondict=getMap();dict.put("key",["1",2,3])
it corrupts the data inside the map values

and another related Java-Python-Interop question:

context.getBindings("python").putMember("getMap",(Supplier<Map>)HashMap::new);
try{context.eval("python","dict=getMap();dict[\"a\"]");}catch(Throwable e){e.printStackTrace();}
try{context.eval("python","dict=getMap();dict[\"a\"]=1");}catch(Throwable e){e.printStackTrace();}

is there a way to convert Java Map to PolyglotMap so we can use it like a dict in Python?
currently I'm recursively wrapping them to ProxyObject, but it's cumbersome and pollutes map values

KeyError: invalid key a
	at <python> <module 'Unnamed'>(Unnamed:1:14-22)
	at org.graalvm.polyglot.Context.eval(Context.java:340)
	at BugReport.main(BugReport.java:21)
AttributeError: JavaObject[{} (java.util.HashMap)] instance has no attribute '__setitem__'
	at <python> <module 'Unnamed'>(Unknown)
	at org.graalvm.polyglot.Context.eval(Context.java:340)
	at BugReport.main(BugReport.java:22)