yglukhov / nimpy Goto Github PK

View Code? Open in Web Editor NEW

1.4K 30.0 61.0 293 KB

Nim - Python bridge

License: MIT License

Nim 93.14% Python 6.86%

nimpy's Introduction

nimpy

Native language integration with Python has never been easier!

Implementing a Python Module in Nim

# mymodule.nim - file name should match the module name you're going to import from python
import nimpy

proc greet(name: string): string {.exportpy.} =
  return "Hello, " & name & "!"

# Compile on Windows:
nim c --app:lib --out:mymodule.pyd --threads:on --tlsEmulation:off --passL:-static mymodule
# Compile on everything else:
nim c --app:lib --out:mymodule.so --threads:on mymodule

# test.py
import mymodule
assert mymodule.greet("world") == "Hello, world!"
assert mymodule.greet(name="world") == "Hello, world!"

Calling Python From Nim

import nimpy
let os = pyImport("os")
echo "Current dir is: ", os.getcwd().to(string)

# sum(range(1, 5))
let py = pyBuiltinsModule()
let s = py.sum(py.range(0, 5)).to(int)
assert s == 10

Note: here nimpy relies on your local python installation.

Importing Nim Extensions Directly

For a convenient way to import your Nim extension modules directly, you can use Nimporter.

Misc

The library is designed with ABI compatibility in mind. That is the compiled module doesn't depend on particular Python version, it should properly work with any. The C API symbols are loaded in runtime from whichever process has launched your module.

Troubleshooting, Q&A

Importing the compiled module from Python fails

If you're getting ImportError: dynamic module does not define module export function ... make sure that the module you're importing from Python has exactly the same name as the nim file which the module is implemented in.

Nimpy fails to find (proper) libpython

The most reliable way to find libpython is find_libpython python package:

pip3 install find_libpython
python3 -c 'import find_libpython; print(find_libpython.find_libpython())'

Then you can specify path to libpython using nimpy.py_lib.pyInitLibPath. Tracking issue: #171.

Nim strings are converted to Python bytes instead of string

nimpy converts Nim strings to Python strings usually, but since Nim strings are encoding agnostic and may contain invalid utf8 sequences, nimpy will fallback to Python bytes in such cases.

Is there any numpy compatibility?

nimpy allows manipulating numpy objects just how you would do it in Python, however it is not much more efficient. scinim offers API for performance critical numpy interop, and it is advised to consider it first.

Nimpy also exposes lower level Buffer protocol, see raw_buffers.nim. tpyfromnim.nim contains a very basic test for this.

Does nim default garbage collector (GC) and ARC/ORC work?

Yes. nimpy internally does everything needed to run the GC properly (keeps the stack bottom actual, and appropriate nim references alive), and doesn't introduce any special rules on top. So the GC question boils down to proper GC usage in nim shared libraries, you'd better lookup elsewhere. The following guidelines are by no means comprehensive, but should be enough for the quick start:

If it's known there will be only one nimpy module in the process, you should be fine.
If there is more than one nimpy module, it is recommended to move nim runtime out to a separate shared library. However it might not be needed if nim references are known to never travel between nim shared libraries.
If you hit any GC problems with nimpy, whether you followed these guidelines or not, please report them to nimpy tracker :)

Windows, threads and MinGW

When compiling with --threads:on Nim will imply --tlsEmulation:on (Windows only) which prevents Nim runtime from initing properly when being called from a foreign thread (which is always the case in case of Python module).

Adding --tlsEmulation:off when using MinGW toolchain (Nim's default on Windows) will introduce a dependency on libgcc_s_seh-*.dll, that newer python versions are often unable to find.

One way to overcome this is to link with libgcc statically, by passing -static to linker, or --passL:-static to Nim.

Exporting Nim types as Python classes

Warning! This is experimental.

An exported type should be a ref object and inherit PyNimObjectExperimental directly or indirectly.
The type will only be exported if at least one exported "method" is defined.
A proc will be exported as python type method only if it's first argument is of the corresponding type and is called self. If the first argument is not called self, the proc will exported as a global module function.

# mymodule.nim
type TestType = ref object of PyNimObjectExperimental
  myField: string

proc setMyField(self: TestType, value: string) {.exportpy.} =
  self.myField = value

proc getMyField(self: TestType): string {.exportpy.} =
  self.myField

# test.py
import mymodule
tt = mymodule.TestType()
tt.setMyField("Hello")
assert(tt.getMyField() == "Hello")

Future directions

High level buffer API

Stargazers over time

nimpy's People

Contributors

Stargazers

Watchers

nimpy's Issues

garbage collectors

I read about nimpy here: https://robert-mcdermott.gitlab.io/posts/speeding-up-python-with-nim/ . At the bottom of that page there is this remark:

"UPDATE 2019-01-23: On the Nim forum there was a comment about a potential problem with the Nim and Python garbage collectors interfering with each other. The creator of Nim, Andreas Rumpf, responded that compiling with the “–gc:regions” flag when creating an extension module will eliminate this risk. I’ve updated the article to include this detail."

If it's true, then "--gc:regions" should be added to the README.

numpy array integration [enhacement suggestion]

Hi,

I recently discovered the package after failing to get pymod working with nim 18.0.
I am liking the simplicity of the nimpy but do find myself missing integration with Numpy arrays.

Is this something that you're planning on integrating?
I see a strong usecase for Nim in numerical computing / data science but integration with python/numpy is quite important.

I would be willing to help where I can but I am not very experienced in Nim yet.

Thank you for the work so far!

Type Mismatch when trying to pass seq[int] to Python

Hi,
I would like to pass a sequence to Python after processing. I wrote a simple nim using one of the expressions in your nimfrompy.nim file

import nimpy
proc reverseArray(a: seq[int]): seq[int] {.exportpy.} = a.reversed()

When I compile it with

nim c --app:lib --out:nimpy_test.so nimpy_test

I get
nimpy_test.nim(7, 58) Error: type mismatch: got <seq[int]>
but expected one of:
template .()(o: PyObject; field: untyped; args: varargs[untyped]): PyObject
template .(o: PyObject; field: untyped): PyObject

expression: reversed(a)

What can be the problem?
Thanks
Zafer

Is there a way to use tuples?

When I try to do something like this:

proc simpleDifference(a, b: tuple[x, y: float]): tuple[x, y: float] {.exportpy.} =
  result = (a.x - b.x, a.y - b.y)

I get:

Error: type mismatch: got <tuple[x: float, y: float]>
but expected one of:
proc unknownTypeCompileError()

expression: unknownTypeCompileError(v)

What am I doing wrong?
Can I use Nim's tuples and/or interact with Python's tuples?

Documentation build failure on nimble.directory CI service

https://nimble.directory/ci/badges/nimpy/nimdevel/doc_build_output.html

Build output from nimpy

Built on 2018-08-29T08:39:22+00:00

/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim

nim doc --index:on /var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim

Hint: used config file '/etc/nim.cfg' [Conf]
Hint: used config file '/etc/nimdoc.cfg' [Conf]
Hint: system [Processing]
Hint: nimpy [Processing]
Hint: dynlib [Processing]
Hint: strutils [Processing]
Hint: parseutils [Processing]
Hint: math [Processing]
Hint: algorithm [Processing]
Hint: macros [Processing]
Hint: ospaths [Processing]
Hint: complex [Processing]
Hint: sequtils [Processing]
Hint: typetraits [Processing]
Hint: tables [Processing]
Hint: hashes [Processing]
nimpy.nim(1160, 9) Hint: 'modulename' is declared but not used [XDeclaredButNotUsed]
nimpy.nim(1329, 32) Error: undeclared identifier: 'nnkTupleConstr'
nimpy.nim(909, 14) Hint: 'conversionTypeCheck' is declared but not used [XDeclaredButNotUsed]
nimpy.nim(906, 14) Hint: 'conversionTypeCheck' is declared but not used [XDeclaredButNotUsed]
nimpy.nim(909, 14) Hint: 'conversionTypeCheck' is declared but not used [XDeclaredButNotUsed]
nimpy.nim(445, 5) Hint: 'Py_TPFLAGS_TUPLE_SUBCLASS' is declared but not used [XDeclaredButNotUsed]
nimpy.nim(433, 5) Hint: 'Py_TPFLAGS_VALID_VERSION_TAG' is declared but not used [XDeclaredButNotUsed]
nimpy.nim(446, 5) Hint: 'Py_TPFLAGS_STRING_SUBCLASS' is declared but not used [XDeclaredButNotUsed]
nimpy.nim(447, 5) Hint: 'Py_TPFLAGS_UNICODE_SUBCLASS' is declared but not used [XDeclaredButNotUsed]
nimpy.nim(416, 5) Hint: 'Py_TPFLAGS_READY' is declared but not used [XDeclaredButNotUsed]
nimpy.nim(449, 5) Hint: 'Py_TPFLAGS_BASE_EXC_SUBCLASS' is declared but not used [XDeclaredButNotUsed]
nimpy.nim(419, 5) Hint: 'Py_TPFLAGS_READYING' is declared but not used [XDeclaredButNotUsed]
nimpy.nim(436, 5) Hint: 'Py_TPFLAGS_IS_ABSTRACT' is declared but not used [XDeclaredButNotUsed]
nimpy.nim(413, 5) Hint: 'Py_TPFLAGS_BASETYPE' is declared but not used [XDeclaredButNotUsed]
nimpy.nim(450, 5) Hint: 'Py_TPFLAGS_TYPE_SUBCLASS' is declared but not used [XDeclaredButNotUsed]
nimpy.nim(462, 5) Hint: 'Py_TPFLAGS_DEFAULT_CORE' is declared but not used [XDeclaredButNotUsed]
nimpy.nim(395, 5) Hint: 'Py_TPFLAGS_CHECKTYPES' is declared but not used [XDeclaredButNotUsed]
nimpy.nim(448, 5) Hint: 'Py_TPFLAGS_DICT_SUBCLASS' is declared but not used [XDeclaredButNotUsed]
nimpy.nim(410, 5) Hint: 'Py_TPFLAGS_HEAPTYPE' is declared but not used [XDeclaredButNotUsed]
nimpy.nim(444, 5) Hint: 'Py_TPFLAGS_LIST_SUBCLASS' is declared but not used [XDeclaredButNotUsed]
nimpy.nim(422, 5) Hint: 'Py_TPFLAGS_HAVE_GC' is declared but not used [XDeclaredButNotUsed]
nimpy.nim(439, 5) Hint: 'Py_TPFLAGS_HAVE_NEWBUFFER' is declared but not used [XDeclaredButNotUsed]

/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/tests/tpyfromnim.nim

nim doc --index:on /var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/tests/tpyfromnim.nim

Hint: used config file '/etc/nim.cfg' [Conf]
Hint: used config file '/etc/nimdoc.cfg' [Conf]
Hint: system [Processing]
Hint: tpyfromnim [Processing]
Hint: nimpy [Processing]
Hint: dynlib [Processing]
Hint: strutils [Processing]
Hint: parseutils [Processing]
Hint: math [Processing]
Hint: algorithm [Processing]
Hint: macros [Processing]
Hint: ospaths [Processing]
Hint: complex [Processing]
Hint: sequtils [Processing]
Hint: typetraits [Processing]
Hint: tables [Processing]
Hint: hashes [Processing]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(1160, 9) Hint: 'modulename' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(1329, 32) Error: undeclared identifier: 'nnkTupleConstr'
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(909, 14) Hint: 'conversionTypeCheck' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(906, 14) Hint: 'conversionTypeCheck' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(909, 14) Hint: 'conversionTypeCheck' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(445, 5) Hint: 'Py_TPFLAGS_TUPLE_SUBCLASS' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(433, 5) Hint: 'Py_TPFLAGS_VALID_VERSION_TAG' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(446, 5) Hint: 'Py_TPFLAGS_STRING_SUBCLASS' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(447, 5) Hint: 'Py_TPFLAGS_UNICODE_SUBCLASS' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(416, 5) Hint: 'Py_TPFLAGS_READY' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(449, 5) Hint: 'Py_TPFLAGS_BASE_EXC_SUBCLASS' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(419, 5) Hint: 'Py_TPFLAGS_READYING' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(436, 5) Hint: 'Py_TPFLAGS_IS_ABSTRACT' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(413, 5) Hint: 'Py_TPFLAGS_BASETYPE' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(450, 5) Hint: 'Py_TPFLAGS_TYPE_SUBCLASS' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(462, 5) Hint: 'Py_TPFLAGS_DEFAULT_CORE' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(395, 5) Hint: 'Py_TPFLAGS_CHECKTYPES' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(448, 5) Hint: 'Py_TPFLAGS_DICT_SUBCLASS' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(410, 5) Hint: 'Py_TPFLAGS_HEAPTYPE' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(444, 5) Hint: 'Py_TPFLAGS_LIST_SUBCLASS' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(422, 5) Hint: 'Py_TPFLAGS_HAVE_GC' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(439, 5) Hint: 'Py_TPFLAGS_HAVE_NEWBUFFER' is declared but not used [XDeclaredButNotUsed]
Hint: os [Processing]
Hint: times [Processing]
Hint: posix [Processing]
tpyfromnim.nim(5, 15) template/generic instantiation from here

/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/tests/nimfrompy.nim

nim doc --index:on /var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/tests/nimfrompy.nim

Hint: used config file '/etc/nim.cfg' [Conf]
Hint: used config file '/etc/nimdoc.cfg' [Conf]
Hint: system [Processing]
Hint: nimfrompy [Processing]
Hint: nimpy [Processing]
Hint: dynlib [Processing]
Hint: strutils [Processing]
Hint: parseutils [Processing]
Hint: math [Processing]
Hint: algorithm [Processing]
Hint: macros [Processing]
Hint: ospaths [Processing]
Hint: complex [Processing]
Hint: sequtils [Processing]
Hint: typetraits [Processing]
Hint: tables [Processing]
Hint: hashes [Processing]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(1160, 9) Hint: 'modulename' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(1329, 32) Error: undeclared identifier: 'nnkTupleConstr'
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(909, 14) Hint: 'conversionTypeCheck' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(906, 14) Hint: 'conversionTypeCheck' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(909, 14) Hint: 'conversionTypeCheck' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(445, 5) Hint: 'Py_TPFLAGS_TUPLE_SUBCLASS' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(433, 5) Hint: 'Py_TPFLAGS_VALID_VERSION_TAG' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(446, 5) Hint: 'Py_TPFLAGS_STRING_SUBCLASS' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(447, 5) Hint: 'Py_TPFLAGS_UNICODE_SUBCLASS' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(416, 5) Hint: 'Py_TPFLAGS_READY' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(449, 5) Hint: 'Py_TPFLAGS_BASE_EXC_SUBCLASS' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(419, 5) Hint: 'Py_TPFLAGS_READYING' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(436, 5) Hint: 'Py_TPFLAGS_IS_ABSTRACT' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(413, 5) Hint: 'Py_TPFLAGS_BASETYPE' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(450, 5) Hint: 'Py_TPFLAGS_TYPE_SUBCLASS' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(462, 5) Hint: 'Py_TPFLAGS_DEFAULT_CORE' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(395, 5) Hint: 'Py_TPFLAGS_CHECKTYPES' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(448, 5) Hint: 'Py_TPFLAGS_DICT_SUBCLASS' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(410, 5) Hint: 'Py_TPFLAGS_HEAPTYPE' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(444, 5) Hint: 'Py_TPFLAGS_LIST_SUBCLASS' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(422, 5) Hint: 'Py_TPFLAGS_HAVE_GC' is declared but not used [XDeclaredButNotUsed]
/var/lib/nim_package_directory/cache/nimpy/pkgs/nimpy-0.1.0/nimpy.nim(439, 5) Hint: 'Py_TPFLAGS_HAVE_NEWBUFFER' is declared but not used [XDeclaredButNotUsed]
Hint: tpyfromnim [Processing]
Hint: os [Processing]
Hint: times [Processing]
Hint: posix [Processing]
tpyfromnim.nim(5, 15) template/generic instantiation from here

`nimble test` fails on mac:

I'm on OSX
python --version
Python 2.7.15
python -c 'import matplotlib; print(matplotlib.__version__)'
2.2.2
which python
/Users/timothee/homebrew//bin/python

possibly related to this: timotheecour/vitanim#5 (comment) /cc @Vindaar

$nim_D/nimpy master 2.900 $ nimble test
   Warning: Using env var NIM_LIB_PREFIX: /Users/timothee/git_clone//nim//Nim/
  Executing task test in /Users/timothee/git_clone/nim/nimpy/nimpy.nimble
Hint: used config file '/Users/timothee/git_clone/nim/Nim/config/nim.cfg' [Conf]
Hint: used config file '/Users/timothee/.config/nim/nim.cfg' [Conf]
Hint: used config file '/Users/timothee/git_clone/nim/Nim/config/config.nims' [Conf]
/Users/timothee/git_clone/nim/nimpy/tests/nimfrompy.nim(1, 1) template/generic instantiation from here
/Users/timothee/git_clone/nim/nimpy/nimpy.nim(1, 24) Warning: import os.nim instead; ospaths is deprecated [Deprecated]
/Users/timothee/git_clone/nim/nimpy/tests/nimfrompy.nim(1, 1) template/generic instantiation from here
/Users/timothee/git_clone/nim/nimpy/nimpy.nim(10, 30) Warning: type pragmas follow the type name; this form of writing pragmas is deprecated [Deprecated]
/Users/timothee/git_clone/nim/nimpy/tests/nimfrompy.nim(1, 1) template/generic instantiation from here
/Users/timothee/git_clone/nim/nimpy/nimpy.nim(805, 34) Warning: type pragmas follow the type name; this form of writing pragmas is deprecated [Deprecated]
Hint:  [Link]
Hint: operation successful (58055 lines compiled; 1.496 sec total; 91.98MiB peakmem; Debug Build) [SuccessX]
Hint: used config file '/Users/timothee/git_clone/nim/Nim/config/nim.cfg' [Conf]
Hint: used config file '/Users/timothee/.config/nim/nim.cfg' [Conf]
Hint: used config file '/Users/timothee/git_clone/nim/Nim/config/config.nims' [Conf]
/Users/timothee/git_clone/nim/nimpy/tests/custommodulename.nim(1, 1) template/generic instantiation from here
/Users/timothee/git_clone/nim/nimpy/nimpy.nim(1, 24) Warning: import os.nim instead; ospaths is deprecated [Deprecated]
/Users/timothee/git_clone/nim/nimpy/tests/custommodulename.nim(1, 1) template/generic instantiation from here
/Users/timothee/git_clone/nim/nimpy/nimpy.nim(10, 30) Warning: type pragmas follow the type name; this form of writing pragmas is deprecated [Deprecated]
/Users/timothee/git_clone/nim/nimpy/tests/custommodulename.nim(1, 1) template/generic instantiation from here
/Users/timothee/git_clone/nim/nimpy/nimpy.nim(805, 34) Warning: type pragmas follow the type name; this form of writing pragmas is deprecated [Deprecated]
Hint:  [Link]
Hint: operation successful (57710 lines compiled; 1.042 sec total; 73.898MiB peakmem; Debug Build) [SuccessX]
Tests complete!
Tests complete!
Hint: used config file '/Users/timothee/git_clone/nim/Nim/config/nim.cfg' [Conf]
Hint: used config file '/Users/timothee/.config/nim/nim.cfg' [Conf]
Hint: used config file '/Users/timothee/git_clone/nim/Nim/config/config.nims' [Conf]
/Users/timothee/git_clone/nim/nimpy/tests/tpyfromnim.nim(1, 1) template/generic instantiation from here
/Users/timothee/git_clone/nim/nimpy/nimpy.nim(1, 24) Warning: import os.nim instead; ospaths is deprecated [Deprecated]
/Users/timothee/git_clone/nim/nimpy/tests/tpyfromnim.nim(1, 1) template/generic instantiation from here
/Users/timothee/git_clone/nim/nimpy/nimpy.nim(10, 30) Warning: type pragmas follow the type name; this form of writing pragmas is deprecated [Deprecated]
/Users/timothee/git_clone/nim/nimpy/tests/tpyfromnim.nim(1, 1) template/generic instantiation from here
/Users/timothee/git_clone/nim/nimpy/nimpy.nim(805, 34) Warning: type pragmas follow the type name; this form of writing pragmas is deprecated [Deprecated]
Hint:  [Link]
Hint: operation successful (57945 lines compiled; 1.226 sec total; 73.891MiB peakmem; Debug Build) [SuccessX]
Hint: /Users/timothee/git_clone/nim/nimpy/tests/tpyfromnim  [Exec]
/Users/timothee/git_clone/nim/nimpy/tests/tpyfromnim.nim(199) tpyfromnim
/Users/timothee/git_clone/nim/nimpy/tests/tpyfromnim.nim(62) test
/Users/timothee/git_clone/nim/nimpy/nimpy.nim(1018) pyImport
/Users/timothee/git_clone/nim/nimpy/nimpy/py_utils.nim(98) raisePythonError
Error: unhandled exception: <type 'exceptions.ImportError'>: cannot import name _remove_dead_weakref [Exception]
Error: execution of an external program failed: '/Users/timothee/git_clone/nim/nimpy/tests/tpyfromnim '
stack trace: (most recent call last)
/Users/timothee/git_clone/nim/nimpy/nimpy.nimble(34, 18) testTask
/Users/timothee/git_clone/nim/Nim/lib/system/nimscript.nim(241, 7) exec
/Users/timothee/git_clone/nim/Nim/lib/system/nimscript.nim(241, 7) Error: unhandled exception: FAILED: nim c -r tests/tpyfromnim.nim

same error after:

pip install --upgrade matplotlib
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Requirement already up-to-date: matplotlib in /Users/timothee/homebrew/lib/python2.7/site-packages (2.2.3)
Requirement already satisfied, skipping upgrade: kiwisolver>=1.0.1 in /Users/timothee/homebrew/lib/python2.7/site-packages (from matplotlib) (1.0.1)
Requirement already satisfied, skipping upgrade: numpy>=1.7.1 in /Users/timothee/homebrew/lib/python2.7/site-packages (from matplotlib) (1.15.4)
Requirement already satisfied, skipping upgrade: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /Users/timothee/homebrew/lib/python2.7/site-packages (from matplotlib) (2.2.0)
Requirement already satisfied, skipping upgrade: backports.functools-lru-cache in /Users/timothee/homebrew/lib/python2.7/site-packages (from matplotlib) (1.5)
Requirement already satisfied, skipping upgrade: six>=1.10 in /Users/timothee/homebrew/lib/python2.7/site-packages (from matplotlib) (1.11.0)
Requirement already satisfied, skipping upgrade: pytz in /Users/timothee/homebrew/lib/python2.7/site-packages (from matplotlib) (2018.4)
Requirement already satisfied, skipping upgrade: subprocess32 in /Users/timothee/homebrew/lib/python2.7/site-packages (from matplotlib) (3.5.1)
Requirement already satisfied, skipping upgrade: cycler>=0.10 in /Users/timothee/homebrew/lib/python2.7/site-packages (from matplotlib) (0.10.0)
Requirement already satisfied, skipping upgrade: python-dateutil>=2.1 in /Users/timothee/homebrew/lib/python2.7/site-packages (from matplotlib) (2.7.3)
Requirement already satisfied, skipping upgrade: setuptools in /Users/timothee/homebrew/lib/python2.7/site-packages (from kiwisolver>=1.0.1->matplotlib) (39.2.0)

diagnostic

let o = pyLib.PyImport_ImportModule(moduleName) fails

Error: Expected a node of kind nnkProcDef, got nnkFuncDef

Nim:

import nimpy

func foo(): int {.exportpy.} =
  42

Bash:

nim c -d:lib --out:module.so module.nim

Error:

stack trace: (most recent call last)
../../.nimble/pkgs/nimpy-0.1.0/nimpy.nim(879) exportpy
../../.choosenim/toolchains/nim-0.19.0/lib/core/macros.nim(523) expectKind
foo.nim(3, 1) Error: Expected a node of kind nnkProcDef, got nnkFuncDef

🙂

Support nim default arguments

This fails to compile at the moment.

import strformat
import nimpy

proc greet(person: string, greeting = "Hello"): string {. exportpy .} =
    return &"{greeting} {person}"

It throws the following error:

test.nim(4, 28) Error: type mismatch: got <void>
but expected one of:
template valueTypeForArgType(t: typedesc): typedesc

expression: valueTypeForArgType()

Also, the following does not allow using the default argument in Python.

import strformat
import nimpy

proc greet(person: string, greeting: string = "Hello"): string {. exportpy .} =
    return &"{greeting} {person}"

The following throws an error:

In [1]: import test

In [2]: test.greet("Jane")
--------------------------------------------------------------------
TypeError                          Traceback (most recent call last)
<ipython-input-2-315d7f41b8d6> in <module>()
----> 1 test.greet("Jane")

TypeError: greet() takes exactly 2 arguments (1 given)

Numpy interop

Thanks for starting what looks like a very promising project. I very much believe that python owes much of its popularity by its easy of integration with a then dominant language, C. And for nims sake easy interop with the currently dominant python ecosystem seems vital to me!

Is there planned support for passing numpy arrays between nim and python?

Note that this is far more important to many users in practice (and I suspect far simpler to implement) than marshalling arbitrary datatypes back and forth. Typically you keep your interfaces between languages small anyway; but that does not mean you dont want to pass lots of data around.

From what it seems nim does not have much in the way of an ndarray object yet; but for instance even being able to pass an [n, 3] ndarray to a nim array of xyz points, and vice versa, with autogenerated runtime checks that this is a valid operation, and proper handling of memory management, would be a killer feature to me.

Mapping nim names to different python names

I am trying to create a nim exe and python module with the same name. I want to expose snake case names in the python module and camel case in the nim code. I also want the python module to contain a __version__ attribute.

I've tried many different ways to achieve this but I haven't found something that works. I made a simple example to show what I am trying to do.

If the exportpy pragma supported symbol renaming like the exportc pragma does, I think that would solve my case. Do you have plans to support this?

The commented out lines so what I want to do.

$ cat t.nim

import nimpy

# const version {.exportpy: "__version__".} = "1.2.3"
const version = "1.2.3"

# proc fooBar*(): string {.exportpy: "foo_bar".} = 
#   result = "fooBar"

proc fooBar*(): string {.exportpy.} = 
  result = "fooBar"
  
when not defined(buidingLib):
  when isMainModule:
    echo fooBar()

$ cat test.py

import t
# print t.__version__
# print t.foo_bar()
print t.fooBar()

$ cat t.nimble 

# Package

version       = "1.2.3"
author        = "Steve Flenniken"
description   = "Example nim exe and python module with the same name."
license       = "MIT"

# Dependencies

requires "nim >= 0.17.2"

task b, "Build exe and python module":
  exec "nim c -d:buidingLib --threads:on --tlsEmulation:off --app:lib --out:t.so t"
  exec "nim c -r t"
  exec "python test.py"

Windows support?

Hi,
I tried out the example on Windows, but i get:

ModuleNotFoundError: No module named 'mymodule'

I tried compiling with the output as .dll or .pyd (like cython), or bootstrapping the .so myself, then i get:

ImportError: dynamic module does not define module export function (PyInit_mymodule)

I guess this is a windows issue as .dll files need a dllexport instruction. Can that be added as an option for windows compatibility?

stack bottom related crashes

Sometimes CC merges the stack frames of an exported function and its pywrapper resulting in a wrong stack bottom deduction. The following hacks are considered:

increase tolerance even more. Not really reliable, and to what extent?
prevent inlining with noinline. potentially can be ignored by CC.
prevent inlining with volatile fn trick as nim does for NimMain. Likely the most reliable way.

Really don't want to prevent inlining though...

Crash on exit in py 3.7.1 on MacOS

@yglukhov - I have a bug report from a user who's reporting a crash. For reference, this is from a nvim plugin that loads a python module written in nim using nimpy.

raghur/fruzzy#16

I don't have any other similar reports and I'm not sure there's much I can do either. There's a crash log from the user - https://pastebin.com/eRaqjzSR

I'm hoping you can look at the crash log and make some sense out of it.

On a separate note - do you (or others here) have nim modules on py3.7 on a mac behave similarly?

Error: invalid pragma: exportpy, with hello world

I'm getting an error when trying out hello world. I must have something setup wrong. Here is what I did.

I installed it with nimble.

nimble install python
nimble list -i

docopt  [0.6.5]
python  [1.2]

I created a file called mymodule.nim with this:

import python
proc greet(name: string): string {.exportpy.} =
  return "Hello, " & name & "!"

I compiled the module with this:

nim c --threads:on --tlsEmulation:off --app:lib --out:mymodule.so mymodule

I get the error: Error: invalid pragma: exportpy

$ nim c --threads:on --tlsEmulation:off --app:lib --out:mymodule.so mymodule
Hint: used config file '/usr/local/Cellar/nim/0.17.2/nim/config/nim.cfg' [Conf]
Hint: system [Processing]
Hint: mymodule [Processing]
Hint: python [Processing]
Hint: dynlib [Processing]
Hint: strutils [Processing]
Hint: parseutils [Processing]
Hint: math [Processing]
Hint: algorithm [Processing]
mymodule.nim(2, 36) Error: invalid pragma: exportpy

using using

Nim:

import nimpy

using argument: string

func foo(argument): int {.exportpy.} =
  42

Bash:

nim c -d:lib --out:module.so module.nim

Error:

stack trace: (most recent call last)
../../../.nimble/pkgs/nimpy-0.1.0/nimpy.nim(689, 15) template/generic instantiation from here
../../../.nimble/pkgs/nimpy-0.1.0/nimpy.nim(400, 32) Error: type mismatch: got <>
but expected one of:
proc unknownTypeCompileError()
expression: unknownTypeCompileError(v)

I dont know if its a limitation that this module will have, or is just a Bug,
whatsoever is not Documented, its not like super important but still a comfy Nim feature.
If it will be a well known limitation of this module, just mention it on the Docs and close this bug.
Thanks for the great work ❕
🙂

Passing data back to Python

In my use case, I have a function that creates some kind of object that eventually needs to be passed back to python. There is no need be able to do anything with in python, except pass it back to other nim functions. Something like this:

# module.nim
import nimpy

type MyObject = object
  a, b, c: int
  d: string

proc makeMyObject(a, b, c: int, d: string): MyObject {.exportpy.} =
  MyObject(a: a, b: b, c: c, d: d)

proc doSomethingWithMyObject(obj: MyObject) {.exportpy.} =
  echo obj.d

import module

x = module.makeMyObject(1, 2, 3, "hi")
module.doSomethingWithMyObject(x)

Unfortunately trying to compile the nim file gives

usage of makeMyObjectPy_wrapper is a user-defined error

I just need to return the result as an opaque type from the Python point of view. Is there a way to do something like that?

Raise equivalent Nim-Exception

It would be helpful to have the equivalent nim exception raised like the python one if there is an equivalent (like IOError, ValueError,...) and fallback to a generic Exception.
I saw the Exception type is also delivered from the python side (raise newException(Exception, typns & ": " & valns), typns here) so that would be possible with a small case, but of course low prio.

`eval` doesn't seem to work

If the following code is the correct way to call eval, it doesn't work (Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 17:00:18) [MSC v.1900 64 bit (AMD64)] on win32). Not really use this, but maybe some Python modules do.

import nimpy

let py = pyBuiltinsModule()
echo py.eval("3+3").to(int)

nimpy.nim(1185)          callMethod
nimpy.nim(1182)          callMethodAux
nimpy.nim(1165)          raisePythonError
Error: unhandled exception: <class 'SystemError'>: frame does not exist [Exception]

[question] Python dependency?

Take this example:

import nimpy
let os = pyImport("os")
echo "Current dir is: ", os.getcwd().to(string)

If I use it in my code, will my binary depend on a Python installation? The big advantage of Nim is that it produces stand-alone executables. What happens with the stand-alone status if we use the code snippet above? The answer should be incorporated in the README. Thanks.

unhandled exception: Symbol not loaded: PyObject_GetBuffer [Exception]

I use your code

import nimpy
let os = pyImport("os")
echo "Current dir is: ", os.getcwd().to(string)

# sum(range(1, 5))
let py = pyBuiltinsModule()
let s = py.sum(py.range(0, 5)).to(int)
assert(s == 10)

with

Nim Compiler Version 0.19.9 [Windows: amd64]
Compiled at 2018-12-13
Copyright (c) 2006-2018 by Andreas Rumpf

in MSYS2 on win7 64 bits

when I copy the EXE file into python 64bits directory

E:\prg\py\Anaconda3_64>a
a.nim(2)                 a
nimpy.nim(1002)          pyImport
py_lib.nim(404)          initPyLibIfNeeded
py_lib.nim(369)          initPyThreadFrame
py_lib.nim(364)          initPyLib
py_lib.nim(152)          loadPyLibFromModule
py_lib.nim(133)          symNotLoadedErr
Error: unhandled exception: Symbol not loaded: PyObject_GetBuffer [Exception]

`module 'sys' has no attribute 'argv'` is raised by some modules

It seems it's a common bug in embedded Python (see: googleapis/oauth2client#642) that the number of default arguments passed to the interpreter is wrong. Some libs assume the default parameters and fail.

Either there is a way to pass the default parameters, or the following workaround would fix sys.argv, this code should be executed at every init:

import nimpy

let py = pyBuiltinsModule()
let sys = pyImport("sys")
if not py.hasattr(sys, "argv").to(bool):
  let argv = py.list()
  discard argv.append("")
  discard py.setattr(sys, "argv", argv)

Otherwise these libs would raise:
Error: unhandled exception: <class 'AttributeError'>: module 'sys' has no attribute 'argv' [Exception]

Publish nimpy?

I think you should publish your package to the standard package manager. You can use the nimble publish command. That way you stake your claim on your cool name and it will become visible to more people.

NimPy Compile Error on Nim 0.19.1

tried test code:

import nimpy
let os = pyImport("os")
echo "Current dir is: ", os.getcwd().to(string)

and got this on compile:
../../.nimble/pkgs/nimpy-0.1.0/nimpy/py_lib.nim(114, 5) Error: invalid type: 'Complex' in this context: 'PyLib' for var

Can the datas exported be used in another thread in python?

This project is very nice.
But I am wandering if the datas structures exported from nim to python can be passed to another thread?
If not, I think it will be easy to shoot my foot in a multi-threaded program.

`SIGSEGV: Illegal storage access.` when function called too often

I try to use a python function to support my test cases, but when I call the function (it should be called ~200_000 times) in a loop it will always raises the error mentioned below. It doesn't matter if the code is pure python or a linked library.
If I call it not in a loop it works as expected.

The file icuCol.py in the code below is the same as described here.

import nimpy
let op = pyImport("operator")
let pyicu = pyImport("icuCol")
let icuCollator = pyicu.collator()

proc cmpUcaPyicu*(a, b: string): int =
  let keyA = icuCollator.getSortKey(a)
  let keyB = icuCollator.getSortKey(b)
  return op.sub(op.gt(keyA, keyB), op.gt(keyB, keyA)).to(int)

Traceback (most recent call last)
tests_uca.nim(35)        tests_uca
pyTest.nim(13)           cmpUcaPyicu
nimpy.nim(908)           nimValueToPy
nimpy.nim(499)           incRef
SIGSEGV: Illegal storage access. (Attempt to read from nil?)
Error: execution of an external program failed:

Python version: Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 17:00:18) [MSC v.1900 64 bit (AMD64)] on win32
Nim version:

Nim Compiler Version 0.18.0 [Windows: amd64]
Copyright (c) 2006-2018 by Andreas Rumpf

git hash: 5ee9e86c87d831d32441db658046fc989a197ac9
active boot switches: -d:release

nim keyword arguments in python

Right now I cant pass keyword args from python to nim proc.

import nimpy
import strformat

proc greet(name: string, greeting: string = "hi"): string {.exportpy.} =
    ## Say a greeting to a name
    return &"{greeting} {name}"

python throws the following when trying to pass keyword args

>>> import test
>>> test.greet(name="bob")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: greet() takes no keyword arguments

The help message also does not reflect the arguments:

>>> help(test.greet)
Help on built-in function greet in module test:

greet(...)
    Say a greeting to a person

Mark `callMethod` as `discardable`

In order to avoid a slew of useless discard I'd suggest to mark the callMethod returning a PyObject as discardable.

PyList_New is nil

I have a function that returns a sequence, and when trying to access these values from python I get a SIGSEGV: Illegal storage access. (Attempt to read from nil?).

In trying to debug this, I added some debug information to nimpy, and it turns out that this happens in nimArrToPy, where pyLib.PyList_New is nil.

Strangely enough, other functions from pyLib, such as Py_BuildValue, are not nil.

I struggle to provide an example, because when I try to find a minimal case, the issue disappears. Still, maybe you have some idea where I should start looking to understand whether pyLis is initialized correctly.

What is especially confusing is that these symbols seem to be all loaded with a template load which reads

template load(v: untyped, name: cstring) =
  maybeLoad(v, name)
  if pyLib.v.isNil:
    raise newException(Exception, "Symbol not loaded: " & $name)

so apparently it shouldn't be possible that pyLib is partially loaded :-?

Hand Nim proc to Python func taking a function

So recently I wanted to call scipy.curve_fit, which takes a Python function as the first argument. I thought I'd just define a Nim proc with the exportpy pragma, but that doesn't seem to work.

A simple example:

# test.py
def myProc(ar, op):
    result = []
    for x in ar:
        result.append(op(x))
    return result

# handproc.nim
import nimpy

proc testProc(x: float): float {.exportpy.} =
  result = x * 1.5

# without the following Python doesn't find `test.py` :/
let sys = pyImport("sys")
discard sys.path.append("./")

let pfn = pyImport("test")

let a = @[1'f64, 2, 3, 4, 5, 6]
let b = pfn.myProc(a, testProc)

Unfortunately this results in a call to unknownTypeCompileError.

I suppose nimValueToPy needs to grow a elif T is proc branch. But I don't know how to get started with that. :/

Does not compile with latest Nim

Trying to use nimpy wiht latest Nim devel gives

/Users/andrea/.nimble/pkgs/nimpy-0.1.0/nimpy.nim(868, 14) Error: usage of 'isNil' is a user-defined error

[question] GC/GIL

I cant find this info on Docs, so I ask here.

What GC works on a compiled module?, Python GC only?, Nim GC Only?, Both?.
Compile options like --gc:markAndSweep have its effect?.
Pythons GIL is active inside the compiled module?.

Feel free to add the info on the Docs and close this bug.

ImportError: dynamic module does not define module export function (PyInit_mymod)

Hi Yuriy, thanks a lot for this cool project.

I am experimenting with creating a Jupyter cell magic for Nim based on nimpy (similar to %%cython but for my favorite language :-P ), but am getting the above stated error.

The example from the Readme works fine when I put it in a file and compile the module from the command line.

But when I use the same code in a Jupyter notebook cell and compile it from there using Python3's subprocess.run, importing the resulting .so file gives the above error. I have no idea what's going on.
Trying to import the .so file generated by the Jupyter notebook in a new Python console session, gives the same error.

The Jupyter code (experimental, POC) is in this gist:

Do you have an idea how to fix this?

Many thanks in advance for your help.

Kind regards,
Axel

how to release a nim app which calls python inside?

so that I can give the app to user who does not have python installed. I mean 3rd packages are used, for example scipy, pandas, matplotlib.

Load python 3.6 x86 error.

Because python3.dll with python 3.6 x86 version doesn't export function PyObject_GetBuffer, so it cannot load the python3.dll.

t93.nim(4)               t93
nimpy.nim(1039)          pyBuiltinsModule
py_lib.nim(404)          initPyLibIfNeeded
py_lib.nim(369)          initPyThreadFrame
py_lib.nim(364)          initPyLib
py_lib.nim(152)          loadPyLibFromModule
py_lib.nim(133)          symNotLoadedErr
Error: unhandled exception: Symbol not loaded: PyObject_GetBuffer [Exception]
Error: execution of an external program failed: 't93.exe '

If we can put the python version like:

    for v in ["3.6", "3.7", "3", "3.5", "", "2", "2.7"]:
        when defined(macosx):
            yield "libpython" & v & ".dylib"
            yield "libpython" & v & "m.dylib"

The problem can be fixed, and will not arise any issue.

Dunder methods

The magic/dunder methods rely on identifiers with underscores incompatible with Nim identifiers.
Below is an example of a common interaction with Python context managers where a slight workaround is needed to invoke such methods.

import macros

import nimpy

let None = pyBuiltins().None

template with_py(context: typed, name: untyped, body: untyped) =
  try:
    # Can't use the `.` template
    let name = callMethod(context, "__enter__")  # context.__enter__()
    body
  finally:
    discard callMethod(context, "__exit__", None, None, None)

let os = pyImport("os")

os.scandir(os.getcwd()).with_py(context):
  for entry in context:
    echo entry.name.to(string)

This works just fine, but it seems like this could be a spot for some convenience wrapper/convention for dealing with Python context managers. Any thoughts on anything nimpy might want to support?

How to raise corresponding Python exception from Nim

If I have the following function that I expose into Python,

proc function(path_to_file: string): string {. exportpy .} =
    try:
        some_nim_function(path_to_file)
    except:
        raise newException(Exception, "Unable to load file from " & path_to_file)

how do I get it to raise a Exception in Python if some_nim_function raises a Exception in Nim? Everything I've tried so far has resulted in Python interpreter dying.

ModuleNotFoundError

Getting this runtime error

nimpy.nim(1004)          pyImport
py_utils.nim(98)         raisePythonError
Error: unhandled exception: <class 'ModuleNotFoundError'>: No module named 'fdb' [Exception]

pip3 installed modules are not found anymore , code like pyos=pyImport("os") work fine
I still have compiled code where this works so not sure if this is a nimpy or Nim issue or something else.
System is debian with python3.6.7 and latest Nim devel

Edit
After further tests on several systems , this error seems to show itself only if there is a mixture of
python3.7.x standard libraries and python 3.6.x libraries on the system

[question] How to create tuple inside of nim ?

Sorry, maybe stupid question, but can't figure out how to create tuple ?
Tried many-many ways, but unsuccessfully...

I'm newbie in Nim.

Unable to install nimpy from head

When I run the command below it errors out

$ nimble install "https://github.com/yglukhov/nimpy"
Downloading https://github.com/yglukhov/nimpy using git
       Tip: 11 messages have been suppressed, use --verbose to show them.
     Error: nimble.nim(1106)         nimble
        ... nimble.nim(1044)         doAction
        ... nimble.nim(481)          install
        ... download.nim(256)        downloadPkg
        ... packageparser.nim(410)   getPkgInfo
        ... packageparser.nim(398)   getPkgInfoFromFile
        ... packageparser.nim(359)   readPackageInfo
        ... Could not read package info file in /tmp/nimble_3655/githubcom_yglukhovnimpy/nimpy.nimble;
        ...   Reading as ini file failed with:
        ...     Invalid section: .
        ...   Evaluating as NimScript file failed with:
        ...     /Users/$USER/.choosenim/toolchains/nim-0.19.0/lib/posix/posix_other.nim(564, 3) Error: cannot 'importc' variable at compile time.

Fails to build

nimpy fails to build with Nim 0.18.0
nimpy.nim(1326, 32) Error: undeclared identifier: 'nnkTupleConstr'

In nimpy.nimble the requirement is nim >= 0.17.0, but nnkTupleConstr was only introduced recently.
Here's the relevant Nim commit, and the relevant nimpy commit acd4f10.
Building versions before this commit works.

Windows: cannot find pythonxx.dll

Trying the example in the readme gives me

HIBAML60075 :: » nim c --app:lib --out:ex.pyd ex                                      
Hint: used config file 'D:\sdks\nim-0.18.0\config\nim.cfg' [Conf]                                                  
Hint: system [Processing]                                                                                          
Hint: ex [Processing]                                                                                              
Hint: nimpy [Processing]                                                                                           
Hint: dynlib [Processing]                                                                                          
Hint: strutils [Processing]                                                                                        
Hint: parseutils [Processing]                                                                                      
Hint: math [Processing]                                                                                            
Hint: algorithm [Processing]                                                                                       
Hint: macros [Processing]                                                                                          
Hint: ospaths [Processing]                                                                                         
Hint: winlean [Processing]                                                                                         
Hint: complex [Processing]                                                                                         
Hint: sequtils [Processing]                                                                                        
Hint: typetraits [Processing]                                                                                      
Hint: tables [Processing]                                                                                          
Hint: hashes [Processing]                                                                                          
CC: ex                                                                                                             
CC: stdlib_system                                                                                                  
CC: nimpy_nimpy                                                                                                    
CC: stdlib_strutils                                                                                                
CC: stdlib_typetraits                                                                                              
Hint:  [Link]                                                                                                      
Hint: operation successful (24605 lines compiled; 2.360 sec total; 46.313MiB peakmem; Debug Build) [SuccessX]      
HIBAML60075 :: » ipython                                                              
Python 3.6.1 (v3.6.1:69c0db5, Mar 21 2017, 18:41:36) [MSC v.1900 64 bit (AMD64)]                                   
Type 'copyright', 'credits' or 'license' for more information                                                      
IPython 6.3.1 -- An enhanced Interactive Python. Type '?' for help.                                                
                                                                                                                   
In [1]: import ex                                                                                                  
nimpy.nim(797)           initModule3                                                                               
nimpy.nim(748)           initCommon                                                                                
nimpy.nim(710)           pythonLibHandleForThisProcess                                                             
nimpy.nim(575)           findPythonDLL                                                                             
Error: unhandled exception: Could not find pythonXX.dll [Exception]

I've tried copying python35.dll and python36.dll into the same folder as the pyd but to no avail. Any tips/hints on how to get past this?

[question] proc dict argument

Hi, thanks for the epic work, it works nicely.

I need to create a proc that needs to take a Python dict().

dict is of unknown length, all dict item values are JSON-compatible like int, bool, str, float, etc,

I have search the doc, examples, and tests and cant find which type to use on the Nim code.

Nim

import nimpy

proc needs_dict_argument( dict_arg: WhatToPutHere ) {.exportpy.} =
  # does stuff with dict_arg

Python

from nimpy_module import needs_dict_argument

needs_dict_argument( {"foo": 666, "bar": False, "baz": 42.0} )

WhatToPutHere is what I dont understand. Ive tried seq, openArray & Table but didnt work. Thanks ❕
🐱

Cannot import module on Windows

I use Windows10 64bit Python3.5

I create xx.pyd, but cannot import on console.

import nimpy

proc greeting (name: string): string {.exportpy.} =
    return "Hello nimpy." & name

nim c --threads:on --tlsEmulation:off --app:lib --out:mymodule.pyd nimpy_test.nim
ImportError: dynamic module does not define module export function (PyInit_mymodule)

I see Windows support? #4, Do I create for Py2?
But I don't recognize what command or progma needed.
Please help how to use.

Add an option to specify which Python to use?

I've tried calling Python from Nim (as shown in the Readme), just to realize it calls system's python, not the version I use for developing (installed via conda) which has all the libraries I usually use (e.g. numpy, matplotlib, etc.)

Could there be a way to specify which Python to use?

if nothing is specified, use the system Python,
if user provides a path, use that version of Python.

Maybe something like this?

import nimpy

pyPath("/path/to/my/python")
let np = pyImport("numpy")

Unclear how to call function with class name

I tried to save the collator from this example.
The problem is, that I didn't see how it works, when there is a class name involved (other functions without class name work).

When I write this code

import icu
col = icu.Collator.createInstance()

def collator():
  return col

and save it in the same folder of my nim file as icuCol.py, I can workaround this by importing and accessing the now defined function collator like this

let pyicu = pyImport("icuCol")
let icuCollator = pyicu.collator()

but I thought there is a direct way.

Update:
Thanks for the fix, it works perfect now!

Exporting in sub modules

Do all your python exports need to be in the main file? The procedures I mark with exportpy in imported nim modules to not appear in the python module. No compiler errors.

GC Seg fault

So I wrote a small module and then when I call it from python, it works fine if it's just a single call.

OTOH, if I call it in a loop - even say just 100 times, then after the 20th time there's a crash [on windows, the debug window opens, on linux I get a message about the GC]

I also tried building with --gc:none and in this case, the program completes successfully.

Are there guidelines/dos & donts in terms of what data can be returned from nim?

Happy to share code if you want to drop me an email at raghu dot nospam at gmail

Error checking for pyObjToNim conversions

A small example:

let py = pyBuiltinsModule()
echo py.chr(65).to(int)

The chr function returns a string but the to(int) tries a conversion to int and PyLong_AsLongLong returns -1 to signal this. Checking for PyErr_Occurred or using the <type>_Check procedures may help here.

Error: usage of 'isNil' is a user-defined error

I'm updating my code from nim 0.18 to 0.19 and am running into this error:

../../nimpy/nimpy.nim(740, 14) Error: usage of 'isNil' is a user-defined error

Nim changed the way they handle nil for 0.19, does nimpy need to be updated?

New Error with Nimpy on Compile

Same Test compile Script new error.

import nimpy
let os = pyImport("os")
echo "Current dir is: ", os.getcwd().to(string)

../../.nimble/pkgs/nimpy-0.1.0/nimpy.nim(594, 19) template/generic instantiation of `pyObjToNim` from here
../../.nimble/pkgs/nimpy-0.1.0/nimpy.nim(355, 18) Error: attempting to call undeclared routine: 're='