jetbrains-research / npy Goto Github PK
View Code? Open in Web Editor NEWNPY and NPZ support for the JVM
License: MIT License
NPY and NPZ support for the JVM
License: MIT License
import numpy as np
npz = np.load('/Users/oleg/Desktop/zinbra_peaks_200_1.0E-4.npz')
npz.files
npz = np.load('/Users/oleg/Desktop/zinbra_peaks_200_1.0E-4.npz')
npz.files
Out[31]:
['H4K20me1',
'H3K4me1',
'H3K9ac',
'H3K4me3',
'H3K4me2',
'H3K27ac',
'Input_Sonicated',
'CTCF',
'H3K36me3',
'H3K27me3']
In [32]:
npz['CTCF']
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-32-56170b6f17ad> in <module>()
----> 1 npz['CTCF']
/Users/oleg/miniconda3/lib/python3.5/site-packages/numpy/lib/npyio.py in __getitem__(self, key)
222 return format.read_array(bytes,
223 allow_pickle=self.allow_pickle,
--> 224 pickle_kwargs=self.pickle_kwargs)
225 else:
226 return self.zip.read(key)
/Users/oleg/miniconda3/lib/python3.5/site-packages/numpy/lib/format.py in read_array(fp, allow_pickle, pickle_kwargs)
621 version = read_magic(fp)
622 _check_version(version)
--> 623 shape, fortran_order, dtype = _read_array_header(fp, version)
624 if len(shape) == 0:
625 count = 1
/Users/oleg/miniconda3/lib/python3.5/site-packages/numpy/lib/format.py in _read_array_header(fp, version)
504 not numpy.all([isinstance(x, (int, long)) for x in d['shape']])):
505 msg = "shape is not valid: %r"
--> 506 raise ValueError(msg % (d['shape'],))
507 if not isinstance(d['fortran_order'], bool):
508 msg = "fortran_order is not a valid bool: %r"
ValueError: shape is not valid: 550581
And also investigate why it is failing on windows.
In case windows built is not available on TC, change https://ci.appveyor.com build account.
Currently the API only allows to use ByteOrder.nativeOrder()
when creating an NPY file.
It looks like the library can't be used with Maven at the moment.
The Kotlin dependency is declared using Gradle-style wildcards (1.0.+
), which Maven doesn't support. This results in Maven trying to find a version that is literally named 1.0.+
.
What's even worse, this makes the downstream dependencies of npy
(e.g. viktor
) unusable with Maven.
Proposed solution: replace the wildcard dependency with a fixed one or a range.
Do you have any plans to upgrade the Kotlin version?
java.lang.IllegalAccessError: tried to access method kotlin.LazyKt.getValue(Lkotlin/Lazy;Ljava/lang/Object;Lkotlin/reflect/KProperty;)Ljava/lang/Object; from class org.jetbrains.bio.npy.NpyFile$Header
at org.jetbrains.bio.npy.NpyFile$Header.getMeta(Npy.kt)
at org.jetbrains.bio.npy.NpyFile$Header.allocate(Npy.kt:78)
at org.jetbrains.bio.npy.NpyFile$Companion.allocate$npy_compileKotlin(Npy.kt:256)
at org.jetbrains.bio.npy.NpzFile$Writer.write(Npz.kt:103)
at org.jetbrains.bio.histones.GenomeCoverage.save(GenomeCoverage.kt:79)
at org.jetbrains.bio.histones.GenomeCoverageTest.testPartialLoading(GenomeCoverageTest.kt:141)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:105)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:56)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:64)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:50)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
at org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:106)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:360)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
JFrog is sunsetting Bintray, so we need to move.
We already have credentials and deploy permissions to org.jetbrains.bio
, so all that remains is:
When trying to load a very large matrix (8312274035 elements) stored in npy format, I get a "NegativeArraySizeException".
It seems, that the operation to calculate the size being int32 limits the maximal number of elements (line 184 in NpyFile.class, int32(8312274035) = -277660557).
Maybe, there is a possibility to make this fit for large arrays?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.