jetbrains-research / npy Goto Github PK

View Code? Open in Web Editor NEW

53.0 53.0 7.0 341 KB

NPY and NPZ support for the JVM

License: MIT License

Kotlin 100.00%

npy's People

Contributors

Stargazers

Watchers

Forkers

roblovelock sa501428 originrose chenzhihuai1990 tobidelbruck jfftonsic

npy's Issues

Compare npy read/write speed with hdf5 for single thread mode

Numpy cannot load created file.

import numpy as np
npz = np.load('/Users/oleg/Desktop/zinbra_peaks_200_1.0E-4.npz')
npz.files
npz = np.load('/Users/oleg/Desktop/zinbra_peaks_200_1.0E-4.npz')
npz.files
Out[31]:
['H4K20me1',
 'H3K4me1',
 'H3K9ac',
 'H3K4me3',
 'H3K4me2',
 'H3K27ac',
 'Input_Sonicated',
 'CTCF',
 'H3K36me3',
 'H3K27me3']
In [32]:

npz['CTCF']
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-32-56170b6f17ad> in <module>()
----> 1 npz['CTCF']

/Users/oleg/miniconda3/lib/python3.5/site-packages/numpy/lib/npyio.py in __getitem__(self, key)
    222                 return format.read_array(bytes,
    223                                          allow_pickle=self.allow_pickle,
--> 224                                          pickle_kwargs=self.pickle_kwargs)
    225             else:
    226                 return self.zip.read(key)

/Users/oleg/miniconda3/lib/python3.5/site-packages/numpy/lib/format.py in read_array(fp, allow_pickle, pickle_kwargs)
    621     version = read_magic(fp)
    622     _check_version(version)
--> 623     shape, fortran_order, dtype = _read_array_header(fp, version)
    624     if len(shape) == 0:
    625         count = 1

/Users/oleg/miniconda3/lib/python3.5/site-packages/numpy/lib/format.py in _read_array_header(fp, version)
    504             not numpy.all([isinstance(x, (int, long)) for x in d['shape']])):
    505         msg = "shape is not valid: %r"
--> 506         raise ValueError(msg % (d['shape'],))
    507     if not isinstance(d['fortran_order'], bool):
    508         msg = "fortran_order is not a valid bool: %r"

ValueError: shape is not valid: 550581

Investigate possibility to configure multi platform builds on TeamCity

And also investigate why it is failing on windows.
In case windows built is not available on TC, change https://ci.appveyor.com build account.

Add a way of writing NPY in LE or BE

Currently the API only allows to use ByteOrder.nativeOrder() when creating an NPY file.

Impossible to use with Maven?

It looks like the library can't be used with Maven at the moment.

The Kotlin dependency is declared using Gradle-style wildcards (1.0.+), which Maven doesn't support. This results in Maven trying to find a version that is literally named 1.0.+.

What's even worse, this makes the downstream dependencies of npy (e.g. viktor) unusable with Maven.

Proposed solution: replace the wildcard dependency with a fixed one or a range.

Upgrade the Kotlin version to 1.6.1

Do you have any plans to upgrade the Kotlin version?

npy build for teamcity

https://teamcity.jetbrains.com/project.html?projectId=Epigenome_Tools&tab=projectOverview

Kotlin glitch

java.lang.IllegalAccessError: tried to access method kotlin.LazyKt.getValue(Lkotlin/Lazy;Ljava/lang/Object;Lkotlin/reflect/KProperty;)Ljava/lang/Object; from class org.jetbrains.bio.npy.NpyFile$Header
    at org.jetbrains.bio.npy.NpyFile$Header.getMeta(Npy.kt)
    at org.jetbrains.bio.npy.NpyFile$Header.allocate(Npy.kt:78)
    at org.jetbrains.bio.npy.NpyFile$Companion.allocate$npy_compileKotlin(Npy.kt:256)
    at org.jetbrains.bio.npy.NpzFile$Writer.write(Npz.kt:103)
    at org.jetbrains.bio.histones.GenomeCoverage.save(GenomeCoverage.kt:79)
    at org.jetbrains.bio.histones.GenomeCoverageTest.testPartialLoading(GenomeCoverageTest.kt:141)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:105)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:56)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:64)
    at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:50)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
    at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
    at org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
    at org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
    at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:106)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
    at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
    at org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:360)
    at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
    at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Migrate to Maven Central

JFrog is sunsetting Bintray, so we need to move.

We already have credentials and deploy permissions to org.jetbrains.bio, so all that remains is:

add Javadoc JAR to artifacts
implement artifact signing
provide sufficient metadata
configure Gradle to upload to OSSRH
configure a TeamCity build to upload to OSSRH with signing and deployment credentials

NegativeArraySizeException for very large arrays

When trying to load a very large matrix (8312274035 elements) stored in npy format, I get a "NegativeArraySizeException".
It seems, that the operation to calculate the size being int32 limits the maximal number of elements (line 184 in NpyFile.class, int32(8312274035) = -277660557).

Maybe, there is a possibility to make this fit for large arrays?

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.