wg / scrypt Goto Github PK

There is native support for Darwin, FreeBSD and Linux, but not Windows. Performance on Windows, consequently, is poor.

Windows support would be appreciated!

Hi Will,
I've been wondering how to handle the license information. Can you include it with Maven?
For example: http://jeffbadge.blogspot.com/2010/07/telling-maven-to-include-license-info.html.

Thanks!

Hi all,

I added the libscrypt.so to my Android application and when uploading a newer version to the Play Store, I get 0 (zero) supported devices.

Supported Android devices: 0 devices (6896 removed)
Native platforms: arm5, x86_64 

The it seems that it understands that there are native libraries but for some reason the PlayStore shows the APK as incompatible.

When I try with my Nexus, it works just fine.

Here are the sources for reference https://github.com/Coinomi/coinomi-android

Thanks

When downloading the maven jar, it seems to consist of three sub-jars:

$ jar tf ~/.ivy2/cache/com.lambdaworks/scrypt/jars/scrypt-1.1.0.jar
META-INF/
META-INF/MANIFEST.MF
pom.xml
scrypt-1.1.0.jar
scrypt-1.1.0.pom.asc
scrypt-1.1.0-javadoc.jar
scrypt-1.1.0-sources.jar
scrypt-1.1.0-javadoc.jar.asc
scrypt-1.1.0-sources.jar.asc

I've never seen jars like this and the tools I'm using (Eclipse, Ivy, sbt, etc.) don't get to use this the right way. Would be great if you could fix this (these sub-jars should be hoisted up out of the parent jar and into the hosted Maven directory). Thanks!

The password parameters of SCryptUtil.scrypt() and check() are strings, which should not be used for passwords as explained here. For developers that care about these nuances, it would be great if you provided additional overloads that allow using a char[]:

public static String scrypt(char[] passwd, int N, int r, int p)
public static boolean check(char[] passwd, String hashed)

Note that I consider the hash to be harmless and kept it inside a String; one might argue to use char[] here for consistency.

Is there a precompiled 64 bit version of the native library?

If not, is the code 64 bit compatible and it's just a matter of rebuilding it, or would there be code changes needed too?

please check!!

os: windows 10

It would be great if you could build both w/ ABI and w/o on mvn as installing gives a INSTALL_FAILED_NO_MATCHING_ABIS error.

The method SCryptUtil#check produces a different result when the parameter passwd is an empty string depending on whether the native library or pure Java implementation of scrypt is used.

The native library (lib.x86_64/linux/libscrypt.so) works as expected.

The pure Java implementation throws java.lang.IllegalArgumentException: Empty key.

Steps to reproduce:

System.setProperty("com.lambdaworks.jni.loader", "nil");
boolean result = SCryptUtil.check("", "$s0$a0801$uk1+u+tylCxLj/l3I8smTA==$6SaqEZxC6fO89qGEjR25sYym+jXd6491g+1Ufa36sQw=");
assertTrue(result);

I suggest creating the following test case:

byte[] passwd = {};
byte[] salt = {1, 2, 3, 4};
byte[] n = SCrypt.scryptN(passwd, salt, 1024, 8, 1, 64);
byte[] j = SCrypt.scryptJ(passwd, salt, 1024, 8, 1, 64);
assertArrayEquals(j, n);

The JarLibraryLoader has apparently a problem while loading if placed in a directory that contains special characters which have to be URL encoded.

For example, if the directory path contains "web#app" java.security.CodeSource.getLocaction().getPath() in line 59 will encode the '#' and return "web%23app". However, the java.util.jar.JarFile constructor expects a (not encoded) filename and consequently fails to load from the nonexistent directory "web%23app". A simple java.net.URLDecoder.decode() in line 59 would fix this issue.

I can create a pull request upon request.

I noticed that a customer's application increased in memory usage every time some users logged in and the memory usage would not go down again even when everything was quiet. I profiled using async-profiler and found that the memory was being allocated by Scrypt. I managed to reduce it to the following reproducer:

public static void main(String[] args) throws InterruptedException {
    int NUM_THREADS = 10;
    int HASHES_PER_THREAD = 10;
    for (int i = 0; i < NUM_THREADS; i++) {
        new Thread(() -> {
            for (int j = 0; j < HASHES_PER_THREAD; j++) {
                //High memory usage during this loop is understandable
                //since we're actively using the native buffers
                SCryptUtil.scrypt("pwd", 16384, 8, 1);
            }
            //Once we're done, memory should be freed
            //However, it stays allocated until the JVM exists
            try {
                Thread.sleep(30000);
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
            }
        }).start();
    }
    Thread.sleep(30000);
}

Just run this and watch the process' memory usage. It spikes up when the threads invoke scrypt and never goes down again. It gets worse the more threads you have. The hashes per thread also seem to play a role, but only to a certain extent. If I set that parameter to 1, I get ~50MB memory usage. If I set it to 10, I get 500MB, but setting it to 100 does not further increase it.

I played around with the parameters a bit and if I increase N, the problem goes away. E.g. with 2^15 the memory usage spike up and then go down again. I'm afraid I don't know enough C to be of assistance in figuring out why :)

I ran the reproducer on Linux 5.4.0-72 if that helps.

I looked at all the current issues, pull requests and most of the forks (on GitHub). I noticed almost all changes deal with native code support. Either they want to get rid of specific native libs (because they cause crashes) or they want to support currently unsupported architectures. One PR even wants to get rid of the Java impl such that only native code can be used.

I think a clean way of handling all these cases would be splitting the project into modules:

• scrypt-api: contains only the interfaces.
• scrypt-impl-java: cointains the Java implementation.
• scrypt-impl-android: contains native impls for relevant Android architectures (these days: arm+intel 32+64bit).
• scrypt-impl-desktop: contains native impls for relevant Desktop OSes (Linux, Windows, MacOS).

Of course variations are possible, for example scrypt-impl-java could stay with scrypt-api, or scrypt-impl-desktop could be split into separate scrypt-impl-linux, scrypt-impl-windows and scrypt-impl-macos modules.

Ideally all the modules are built from source, rather than including binary blobs like today.

The idea is for example an Android app would depend on scrypt-impl-android and get all it needs (but not more).

@wg Would you be ok with this concept? You don't necessarly need to implement it yourself.

This is not an issue, it's a 'thank you'.

I would like to thank you for your 'Java scrypt implementation'. I'm using it in my project to calculate Cisco Type 9 hashes.
See: https://github.com/videgro/cisco-password-hashes

params - 32-bit hex integer containing log2(N) (16 bits), r (8 bits), and p (8 bits)

should be:

params - 40-bit hex integer containing log2(N) (16 bits), r (8 bits), and p (16 bits)

The following code in SCryptUtil.check only extracts 4 bits for r and p

        int N = (int) Math.pow(2, params >> 16 & 0xff);
        int r = params >> 8 & 0x0f;
        int p = params      & 0x0f;

When I use the native library to crypt an empty string ("") there is a result.
When I use the java code to crypt an empty string ("") I get the Exception
Exception in thread "main" java.lang.IllegalArgumentException: Empty key at java.base/javax.crypto.spec.SecretKeySpec.<init>(SecretKeySpec.java:95) at com.lambdaworks.crypto.SCrypt.scryptJ(SCrypt.java:87) at de.empic.scryptbug.Main.main(Main.java:19)

Example code to reproduce (on linux only):
`import java.security.GeneralSecurityException;
import java.security.SecureRandom;

import com.lambdaworks.crypto.SCrypt;

public class Main
{
private static final int CPU_COST = 16384;
private static final int MEMORY_COST = 8;
private static final int PARALLELIZATION = 1;

public static void main(String[] args) throws Exception
{
byte[] salt = new byte[16];
SecureRandom.getInstance("SHA1PRNG").nextBytes(salt);
final byte[] emptyStringInBytes = "".getBytes("UTF-8");
try
{
byte[] resultJava = SCrypt.scryptJ(emptyStringInBytes, salt, CPU_COST, MEMORY_COST, PARALLELIZATION, 32);
}
catch (GeneralSecurityException e)
{
e.printStackTrace();
}
try
{
byte[] resultNative = SCrypt.scryptN(emptyStringInBytes, salt, CPU_COST, MEMORY_COST, PARALLELIZATION, 32);
}
catch (Exception exception)
{
exception.printStackTrace();
}
}
}`

java.util.concurrent.ExecutionException: com.android.tools.build.bundletool.exceptions.BundleFileTypesException$FileUsesReservedNameException: File 'root/lib/x86_64/darwin/libscrypt.dylib' uses reserved file or directory name 'lib'.

Google has this to say about Android KitKat's release:

"The Scrypt key derivation function is implemented to protect the cryptographic keys used for full-disk encryption."

Looking at the code, the Android source now includes this:

https://android.googlesource.com/platform/external/scrypt

Unfortunately, there isn't really any code reference to this project within the rest of the Android source tree, and I'm not sure how/whether KitKat's native scrypt support is available to applications. This is probably worth looking into!

Also, their code includes an "arm-neon.patch" which supposedly "Adds NEON acceleration for the Salsa20/8 mixing function." Might this be interesting for this project as well?

Hey Will,

On "Scrypt.java" you're calling every time:

public static byte[] scryptJ(byte[] passwd, byte[] salt, ...) {
    ....
    Mac mac = Mac.getInstance("HmacSHA256");
    ...
}

For whatever reason (bad karma on JSSE?) that call roughly takes 50% of the entire SCrypt calculation. Initialise the Mac statically and then call .clone() on it, you get double the performance (according to JProfiler).

private static final Mac SHARED_MAC = Mac.getInstance("HmacSHA256");
public static byte[] scryptJ(byte[] passwd, byte[] salt, ...) {
    ....
    Mac mac = (Mac) SHARED_MAC.clone();
    ...
}

Using arm64-v8a native libraries and android devices with x64 cpu the hash coming out from Scrypt function is different than the result obtained using 32 bit libraries

It would make it easier to redistribute scrypt as a RPM in the standard Fedora repo if each source file contains a license header.

Please see https://bugzilla.redhat.com/show_bug.cgi?id=1230949#c8 for an example.

This was brought to my attention after a TPE enabled grsecurity system gave the notice:

grsec: denied untrusted exec (due to file in world-writable directory) of /tmp/scrypt199390770223188475lib

The java process extracting the jar entry is included in the tpe group with the following sysctl options:

kernel.grsecurity.tpe = 1
kernel.grsecurity.tpe_invert = 1
kernel.grsecurity.tpe_restrict_all = 1

Which means the problem is almost certainly not server-side.

A closer look at

I don't java much or else I'd submit a PR but I suspect that a trivial solution would be to File.createTempDirectory() before File.createTempFile() and apply the appropriate changes to File.createTempFile() so that the file is written into something like /tmp/scrypt instead of /tmp.

Example (suggested umasks):

umask: 0750
/tmp/scrypt/

umask: 0640
/tmp/scrypt/scrypt199390770223188475lib

This particularly effects https://github.com/i2p/i2p.i2p-bote

Nonetheless, thanks for the great implementation!

Hi,

when using scrypt from the maven repo, I have a crash on a debian 64 bit system. I have a strong suspicion that it is related to the jni libs (see below). Since deplyment of sefbuild jars is not so easy in my case, I would prefer to just enforce the pure java version, disabling the native libs. Is there a way to easiliy do that?

----------details---------------
sm@debian64sim:$ java -version
java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11.5) (6b24-1.11.5-1)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
sm@debian64sim:$ uname -a
Linux debian64sim 2.6.32-5-amd64 #1 SMP Sun Sep 23 10:07:46 UTC 2012 x86_64 GNU/Linux
sm@debian64sim:~$ doTheCodeThatCrashes
*** glibc detected *** /usr/lib/jvm/java-6-openjdk-amd64/jre/bin/java: double free or corruption (!prev): 0x0000000002f20d40 ***
======= Backtrace: =========
/lib/libc.so.6(+0x71e16)[0x7f260fe08e16]
/lib/libc.so.6(cfree+0x6c)[0x7f260fe0db8c]
/tmp/scrypt4590374583790695139lib(crypto_scrypt+0x4e1)[0x7f2608a6f1d1]
/tmp/scrypt4590374583790695139lib(scryptN+0xee)[0x7f2608a6f37e]
[0x7f260a99ed28]
======= Memory map: ========
...

Using your library with Android is pretty easy. Add the maven artifact as dependency and insert the precompiled ARM library. Using the native library is necessary, the computation is about 20 times faster on my Nexus 5, because of the slow garbage collector in the pure Java implementation.

I tried to build the library, but I failed. Could you also provide a precompiled version for x86 and MIPS platforms?

All the methods in the SCrypt class are static.

First of all, why? :) Generally speaking, this is a code smell and should be corrected.

Secondly, does this mean that the implementation is not thread-safe? If it is, the Javadoc should state as much.

This is such a big deal to people that Wikipedia actually lists a non-static alternative to your library: http://en.wikipedia.org/wiki/Scrypt#Implementations.2C_wrappers.2C_and_distributions

If a password doesn't match the scrypt format, an IllegalArgumentException is thrown. It would be useful to have something to put in the password field that reliably matches no password, so that passwords can conveniently be disabled.

Thanks for writing this!

When trying to load native scrypt targeting android-23 following Exception will be thrown:

java.lang.UnsatisfiedLinkError: dlopen failed: /data/app/pck.app-2/lib/arm/libscrypt.so: has text relocations

Does it have something to do with this Andriod 6.0 Change?

Hi all,
I launched the mvn install everything look good except when dealing with surefire:

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building scrypt 1.4.0
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ scrypt ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ scrypt ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ scrypt ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 5 resources
[INFO]
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ scrypt ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-surefire-plugin:2.10:test (default-test) @ scrypt ---
[INFO] Surefire report directory: /Users/daN/git/scrypt/target/surefire-reports

T E S T S

java.lang.NoClassDefFoundError: org/junit/runner/notification/RunListener
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2446)
at java.lang.Class.getConstructor0(Class.java:2756)
at java.lang.Class.getConstructor(Class.java:1693)
at org.apache.maven.surefire.util.ReflectionUtils.getConstructor(ReflectionUtils.java:76)
at org.apache.maven.surefire.util.ReflectionUtils.instantiateOneArg(ReflectionUtils.java:129)
at org.apache.maven.surefire.booter.SurefireReflector.instantiateProvider(SurefireReflector.java:247)
at org.apache.maven.surefire.booter.ProviderFactory.createProvider(ProviderFactory.java:81)
at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:171)
at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:107)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68)
Caused by: java.lang.ClassNotFoundException: org.junit.runner.notification.RunListener
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at org.apache.maven.surefire.booter.IsolatedClassLoader.loadClass(IsolatedClassLoader.java:93)
... 11 more

Results :

Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

[INFO]
[INFO] --- maven-jar-plugin:2.3.2:jar (default-jar) @ scrypt ---
[INFO] Building jar: /Users/daN/git/scrypt/target/scrypt-1.4.0.jar
[INFO] META-INF/maven/com.lambdaworks/scrypt/pom.xml already added, skipping
[INFO] META-INF/maven/com.lambdaworks/scrypt/pom.properties already added, skipping
[INFO]
[INFO] --- maven-surefire-plugin:2.10:test (integration-test) @ scrypt ---
[INFO] Surefire report directory: /Users/daN/git/scrypt/target/surefire-reports

T E S T S

java.lang.NoClassDefFoundError: org/junit/runner/notification/RunListener
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2446)
at java.lang.Class.getConstructor0(Class.java:2756)
at java.lang.Class.getConstructor(Class.java:1693)
at org.apache.maven.surefire.util.ReflectionUtils.getConstructor(ReflectionUtils.java:76)
at org.apache.maven.surefire.util.ReflectionUtils.instantiateOneArg(ReflectionUtils.java:129)
at org.apache.maven.surefire.booter.SurefireReflector.instantiateProvider(SurefireReflector.java:247)
at org.apache.maven.surefire.booter.ProviderFactory.createProvider(ProviderFactory.java:81)
at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:171)
at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:107)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68)
Caused by: java.lang.ClassNotFoundException: org.junit.runner.notification.RunListener
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at org.apache.maven.surefire.booter.IsolatedClassLoader.loadClass(IsolatedClassLoader.java:93)
... 11 more

Results :

Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

[INFO]
[INFO] --- maven-jarsigner-plugin:1.2:sign (sign) @ scrypt ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 6.646s
[INFO] Finished at: Sat Feb 01 23:42:52 CET 2014
[INFO] Final Memory: 5M/81M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-jarsigner-plugin:1.2:sign (sign) on project scrypt: The parameters 'alias' for goal org.apache.maven.plugins:maven-jarsigner-plugin:1.2:sign are missing or invalid -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginParameterException

Please help!

Hi there is there a option to disable validation for plain hash? Now if I pass hash to
SCryptUtil.check(enteredHash, storedHash) this is treu, I want to allow only entering on regular value not hashed.