tattdcodemonkey / crc Goto Github PK
View Code? Open in Web Editor NEWCRC library in elixir
License: MIT License
CRC library in elixir
License: MIT License
I get ** (ArgumentError) argument error
when computing the CRC of a binary constituted of 34599 bytes with CRC.ccitt_16
Is it to be expected??
The NIF seems to require specific LDFLAGS
to be set in the Makefile in order to compile. The Makefile uses the ?=
conditional operator to set these flagsβif LDFLAGS
are already configured, this means that the library cannot compile.
This caused us some problems today, because we are using crc
and fast_tls
in the same project, and the latter only compiles for us on M1 Macs if LDFLAGS
include the path to openssl.
We got around this by copying the specific flags from this Makefile into the LDFLAGS that we set for our project, but if another operator such as +=
was used here, it might make it so that crc
could compile without issues, but without conflicting with other projects.
Anyways, hopefully this helps others!
CRC algorithm nifs were released in v0.8.0
and they add lots of new functionality that needs to be documented.
Some of these will require some usage and testing to document best practices. I'm curious if there is any performance gains to using a model's resource over just calling it by name, especially in gen servers that are calculating crc very often.
Originating issue: nerves-project/nerves#444
This is an issue with the crc library. It builds the NIF in the source directory and not under _build. Since the source directory is shared between all targets and environments, make doesn't know that it needs to be rebuilt.
This is a common problem with NIF projects. Could you file an issue with crc or PR an update to its Makefile to build under _build? The Elixir Circuits projects are NIFs that have Makefiles that put object files and libraries in the right place. See https://github.com/elixir-circuits/circuits_spi/blob/master/Makefile#L21 for an example.
It would be really cool if crc
's Makefile were to be altered, so that it is possible to compile for various targets without having to manually clean up the built files in between π .
For small binaries an elixir implementation can be more performant especially for nerves devices.
If we implement non-NIF version for CRC algorithm we could add an optional parameter to use the non-NIF calculation for users who know their payloads are small and would prefer to not load the NIF.
During implementation we have the NIFs to unit test against.
If useful we could also look at adding an optional threshold param to use vs not use the NIF.
Goals:
crc/2
& calculate/2
takes optional param to choose not to use NIF, should default to current behavior of using NIFGenerated elixir_make app
==> crc
C checksum_xor.c
In file included from /app/deps/crc/c_src/nif/checksum_xor.c:4:
/app/deps/crc/c_src/nif/checksum_xor.h:7:10: fatal error: stdlib.h: No such file or directory
7 | #include <stdlib.h>
| ^~~~~~~~~~
compilation terminated.
make: *** [Makefile:163: /app/_build/prod/lib/crc/obj/checksum_xor.o] Error 1
could not compile dependency :crc, "mix compile" failed. You can recompile this dependency with "mix deps.compile crc", update it with "mix deps.update crc" or clean it with "mix deps.clean crc"
In python3,
>>> from binascii import crc32
>>> crc32(b"123", 34) # Provide an initial value
3011637596
In elixir using this library:
iex(36)> CRC.crc(%{extend: :crc_32, init: 34}, "123")
2297841019
Is this a bug?
BREAKING CHANGE
Remove the legacy algorithm specific functions in favor of the single calculate/2
with parameters.
It seems the CRC makefile may not be using the correct compiler when CROSSCOMPILE
is set by Nerves.
$(info "**** CRC PLATFORM set to [$(PLATFORM)] ****")
$(info "**** CRC CROSSCOMPILE set to [$(CROSSCOMPILE)] ****")
>MIX_TARGET=bbb mix deps.compile crc
==> nerves
==> atmos
Nerves environment
MIX_TARGET: bbb
MIX_ENV: dev
==> crc
"**** CRC PLATFORM set to [darwin] ****"
"**** CRC CROSSCOMPILE set to [/Users/taun/.nerves/artifacts/nerves_toolchain_armv7_nerves_linux_gnueabihf-darwin_arm-1.8.0/bin/armv7-nerves-linux-gnueabihf] ****"
C checksum_xor.c
armv7-nerves-linux-gnueabihf-gcc: error: unrecognized command-line option '-arch'; did you mean '-march='?
make: *** [/Users/taun/Development/Elixir/atmos/_build/bbb_dev/lib/crc/obj/checksum_xor.o] Error 1
could not compile dependency :crc, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile crc --force", update it with "mix deps.update crc" or clean it with "mix deps.clean crc"
==> atmos
** (Mix) Could not compile with "make" (exit status: 2).
You need to have gcc and make installed. Try running the
commands "gcc --version" and / or "make --version". If these programs
are not installed, you will be prompted to install them.
It seems the above is indicating, the platform is darwin
which is correct but the crc source should be compiled using the cross compiler armv7-nerves-linux-gnueabihf-gcc
and not compiled using darwin's cc
which is what the makefile attempts.
Hello, I'm try to compile it on Windows 10 using visual studio 2019 pre , Program Maintenance Utility Version is 14.20.27607.1 , something wrong with " cannot open file '..\priv\crc_nif.dll'" and no dll file was generated in floder , logs detail as follows
Copyright (C) Microsoft Corporation. All rights reserved.
echo # Auto-generated as part of Makefile.win, do not modify. > Makefile.auto.win
erl -eval "io:format(\"~s~n\", [lists:concat([\"ERTS_INCLUDE_PATH=\", code:root_dir(), \"/erts-\", erlang:system_info(version), \"/include\"])])" -s init stop -noshell >> Makefile.auto.win
nmake / /F Makefile.win app
Microsoft (R) Program Maintenance Utility Version 14.20.27607.1
Copyright (C) Microsoft Corporation. All rights reserved.
cl /O2 /EHsc /I"c:/Program Files/erl10.1/erts-10.1/include" /LD /MD /Fe"..\priv\crc_nif.dll" nif\xnif_slice.c nif\checksum_xor.c nif\crc_8.c nif\crc_nif.c nif\crc_algorithm.c nif\crc_model.c nif\crc_resource.c
Microsoft (R) C/C++ Optimizing Compiler Version 19.20.27607.1 for x86
Copyright (C) Microsoft Corporation. All rights reserved.
xnif_slice.c
checksum_xor.c
crc_8.c
nif\crc_8.c(37): warning C4333: '>>': right shift by too large amount, data loss
crc_nif.c
crc_algorithm.c
c:\myprojectpath\deps\crc\c_src\nif\crc_algorithm_fast.c.h(60): warning C4333: '>>': right shift by too large amount, data loss crc_model.c
crc_resource.c
Generating Code...
Microsoft (R) Incremental Linker Version 14.20.27607.1
Copyright (C) Microsoft Corporation. All rights reserved.
/dll
/implib:..\priv\crc_nif.lib
/out:..\priv\crc_nif.dll
xnif_slice.obj
checksum_xor.obj
crc_8.obj
crc_nif.obj
crc_algorithm.obj
crc_model.obj
crc_resource.obj
LINK : fatal error LNK1104: cannot open file '..\priv\crc_nif.dll'
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio\2019\Preview\VC\Tools\MSVC\14.20.27607\bin\HostX86\x86\cl.EXE"' : return code '0x2' Stop.
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio\2019\Preview\VC\Tools\MSVC\14.20.27607\bin\HostX86\x86\nmake.EXE"' : return code '0x2'
Stop.
What should I do to fix this ?
Hello,
Using crc
on Linux causes the following problem:
iex(1)> CRC.crc_16("hello")
2017-11-23 12:37:09.102 pid=<0.1289.0> [warn] : The on_load function for module crc_nif returned:
{:error,
{:load_failed,
'Failed to load NIF library: \'/home/user/projects/xyz/_build/dev/lib/crc/priv/crc_nif.so: invalid ELF header\''}}
** (UndefinedFunctionError) function :crc_nif.crc_16/1 is undefined (module :crc_nif is not available)
(crc) :crc_nif.crc_16("hello")
The error is due to crc-0.7.0 package, containing the files c_src/env.mk
and priv/crc_nif.so
. Since these files are built and environment dependent, they must not be included in the package. priv/crc_nif.so
is a MACH format compiled binary, causing the error.
Since the NIFs are load via erlang files and not elixir now, they are being auto loaded. This breaks cross compiling the module for a nerves project.
Lately I get a lot of this kind of error when running my tests locally:
** (UndefinedFunctionError) function :crc_nif.crc_fast/2 is undefined (module :crc_nif is not available)
When re-running with e.g. mix test --failed
, the same tests are green and nothing have been compiled since the error.
I'm not sure if this belongs here. Perhaps it is an issue of elixir_make, elixir or erlang.
I recently updated to erlang 23.0, and I think this is where the problems started.
I have no similar problems on the CI server (that also runs tests) or in my production or staging environments. I'm also not able to reproduce the error. It just comes some times - I see it perhaps around 2-3 times per 8 hour working day.
Has anyone seen something similar or know what might be wrong?
I'm running:
For larger binaries sending the entire value to a NIF could possibly cause issues with the scheduler, if execution time in the NIF is too great.
Some testing should be done to see what a sensible max binary size is. With that information we should limit calls to the NIF functions with binaries smaller than that, and for larger ones chunk calls to the NIF and calculate the CRC as we go. This will require modifications to the NIF functions to take continue a CRC instead of always starting from a seed.
Hello,
According to this paper, it is possible to detect errors with a longer hamming distance with CRC-32 by using other polynomial than the usual one.
Do the implementation use the suggestion of the paper?
Where can I find a documentation of hamming distance errors for the various CRC implemented?
Would it make sense to add that to the README?
Erlang version: 22.1
Elixir version: 1.9.1
crc version: 0.9.2
I'm getting the following error while compiling the dependencies with mix deps.compile
:
could not compile dependency :crc, "mix compile" failed. You can recompile this dependency with "mix deps.compile crc", update it with "mix deps.update crc" or clean it with "mix deps.clean crc"
** (Mix) Could not compile with "make" (exit status: 2).
You need to have gcc and make installed. Try running the
commands "gcc --version" and / or "make --version". If these programs
are not installed, you will be prompted to install them.
I have both gcc
and make
installed and they're added properly in the PATH.
noven/deps/crc on ξ main [!] is π¦ v0.9.2 via π§ 1.11.0 (OTP 23) via ξ± 23.2
π [ 19:21:17 ] β― elixir -v
Erlang/OTP 23 [erts-11.1.4] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [hipe]
Elixir 1.11.0 (compiled with Erlang/OTP 23)
noven/deps/crc on ξ main [!] is π¦ v0.9.2 via π§ 1.11.0 (OTP 23) via ξ± 23.2
π [ 19:21:32 ] β― uname -a
Linux palladium 5.9.14-arch1-1 #1 SMP PREEMPT Sat, 12 Dec 2020 14:37:12 +0000 x86_64 GNU/Linux
noven/deps/crc on ξ main [!] is π¦ v0.9.2 via π§ 1.11.0 (OTP 23) via ξ± 23.2
π [ 19:21:41 ] β― mix compile\
LD crc_nif.so
/usr/bin/ld: cannot find -lerl_interface
collect2: error: ld returned 1 exit status
make: *** [Makefile:151: /home/loictosser/github.com/KalvadTech/noven/deps/crc/priv/crc_nif.so] Error 1
** (Mix) Could not compile with "make" (exit status: 2).
You need to have gcc and make installed. If you are using
Ubuntu or any other Debian-based system, install the packages
"build-essential". Also install "erlang-dev" package if not
included in your Erlang/OTP version. If you're on Fedora, run
"dnf group install 'Development Tools'".
Initially discussed in #11 under Possible Improvements
implementing crc/2
, crc_init/1
, crc_update/2
, and crc_final/1
will allow implementing new algorithms by just passing the required params to crc_init/1
and using the context provided with the other functions. This would allow reduction of C code to a single implementation, instead of separate files for different widths etc.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.