Giter Club home page Giter Club logo

fstcore's People

Contributors

marcusklik avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

fstcore's Issues

`liblz4 was not found` error, which did not occur yesterday, occurred today when source installation of fst

Hello. Thank you for your continued development.

The following error occurred during source installation of the fstcore package on Ubuntu today.

#6 1783.9 * installing *source* package ‘fstcore’ ...
#6 1783.9 ** package ‘fstcore’ successfully unpacked and MD5 sums checked
#6 1783.9 ** using staged installation
#6 1783.9 0.29.1
#6 1783.9 Package liblz4 was not found in the pkg-config search path.
#6 1783.9 Perhaps you should add the directory containing `liblz4.pc'
#6 1783.9 to the PKG_CONFIG_PATH environment variable
#6 1783.9 No package 'liblz4' found
#6 1783.9 Package liblz4 was not found in the pkg-config search path.
#6 1783.9 Perhaps you should add the directory containing `liblz4.pc'
#6 1783.9 to the PKG_CONFIG_PATH environment variable
#6 1783.9 No package 'liblz4' found
#6 1783.9 Using PKG_CFLAGS= and PKG_LIBS=-llz4
#6 1783.9 ------------------------- ANTICONF ERROR ---------------------------
#6 1783.9 Configuration failed because liblz4 was not found. Try installing:
#6 1783.9  * deb: liblz4-dev (Debian, Ubuntu, etc)
#6 1783.9  * rpm: liblz4-devel (Fedora, CentOS, RHEL)
#6 1783.9  * csw: liblz4_dev (Solaris)
#6 1783.9  * OSX: lz4 (macOS)
#6 1783.9 If liblz4 is already installed, check that 'pkg-config' is in your
#6 1783.9 PATH and PKG_CONFIG_PATH contains a liblz4.pc file. If pkg-config
#6 1783.9 is unavailable you can set INCLUDE_LZ4_DIR and LIB_LZ4_DIR manually via:
#6 1783.9 R CMD INSTALL --configure-vars='INCLUDE_LZ4_DIR=... LIB_LZ4_DIR=...'
#6 1783.9 Alternatively, you can build the required libraries directly from bundled code by
#6 1783.9 setting BUILD_INTERNAL_LIBS to true
#6 1783.9 --------------------------------------------------------------------
#6 1783.9 ERROR: configuration failed for package ‘fstcore’

This is the build log of rocker/tidyverse:devel, which is built on time every day.

liblz4-1 is already installed on this image.
I am wondering if the error is due to the new fstcore version 0.9.10 no longer finding liblz4-1 or if I need to add liblz4-dev as a new dependency.

Allow R_xlen_t start and end rows

eRow = *INTEGER(endRow);

Hi, Mark,I found one issue when partially loading data frame that has nrows >= 2^31. In R, INTEGER() only supports maximum integer of 2^31-1 (~18GB * # of column). However, R_xlen_t is way beyond that on x64 architecture.

In your FstStore::fstRead definition,startRow and endRow allow long long, which (I think) is the definition of R_xlen_t, and has limit of 2^63-1.

FstStore::fstRead(IFstTable &tableReader, IStringArray* columnSelection, const long long startRow, const long long endRow,
  IColumnFactory* columnFactory, vector<int> &keyIndex, IStringArray* selectedCols, IStringColumn* col_names)

For example, the following code can break the limit of INTEGER()

f=tempfile()
fst::write_fst(structure(list(V1=1:(2^31+1)), class = 'data.frame', row.names = c(NA, -1L)), path = f)
.Call(fstcore:::`_fstcore_fstretrieve`, normalizePath(f), "V1", as.integer(2^31-10), 2^31+1)

However, setting endRow be NULL allows me to read the final lines.

.Call(fstcore:::`_fstcore_fstretrieve`, normalizePath(f), "V1", as.integer(2^31-10), NULL)

Best.

Forked package leads to rstudio crashes

From @dipterix:

I forked your package and made some experiments on the forked version. I found my R keeps crashing when I use devtools::load_all() or sometimes recompiling in RStudio.

Did you have any similar experience before? I posted an issue here: r-lib/devtools#2277

Basically OpenMP causes this issue, but it's interesting to me that parallel or future package forked clusters worked fine. So I guess users will still use packages fine, but it's really annoying to devs.

It requires some steps to reproduce

  1. compile & load package
  2. Run some R code that eventually runs OpenMP
  3. change cpp scripts (add lines or delete lines, any change)
  4. call devtools::load_all()

Step 2 is required to reproduce this issue. If step 2 is missing, either works, but if step 2 is present, load_all just crash R session.

I disabled OpenMP and it just worked fine. So I guess either devtools is bugged with OpenMP, or OpenMP was not properly turned off when running load_all.

[feature request] fstretrieve buffer

Hi @MarcusKlik ,

I was wondering how hard it'll be to let fstretrieve (or maybe fstretrieve2) to receive pointer(s) that allows the readers to directly write to the pointers?

For example, something like

template <typename T>
SEXP fstretrieve(Rcpp::String fileName, SEXP columnSelection, SEXP startRow, SEXP endRow, T* buffer, SEXP len);

// or

SEXP fstretrieve(Rcpp::String fileName, SEXP columnSelection, SEXP startRow, SEXP endRow, SEXP buffer, SEXP start);

It's ok if this is hard, I'm using memcpy, which is fast enough, but the only disadvantage is gc takes time.

Configure script should have meaningful output

From CRAN: reverse dependency checks have been triggered, but it reports

1.8.0
Found pkg-config cflags and libs for LZ4!
1.8.0
Found pkg-config cflags and libs for ZSTD!

(and it is not clear to the end user what this means exactly)

Configuration failed because libzstd was not found.

Installing (updating) fstcore on Fedora 35 OS yields this error message:

Found pkg-config cflags and libs for LZ4!
Using PKG_CFLAGS=  and PKG_LIBS=-llz4 
1.8.0
Package libzstd was not found in the pkg-config search path.
Perhaps you should add the directory containing `libzstd.pc'
to the PKG_CONFIG_PATH environment variable
Package 'libzstd', required by 'virtual:world', not found
Package libzstd was not found in the pkg-config search path.
Perhaps you should add the directory containing `libzstd.pc'
to the PKG_CONFIG_PATH environment variable
Package 'libzstd', required by 'virtual:world', not found
Using PKG_CFLAGS=  and PKG_LIBS=-lzstd
------------------------- ANTICONF ERROR ---------------------------
Configuration failed because libzstd was not found. Try installing:
 * deb: libzstd-dev (Debian, Ubuntu, etc)
 * rpm: libzstd-devel (Fedora, CentOS, RHEL)
 * csw: libzstd-dev (Solaris)
 * OSX: zstd (macOS)
If libzstd is already installed, check that 'pkg-config' is in your
PATH and PKG_CONFIG_PATH contains a libzstd.pc file. If pkg-config
is unavailable you can set INCLUDE_ZSTD_DIR and LIB_ZSTD_DIR manually via:
R CMD INSTALL --configure-vars='INCLUDE_ZSTD_DIR=... LIB_ZSTD_DIR=...'
Alternatively, you can build the required libraries directly from bundled code by
setting BUILD_INTERNAL_LIBS to true
--------------------------------------------------------------------
ERROR: configuration failed for package ‘fstcore’
* removing ‘/home/yifanliu/R/x86_64-redhat-linux-gnu-library/4.1/fstcore’
* restoring previous ‘/home/yifanliu/R/x86_64-redhat-linux-gnu-library/4.1/fstcore’

I tried to follow the instruction to install liblz4-devel:
sudo dnf install liblz4-devel

but liblz4-devel is not available to Fedora 35. I suspected you mean lz4-devel, so I installed it, but still had the same error message when installing fstcore.

Session information:

R version 4.1.2 (2021-11-01)
Platform: x86_64-redhat-linux-gnu (64-bit)
Running under: Fedora Linux 35 (Workstation Edition)

Matrix products: default
BLAS/LAPACK: /usr/lib64/libflexiblas.so.3.1

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8       
 [4] LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C              
[10] LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_4.1.2 tools_4.1.2 

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.