Comments (3)
I'm not sure how big simpleFeatureCollection
is but one thing to keep in mind with your current approach is that you (probably) get 16 copies of it, one for each worker, and that could be expensive
from furrr.
Are you sure mediumSizeRaster = rast("twoGBraster.tif")
is actually resulting in an object? It looks like it uses a relative path so the working directory on the worker may be different. You could try supplying an absolute path instead.
from furrr.
If the error message for workers > 1
is to be believed, it appears somehow the call to rast()
is getting skipped—even if the statement was executed and rast()
had some silent error leading it to somehow return NULL
instead of failing properly that should still result in the parser adding mediumSizeRaster
as a workspace variable. So it seems like something might be going pretty badly wrong though, given future's limitations for flowing diagnostics from workers back to their caller, we might be stuck. (I find myself often wishing for plan(multithread)
but that's not on furrr.)
If it was a pathing issue, which it presumably isn't since there's no issue with workers = 1
, I'd expect to see something like the usual
Error: [rast] file does not exist: twoGBraster.tif
In addition: Warning message:
twoGBraster.tif: No such file or directory (GDAL error 4)
come back. But future may not be able to route that.
I'm not sure how big
simpleFeatureCollection
is
Good question! It's only a couple MB, so negligible in this context—32 workers would be better but even 8 GB per worker is maybe asking too much (if this approach to the task had worked I was prepared to kill the future_map()
and try with eight workers to get 16 GB DDR per worker if physical memory was going to be exceeded).
from furrr.
Related Issues (20)
- Error indices from purrr() are misleading in furrr()
- Getting the same random result across `purrr::map()` and `furrr::future_map()` HOT 3
- `furrr` much slower than `purrr` (on Windows) HOT 2
- Export a wrapper around purrr `map()` to workers rather than `map()` itself
- Setting `seed = TRUE` / `.progress=TRUE` globally in the R session HOT 1
- Carrier HOT 2
- mgcv::gam(~s(pc = object not found)) when future::plan(multisession, workers > 1) HOT 1
- when large matrix or df split into list, does furrr workers only access list element? HOT 1
- error: external pointer is not valid is not a particularly user actionable message HOT 5
- Understanding memory usage and performance of `furrr::future_apply` HOT 5
- Implement map_vec
- Use of variables in glue() inside furrr loops HOT 1
- Progress bar showing 100% even if not completed
- Error when plotting a {sf} object
- Identical RNG state for each task, despite setting different seeds in different tasks HOT 10
- How to pass to .f, character vectors in .x that reference global values?
- `future_map` not obviously faster than `map` in simple linear regression setting HOT 1
- Multiple calls of future_map() within a single plan() or script result in massive slowdown.
- furrr_options documentation doesn't say what to do with it
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from furrr.