Comments (4)
Thanks for reporting. Yes, it's likely that there's a race condition for updating the internal Rprofmem file. This file is unique per R session;
Lines 202 to 209 in 46f63f0
but it is shared across forked R processes as when using mclapply()
or plan(multicore)
.
It shouldn't be too hard to make it unique to each forked child process. I'll add it to the todo list for the next release.
from profmem.
Forgot to say, fixing this in the 'profmem' package will not fix the closely-related problem in bench::mark()
. The 'bench' package uses 'profmem' only for parsing an Rprofile file (profmem::readRprofmem()
) - it does not rely on the other parts of profmem.
from profmem.
Follow up/for the record: This problem can be reproduced as:
> p <- profmem::profmem(y <- parallel::mclapply(1:10, rnorm))
Error in parse(text = trace) : <text>:1:103: unexpected symbol
1: c("FUN", "lapply", "doTryCatch", "tryCatchOne", "tryCatchList", "tryCatch", "try", "sendMaster", "FUN"chList
^
> traceback()
8: parse(text = trace)
7: eval(parse(text = trace), enclos = baseenv())
6: FUN(X[[i]], ...)
5: lapply(bfr, FUN = function(x) {
bytes <- gsub(pattern, "\\1", x)
what <- rep("alloc", times = length(x))
idxs <- which(bytes == "new page")
if (length(idxs) > 0) {
what[idxs] <- "new page"
bytes[idxs] <- ""
}
bytes <- as.numeric(bytes)
trace <- gsub(pattern, "\\2", x)
trace <- gsub("\" \"", "\", \"", trace, fixed = TRUE)
trace <- sprintf("c(%s)", trace)
trace <- eval(parse(text = trace), enclos = baseenv())
trace <- trace[seq_len(max(0L, length(trace) - drop))]
list(what = what, bytes = bytes, trace = trace)
})
4: readRprofmem(pathname, drop = drop)
3: profmem_suspend()
2: profmem_begin(threshold = threshold)
1: profmem::profmem(y <- parallel::mclapply(1:10, rnorm))
Not surprisingly, it can also be reproduced with plain utils::Rprofmem()
:
> pf <- tempfile()
> local({ utils::Rprofmem(pf); on.exit(utils::Rprofmem("")); y <- parallel::mclapply(1:10, FUN = function(ii) integer(1e6)) })
> p <- profmem::readRprofmem(pf)
Error in parse(text = trace) : <text>:1:18: unexpected symbol
1: c("FUN", "l272 :"get0
^
> traceback()
5: parse(text = trace)
4: eval(parse(text = trace), enclos = baseenv())
3: FUN(X[[i]], ...)
2: lapply(bfr, FUN = function(x) {
bytes <- gsub(pattern, "\\1", x)
what <- rep("alloc", times = length(x))
idxs <- which(bytes == "new page")
if (length(idxs) > 0) {
what[idxs] <- "new page"
bytes[idxs] <- ""
}
bytes <- as.numeric(bytes)
trace <- gsub(pattern, "\\2", x)
trace <- gsub("\" \"", "\", \"", trace, fixed = TRUE)
trace <- sprintf("c(%s)", trace)
trace <- eval(parse(text = trace), enclos = baseenv())
trace <- trace[seq_len(max(0L, length(trace) - drop))]
list(what = what, bytes = bytes, trace = trace)
})
1: profmem::readRprofmem(pf)
As OP says, this is because forked parallel processing is used. This is because R's internal do_Rprofmem()
and more precisely R_InitMemReporting()
/R_EndMemReporting()
are not fork safe; all forked child processes will write to the exact same Rprofmem file as the parent process.
Actions: I'm not sure there is much that can be done in the profmem package. I don't think it's possible to protect against using Rprofmem()
in forked processing. Maybe one could give a more informative error message, e.g. "Error: ... Was memory profiling run with forked processing? If so, that is unfortunately not safe".
from profmem.
Here's an example of intertwined Rprofmem output from multiple forked processes:
192 :"FUN" "lapply" "<Anonymous>" "<Anonymous>"
328 :"FUN" "lapply" "<Anonymous>" "<Anonymous>"
192 :"FUN" "lapply" "<Anonynew page:"tolower" ".check_ncores" "<Anonymous>" "<Anonymous>"
new page:"seq.default" "seq" "FUN" "lapply" "<Anonymous>" "<Anonymous>"
new page:"seq.default" "seq" "FUN" "lapply" "<Anonymous>" "<Anonymous>"
Note that line with '... "<Anonynew page:"tolower" ...'. This gives:
Error in parse(text = trace) : <text>:1:37: unexpected symbol
1: c("FUN", "lapply", "<Anonynew page:"tolower
One option could be the replace non-parsable entries with a <non-parsable>
result and produce a warning that the captured and parsed Rprofmem results might not be correct. One could also give more control to the user on what should happen via an onParseError=c("error", "warning", "ignore")
argument to readRprofmem()
.
from profmem.
Related Issues (20)
- Don't display NA by default HOT 1
- DON"T FORGET: What causes vignette to use threshold = 1000? HOT 1
- profmem_status(): "inactive", "active", "suspended"
- profmem_depth(): current depth >= 0
- DOCS: Document the Rprofmem data.frame
- Option profmem.print.expr=TRUE/FALSE
- ROBUSTNESS: Add explicit 'stringsAsFactors' arguments [data.frame]
- ROBUSTNESS: Add explicit 'stringsAsFactors' arguments [cbind, rbind] HOT 1
- Shiny-based example HOT 3
- Different result first time I run profmem... HOT 1
- Errors are silenced
- print() for Rprofmem should report if there was an error while profiling
- VIGNETTE: Integers `5000L` in code snippets are displayed as `5000` HOT 2
- Memory profiling as of R 3.2.0 only? HOT 1
- profmem_begin() and profmem_end()
- Support for nested `profmem()` calls
- VIGNETTE: Was R updated such that example is no longer valid? HOT 3
- Export profmem_suspend() and profmem_resume()
- The threshold argument
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from profmem.