Related to <a class="issue-link js-issue-link" data-error-text="Failed to load title"

Figured it out. This is the culprit, which lives in Base: <div class="highlight h

Best practices for high online statistics for many parameters about onlinestats.jl HOT 4 CLOSED

vandenman commented on June 19, 2024

Best practices for high online statistics for many parameters

from onlinestats.jl.

Comments (4)

joshday commented on June 19, 2024 1

A few things:

1) Thanks for the report! I was able to make a simpler example:

julia> typeof(Group([Mean() for _ in 1:1375]...))
Group{NTuple{1375, Mean{Float64, EqualWeight}}, Union{NTuple{1375, Number}, NamedTuple{names, R} where R<:NTuple{1375, Number}, AbstractVector{<:Number}} where names}

julia> typeof(Group([Mean() for _ in 1:1376]...))
ERROR: StackOverflowError:
Stacktrace:
 [1] promote_type(::Type, ::Type, ::Type, ::Type, ::Vararg{Type}) (repeats 1368 times)
   @ Base ./promotion.jl:293
 [2] Group(stats::NTuple{1376, Mean{Float64, EqualWeight}})
   @ OnlineStatsBase ~/.julia/dev/OnlineStatsBase/src/stats.jl:368
 [3] Group(::Mean{Float64, EqualWeight}, ::Vararg{Mean{Float64, EqualWeight}})
   @ OnlineStatsBase ~/.julia/dev/OnlineStatsBase/src/stats.jl:372

I am at a loss as to why 1375 would be different than 1376. However, there is a one-line fix in OnlineStatsBase that I'll add:

# don't do this.  See below
julia> typeof(Group([Mean() for _ in 1:999999]...))
Group{NTuple{999999, Mean{Float64, EqualWeight}}, Union{Tuple{Number}, NamedTuple{names, R} where R<:Tuple{Number}, AbstractVector{<:Number}} where names}

2) For large numbers of stats in a `Group`, use a `Vector`:

julia> typeof(Group([Mean() for _ in 1:999999]))
Group{Vector{Mean{Float64, EqualWeight}}, Union{Tuple{Number}, NamedTuple{names, R} where R<:Tuple{Number}, AbstractVector{<:Number}} where names}

I'll have to do some benchmarking. There may be no benefit for using tuples even for a smaller number of stats.

3) If you already have a `Variance`, don't add in a `Mean`

A Variance needs to calculate the mean internally, so you're adding unnecessary compute by including a Mean as well. You can do:

o = fit!(Variance(), randn(100))

mean(o)

from onlinestats.jl.

joshday commented on June 19, 2024

Eh, I lied. My "fix" broke other stuff. This may be some internal Julia limitation on tuple sizes. I'll look into it.

from onlinestats.jl.

joshday commented on June 19, 2024

Figured it out. This is the culprit, which lives in Base:

promote_type(T, S, U, V...) = (@inline; promote_type(T, promote_type(S, U, V...)))

For a large number of arguments, e.g. promote_type(many_things...), this method is called over and over so we hit a stack overflow. Fortunately, changing some OnlineStatsBase code from promote_type(types...) to reduce(promote_type, types) fixes everything.

Re: Vectors vs. Tuples as the Group container: Tuples are faster for a small number of items.

from onlinestats.jl.

joshday commented on June 19, 2024

new OnlineStatsBase release is pending.

from onlinestats.jl.

Best practices for high online statistics for many parameters about onlinestats.jl HOT 4 CLOSED

Comments (4)

1) Thanks for the report! I was able to make a simpler example:

2) For large numbers of stats in a `Group`, use a `Vector`:

3) If you already have a `Variance`, don't add in a `Mean`

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

Comments (4)

1) Thanks for the report! I was able to make a simpler example:

2) For large numbers of stats in a Group, use a Vector:

3) If you already have a Variance, don't add in a Mean

Related Issues (20)

Recommend Projects

Recommend Topics

Recommend Org

2) For large numbers of stats in a `Group`, use a `Vector`:

3) If you already have a `Variance`, don't add in a `Mean`