Hi there, I have been redirected from Slack to post an issue here. I

Just to add an MWE: <div class="snippet-clipboard-content notranslate position-rel

Ok, merged <a class="issue-link js-issue-link" data-error-text="Failed to load title"

Reading a large CSV file with BigFloats takes a long time about parsers.jl HOT 4 CLOSED

juliadata commented on September 20, 2024

Reading a large CSV file with BigFloats takes a long time

from parsers.jl.

Comments (4)

nilshg commented on September 20, 2024 1

Just to add an MWE:

julia> using CSV, DataFrames

julia> df = DataFrame(rand(2_000_000, 4));

julia> CSV.write("test_out.csv", df);

julia> @time CSV.read("test_out.csv", DataFrame);
  0.569546 seconds (154.47 k allocations: 75.956 MiB, 3.24% gc time)

julia> df = DataFrame(BigFloat.(rand(2_000_000, 4)));

julia> CSV.write("test_out.csv", df);

julia> @time CSV.read("test_out.csv", DataFrame);
629.696438 seconds (6.00 G allocations: 97.848 GiB, 23.51% gc time)

(@v1.5) pkg> st CSV
Status `C:\Users\ngudat\.julia\environments\v1.5\Project.toml`
  [336ed68f] CSV v0.7.7

from parsers.jl.

quinnj commented on September 20, 2024 1

Ok, merged #76 and the perf should be much much better now

from parsers.jl.

quinnj commented on September 20, 2024

Looks like a Parsers.jl issue, I'll dig in to see what we can do to make it faster:

julia> @btime parse(Float64, "0.8720837370748981687285095176775939762592315673828125")
  452.393 ns (0 allocations: 0 bytes)
0.8720837370748982

julia> @btime Parsers.parse(Float64, "0.8720837370748981687285095176775939762592315673828125")
  20.607 μs (654 allocations: 13.08 KiB)
0.8720837370748982