HI:
saddle is perfect tools read csv in scala like pandas,but for huge num rows csv,we need multiple threads to read the csv file to increase the load the data speed. in one thread read csv is no problem, but for multiple thread the saddle can not read the csv file, and meet the error [ CharsetDecoder Current state = RESET, new state = FLUSHED ] in val dataCsvOpt = csv.CsvParser.parseFile[String](csvFileHandle,fieldSeparator='\t',recordSeparator = "\n").toOption

Placeholder issue for the scala 3 support effort.
Have some challenges been already identified for this migration?

Frame(
  Series(0 -> 1,2 -> 2, 1 -> 3, 0 -> 4),
  Series(1 -> 1,2 -> 2, 0 -> 3, 0 -> 4),
  Series(0 -> 1,1 -> 2, 2 -> 3, 0 -> 4)
)

yields

val res3: org.saddle.Frame[Int,Int,Int] =
[10 x 3]
      0  1  2
     -- -- --
0 ->  1  3  1
0 ->  1  3  4
0 ->  1  4  1
0 ->  1  4  4
0 ->  4  3  1
0 ->  4  3  4
0 ->  4  4  1
0 ->  4  4  4
2 ->  2  2  3
1 ->  3  1  2

If the row indices are not the same and they contain duplicates then it is doing a series of cross products. This is looses no data and from a certain pespective this is correct. However it is rarely the expected behavior and on any medium sized data it blows up.

A few questions come up when browsing saddle doc.

How does saddle achieve bin operator vectorization? What is the difference between regular operators and macro operators?
The original documentation has a short section on the topic without much explanation: https://saddle.github.io/doc/quickstart.html#a-note-on-optimization

I didn't find these answers easily/at all in the current doc. A doc section on this topic would be great.

I noticed mixed used of present and imperative tense in the scala doc.
Is there a preference?

Currently, saddle uses one value of each primitive type to represent NA. For floating point numbers, this is straightforward as they already include such a value. For other types (Boolean, Byte, Int, etc), it isn't straightforward and an arbitrary value must be used. Currently the minimum value is used (Byte.MinValue, Short.MinValue, etc).

I think this approach has important drawbacks:

• users are unlikely to know about that encoding and could use the min values. That leads to surprising behavior.
• operations resulting in the MinValue would result in a missing value.
• the binary operations on collections lose in simplicity. Implementation such as if (tag.isMissing(v1)) v1 else v1 + 2 might prevent loop optimizations of the jvm to kick-in.
• the .raw(i)-like api exposes unnecessary complexity to the user.

An alternative approach would be to use a mask-based implementation for the integer-based Vec[T]s. That is, the vector stores a companion Array[Boolean] indicating missing value. This approach is used by pandas.

HI ,when I create the Frame[Int,int,string] by saddle ,I want to compute one field column every element count, like

frame.columnAt(index).groupby( ele => ele.count), but I found can not coding like this, how to do it,thanks

  "Outer join of same non-unique indexes " in {
      val ix1 = Index(0, 0)
      val ix2 = Index(0, 0)
      val res = ix1.join(ix2, how = index.OuterJoin)
      /*
        Outer join two columns with repeated values should generate all possible pairs

        Correct assertions:
      res.index must_== Index(0, 0, 0, 0)
      res.lTake.get must_== Array(0, 0, 1, 1)
      res.rTake.get must_== Array(0, 1, 0, 1)

      However the current implementation has a shortcut for identical indexes and passes these:
       */
      res.index must_== Index(0, 0)
      res.lTake must_== None
      res.rTake must_== None

    }

This is because in the implementation there is a shortcut:

def join(left: Index[T], right: Index[T], how: JoinType): ReIndexer[T] = {
    if (left == right) {
      ReIndexer(None, None, right)
    } else if..

The above condition should check for uniqueness.

org.saddle.Vec(()).map(println) throws

java.lang.IllegalArgumentException
  at java.lang.reflect.Array.newArray(Native Method)
  at java.lang.reflect.Array.newInstance(Array.java:75)
  at scala.reflect.ClassTag.newArray(ClassTag.scala:66)
  at scala.reflect.ClassTag.newArray$(ClassTag.scala:65)
  at org.saddle.scalar.ScalarTagAny.newArray(ScalarTagAny.scala:23)
  at scala.collection.IterableOnceOps.toArray(IterableOnce.scala:1278)
  at scala.collection.IterableOnceOps.toArray$(IterableOnce.scala:1276)
  at scala.collection.AbstractIterable.toArray(Iterable.scala:919)
  at org.saddle.Vec$.apply(Vec.scala:52)

Even if not very useful, it should not blow up.

Unexpected: ScalarTagChar.isMissing(na.to[Char]) returns false.

I noticed this by by testing Vec[Char]('a', 'b', na).fillFoward() which didn't yield the result expected.

The current hugo theme does not support page tocs. Switching to a doc theme that supports it would improve the doc navigability.
Example of page toc here: https://doks.netlify.app/docs/prologue/introduction/

Two potential alternatives:

• switch to an hugo template that supports it. This one or this one are great candidates.
• switch to docausaurus and leverage mdoc integrations.

Cherry on the cake: all these doc templates have search bars.

My suggestion is a switch to docausaurus and mdoc integration because many other reference libraries in the scala ecosystem use it.

I'd be glad to take this task.

Scala 3 support is blocked on the lack of @ specialized annotation. Boxing would turn a fast and terminating program into a slow and non-terminating (OOM) program.

Hotspot has support for auto-vectorization. That is, the jvm will identify certain looping patterns on arrays and generate vectorized instructions (SIMD).

Investigate whether saddle triggers these optimizations by inspecting the produced x86_64 assembly. Good candidates for the investigations are the binary operations implemented in VecSclrElemOp, VecVecElemOp, etc.

import org.saddle._
Panel(Vec(1, 2, 3), Vec("hello", "world", "!")).T

throws

java.lang.ArrayStoreException
	at java.lang.System.arraycopy(Native Method)
	at org.saddle.array.package$.$anonfun$flatten$2(package.scala:623)
	at org.saddle.array.package$.$anonfun$flatten$2$adapted(package.scala:621)
	at scala.collection.immutable.Vector.foreach(Vector.scala:1856)
	at org.saddle.array.package$.flatten(package.scala:621)
	at org.saddle.scalar.ScalarTagAny.concat(ScalarTagAny.scala:64)
	at org.saddle.Frame.toMat(Frame.scala:1426)
	at org.saddle.Frame.T(Frame.scala:168)
	at repl.MdocSession$App.<init>(scalar.worksheet.sc:11)
	at repl.MdocSession$.app(scalar.worksheet.sc:3)

Hi @pityka,

Do you have some tasks in mind for the project? Especially smaller ones that new contributors can potentially handle?
If so, don't hesitate to document them somewhere and/or create github issues :).

Cheers

scoverage+codecov was removed a while back for two reasons:

1. scoverage did not support some Scala 2.12.x versions. This is not an issue today any more.
2. https://about.codecov.io/security-update/

• drop implicit param of Series.proxyWith
• fix Melter.melt3_2 (type params A,B,C,D,E and not A, B, C, D)
• remove with AnyRef in Scalar.ord
• widen all implicit defs of Slice and ScalarTag. Don't use subtypes or object types. Review all implicit defs.
• ...

Add a method Frame.get(r,c) similar to Series.get(k).

Current way is Frame.first(r).get(c).

There is a bug in the new csv parser which manifest if the buffer ends around a line break. Can be fixed by using a larger buffer. Needs further investigation to find the specific cause.