Giter Club home page Giter Club logo

Comments (3)

braaannigan avatar braaannigan commented on June 1, 2024 1

Closed by #211

from polars-book.

Trippy3 avatar Trippy3 commented on June 1, 2024

I have investigated the above.
About "projection-pushdown", it seems to be simply a matter of specifying the code path.

-![](./../outputs/projection_pushdown/graph.png)
+![](../../outputs/projection_pushdown/graph.png)

About "predicate-pushdown", I am wondering if the return value of to_dot is as expected.
I did a debug print in frame.py and the results are below. 'graph polars_query {\n\n\n}'

        dot = _ldf.to_dot(optimized)
        print(f"{dot=}")
(.venv) $ python -m user_guide.src.examples.predicate_pushdown
dot='graph  polars_query {\n"FILTER BY col(\\"name\\").str.contains(... [(0, 0)]" -- "FILTER BY (col(\\"link_karma\\")) > (0i... [(0, 1)]"\n"FILTER BY (col(\\"link_karma\\")) > (0i... [(0, 1)]" -- "FILTER BY (col(\\"comment_karma\\")) > ... [(0, 2)]"\n"FILTER BY (col(\\"comment_karma\\")) > ... [(0, 2)]" -- "CSV SCAN data/reddit.csv;\nπ */6;\nσ -; [(0, 3)]"\n\n"FILTER BY (col(\\"link_karma\\")) > (0i... [(0, 1)]"[label="FILTER BY (col(\\"link_karma\\")) > (0i..."]\n"CSV SCAN data/reddit.csv;\nπ */6;\nσ -; [(0, 3)]"[label="CSV SCAN data/reddit.csv;\nπ */6;\nσ -;"]\n"FILTER BY col(\\"name\\").str.contains(... [(0, 0)]"[label="FILTER BY col(\\"name\\").str.contains(..."]\n"FILTER BY (col(\\"comment_karma\\")) > ... [(0, 2)]"[label="FILTER BY (col(\\"comment_karma\\")) > ..."]\n\n}'
dot='graph  polars_query {\n\n\n}'
dot='graph  polars_query {\n"FILTER BY (((col(\\"comment_karma\\")) ... [(0, 0)]" -- "CSV SCAN data/reddit.csv;\nπ */6;\nσ -; [(0, 1)]"\n\n"FILTER BY (((col(\\"comment_karma\\")) ... [(0, 0)]"[label="FILTER BY (((col(\\"comment_karma\\")) ..."]\n"CSV SCAN data/reddit.csv;\nπ */6;\nσ -; [(0, 1)]"[label="CSV SCAN data/reddit.csv;\nπ */6;\nσ -;"]\n\n}'
dot='graph  polars_query {\n\n\n}'

I guess I need to trace to_dot in dot.rs any further.

from polars-book.

Trippy3 avatar Trippy3 commented on June 1, 2024

I did some more research on the above.　This issue seems to be relevant as well. pola-rs/polars#5570

Print Logical Plan

polars/polars-lazy/src/dot.rs: println!("{:?}", &logical_plan);

  • optimized=False
  FILTER col("name").str.contains() FROM
    FILTER [(col("link_karma")) > (0i64)] FROM
      FILTER [(col("comment_karma")) > (0i64)] FROM
        CSV SCAN data/reddit.csv
        PROJECT */6 COLUMNS
        SELECTION: None
  • optimized=True
  CSV SCAN data/reddit.csv
  PROJECT */6 COLUMNS
  SELECTION: Some([([(col("name").str.contains()) & ([(col("link_karma")) > (0i64)])]) & ([(col("comment_karma")) > (0i64)])])

dot

Get a dot language representation of the LogicalPlan.
When id and branch are 0, node evaluation is not performed.

polars/polars-lazy/polars-plan/src/dot.rs: write_dot

        if current_node.id == 0 && current_node.branch == 0 {
            writeln!(acc_str, "graph  polars_query {{")
        } else {
            let fmt_prev_node = prev_node.fmt.replace('"', r#"\""#);
            let fmt_current_node = current_node.fmt.replace('"', r#"\""#);


            let id_prev_node = format!(
                "\"{} [{:?}]\"",
                &fmt_prev_node,
                (prev_node.branch, prev_node.id)
            );
            let id_current_node = format!(
                "\"{} [{:?}]\"",
                &fmt_current_node,
                (current_node.branch, current_node.id)
            );


            writeln!(acc_str, "{} -- {}", &id_prev_node, &id_current_node)?;


            id_map.insert(id_current_node, fmt_current_node);
            id_map.insert(id_prev_node, fmt_prev_node);


            Ok(())
        }

If the first Plan is Selection, then write_dot will call dot recursively after write_dot, so the "dot language" is correctly represented.

                self.write_dot(acc_str, prev_node, current_node, id_map)?;
                input.dot(acc_str, (branch, id + 1), current_node, id_map)
            }

On the other hand, if the first Plan is CsvScan, write_dot is called only once and returns only "graph polars_query {{".

                self.write_dot(acc_str, prev_node, current_node, id_map)
            }

Therefore, I think that either of the following should be addressed

  • Fix write_dot.
  • Fix node_to_lp (if an unexpected Logical Plan is returned in the first place)

from polars-book.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.