Giter Club home page Giter Club logo

activerecord-hierarchical_query's Introduction

ActiveRecord::HierarchicalQuery

Code Climate Coverage Status Gem Version

Create hierarchical queries using simple DSL, recursively traverse trees using single SQL query.

If a table contains hierarchical data, then you can select rows in hierarchical order using hierarchical query builder.

Requirements

  • ActiveRecord >= 5.0, < 8
  • PostgreSQL >= 8.4
  • Postgres Gem >= 0.21, < 1.5

Note that though PostgresSQL 8.4 and up should work, this library is tested on PostgresSQL 15.1.

In a nutshell

Traverse trees

Let's say you've got an ActiveRecord model Category that related to itself:

class Category < ActiveRecord::Base
  belongs_to :parent, class_name: 'Category'
  has_many :children, foreign_key: :parent_id, class_name: 'Category'
end

# Table definition
# create_table :categories do |t|
#   t.integer :parent_id
#   t.string :name
# end

Traverse descendants

Category.join_recursive do |query|
  query.start_with(parent_id: nil)
       .connect_by(id: :parent_id)
       .order_siblings(:name)
end # returns ActiveRecord::Relation instance

Traverse ancestors

Category.join_recursive do |query|
  query.start_with(id: 42)
       .connect_by(parent_id: :id)
end

Show breadcrumbs using single SQL query

records = Category.join_recursive do |query|
  query
    # assume that deepest node has depth=0
    .start_with(id: 42) { select('0 depth') }
    # for each ancestor decrease depth by 1, do not apply
    # following expression to first level of hierarchy
    .select(query.prior[:depth] - 1, start_with: false)
    .connect_by(parent_id: :id)
end.order('depth ASC')

# returns a regular ActiveRecord::Relation instance
# so methods like `pluck` all work as expected.

crumbs = records.pluck(:name).join(' / ')

Installation

Add this line to your application's Gemfile:

gem 'activerecord-hierarchical_query'

And then execute:

$ bundle

Or install it yourself as:

$ gem install activerecord-hierarchical_query

You'll then need to require the gem:

require 'active_record/hierarchical_query'

Alternatively, the require can be placed in the Gemfile:

gem 'activerecord-hierarchical_query', require: 'active_record/hierarchical_query'

Usage

Let's say you've got an ActiveRecord model Category with attributes id, parent_id and name. You can traverse nodes recursively starting from root rows connected by parent_id column ordered by name:

Category.join_recursive do
  start_with(parent_id: nil).
  connect_by(id: :parent_id).
  order_siblings(:name)
end

Hierarchical queries consist of these important clauses:

  • START WITH clause

    This clause specifies the root row(s) of the hierarchy.

  • CONNECT BY clause

    This clause specifies relationship between parent rows and child rows of the hierarchy.

  • ORDER SIBLINGS clause

    This clause specifies an order of rows in which they appear on each hierarchy level.

These terms are borrowed from Oracle hierarchical queries syntax.

Hierarchical queries are processed as follows:

  • First, root rows are selected -- those rows that satisfy START WITH condition in order specified by ORDER SIBLINGS clause. In example above it's specified by statements query.start_with(parent_id: nil) and query.order_siblings(:name).

  • Second, child rows for each root rows are selected. Each child row must satisfy condition specified by CONNECT BY clause with respect to one of the root rows (query.connect_by(id: :parent_id) in example above). Order of child rows is also specified by ORDER SIBLINGS clause.

  • Successive generations of child rows are selected with respect to CONNECT BY clause. First the children of each row selected in step 2 selected, then the children of those children and so on.

START WITH

This clause is specified by start_with method:

Category.join_recursive { start_with(parent_id: nil) }
Category.join_recursive { start_with { where(parent_id: nil) } }
Category.join_recursive { start_with { |root_rows| root_rows.where(parent_id: nil) } }

All of these statements are equivalent.

CONNECT BY

This clause is necessary and specified by connect_by method:

# join parent table ID columns and child table PARENT_ID column
Category.join_recursive { connect_by(id: :parent_id) }

# you can use block to build complex JOIN conditions
Category.join_recursive do
  connect_by do |parent_table, child_table|
    parent_table[:id].eq child_table[:parent_id]
  end
end

ORDER SIBLINGS

You can specify order in which rows on each hierarchy level should appear:

Category.join_recursive { order_siblings(:name) }

# you can reverse order
Category.join_recursive { order_siblings(name: :desc) }

# arbitrary strings and Arel nodes are allowed also
Category.join_recursive { order_siblings('name ASC') }
Category.join_recursive { |query| query.order_siblings(query.table[:name].asc) }

WHERE conditions

You can filter rows on each hierarchy level by applying WHERE conditions:

Category.join_recursive do
  connect_by(id: :parent_id).where('name LIKE ?', 'ruby %')
end

You can even refer to parent table, just don't forget to include columns in SELECT clause!

Category.join_recursive do |query|
  query.connect_by(id: :parent_id)
       .select(:name).
       .where(query.prior[:name].matches('ruby %'))
end

Or, if Arel semantics does not fit your needs:

Category.join_recursive do |query|
  query.connect_by(id: :parent_id)
       .where("#{query.prior.name}.name LIKE ?", 'ruby %')
end

NOCYCLE

Recursive query will loop if hierarchy contains cycles (your graph is not acyclic). NOCYCLE clause, which is turned off by default, could prevent it.

Loop example:

node_1 = Category.create
node_2 = Category.create(parent: node_1)

node_1.parent = node_2
node_1.save

node_1 and node_2 now link to each other, so the following query will not terminate:

Category.join_recursive do |query|
  query.connect_by(id: :parent_id)
       .start_with(id: node_1.id)
end

#nocycle method will prevent endless loop:

Category.join_recursive do |query|
  query.connect_by(id: :parent_id)
       .start_with(id: node_1.id)
       .nocycle
end

DISTINCT

By default, the union term in the Common Table Expression uses a UNION ALL. If you want to SELECT DISTINCT CTE values, add a query option for distinct:

Category.join_recursive do |query|
  query.connect_by(id: :parent_id)
       .start_with(id: node_1.id)
       .distinct
end

If you want to join CTE terms by UNION DISTINCT, pass an option to join_recursive:

Category.join_recursive(union_type: :distinct) do |query|
  query.connect_by(id: :parent_id)
       .start_with(id: node_1.id)
end

Generated SQL queries

Under the hood this extensions builds INNER JOIN to recursive subquery.

For example, this piece of code

Category.join_recursive do |query|
  query.start_with(parent_id: nil) { select('0 LEVEL') }
       .connect_by(id: :parent_id)
       .select(:depth)
       .select(query.prior[:LEVEL] + 1, start_with: false)
       .where(query.prior[:depth].lteq(5))
       .order_siblings(:position)
       .nocycle
end

Would generate following SQL:

SELECT "categories".*
FROM "categories" INNER JOIN (
    WITH RECURSIVE "categories__recursive" AS (
        SELECT depth,
               0 LEVEL,
               "categories"."id",
               "categories"."parent_id",
               ARRAY["categories"."position"] AS __order_column,
               ARRAY["categories"."id"] AS __path
        FROM "categories"
        WHERE "categories"."parent_id" IS NULL

        UNION ALL

        SELECT "categories"."depth",
               "categories__recursive"."LEVEL" + 1,
               "categories"."id",
               "categories"."parent_id",
               "categories__recursive"."__order_column" || "categories"."position",
               "categories__recursive"."__path" || "categories"."id"
        FROM "categories" INNER JOIN
             "categories__recursive" ON "categories__recursive"."id" = "categories"."parent_id"
        WHERE ("categories__recursive"."depth" <= 5) AND
              NOT ("categories"."id" = ANY("categories__recursive"."__path"))
    )
    SELECT "categories__recursive".* FROM "categories__recursive"
) AS "categories__recursive" ON "categories"."id" = "categories__recursive"."id"
ORDER BY "categories__recursive"."__order_column" ASC

If you want to use a LEFT OUTER JOIN instead of an INNER JOIN, add a query option for outer_join_hierarchical. This option allows the query to return non-hierarchical entries:

  .join_recursive(outer_join_hierarchical: true)

If, when joining the recursive view to the main table, you want to change the foreign_key on the recursive view from the primary key of the main table to another column:

  .join_recursive(foreign_key: another_column)

Related resources

Contributing

Read through the short contributing guide.

activerecord-hierarchical_query's People

Contributors

ahacop avatar anfinil avatar artplan1 avatar ebakan avatar felipemaier avatar healpay-mike avatar keithmgould avatar susanblueapron avatar take-five avatar walski avatar zachaysan avatar zachaysan-pickups avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

activerecord-hierarchical_query's Issues

recursive has_many queries

We have a hierarchical tree of X which this works well for but each leaf on that tree has_many Y relations and we need a way to recursively select all of the Y from all descendants of a particular instance x of X

So / in pseudocode

class X < ActiveRecord
  belongs_to :parent, class_name: "X", optional: true, touch: true
  has_many :children, inverse_of: :parent, class_name: "X", foreign_key: :parent

  has_many :y
end

class Y < ActiveRecord
  belongs_to :x
end

Then we would want

x = X.all.sample

ys = x.all_child_ys # this is all elements of Y which are in z.ys for all z which are of type X and children of x

Is there a way to do this now or some ( reasonable ) way to add it as a new feature ?

Rails 5.1 support

Hi there,

Thank you for making this. I know you do not have a lot of time so I've created a branch for Rails 5.1 support and I've placed it here:

https://github.com/zachaysan/activerecord-hierarchical_query/tree/rails-5.1

If you would like me to created a pull request you can either add me to the repo and I can create the same branch for you or you can create the branch yourself and I'll issue the pull request. It's up to you.

All the tests pass after updating some gems and I verified that basic features work with my rails project. I try to keep my rails projects as up to date as possible so I will continue to upgrade this gem as I need it.

For other using rails-5.1 the following code in your Gemfile will do the trick as a temporary measure:

# TODO: Check back at https://github.com/take-five/activerecord-hierarchical_query to replace gem
gem "activerecord-hierarchical_query",
    ">= 1.0.0",
    git: "https://github.com/zachaysan/activerecord-hierarchical_query",
    branch: "rails-5.1"

Column name 'parent' replaces automatically to 'parent_id'

Schema:

  create_table "products", force: :cascade do |t|
    t.string   "title"
    t.text     "description"
    t.integer  "parent" #Foreign key
    t.boolean  "leaf"
  end

code:
Product.join_recursive { start_with(parent: nil).connect_by(id: :parent).order_siblings(:title) }

Error:
PG::UndefinedColumn: ERROR: column products.parent_id does not exist

NOCYCLE query with DISTINCT clause returns extremely long result set

I'm trying to traverse a table with circular references. The table looks like this:

create_table "umbrella_relationships" do |t|
  t.integer "umbrella_id"
  t.integer "underlying_id"
  t.index ["umbrella_id"], name: "index_umbrella_id_on_umbrellables", using: :btree
  t.index ["underlying_id"], name: "index_underlying_id_on_umbrellables", using: :btree
end

There are umbrella_relationships record pairs where the foreign keys permutate (i.e. for a record with { umbrella_id: 1, underlying_id: 2 }, it'd be usual to see another record with { umbrella_id: 2, underlying_id: 1 })

To retrieve umbrella_relationships that "stack" above a given underlying_id, I wrote the following:

UmbrellaRelationship.join_recursive do |query|
  query.
    start_with(underlying_id: id).
    connect_by(umbrella_id: :underlying_id).
    nocycle.
    distinct
end

However, for an example that expects to get 13 records back, this returns a 13310-row result set, with many duplications of the expected 13 records. My first guess is that the query would iterate multiple times through all relationships, using the nocycle method to detect where to stop, but, unless I'm not understanding something correctly, the call to distinct should apply to CTEs, so I fail to see why am I receiving so many records. This scales to be a major problem when expecting longer stacks, where I keep running into OOM errors on the RDBMS's end. Could you please advise on how to minimize memory usage on a query like this?

I have tested this in Postgres 9.4, 9.6 and 12.2, in case it helps. Thanks!

Error: undefined method `bound_attributes` for ActiveRecord::Relation

ruby 2.1.2, rails 3.2.14, gem version 1.0.0
When traversing descendants, i've got error from .rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/activerecord-hierarchical_query-1.0.0/lib/active_record/hierarchical_query/cte/non_recursive_term.rb:18:in `bind_values'

def bind_values
  scope.bound_attributes
end

I've looked into source here, at github, and have found that in master branch you have next lines in that file:

def bind_values
  scope.bind_values
end

When i replaced the line in this local file, i've got similar error for file .rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/activerecord-hierarchical_query-1.0.0/lib/active_record/hierarchical_query/cte/recursive_term.rb
That error had been bypassed the same way and traversing started to work as expected.
Help me, please, to use your very handful gem without that local files editings.

pg-1.4

Is there a reason to not support it? Thanks

PG 1.2.x Support

Hi,

I'd like to start using PG 1.2, and found this incompatibility:

Bundler could not find compatible versions for gem "pg":
  In Gemfile:
    pg (~> 1.2)

    activerecord-hierarchical_query (= 1.2.2) was resolved to 1.2.2, which depends on
      pg (< 1.2, >= 0.21)

Could activerecord-hierarchical_query be updated to support pg 1.2? I'm happy to help if needed, but I thought I'd bring it up in an issue first.

Thanks, and happy new year!

Rails 6

Is there an issue supporting it? Thanks

Error: bind message supplies 2 parameters, but prepared statement requires 1

Hi,

Thanks for writing this gem, it is very helpful and has saved me some headaches. Great work!

There was an issue in one of my apps today where I got a PG::ProtocolViolation when trying to nest a recursive query in the where clause of another query. There is a workaround, but here is a description of the problem:

# Query for the user's organization with all of its parent organizations recursively
organizations = Organization.join_recursive do |query|
  query.start_with(id: current_user.organization.id).connect_by(parent_organization_id: :id)
end

# Query for all licenses that belong to these organization and belong to this app
licenses = License.where(organization_id: organizations, license_app_id: license_app.id)

This fails because the last where clause has a $1 in the prepared statement, but the nested select from the recursive join also has a $1. Full example:

PG::ProtocolViolation: ERROR: bind message supplies 2 parameters, but prepared statement "a5" requires 1 : SELECT "licenses".* FROM "licenses" WHERE "licenses"."organization_id" IN (SELECT "organizations"."id" FROM "organizations" INNER JOIN (WITH RECURSIVE "organizations__recursive" AS ( SELECT "organizations"."id", "organizations"."parent_organization_id" FROM "organizations" WHERE "organizations"."id" = $1 UNION ALL SELECT "organizations"."id", "organizations"."parent_organization_id" FROM "organizations" INNER JOIN "organizations__recursive" ON "organizations__recursive"."parent_organization_id" = "organizations"."id" ) SELECT "organizations__recursive".* FROM "organizations__recursive") AS "organizations__recursive" ON "organizations"."id" = "organizations__recursive"."id") AND "licenses"."license_app_id" = $1

Notice that $1 appears twice. The solution was to do a separate query for the list of IDs like so:

licenses = License.where(organization_id: organizations.pluck(:id)).where(license_app_id: license_app.id)

Do you know of a way this can be fixed?

Thanks,
Justin

to_hash

is there anyway to convert results into nested hashes representing a tree of results?

Rails 7?

Hi,

Is rails 7 support coming?

Having syntax errors with this gem

Hello,

Currently, I'm using this gem to try it out on my data model, and I'm having Postgres SQL syntax errors on the most basic recursive queries. This is the table in question:
create_table "hierarchy_level_elements", force: :cascade do |t| t.string "name" t.bigint "parent_id" t.bigint "level_id", null: false t.datetime "created_at", precision: 6, null: false t.datetime "updated_at", precision: 6, null: false t.index ["level_id"], name: "index_hierarchy_level_elements_on_level_id" t.index ["parent_id"], name: "index_hierarchy_level_elements_on_parent_id" end

And this is my query:
HierarchyLevelElement.join_recursive do |query| query.start_with(parent_id: nil) .connect_by(id: :parent_id) .order_siblings(:name) end

What I get is the following error:
ActiveRecord::StatementInvalid (PG::SyntaxError: ERROR: syntax error at or near ""( SELECT "")

The query that the gem is generating is the following:
SELECT "hierarchy_level_elements".* FROM "hierarchy_level_elements" INNER JOIN (WITH RECURSIVE "hierarchy_level_elements__recursive" AS "( SELECT "hierarchy_level_elements"."id", "hierarchy_level_elements"."parent_id", ARRAY[ROW_NUMBER() OVER (ORDER BY "hierarchy_level_elements"."name" ASC)] AS "__order_column" FROM "hierarchy_level_elements" WHERE "hierarchy_level_elements"."parent_id" IS NULL UNION ALL SELECT "hierarchy_level_elements"."id", "hierarchy_level_elements"."parent_id", "hierarchy_level_elements__recursive"."__order_column"||ROW_NUMBER() OVER (ORDER BY "hierarchy_level_elements"."name" ASC) FROM "hierarchy_level_elements" INNER JOIN "hierarchy_level_elements__recursive" ON "hierarchy_level_elements__recursive"."id" = "hierarchy_level_elements"."parent_id" )" SELECT "hierarchy_level_elements__recursive".* FROM "hierarchy_level_elements__recursive") AS ""hierarchy_level_elements__recursive"" ON "hierarchy_level_elements"."id" = "hierarchy_level_elements__recursive"."id" ORDER BY "hierarchy_level_elements__recursive"."__order_column" ASC LIMIT $1 [["LIMIT", 11]]

Can someone help me? Am I doing something wrong?

Thanks in advance

ActiveRecord::StatementInvalid errors with v1.0.0 and Rails 5

Hi,

I'm trying to use v1.0.0 with Rails 5, and my queries don't quite work. I could really use some help.

This does what is expected (gets the ancestors as a relation):

relation = self.class.join_recursive do |query|
  query.start_with(id: id).connect_by(parent_organization_id: :id)
end

and if I call #to_a on that, I get the correct results. However, if I do any of these things:

relation.first, relation.where(...), relation.limit(...), relation.order(..) I get an error like this:

ActiveRecord::StatementInvalid: PG::ProtocolViolation: ERROR:  bind message supplies 1 parameters, but prepared statement "a3" requires 2

Here's an example of the generated SQL when it fails to on relation.first:

SELECT "organizations".* FROM "organizations"
INNER JOIN
  (WITH RECURSIVE "organizations__recursive" AS
     (SELECT "organizations"."id", "organizations"."parent_organization_id"
      FROM "organizations"
      WHERE "organizations"."deleted_at" IS NULL
        AND "organizations"."id" = $1
        UNION ALL
        SELECT "organizations"."id",
               "organizations"."parent_organization_id"
        FROM "organizations"
        INNER JOIN "organizations__recursive" ON "organizations__recursive"."parent_organization_id" = "organizations"."id" WHERE "organizations"."deleted_at" IS NULL ) SELECT "organizations__recursive".*
   FROM "organizations__recursive") AS "organizations__recursive" ON "organizations"."id" = "organizations__recursive"."id"
WHERE "organizations"."deleted_at" IS NULL
ORDER BY "organizations"."id" ASC
LIMIT $2

Any ideas? Thanks in advance!

LTree Support?

Does this gem support LTree, and if not, are there plans to add support for it?

pg ~>1.0 support

Is there a specific reason why the pg dependency is set to ~> 0.21 in rails5 branch? The pg gem is at 1.1.3 currently.

No tag for 1.4.3 release

There's a new gem version, but no tag for the release. Could you add one, so we can see what is in the new version?

require statement

Initially including the gemfile did not allow me to use the join_recursive function immediately in the console I had to do:

 require 'active_record/hierarchical_query'

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.