Giter Club home page Giter Club logo

jpa-streamer's Introduction

JPAstreamer

JPAstreamer Logo

badge badge License LGPL%20v2.1 blue Join%20Chat

JPAstreamer is a lightweight extension for any JPA provider that allows creation of Java Streams from database content.With a single dependency, your application can immediately operate on database elements using standard Stream operators e.g. filter(), sort() and map().

The following example assumes there is an existing JPA Entity that represents a table containing films as follows:

@Entity
@Table(name = "film", schema = "sakila")
public class Film implements Serializable {

    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    @Column(name = "film_id", nullable = false, updatable = false, columnDefinition = "smallint(5)")
    private Integer filmId;

    @Column(name = "title", nullable = false, columnDefinition = "varchar(255)")
    private String title;

    @Column(name = "length", columnDefinition = "smallint(5)")
    private Integer length;

    @Column(name = "rating", columnDefinition = "enum('G','PG','PG-13','R','NC-17')")
    private String rating;

}

To operate on the elements of the table, JPAstreamer is first initialized with a simple builder (in this case using a persistence unit named "sakila"):

JPAStreamer jpaStreamer = JPAStreamer.of("sakila");

The obtained streamer is then used to create Streams that are rendered to database queries through JPA. For example:

jpaStreamer.stream(Film.class) // Film.class is the @Entity representing the film-table
    .filter(Film$.rating.equal("G"))
    .sorted(Film$.length.reversed().thenComparing(Film$.title.comparator()))
    .skip(10)
    .limit(5)
    .forEach(System.out::println);

This will print films rated G in reversed length order (where films of equal length will be in title order) but skipping the first ten and then printing only the following five films.(Film$ is automatically generated from the Film-table Entity at compile-time by JPAstreamer).

In order to minimize the burden on the JVM and the database, JPAstreamer eventually renders the Stream to SQL like so (example from Hibernate/MySQL):

select
    film0_.film_id as film_id1_1_,
    film0_.description as descript2_1_,
    film0_.language_id as languag11_1_,
    film0_.last_update as last_upd3_1_,
    film0_.length as length4_1_,
    film0_.rating as rating5_1_,
    film0_.rental_duration as rental_d6_1_,
    film0_.rental_rate as rental_r7_1_,
    film0_.replacement_cost as replacem8_1_,
    film0_.special_features as special_9_1_,
    film0_.title as title10_1_
from
    film film0_
where
    film0_.rating=?
order by
    film0_.length desc,
    film0_.title asc
limit ?, ?
["G", 10, 5]

More examples are available in the JPAstreamer documentation.

Install

Since JPAstreamer acts merely as an API extension for existing JPA providers it requires minimal installation and configuration efforts. You only need to specify that the JPAstreamer dependency is required to compile your source code.

Note
JPAStreamer requires the use of Java 11 or later.

Maven

In Maven, the following JPAstreamer dependency is added to the project’s pom.xml-file.

<dependencies>
    <dependency>
        <groupId>com.speedment.jpastreamer</groupId>
        <artifactId>jpastreamer-core</artifactId>
        <version>${jpa-streamer-version}</version>
    </dependency>
</dependencies>


<plugins>
    <!-- Needed by some IDEs -->
    <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>build-helper-maven-plugin</artifactId>
        <version>3.2.0</version>
        <executions>
            <execution>
                <phase>generate-sources</phase>
                <goals>
                    <goal>add-source</goal>
                </goals>
                <configuration>
                    <sources>
                        <source>${project.build.directory}/generated-sources/annotations</source>
                    </sources>
                </configuration>
            </execution>
        </executions>
    </plugin>
</plugins>

Gradle

In Gradle, the following JPAstreamer dependency is added to the project’s build.gradle-file (replace "version" with the latest JPAstremer version):

repositories {
    mavenCentral()
}

dependencies {
    compile "com.speedment.jpastreamer:jpastreamer-core:version"
    annotationProcessor "com.speedment.jpastreamer:fieldgenerator-standard:version"
}

/* Needed by some IDEs */
sourceSets {
    main {
        java {
            srcDir 'src/main/java'
            srcDir 'target/generated-sources/annotations'
        }
    }
}

Contributing

We gladly welcome any form of contributions, whether it is comments and questions, filed issues or pull requests.

Before we can accept your patches we need to establish a common legal ground to protect your rights to your contributions and the users of JPAstreamer. This is done by signing a Contributor License Agreement (CLA) with Speedment, Inc. The details of this process is laid out here.

License

JPAstreamer is released under the LGPL 2.1 License.

Short Video

Here is a short 1-minute video describing JPAstreamer:

video tn

jpa-streamer's People

Contributors

benjaminjacober avatar dekmm avatar dependabot[bot] avatar frascu avatar julgus avatar kuckjwi0928 avatar minborg avatar nitroin avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

jpa-streamer's Issues

Implement filter optimization

Filter intermediate operations can be merged into JPA queries if they use Speedment Predicates to describe filter conditions.

Depends on #10

Count optimization

Starting from the end, the longest string of Intermediate operators that preserve the Size property can safely be removed.

Add support for Predicate mappings

We need a way to map Speedment Predicates to JPA Predicates. Here's a list of Speedment Predicate types for a easier tracking:

  • Constants

  • ALWAYS_TRUE / ALWAYS_FALSE

  • Reference

  • IS_NULL / IS_NOT_NULL

  • Comparable

  • EQUAL / NOT_EQUAL

  • GREATER_THAN

  • GREATER_OR_EQUAL

  • LESS_THAN

  • LESS_OR_EQUAL

  • BETWEEN / NOT_BETWEEN

  • IN / NOT_IN

  • String

  • EQUAL_IGNORE_CASE / NOT_EQUAL_IGNORE_CASE

  • STARTS_WITH / NOT_STARTS_WITH

  • STARTS_WITH_IGNORE_CASE / NOT_STARTS_WITH_IGNORE_CASE

  • ENDS_WITH / NOT_ENDS_WITH

  • ENDS_WITH_IGNORE_CASE / NOT_ENDS_WITH_IGNORE_CASE

  • CONTAINS / NOT_CONTAINS

  • CONTAINS_IGNORE_CASE / NOT_CONTAINS_IGNORE_CASE

  • IS_EMPTY / IS_NOT_EMPTY

Fieldgenerator: Imports of types with duplicate simple name yields generated code to render erroneously

If entity has columns using the same simple name, as:

    @Column(name = "release_date", nullable = false, columnDefinition = "DATE")
    @Temporal(TemporalType.DATE)
    private java.util.Date releaseDate;

    @Column(name = "release_time", nullable = false, columnDefinition = "DATE")
    private java.sql.Date releaseTime;

The generated code cannot be compiled as the reference type is not rendered properly.

public static final Date> releaseDate = ComparableField.create(
     Film.class,
     "release_date",
     Film::getReleaseDate,
     false
); 

What I know so far is that this error cannot be derived to the Speedment generator nor the parsing of types.

Fields: Add custom fields for Collections and Timestamps etc.

Currently all Java Collections are rendered to ReferenceField and Timestamps are generated as ComparableField. These Fields contain methods which may not be meaningful in the given context, hence we should implement specialized Field to better suit the properties of the column e.g. CollectionField.

General pipeline optimizations

If at least one of the following conditions are true:

  • the stream is unordered (e.g. the Stream::unordered has been called)
  • the terminal operation does not require the order to be preserved (for example forEach and min)
  • the stream is parallel and terminal operation does not require the order to be preserved for parallel streams

Then, the longest consecutive string of intermediate operations starting from the back that only affects the order of elements (i.e. Sorted) can be removed.

Implement skip/limit optimization

Skip and limit intermediate operations can be directly merged into the JPA Query when they are in the following forms:

  • .skip(x) -> OFFSET x
  • .limit(x) -> LIMIT x
  • .skip(x).limit(y) -> OFFSET x LIMIT y

.limit(x).skip(y) should be rendered to LIMIT x as the semantics are not the same as .skip(x).limit(y)

Add support for IntStream, LongStream and DoubleStream

Exception in thread "main" java.lang.UnsupportedOperationException
	at com.speedment.jpastreamer.builder.standard.internal.StreamBuilder.mapToInt(StreamBuilder.java:84)
	at com.speedment.jpastreamer.autoclose.standard.internal.AutoClosingStream.mapToInt(AutoClosingStream.java:64)
	at com.speedment.jpastreamer.demo.internal.MapToInt.main(MapToInt.java:20)

Write unit tests for provider modules

The amount of unit tests we currently have is quite low. Postponing this could create a big backlog.

Modules:

  • application-standard
  • autoclose-standard
  • builder-standard
  • criteria-standard
  • merger-standard
  • pipeline-standard
  • preoptimizer-standard
  • renderer-standard
  • typeparser-standard

Reorganize module.info classes in almost all modules

The following order shall be used:

  1. requires
  2. exports
  3. uses
  • application
  • autoclose
  • builder
  • criteria
  • exception
  • field
  • javanine
  • merger
  • pipeline
  • preoptimizer
  • renderer
  • rootfactory

Provider

  • application-standard
  • autoclose-standard
  • builder-standard
  • criteria-standard
  • fieldgenerator-standard
  • merger-standard
  • pipeline-standard
  • preoptimizer-standard
  • renderer-standard
  • typeparser-standard

Fields: Eliminate dependencies

Ideally, we should remove speedment dependencies.

module jpastreamer.field {
    requires com.speedment.common.invariant;
    requires transitive java.persistence;
    requires transitive com.speedment.common.tuple;
    requires transitive com.speedment.common.function;
    requires transitive com.speedment.common.annotation;
    requires transitive com.speedment.runtime.config;
    requires transitive com.speedment.runtime.compute;
    requires transitive com.speedment.runtime.typemapper;

    exports com.speedment.jpastreamer.field;
    exports com.speedment.jpastreamer.field.collector;
    exports com.speedment.jpastreamer.field.comparator;
    exports com.speedment.jpastreamer.field.exception;
    exports com.speedment.jpastreamer.field.expression;
    exports com.speedment.jpastreamer.field.method;
    exports com.speedment.jpastreamer.field.predicate;
    exports com.speedment.jpastreamer.field.predicate.trait;
    exports com.speedment.jpastreamer.field.trait;
    exports com.speedment.jpastreamer.field.util;
}

Add jpastreamer-core module/dependency

Applications that want to make use of JPA Streamer should have a single dependency that they can use to pull in the most relevant modules:

        <dependency>
            <groupId>com.speedment.jpastreamer</groupId>
            <artifactId>jpastreamer-core</artifactId>
            <version>0.1.0</version>
        </dependency>

Optimize count termination

Due to the way JPA Criteria works, we can only fully optimize count terminators if all intermediate operations can be merged into the JPA Criteria. If there are leftover intermediate operations after all optimizations are performed, we can either ignore the leftover operations or count termination has to be performed on the JVM side.

Make rootfactory run under the module system

If run under the module system, the rootfactory cannot load services because it does not declare uses on any and all modules. We need to provide some kind of "proxy" to do this.

Anonymous Lambda reordering

Anonymous Lambdas should be moved down the order as much as possible to allow as much as possible of the stream to be executed on the database side.

field-generator: Bootstrap issue when parsing attribute type which is uncompiled

I ran into a troublesome issue when trying to build the demo repository. It contains three entities; Film, Actor and Language. Film has an attribute named language of type Language as shown below:

@Entity
@Table(name = "film", schema = "sakila")
public class Film implements Serializable {

    /... 

    @ManyToOne
    @JoinColumn(name="language_id", nullable = false)
    private Language language;

    // ... getters and setters 

}

When running mvn clean install, the field-generator throws the following exception:

com.speedment.jpastreamer.fieldgenerator.exception.FieldGeneratorProcessorException: Type with name com.speedment.jpastreamer.test.model.Language was not found.

This is thrown as com.speedment.jpastreamer.test.model.Language has not yet been compiled, hence cannot be found by the class parser. This means we are left with a bootstrap issue. We need a compiled version of the JPA Entity-model before we run the annotation-processor, yet the application logic relies on the output of the annotation-processor and cannot be compiled until we have generated the JPAstreamer metamodel.

Optimize Streams that yield Join-operations

As JPAstreamer makes it easy to stream any database table there are many operations which can be added that results in a non-optimized stream. That is for example the case when accessing fields that depend on other tables. See for example:

        long count = jpaStreamer.stream(Film.class)
                .filter(f -> f.getActors()
                        .stream()
                        .filter(Actor$.lastName.startsWith("A"))
                        .count() > 0)
                .count();

Here, the predicate on the fourth row (actors last names starts with A) should optimally be included as a where-clause in the original Join. As of now, we have to materialize every actor even though we are only interested in a subset of them.

It would be good to find a way to optimize similar queries.

Add documentation and host on GitHub Pages

We have earlier decided to write the documentation in AsciiDocs and version handle it through GitHub. The documents are then rendered with Antora and hosted on Github Pages.

  • Add documentation folder with correct structure in JPA Streamer project
  • Set up Github Pages correctly
  • Integrate Antora
  • Fix branding
  • ManyToOne
  • OneToMany
  • ManyToMany
  • Pivot Data
  • Transactions

The Antora build process is later to be automated.

Fields: Investigate if database type is required

Field such as ComparableField<ENTITY, D, V> and ReferenceField<ENTITY, D, V> require the database column type as an argument. We should investigate if this information is required for JPAStreamer.

fieldgenerator fails under client running with JPMS

Reproduce:

cd jpa-streamer-test
mvn clean install -e

Produces:

Caused by: java.lang.ClassNotFoundException: com.speedment.jpastreamer.field.ComparableField
    at java.net.URLClassLoader.findClass (URLClassLoader.java:471)
    at java.lang.ClassLoader.loadClass (ClassLoader.java:588)
    at java.lang.ClassLoader.loadClass (ClassLoader.java:521)
    at com.speedment.jpastreamer.fieldgenerator.standard.internal.InternalFieldGeneratorProcessor.referenceType (InternalFieldGeneratorProcessor.java:236)
    at com.speedment.jpastreamer.fieldgenerator.standard.internal.InternalFieldGeneratorProcessor.addFieldToClass (InternalFieldGeneratorProcessor.java:145)
    at com.speedment.jpastreamer.fieldgenerator.standard.internal.InternalFieldGeneratorProcessor.lambda$generatedEntity$3 (InternalFieldGeneratorProcessor.java:126)
    at java.lang.Iterable.forEach (Iterable.java:75)
    at com.speedment.jpastreamer.fieldgenerator.standard.internal.InternalFieldGeneratorProcessor.generatedEntity (InternalFieldGeneratorProcessor.java:125)
    at com.speedment.jpastreamer.fieldgenerator.standard.internal.InternalFieldGeneratorProcessor.generateFields (InternalFieldGeneratorProcessor.java:108)
    at com.speedment.jpastreamer.fieldgenerator.standard.internal.InternalFieldGeneratorProcessor.lambda$process$1 (InternalFieldGeneratorProcessor.java:75)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.accept (ForEachOps.java:183)
    at java.util.stream.ReferencePipeline$2$1.accept (ReferencePipeline.java:177)
    at java.util.Iterator.forEachRemaining (Iterator.java:133)

Remove Tuple dependency

Field predicates could implement HasArg0<T0>,HasArg1<T1> and HasArg2<T2> instead.

interface HasArg0<T0> {
   T0 get0();
}

Retaining the Tuple getter names would require minimum effort.

Change field signatures

The current fields need to change to fit JPA. For example,

    static <ENTITY, D, V extends Comparable<? super V>>
    ComparableField<ENTITY, D, V> create(
            ColumnIdentifier<ENTITY> identifier,
            ReferenceGetter<ENTITY, V> getter,
            ReferenceSetter<ENTITY, V> setter,
            TypeMapper<D, V> typeMapper,
            boolean unique) {
        
        return new ComparableFieldImpl<>(
            identifier, getter, setter, typeMapper, unique
        );
    }

should be

    static <ENTITY, D, V extends Comparable<? super V>>
    ComparableField<ENTITY, D, V> create(
            Class<ENTITY> root, // Changed
            ReferenceGetter<ENTITY, V> getter,
            ReferenceSetter<ENTITY, V> setter,
            Class<AttributeConverter<V, D>> attributeConverter, // Changed
            boolean unique) {
        
        return new ComparableFieldImpl<>(
            root, getter, setter, attributeConverter, unique
        );
    }

Squash sequential operations of same type

Intermediate operations appearing in sequential order that are of the same type can be squashed into a single operation, making them easier to optimize later.

Examples:

  • .skip(5).skip(5) can be squashed into .skip(10)
  • .filter(_Film.filmId.between(1, 100)).filter(_Film.filmId.greaterThan(100)) can be squashed into .filter(_Film.filmId.between(1, 100).and(_Film.filmId.greaterThan(100))

JPA range inclusion

JPA's CriteriaBuilder::between method is inclusive on both ends (c >= x AND c <= y) and it seems that this behaviour can't be modified. This can cause problems when mapping our Predicates that contain an inclusion strategy different from the one JPA provides.

The between method accepts Comparable objects to determine the start and end of the range. While we can mimic different inclusions for Number types by incrementing/decrementing the provided values, we can't really account for all possible Comparable types that might be provided.

Fields: Todos

remove annotation dependency
Swap order of D and V parameters to conform to AttributeConverter
Remove speedment dependencies

Add support for preoptimizer priority

We need a way to guarantee certain pre-optimizers (operation reordering) will run before others. The pre-optimizer factory should be rewritten to support prioritization.

Add descriptions of advanced features in manual

There are several advanced topics covered in the Speedment manual which are yet to be covered in the JPAstreamer user guide:

  • Automatic Closing
  • Iterator and Spliterator
  • Parallelism
  • Parallel Strategy
  • Parallel Thread Pool
  • Connection Pooling
  • Custom Stream Suppliers

Optimize terminating operations

Some terminal operations can be expressed as a combination of multiple intermediate operations, making them easier to represent in SQL:

  • AnyMatch
    .anyMatch(p) can be expressed as .filter(p).limit(1).count() > 0)
  • NoneMatch
    .noneMatch(p) can be expressed as .filter(p).limit(1).count() = 0

Some terminal operations can be enhanced with certain intermediate operations in order to optimize the SQL query that is being executed:

  • FindFirst
    .findFirst() can be expressed as .limit(1).findFirst() to restrict the size of the ResultSet
  • FindAny can be expressed as .unordered().limit(1).findAny()

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.