elixir-ecto / ecto Goto Github PK

We need to start writing full stack tests. For example, that Repo.create actually creates a record in the database and that a query can get records back. We have a couple challenges here:

1. How to setup the database before the test suite;
2. How to clean up the database after each test. Rails, for example, wraps each test in a transaction, so there is no need for a clean up (ExUnit.CaseTemplate could be useful here);
3. How to organize the tests independency. In the future, when we have more than one adapter, we should be able to run just postgresql tests, or just mysql tests. Maybe they should live in a special directory and we would have specific tasks for each of them, like mix test.pg. Unsure.

Since we will import Ecto.Query, internal functions like normalize, merge and validate need to be moved elsewhere.

Repo.update_all/2 will receive a query expression (a Queryable) as first parameter and the values to update as the second one:

q = from p in Post, where: p.published == false
Repo.update_all(q, published: true)

Repo.delete_all/1 receives just a query. We will probably need to extend the validator to check for update and delete queries (the validator could likely have three functions, select, update and delete, one for each operation except create).

For fast access/modification of primary key without first looking up field name for primary key.

We use Ecto.Query around the code base and it would be recommend to tag it whenever possible so the record optimizations can kick in.

So that we can test entity validation in Repo. Related to #26.

Proposed syntax:

  from p in Post,
order: [p.posted, p.last_comment]

  from p in Post,
order: p.posted,
order: p.last_comment

  from p in Post,
order: [asc: p.posted, asc: p.last_comment]

The above queries are all equivalent.

The normalizer will then get all entities and store in a tuple.

Could the repository automatically validate and normalize queries before passing to the adapter so the adapter is not the one responsible for calling it?

Receives an entity and updates it based on its primary key. If the entity has no id or no primary key, it should raise.

It is unclear how validations and callbacks will integrate this feature, so it should not be a concern for now.

Fetches a single entity by primary key. id can be a string that will be converted to integer for convenience.

Right now it's only supported as the first field.

A lot of our primary keys are natural primary keys (strings) like e.g. "CA" or "TX".

Some data stores allow atoms (ETS), but some don't (SQL). So we need a common validator that allow atoms and a custom validator for SQL that doesn't allow them.

It will be the place where we will document the functions available in the Repo. The Ecto.Adapter is more focused around the adapter API.

We have the ordinary keyword queries that are keyword lists of query expressions. I think "keyword query" is a good name for them. A keyword query compiles to a recursive function application on the underlying query functions.

Example:

from c in City,
where: c.population > 10_000,
select: c.name

from(c in City) |> where([c], c.population > 10_000) |> select([c], c.name)

select(where(from(c in City), [c], c.population > 10_000), [c], c.name)

The keyword syntax will be the main syntax that most people will use. We still need to document the query functions, because some may use them, and mainly because that is where we will document all the different query expressions that can go in the keyword query.

Suggestions:
query unit
query fragment
query expression

There is no documentation (or maybe I missed it) on how the from(..., where: ...) syntax translates underlying. How to add new keywords to the query? The docs for Ecto.Query needs to mention this and give more examples of queries, including dynamically creating queries (if some_value do query = extend(...) end).

We also need to document each of extend, where, select, with examples both in the keywords syntax as in the binding syntax.

See comment here:

20db398#commitcomment-3707472

Today:

select(p in query, (1 + 2) + p.x) # => "SELECT (1 + 2) + p.x ..."

Future:

select(p in query, (1 + 2) + p.x) # => "SELECT 3 + p.x ..."

There should be an Ecto.Query.Helpers function that will force an expression to be escaped:

select(p in query, escape(1 + 2) + p.x) # => "SELECT (1 + 2) + p.x ..."

Today we have extend(query, [p], select: p.title) the shorthand should be extend(p in query, select: p.title). Will only work when a single variable is bound in the query.

We will talk about this later @ericmj!

Do it for Repo.all / create / update / delete. Basically all functions that will send queries to the database.

  from p in Post,
where: p.id < 10

# is equivalent to

   from p in Post,
 where: p.id < 10
select: p

This would only work if there is a single from expression. Thoughts?

Check if we get the same thing back that we put in and that Postgres doesn't error on format or encoding issues.

Before we couldn't combine multiple wheres with and in the normalizer because they used different bindings. Now that we don't have named bindings internally we can combine them.

Same goes for order_by etc. order_by and friends can probably be combined in the query merger.

Also, only fetch primary key on create.

All type validation happens at the repo and the adapter should receive the "digested" information. This makes it easy for us to test the adapter with no dependency on the entity.

Will translate to BETWEEN SQL.

Subqueries and arrays haven't been discussed yet so don't implement in for them yet.

Today, we can't call:

require Ecto.Query
Ecto.Query.from p in Post, where: p.published_at != nil

Nor can we write:

import Ecto.Query, only: [from: 2]
Ecto.Query.from p in Post, where: p.published_at != nil

We should add a small heuristic:

1. When we see a keyword, loop __CALLER__.macros looking for an import of that keyword with arity of 3. If one is found, we generate a local call (like today);
2. If not, call Module.defines? (or something of sorts), looking for a public or private macro with the same name and arity. If one is found, we generate a local call (like today);
3. If not, generate a full dispatch to Ecto.Query.unquote(keyword).

Notice I have mentioned we should lookup just for macros. I am not sure if we should lookup for functions as well. The reason for not looking up for functions is that they will always raise due to the binding. So it is a matter of automatically ignoring them or using them but letting them fail. Right now, I am more inclined in looking up just for macros.

We keep the semantics close to SQL's.

from(w in Weather,
group_by: w.city,
having: max(w.temp) > 25,
select: { w.city, max(w.temp) }

"""
SELECT w.city, max(w.temp)
FROM weather AS w
GROUP BY w.city
HAVING max(w.temp) > 20
"""

group_by takes a single field or a list of fields. having takes an expression that have to evaluate to a boolean value.

If fields that weren't expressed in the group by and non-aggregate functions appear in the select or having query expression validation will fail.

(Query functions only used here for illustration. Will be introduced in a separate issue.)

There are two kinds of documentation. The API documentation and code documentation, for developers working with the code. We are working on the API documentation but the code documentation is non existent.

For example, what is the responsibility of escape in where builder? And in order build? What is the normalization step in the normalizer?

Tip: even if the module has @moduledoc false, we can still add @doc ... clauses to it and it won't show up in exdoc.

Proposed syntax:

   from p in Post,
offset: 40,
 limit: 10,
select: p

Only allow numbers in the expressions.

All those syntaxes are valid:

from(query, limit: 10)
from(x in query, where: x.published_at == nil)
from([x, p] in query, where: ...)

We are going to invoke to Ecto.Queryable on the argument and the number of bindings needs to be less than or equal the bindings on the query.