foursquare / rogue Goto Github PK

Hey,

I can't find the current Release (1.1.5/1.1.6) on sonatype repository (latest is 1.1.3)
There is also only an old Snapshot Version available: (1.0.30-SNAPSHOT)

https://oss.sonatype.org/content/repositories/snapshots/com/foursquare/rogue_2.9.1/

Latest Build (beta21) Rogue.scala lacks a conversion from Calendar to a QueryField. It still exists in master branch (beta14).

Hi!

I'm working with GridFS and using Rogue for querying file records. GridFS has one issue - its record doesn't support partial state.
MongoHelpers has such code:

val sel = queryClause.select.map(buildSelect) getOrElse
                buildSelectFromNames(queryClause.meta.metaFields.view.map(_.name))

So DBCursor will always created in partial state (because list of fields is present). So it is impossible to work with gridfs with Rogue.
Can sel be set to null if select() is not specified?
Thank you!

Hi, I'm using the latest 2.2.0 version and it seems I found a bug. Here is how to reproduce it:

import net.liftweb.mongodb.record.BsonRecord
import net.liftweb.record.field.EnumField
import net.liftweb.mongodb.record.BsonMetaRecord
import net.liftweb.mongodb.record.MongoRecord
import net.liftweb.mongodb.record.field.ObjectIdPk
import net.liftweb.mongodb.record.field.BsonRecordField
import wws.db.connection.MongoConfig
import net.liftweb.mongodb.record.MongoMetaRecord
import com.foursquare.rogue.LiftRogue._

// Enum that "causes throuble"
object TestEnum extends Enumeration {
  type TestEnum = Value
  val ONE, TWO = Value
}

import TestEnum._

// Model classes
class NestedDoc extends BsonRecord[NestedDoc] {
  def meta = NestedDoc
  // the same field type as 'goodField' but Rogue converts it to String in queries
  object thisFieldIsNotAString extends EnumField(this, TestEnum)
}
object NestedDoc extends NestedDoc with BsonMetaRecord[NestedDoc]

class TestDbClass extends MongoRecord[TestDbClass] with ObjectIdPk[TestDbClass] {
  override def meta = TestDbClass
  object goodField extends EnumField(this, TestEnum) // this field works just fine
  object badField extends BsonRecordField(this, NestedDoc)
}
object TestDbClass extends TestDbClass with MongoMetaRecord[TestDbClass] {
  override def mongoIdentifier = MongoConfig.identifier.vend
}

// The app that tests this scenario
object TestDbApp extends App {

  def printDocWithFields(doc: TestDbClass): Unit = {
    println(doc)
    println(doc.goodField.get.getClass)
    println(doc.badField.get.thisFieldIsNotAString.get.getClass)
  }

  MongoConfig.init()

  val doc = TestDbClass.createRecord
    .goodField(ONE)
    .badField(NestedDoc.createRecord.thisFieldIsNotAString(ONE))
  println("\n\nDoc before Save:")
  printDocWithFields(doc)
  val docDb = doc.saveTheRecord().get
  val docId = docDb.id.get
  println("\n\nDoc after Save:")
  printDocWithFields(docDb)

  val query = TestDbClass.where(_.goodField eqs ONE)
    .and(_.badField.subfield(_.thisFieldIsNotAString) eqs TWO)
  println("\n\nQuery:")
  println(query)
  println("Good field type: " + query.asDBObject.get("goodField").getClass)
  println("Bad field type: " +     query.asDBObject.get("badField.thisFieldIsNotAString").getClass)

  val docDbRefetch = TestDbClass.find(docId).head
  println("\n\nDoc fetched with Java driver:")
  printDocWithFields(docDbRefetch)

  val docDbFetchWithRogue = TestDbClass.where(_.id eqs docId)
    .and(_.goodField eqs ONE).fetch().head
  println("\n\nDoc fetched with Rogue using good field:")
  printDocWithFields(docDbFetchWithRogue)

  val docDbFetchWithRogueBadField = TestDbClass.where(_.id eqs docId)
    .and(_.badField.subfield(_.thisFieldIsNotAString) eqs ONE).fetch()
  println("\n\nDoc fetched with Rogue using bad field:")
  docDbFetchWithRogueBadField match {
    case expected :: _ => printDocWithFields(expected) // Desired behavior
    case List() => println("Didn't find the doc!!!") // Current behavior (bug?)
  }
}

I've noticed these comments in src/test/scala/com/foursquare/rogue/EndToEndTest.scala:

// This behavior is broken because we get a String back from mongo, and at
// that point we only have a DummyField for the subfield, and that doesn't
// know how to convert the String to an Enum.

However, this seems to be a different issue, could you confirm?

Thanks

p.s. I can write a test and submit a pull request if it helps.

Job.where(_.jobId eqs foo).or(_.where(_._id eqs ObjectId(foo))).signature results in:

db.jobs.find({ "JOB_ID" : 0 , "$or" : [ { "_id" : 0}]})

instead of

db.jobs.find({ "$or": [{"JOB_ID" : 0} , { "_id" : 0}]})

on closer inspection, looks like in order to get a correct $or query out of rogue, one has to do this instead:

Job.or(
  _.where(_.jobId eqs foo),
  _.where(_._id eqs new org.bson.types.ObjectId(foo)))

which yields the correct

db.jobs.find({ "$or" : [ { "JOB_ID" : 0} , { "_id" : 0}]})

however, https://github.com/foursquare/rogue/blob/master/rogue-lift/src/test/scala/com/foursquare/rogue/QueryTest.scala says:

Venue.where(_.mayor eqs 1).or(_.where(_._id eqs oid)).signature() must_== """db.venues.find({ "mayor" : 0 , "$or" : [ { "_id" : 0}]})"""

After upgrading to RC-4 i get, using Lift 2.5-RC5:

[error] bad symbolic reference. A signature in LiftRogue.class refers to type IndexBuilder
[error] in package com.foursquare.rogue.index which is not available.
[error] It may be completely missing from the current classpath, or the version on
[error] the classpath might be incompatible with the version used when compiling LiftRogue.class.

Note for other weary travelers: I've noticed IntelliJ's Scala plugin failing to understand Rogue sub-field queries:

Full error:

Assuming that IntelliJ was correct, I almost gave up and assumed that Rogue subfield queries don't work (this is at the v3.0.0-beta13.1 tag, which seems to have a pretty different syntax for subfield queries than what is documented?), but luckily I noticed that my running sbt ~compile was finishing successfully.

I'll file this against IntelliJ or the Scala plugin.. "soon", but wanted to leave a note here about it too.

Hey guys,
seems we encountered a bug in building a specific query, example bellow:

MyMeta
      .or(
        _.where(_.things in List(thing)),
        _.whereOpt(additionalThing))(_.additionalThings in List(_))
      )

Which generates such query (additionalThing = None):

... "$or": [ { "things" : { "$in" : [ "thingA"]}} , { }]},  ...

What we don't like here is the { } part in the or. We'd rather just have it left out.
This makes the query fetch the entire collection so we had to fallback to manual building of or if it's needed.

Cheers and thanks for the great lib :)
Konrad

 "com.foursquare" %% "rogue-field" % "2.1.0",
 "com.foursquare" %% "rogue-core" % "2.1.0",
 "com.foursquare" %% "rogue-lift" % "2.1.0"

These depend on Lift 2.5-SNAPSHOT when using Scala 2.10. I believe published Lift artefacts have changed naming policy, as SBT is unable to resolve Lift 2.5-SNAPSHOT using all the below channels:

resolvers ++= Seq(
    "Sonatype repo"      at "https://oss.sonatype.org/content/groups/scala-tools/",
    "Sonatype releases"  at "https://oss.sonatype.org/content/repositories/releases",
    "Sonatype snapshots" at "https://oss.sonatype.org/content/repositories/snapshots",
    "Sonatype staging"       at "http://oss.sonatype.org/content/repositories/staging"

Failure was detected on a couple servers and machines without explicitly changing setup anywhere. Our CI servers reported build failures starting with last night.

Add support for $position modifier as described here http://docs.mongodb.org/manual/reference/operator/update/position/

We probably just need to something like

class ModifyPushEachPositionClause(fieldName: String, position: Int, values: Traversable[_])
    extends ModifyClause(ModOps.Push) {
  override def extend(q: BasicDBObjectBuilder): Unit = {
    q.push(fieldName).add("$each", QueryHelpers.list(values)).add("$position", position).pop
  }
}

class ModifyPushEachSliceClause(fieldName: String, slice: Int, position: Int, values: Traversable[_])
    extends ModifyClause(ModOps.Push) {
  override def extend(q: BasicDBObjectBuilder): Unit = {
    q.push(fieldName).add("$each", QueryHelpers.list(values)).add("$slice", slice).add("$position", position).pop
  }
}

Is this possible?

Venue where (_.mayor eqs 1) rawOpt(Some(3)) (_.add("$where", "this.a > %d".format(_))) toString() must_== """db.venues.find({ "mayor" : 1 , "$where" : "this.a > 3"})"""
Venue where (_.mayor eqs 1) rawOpt(None) (_.add("$where", "this.a > %d".format(_))) toString() must_== """db.venues.find({ "mayor" : 1})"""

It seems that Rogue does not currently have support for Scala 2.11 — are there any plans to add it?

A Scala 2.9.2 build is missing

As highlighted in thread: https://groups.google.com/forum/#!topic/rogue-users/VUQFeK4LSbE

We're querying for the _id, but have no way to access it anyway, when using the select(_.name) => (String)syntax, so we could try not querying for it - which would require using the mongofind()differently, not via mongo'sselect().

Hi Jason,

In mongo, it is possible to provide multiple top-level $or queries, the results of which are implicitly AND'ed together, right?

For example, imagine that you have two renamed fields, and you want to query both the old and new fields, you might write:

db.coll.find{ "$or": [ { "newFieldOne": "fieldValueOne" }, { "oldFieldOne": "fieldValueOne" } ],
                      "$or": [ { "newFieldTwo": "fieldValueTwo" }, { "oldFieldTwo": "fieldValueTwo" } ],
                      "otherField": { "$lt": 2 } };

This should return these rows:

{ "newFieldOne": "fieldValueOne", "oldFieldTwo": "fieldValueTwo", "otherField": 1 },
{ "oldFieldOne": "fieldValueOne", "newFieldTwo": "fieldValueTwo", "otherField": 1 }

But should not return these rows:

{ "newFieldOne": "fieldValueOne", "oldFieldTwo": "fieldValueTwo", "otherField": 3 },  // otherField is not less than 2
{ "oldFieldOne": "foo", "newFieldTwo": "fieldValueTwo", "otherField": 1 } // neither oldFieldOne nor newFieldOne equals fieldValueOne

AFAICT, this query form is supported in Mongo since at least 1.6.x, but I cannot find any way to express this query in Rogue. Is it possible?

Thanks,
-Marc

Putting "com.foursquare" %% "rogue" % "1.0.15" in my build.sbt produces

`sbt.ResolveException: unresolved dependency: net.liftweb#lift-mongodb-record_2.9.0;2.4-SNAPSHOT: not found`

This happens with 1.0.15-1.0.18 and 1.0.19-SNAPSHOT.

Looking through scala-tools.org, it appears that there is no lift-mongodb-record in the snapshot (http://scala-tools.org/repo-snapshots/net/liftweb/lift-mongodb_2.9.0/2.4-SNAPSHOT/). Maybe 2.4-M3 should be used instead as it is more stable.

I have a collection whose documents have an array field. I want to query against that field explicitly comparing to an array argument. In this case, I want to compare it to an empty list, so I can pull back only documents that have non-empty array values. But when I try to write this query against Rogue, I get an error when mongodb JSON tries to serialize the List argument:

scala> MyCollection.scan(.myArrayField exists true).scan(.myArrayField neqs Nil)
java.lang.RuntimeException: json can't serialize type : class scala.collection.immutable.Nil$
at com.mongodb.util.JSON.serialize(JSON.java:262)
at com.mongodb.util.JSON.serialize(JSON.java:141)
at com.mongodb.util.JSON.serialize(JSON.java:141)
at com.mongodb.util.JSON.serialize(JSON.java:58)
at com.mongodb.BasicDBObject.toString(BasicDBObject.java:84)
at com.foursquare.rogue.MongoHelpers$MongoBuilder$.buildQueryString(MongoHelpers.scala:97)
at com.foursquare.rogue.Query.toString(Query.scala:225)

The equivalent mongo query run directly against mongo console works and returns the expected results:
db.MyCollection.find({myArrayField:{$ne: [], $exists: true}})

Changes since early 2.0.0 betas have not been merged into master -- is this on purpose?

It would be great if this project adopted the practice of pushing a (git tag)[http://learn.github.com/p/tagging.html] for each released version -- it makes it super easy to go to the source version that corresponds to a released version.

It's such a simple step, and it would be a great help to us users of the library.

Hi,

In mongo is possible to write query with two or's like:

db.someCollection.find({ $or : [ { "system" : "test1" } , { "system" : "test2" } ] ,
$or : [ { "something" : true } , { "something" : false } , { "something" : { $exists : false } }]
})

( I know that this specific query could be wrote in one or but I have much much more complicated query and I need to user or parameter two times ).

But lift rogue when compiling shows me and error:

Query must be HasNoOrClause, but it's actually com.foursquare.rogue.Unselected with com.foursquare.rogue.ShardKeyNotSpecified with com.foursquare.rogue.Unordered with com.foursquare.rogue.Unskipped with com.foursquare.rogue.Unlimited with com.foursquare.rogue.HasOrClause

is it possible in lift rogue write such query ?
Thanks!

This

Job.where(_._id eqs search).toString

yields

db.jobs.find({ "_id" : { "$oid" : "52f064dcef86abdf52155300"}})

Executing that on the Mongo shell:

> db.jobs.find({ "_id" : { "$oid" : "52f064dcef86abdf52155300"}})
error: { "$err" : "invalid operator: $oid", "code" : 10068 }

...because it should really be db.jobs.find({ "_id" : ObjectId("52f064dcef86abdf52155300")}).

fetch-ing this one works fine though... It's just that sometimes for debugging it's good to get the raw query and be able to execute it manually.

I tried to understand how the query is built prior to being executed but ExecutableQuery/QueryExecutor/LiftQueryExecutor/etc is just too complicated to understand in less than a couple of ours with no prior familiarity... However, Query#toString definitely doesn't seem to be using the same mechanism for JSON'izing the query as Query#fetch.

Support $setOnInsert, docs: http://docs.mongodb.org/manual/reference/operator/setOnInsert

Original request: https://groups.google.com/forum/#!topic/rogue-users/8lMY7fKt1aM

It has been brought up again on the mailing list, and I remembered we've needed this feature also once in the past, so I decided to go ahead implementing it -- pull request will follow during the day :-)

Hi!

I have such query:

Model where (_.field matches pattern) and (_.field nin excluding)

It becomes:

db.models.find({ "field" : { "$regex" : "\\Qstr\\E" , "$options" : "i"}})

but i'm expecting:

db.models.find({ "field" : { "$regex" : "\\Qstr\\E" , "$options" : "i", "$nin" : ["str1","str2"]}})

As i understand the reason is matches creates EqClause and this clause is alone.

Are there any plans to support the MongoDB aggregation framework and map-reduce queries?

Rogues doesn't appear to support the use of $where clauses, which allow you to add arbitrary javascript functions to mongo querying, as well as using a short-hand SQL-like syntax.
http://www.mongodb.org/display/DOCS/Advanced+Queries#AdvancedQueries-JavascriptExpressionsand{{%24where}}
http://www.mongodb.org/display/DOCS/Server-side+Code+Execution#Server-sideCodeExecution-{{%24where}}ClausesandFunctionsinQueries

Why is this important?

In mongo, there are current use cases which require the use of $where clauses, as the built-in mongo queries cannot handle them. For example, it is impossible to query records for which ALL members of their embedded lists meet a criteria:
http://stackoverflow.com/questions/6038818/mongodb-query-to-match-each-element-in-doc-array-to-a-condition

Cases such as these require using a $where clause with a custom javascript function, which thankfully will only be executed for records which satisfy the build-in mongo queries.

So, for example, I'd like to be able to ask mongo whether ALL of a Venue's tags start with "prefix" -- that being just an example of a constraint that must be true for ALL of the items in an embedded array in the record.

Venue where (_.mayor eqs 1) andJavascript("function() { for (t in obj.tags) { if !(/^prefix/).test(t) return false; }; return true; }")

And this should return all Venue records where (mayor == 1), and ALL elements of the embedded tag list start with "prefix".

Is there currently a way to do this with Rogue?

I previously was using Lift 2.5.1, Rogue 2.2.0, and was trying to upgrade to Lift 2.6 RC1, Rogue 2.4.0.

I'm getting the following error when trying to perform any queries using the upgraded packages:

java.lang.NoSuchMethodError: net.liftweb.mongodb.MongoMeta.mongoIdentifier()Lnet/liftweb/mongodb/MongoIdentifier;
at com.foursquare.rogue.LiftDBCollectionFactory$.getInstanceName(LiftQueryExecutor.scala:27)
at com.foursquare.rogue.MongoJavaDriverAdapter.runCommand(MongoJavaDriverAdapter.scala:27)
at com.foursquare.rogue.MongoJavaDriverAdapter.doQuery(MongoJavaDriverAdapter.scala:274)
at com.foursquare.rogue.MongoJavaDriverAdapter.query(MongoJavaDriverAdapter.scala:169)
at com.foursquare.rogue.QueryExecutor$class.fetch(QueryExecutor.scala:65)
at com.foursquare.rogue.LiftQueryExecutor.fetch(LiftQueryExecutor.scala:36)
at com.foursquare.rogue.ExecutableQuery.fetch(ExecutableQuery.scala:57)
at .(:12)
at .()
at .(:7)
at .()
at $print()
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:734)
at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:983)
at scala.tools.nsc.interpreter.IMain.loadAndRunReq$1(IMain.scala:573)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:604)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:568)
at scala.tools.nsc.interpreter.ILoop.reallyInterpret$1(ILoop.scala:760)
at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:805)
at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:717)
at scala.tools.nsc.interpreter.ILoop.processLine$1(ILoop.scala:581)
at scala.tools.nsc.interpreter.ILoop.innerLoop$1(ILoop.scala:588)
at scala.tools.nsc.interpreter.ILoop.loop(ILoop.scala:591)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:882)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:837)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:837)
at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:837)
at scala.tools.nsc.interpreter.ILoop.main(ILoop.scala:904)
at org.jetbrains.plugins.scala.compiler.rt.ConsoleRunner.main(ConsoleRunner.java:64)

There is a related post along with some of the workarounds here: https://groups.google.com/forum/#!msg/liftweb/QSsjKLjhR9g/PouK-phiuuQJ

but I was wondering if you were going to update the main trunk of Rogue to solve this issue anytime soon. Thanks,