Comments (4)
Hm, I also have no idea. If I can reproduce it later, maybe I'll just make the change you suggest.
from aas.
Hm, that's weird. I don't recall ever seeing it, though it's been a long while since I ran the code. There aren't any other modifications?
There isn't a join in that line of course. Are you sure it's not triggering elsewhere? or definitely happens only as soon as you execute this line?
Or what happens if you execute that line with .explain()
at the end?
from aas.
Sorry for the delayed I did not realize that this issue had a response, this is the explain():
== Physical Plan ==
org.apache.spark.sql.AnalysisException: Detected cartesian product for LEFT OUTER join between logical plans
Project [_1#153 AS artist#160]
+- SerializeFromObject [assertnotnull(input[0, scala.Tuple2, true])._1 AS _1#153, staticinvoke(class org.apache.spark.sql.catalyst.expressions.UnsafeArrayData, ArrayType(FloatType,false), fromPrimitiveArray, assertnotnull(input[0, scala.Tuple2, true])._2, true, false) AS _2#154]
+- ExternalRDD [obj#152]
and
Project [_2#144 AS features#147]
+- Filter (UDF(2093760) = _1#143)
+- SerializeFromObject [assertnotnull(input[0, scala.Tuple2, true])._1 AS _1#143, staticinvoke(class org.apache.spark.sql.catalyst.expressions.UnsafeArrayData, ArrayType(FloatType,false), fromPrimitiveArray, assertnotnull(input[0, scala.Tuple2, true])._2, true, false) AS _2#144]
+- ExternalRDD [obj#142]
Join condition is missing or trivial.
Use the CROSS JOIN syntax to allow cartesian products between these relations.;```
And once I set the toRecommend.cache()
== Physical Plan ==
TakeOrderedAndProject(limit=5, orderBy=[prediction#295 DESC NULLS LAST], output=[artist#250,prediction#295])
+- *(9) Project [artist#250, UDF(features#237, features#247) AS prediction#295]
+- SortMergeJoin [UDF(artist#250)], [id#246], LeftOuter
:- *(6) Sort [UDF(artist#250) ASC NULLS FIRST], false, 0
: +- Exchange hashpartitioning(UDF(artist#250), 200)
: +- *(5) Project [artist#250, features#237]
: +- SortMergeJoin [UDF(user#252)], [id#236], LeftOuter
: :- *(2) Sort [UDF(user#252) ASC NULLS FIRST], false, 0
: : +- Exchange hashpartitioning(UDF(user#252), 200)
: : +- *(1) InMemoryTableScan [artist#250, user#252]
: : +- InMemoryRelation [artist#250, user#252], true, 10000, StorageLevel(disk, memory, deserialized, 1 replicas)
: : +- *(1) Project [_1#243 AS artist#250, 2093760 AS user#252]
: : +- *(1) SerializeFromObject [assertnotnull(input[0, scala.Tuple2, true])._1 AS _1#243, staticinvoke(class org.apache.spark.sql.catalyst.expressions.UnsafeArrayData, ArrayType(FloatType,false), fromPrimitiveArray, assertnotnull(input[0, scala.Tuple2, true])._2, true, false) AS _2#244]
: : +- Scan ExternalRDDScan[obj#242]
: +- *(4) Sort [id#236 ASC NULLS FIRST], false, 0
: +- Exchange hashpartitioning(id#236, 200)
: +- *(3) Project [_1#233 AS id#236, _2#234 AS features#237]
: +- *(3) SerializeFromObject [assertnotnull(input[0, scala.Tuple2, true])._1 AS _1#233, staticinvoke(class org.apache.spark.sql.catalyst.expressions.UnsafeArrayData, ArrayType(FloatType,false), fromPrimitiveArray, assertnotnull(input[0, scala.Tuple2, true])._2, true, false) AS _2#234]
: +- Scan ExternalRDDScan[obj#232]
+- *(8) Sort [id#246 ASC NULLS FIRST], false, 0
+- Exchange hashpartitioning(id#246, 200)
+- *(7) Project [_1#243 AS id#246, _2#244 AS features#247]
+- *(7) SerializeFromObject [assertnotnull(input[0, scala.Tuple2, true])._1 AS _1#243, staticinvoke(class org.apache.spark.sql.catalyst.expressions.UnsafeArrayData, ArrayType(FloatType,false), fromPrimitiveArray, assertnotnull(input[0, scala.Tuple2, true])._2, true, false) AS _2#244]
+- Scan ExternalRDDScan[obj#242]
from aas.
I got the same error and got it to work with the suggested addition of toRecommend.cache()
.
from aas.
Related Issues (20)
- Pyspark implementation of these HOT 1
- Chapter 9: Getting the Data: 403 Forbidden results HOT 7
- Chapter 10 LeftOuterShuffleRegionJoin issue HOT 6
- Importing projects into IntelliJ HOT 2
- [ch-03] match Error with function: buildArtistAlias HOT 3
- transform RDD [(String, String)] to DATASET [ (String, String)] HOT 1
- Ch 03: audioscrobbler data not available HOT 2
- NullPointerException in chapter9 HOT 1
- how to work around "next on empty iterator"in chapter 9th? HOT 11
- ValueError: cannot decompress PACKBITS in chapter11 HOT 4
- java.lang.NoClassDefFoundError: scala/reflect/internal/Trees HOT 1
- [Question] Chapter 2 - about function "scoreMatchData" HOT 5
- but which data set to download from https://ti.arc.nasa.gov HOT 1
- Increased maven memory for build project HOT 1
- Chapter 1, page 46 HOT 13
- Chapter 3. Recommending Music and the Audioscrobbler Dataset HOT 2
- Chapter 3: Convert PySpark DataFrame to Pandas HOT 4
- Chapter 3: ROC HOT 4
- Where is the code for AA with PySpark Book by Akash Tandon? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aas.