Adapter race condition about hugsql HOT 20 CLOSED

I seem to have circumvented the issue by explicitly declaring the adapter while defining the SQL functions, rather than using the default adapter (as discussed @ http://www.hugsql.org/#adapter-other):

(hugs/def-db-fns "sql/queries.sql"
                 {:adapter (adapter/hugsql-adapter-clojure-jdbc)})

from hugsql.

Running into this as well. Happens intermittently with both lein run and in the REPL using component. No AOT compilation.

JDK 1.8
Clojure 1.8.0
HugSQL 0.4.5
postgresql 9.4.1208

Edit: I also get No such var: jdbc/db-do-prepared-return-keys, compiling:(hugsql/adapter/clojure_java_jdbc.clj:12:9) occasionally.

@Leonidas-from-XIV @scott-abernethy @MarketaAdamova did any of you find a bandaid fix for this?

from hugsql.

We also experience this problem, we have (simulant) integration tests that spin up multiple threads and work on one database.

In single thread mode it dissappears.

from hugsql.

That is really strange, but let's see if we can figure it out. I'd like to find a reproducible example if possible.

What version of Clojure?
What version of HugSQL?

Is this happening in dev, tests, production?
Is this at the REPL?
Are you reloading namespaces in any way?
Is your jar AOT compiled?
Do you have any compiler settings (like dynamic linking)?
Do you have any global settings for read-eval set?

from hugsql.

I think this may have to do with the thread safety of either require and/or alter-var-root. Early versions of HugSQL used an atom to set/hold the adapter, but I switched it out with alter-var-root. However, if require is not thread safe, then I may need to synchronize around both.

from hugsql.

Thanks for the quick response!

What version of Clojure?

1.8.0

What version of HugSQL?

0.4.7 I'm also using clojure/java.jdbc 0.4.2 since Yesql depends on that, but could potentially bump it.

Is this happening in dev, tests, production?

It happens in production and not often but I've seen it in some tests as well (I remember seeing it when the query in question was broken but didn't think about it much since the query was incorrect in any case and fixing the query made this issue disappear).

Is this at the REPL?

No, I haven't seen it there, but I rarely use the REPL so I wouldn't exclude it.

Is your jar AOT compiled?

No, it isn't. Could that that potentially help?

Do you have any compiler settings (like dynamic linking)? Do you have any global settings for read-eval set?

No, it's a pretty standard Leiningen application that get's uberjar'ed and deployed to production.

from hugsql.

Bump. I just experienced this in a production setup also.

JDK 1.8
Clojure 1.8.0
HugSQL 0.4.7
Java.jdbc 0.5.8

Thanks.

from hugsql.

@scott-abernethy Thanks for the report. It adds to the mix: I definitely think this is an issue, but I'm unable to reproduce it to ensure a proper fix.

from hugsql.

Hi,
we sometimes (though very rarely) experience similar issue.
Dependency setup is the same as @scott-abernethy mentioned with the difference we do aot compilation.

I do use repl but never experienced the issue there.

Thanks

from hugsql.

FWIW, I see this constantly when (re)deploying our app.
We have an ELB in front of our app which sends pings all the time, and it seems like some pings are arriving before the app is fully stable.
Some of our reps:

[org.clojure/clojure "1.8.0"]
[org.clojure/java.jdbc "0.3.6"]
[com.layerware/hugsql "0.4.7"]
[org.postgresql/postgresql "9.3-1102-jdbc41"]
[clojure.jdbc/clojure.jdbc-c3p0 "0.3.2"]

from hugsql.

We experience the same issue in an application that runs DB logic in multiple threads. Reducing the number of threads to 1 makes the error disappear. Could there be a race between different threads running the adapter fallback code at the same time?

from hugsql.

Same issue here in a luminus application after executing lein run in console. Never seen in REPL (used a lot) or in the packaged war file in production.

[org.clojure/clojure "1.8.0"]
[org.clojure/java.jdbc "0.6.1"]
[com.layerware/hugsql "0.4.7"]
[hikari-cp "1.7.3"]
[org.postgresql/postgresql "9.4.1209"]

from hugsql.

This is also happening for me intermittently when AOT compiling. It's frustrating because you don't get the error until the app is loading. Any update on a potential fix? Also using conman so the hacks above don't work.

from hugsql.

I'm now adding this before any query loading:

(hugsql.core/set-adapter! (adp/hugsql-adapter-clojure-java-jdbc))

from hugsql.

I would guess the "I add functions in the namespace" that def-db-fns is doing isn't completely thread-safe. Hence, when you call your db-fns in your application in another thread, the db-fn can be already callable, but the rest of the stuff def-db-fn is doing isn't completely finished yet / the adapter isn't set yet.

As said before, forcing the adapter before your queries (for instance in component/mount startup, when you build your datasource/connexion) solves the issue.

from hugsql.

e531d5d reverts back to storing the adapter in an atom instead of using alter-var-root.

I'm not entirely certain that this will fix this issue, as I still haven't encountered it on any repeatable basis. If anyone on this thread would like to test this out by checking out the repo and doing a lein install-all from the main hugsql directory, I'd appreciate your feedback. Thanks!

from hugsql.

A modified version of #80 is now helping me trigger this bug and variations of it. This means that storing the adapter in an atom does not solve the issue.

Still thinking on this, but I'm now leaning toward requiring and setting the default adapter (for clojure.java.jdbc) up front if you're using the main hugsql clojar. This will remove the deferred, eval'd (shoulda known better!) adapter require and set-adapter!.

For most users, this will not change anything--but will solve this bug:

• NO CHANGE: Users of the default hugsql clojar and default clojure-java-jdbc adapter should have no changes to make.
• NO CHANGE: Users of different adapters still need to specify hugsql-core and the adapter clojar of choice (no change) and set the adapter.
• CHANGE: For users who are only using *-sqlvec functions, this would load the clojure.java.jdbc adapter and library.

As for setting the default adapter, maybe the adapter should set itself as the adapter. (This would require all adapter libraries to be updated). Last loaded adapter wins, and can still be set manually. If the *-sqlvec use case above is enough of a user issue because of clojure.java.jdbc dependency being brought in, maybe we could have a no-dependency "dummy" adapter.

from hugsql.

With 8a908f7, I've moved the setting of the default adapter (clojure.java.jdbc) out of the get-adapter function to hugsql.core namespace loading. I think this will prevent race conditions encountered upon queries running in multiple threads.

It's still an eval, since I want to keep the benefit of not requiring clojure.java.jdbc if you're using a different adapter library--which is more relevant now for the new next.jdbc adapter. That's a trade-off I'm willing to accept.

from hugsql.

This is released in 0.5.0.

from hugsql.

For those following this, I botched the change in 0.5.0, but thanks to @kanwei's quick feedback in #105, this is fixed in 0.5.1. Thanks!

from hugsql.