- Branches
- Introduction
- Installation
- Quickstart
- Authentication
- Advanced Usage
- Development
- Tests
- Testimonials
- Other Middleware
- License
There are branches for the major version numbers:
- 2.x (no longer maintained except for security issues)
- 3.x (current stable releases)
- master (which is 4.x, unreleased)
clj-http is an HTTP library wrapping the Apache HttpComponents client. This library has taken over from mmcgrana’s clj-http.
The design of clj-http
is inspired by the Ring protocol for Clojure HTTP
server applications.
The client in clj-http.core
makes HTTP requests according to a given Ring
request map and returns Ring response maps corresponding to the resulting HTTP
response. The function clj-http.client/request
uses Ring-style middleware to
layer functionality over the core HTTP request/response implementation. Methods
like clj-http.client/get
are sugar over this clj-http.client/request
function.
clj-http
is available as a Maven artifact from Clojars.
With Leiningen/Boot:
[clj-http "3.8.0"]
If you need an older version, a 2.x release is also available.
[clj-http "2.3.0"]
clj-http supports clojure 1.6.0 and higher.
The main HTTP client functionality is provided by the clj-http.client
namespace.
First, require it in the REPL:
(require '[clj-http.client :as client])
Or in your application:
(ns my-app.core
(:require [clj-http.client :as client]))
The client supports simple get
, head
, put
, post
, delete
, copy
,
move
, patch
, and options
requests. Response are returned as Ring-style
response maps:
(client/head "http://example.com/resource")
(client/head "http://example.com/resource" {:accept :json})
Example requests:
(client/get "http://example.com/resources/id")
(client/get "http://example.com/resources/3" {:accept :json})
;; Specifying headers as either a string or collection:
(client/get "http://example.com"
{:headers {"foo" ["bar" "baz"], "eggplant" "quux"}})
;; Using either string or keyword header names:
(client/get "http://example.com"
{:headers {:foo ["bar" "baz"], :eggplant "quux"}})
;; Set any specific client parameters manually:
(client/post "http://example.com"
{:client-params {"http.protocol.allow-circular-redirects" false
"http.protocol.version" HttpVersion/HTTP_1_0
"http.useragent" "clj-http"}})
;; Completely ignore cookies:
(client/post "http://example.com" {:cookie-policy :none})
;; There are also multiple ways to handle cookies
(client/post "http://example.com" {:cookie-policy :default})
(client/post "http://example.com" {:cookie-policy :netscape})
(client/post "http://example.com" {:cookie-policy :standard})
(client/post "http://example.com" {:cookie-policy :standard-strict})
;; See core/get-cookie-policy to extend with your own policy if desired:
(defn my-cookie-policy [cookie origin] (your-validation cookie origin))
(defmethod get-cookie-policy :custom [req] my-cookie-policy)
(client/post "http://example.com" {:cookie-policy :custom})
;; Need to contact a server with an untrusted SSL cert?
(client/get "https://alioth.debian.org" {:insecure? true})
;; If you don't want to follow-redirects automatically:
(client/get "http://example.com/redirects-somewhere" {:redirect-strategy :none})
;; Only follow a certain number of redirects:
(client/get "http://example.com/redirects-somewhere" {:max-redirects 5})
;; Avoid throwing exceptions if redirected too many times:
(client/get "http://example.com/redirects-somewhere" {:max-redirects 5 :redirect-strategy :graceful})
;; Throw an exception if the get takes too long. Timeouts in milliseconds.
(client/get "http://example.com/redirects-somewhere" {:socket-timeout 1000 :conn-timeout 1000})
;; Query parameters
(client/get "http://example.com/search" {:query-params {"q" "foo, bar"}})
;; "Nested" query parameters
;; (this yields a query string of `a[e][f]=6&a[b][c]=5`)
(client/get "http://example.com/search" {:query-params {:a {:b {:c 5} :e {:f 6}}}})
;; Provide cookies — uses same schema as :cookies returned in responses
;; (see the cookie store option for easy cross-request maintenance of cookies)
(client/get "http://example.com"
{:cookies {"ring-session" {:discard true, :path "/", :value "", :version 0}}})
;; Tell clj-http not to decode cookies from the response header
(client/get "http://example.com" {:decode-cookies false})
;; Support for IPv6!
(client/get "http://[2001:62f5:9006:e472:cabd:c8ff:fee3:8ddf]")
;; Super advanced, your own http-client-context and request-config
(client/get "http://example.com/get"
{:http-client-context my-http-client-context
:http-request-config my-request-config})
The client will also follow redirects on the appropriate 30*
status codes.
The client transparently accepts and decompresses the gzip
and deflate
content encodings.
:trace-redirects
will contain the chain of the redirections followed.
(client/put "http://example.com/api" {:body "my PUT body"})
;; Various options:
(client/post "http://example.com/api"
{:basic-auth ["user" "pass"]
:body "{\"json\": \"input\"}"
:headers {"X-Api-Version" "2"}
:content-type :json
:socket-timeout 1000 ;; in milliseconds
:conn-timeout 1000 ;; in milliseconds
:accept :json})
;; Send form params as a urlencoded body (POST or PUT)
(client/post "http://example.com" {:form-params {:foo "bar"}})
;; Send form params as a json encoded body (POST or PUT)
(client/post "http://example.com" {:form-params {:foo "bar"} :content-type :json})
;; Send form params as a json encoded body (POST or PUT) with options
(client/post "http://example.com" {:form-params {:foo "bar"}
:content-type :json
:json-opts {:date-format "yyyy-MM-dd"}})
;; You can also specify the encoding of form parameters
(client/post "http://example.com" {:form-params {:foo "bar"}
:form-param-encoding "ISO-8859-1"})
;; Send form params as a Transit encoded JSON body (POST or PUT) with options
(client/post "http://example.com" {:form-params {:foo "bar"}
:content-type :transit+json
:transit-opts
{:encode {:handlers {}}
:decode {:handlers {}}}})
;; Send form params as a Transit encoded MessagePack body (POST or PUT) with options
(client/post "http://example.com" {:form-params {:foo "bar"}
:content-type :transit+msgpack
:transit-opts
{:encode {:handlers {}}
:decode {:handlers {}}}})
;; Multipart form uploads/posts
;; takes a vector of maps, to preserve the order of entities, :name
;; will be used as the part name unless :part-name is specified
(client/post "http://example.org" {:multipart [{:name "title" :content "My Awesome Picture"}
{:name "Content/type" :content "image/jpeg"}
{:name "foo.txt" :part-name "eggplant" :content "Eggplants"}
{:name "file" :content (clojure.java.io/file "pic.jpg")}]
;; You can also optionally pass a :mime-subtype
:mime-subtype "foo"})
;; Multipart :content values can be one of the following:
;; String, InputStream, File, a byte-array, or an instance of org.apache.http.entity.mime.content.ContentBody
;; Some Multipart bodies can also support more keys (like :encoding
;; and :mime-type), check src/clj-http/multipart.clj to see all flags
;; Apache's http client automatically retries on IOExceptions, if you
;; would like to handle these retries yourself, you can specify a
;; :retry-handler. Return true to retry, false to stop trying:
(client/post "http://example.org" {:multipart [["title" "Foo"]
["Content/type" "text/plain"]
["file" (clojure.java.io/file "/tmp/missing-file")]]
:retry-handler (fn [ex try-count http-context]
(println "Got:" ex)
(if (> try-count 4) false true))})
A word about flattening nested :query-params
and :form-params
maps. There are essentially three
different ways to handle flattening them:
:ignore-nested-query-string
- Do not handle nested query parameters specially, treat them as the exact text they come in as. Defaults to false.
:flatten-nested-form-params
- Flatten nested (map within a map)
:form-params
before encoding it as the body. Defaults to false, meaning form params are encoded onlyx-www-form-urlencoded
. :flatten-nested-keys
- An advanced way of specifying which keys having nested maps should be
flattened. A middleware function checks the previous two options
(
:ignore-nested-query-string
and:flatten-nested-form-params
) and modifies this to be the list that will be flattened.
(client/delete "http://example.com/resource")
;; :async? in options map need to be true
(client/get "http://example.com"
{:async? true}
;; respond callback
(fn [response] (println "response is:" response))
;; raise callback
(fn [exception] (println "exception message is: " (.getMessage exception))))
All exceptions thrown during the request will be passed to the raise callback.
Calls to the http methods with :async true
return an Apache BasicFuture that you can call .get
or .cancel
on. See the Javadocs for BasicFuture
here. For instance:
(import '(java.util.concurrent TimeoutException TimeUnit))
(let [future (client/get "http://example.com/slow-url"
{:async true :oncancel #(println "request was cancelled")}
#(println :got %) #(println :err %))]
(try
(.get future 1 TimeUnit/SECONDS)
(catch TimeoutException e
;; Cancel the request, it's taken too long
(.cancel future true))))
clj-http allows coercing the body of the request either before it is sent (input coercion), or after it’s received (output coercion) from the server.
;; body as a byte-array
(client/post "http://example.com/resources" {:body my-byte-array})
;; body as a string
(client/post "http://example.com/resources" {:body "string"})
;; :body-encoding is optional and defaults to "UTF-8"
(client/post "http://example.com/resources"
{:body "string" :body-encoding "UTF-8"})
;; body as a file
(client/post "http://example.com/resources"
{:body (clojure.java.io/file "/tmp/foo") :body-encoding "UTF-8"})
;; :length is optional for passing in an InputStream; if not
;; supplied it will default to -1 to signal to HttpClient to use
;; chunked encoding
(client/post "http://example.com/resources"
{:body (clojure.java.io/input-stream "/tmp/foo")})
(client/post "http://example.com/resources"
{:body (clojure.java.io/input-stream "/tmp/foo") :length 1000})
;; The default output is a string body
(client/get "http://example.com/foo.txt")
;; Coerce as a byte-array
(client/get "http://example.com/favicon.ico" {:as :byte-array})
;; Coerce as something other than UTF-8 string
(client/get "http://example.com/string.txt" {:as "UTF-16"})
;; Coerce as json
(client/get "http://example.com/foo.json" {:as :json})
(client/get "http://example.com/foo.json" {:as :json-strict})
(client/get "http://example.com/foo.json" {:as :json-string-keys})
(client/get "http://example.com/foo.json" {:as :json-strict-string-keys})
;; Coerce as Transit encoded JSON or MessagePack
(client/get "http://example.com/foo" {:as :transit+json})
(client/get "http://example.com/foo" {:as :transit+msgpack})
;; Coerce as a clojure datastructure
(client/get "http://example.com/foo.clj" {:as :clojure})
;; Coerce as x-www-form-urlencoded
(client/post "http://example.com/foo" {:as :x-www-form-urlencoded})
;; Try to automatically coerce the output based on the content-type
;; header (this is currently a BETA feature!). Currently supports
;; text, json and clojure (with automatic charset detection)
;; clojure coercion requires "application/clojure" or
;; "application/edn" in the content-type header
(client/get "http://example.com/foo.json" {:as :auto})
;; Return the body as a stream
(client/get "http://example.com/bigrequest.html" {:as :stream})
;; Note that the connection to the server will NOT be closed until the
;; stream has been read
Output coercion with :as :json
, :as :json-strict
, :as :json-strict-string-keys
, :as :json-string-keys
or :as :x-www-form-urlencoded
will only work with an optional dependency, see Optional Dependencies.
JSON coercion defaults to only an “unexceptional” statuses, meaning status codes
in the #{200 201 202 203 204 205 206 207 300 301 302 303 304 307} range. If you
would like to change this, you can send the :coerce
option, which can be set
to:
:always ;; always json decode the body
:unexceptional ;; only json decode when not an HTTP error response
:exceptional ;; only json decode when it IS an HTTP error response
The :coerce
setting defaults to :unexceptional
.
clj-http’s treatment of headers is a little more permissive than the ring spec specifies.
Rather than forcing all request headers to be lowercase strings, clj-http allows strings or keywords of any case. Keywords will be transformed into their canonical representation, so the :content-md5 header will be sent to the server as “Content-MD5”, for instance. String keys in request headers, however, will be sent to the server with their casing unchanged.
Response headers can be read as keywords or strings of any case. If the server responds with a “Date” header, you could access the value of that header as :date, “date”, “Date”, etc.
If for some reason you require access to the original header name that the server specified, it is available by invoking (keys …) on the header map.
This special treatment of headers is implemented in the wrap-header-map middleware, which (like any middleware) can be disabled by using with-middleware to specify different behavior.
There are three different ways that query string parameters for array values can be generated, depending on what the resulting query string should look like, they are:
- A repeating parameter (default)
- Array style
- Indexed array style
Here is an example of the input and output for the :query_string
parameter,
controlled by the :multi-param-style
option:
;; default style, with :multi-param-style unset
:a [1 2 3] => "a=1&a=2&a=3"
;; with :multi-param-style :array, a repeating param with array suffix
;; (PHP-style):
:a [1 2 3] => "a[]=1&a[]=2&a[]=3"
;; with :multi-param-style :indexed, a repeating param with array suffix and
;; index (Rails-style):
:a [1 2 3] => "a[0]=1&a[1]=2&a[2]=3"
HTML 4.01 allows using the tag <meta http-equiv="..." />
and HTML 5 allows
using the tag <meta charset="..." />
to specify a header that should be
treated as an HTTP response header. By default, clj-http will ignore the body of
the response (other than the regular output coercion), but if you need clj-http
to parse the headers out of the body, you can use the :decode-body-headers
option:
;; without decoding body headers (defaults to off):
(:headers (client/get "http://www.yomiuri.co.jp/"))
=> {"server" "Apache",
"content-encoding" "gzip",
"content-type" "text/html",
"date" "Tue, 09 Oct 2012 18:02:41 GMT",
"cache-control" "max-age=0, no-cache",
"expires" "Tue, 09 Oct 2012 18:02:41 GMT",
"etag" "\"1dfb-2686-4cba2686fb8b1\"",
"pragma" "no-cache",
"connection" "close"}
;; with decoding body headers, notice the content-type,
;; content-style-type and content-script-type headers:
(:headers (client/get "http://www.yomiuri.co.jp/" {:decode-body-headers true}))
=> {"server" "Apache",
"content-encoding" "gzip",
"content-script-type" "text/javascript",
"content-style-type" "text/css",
"content-type" "text/html; charset=Shift_JIS",
"date" "Tue, 09 Oct 2012 18:02:59 GMT",
"cache-control" "max-age=0, no-cache",
"expires" "Tue, 09 Oct 2012 18:02:59 GMT",
"etag" "\"1dfb-2686-4cba2686fb8b1\"",
"pragma" "no-cache",
"connection" "close"}
This can be used to have clj-http correctly interpret the body’s charset by using:
(client/get "http://www.yomiuri.co.jp/" {:decode-body-headers true :as :auto})
=> ;; correctly formatted :body (Shift_JIS charset instead of UTF-8)
Note that this feature is currently beta and uses Crouton to parse the body of the request. If you do not want to use this feature, you can include Crouton in addition to clj-http as a dependency like so:
(defproject foo "0.1.0-SNAPSHOT"
:dependencies [[org.clojure/clojure "1.3.0"]
[clj-http "0.6.0"]
[crouton "1.0.0"]])
Note also that HEAD requests will not return a body, in which case this setting will have no effect.
clj-http will automatically disable the :decode-body-headers
option.
clj-http parses any link headers returned in the response, and adds them to the
:links
key on the response map. This is particularly useful for paging RESTful
APIs:
(:links (client/get "https://api.github.com/gists"))
=> {:next {:href "https://api.github.com/gists?page=2"}
:last {:href "https://api.github.com/gists?page=22884"}}
clj-http conforms its behaviour regarding automatic redirects to the RFC.
It means that redirects on status 301
, 302
and 307
are not redirected on
methods other than GET
and HEAD
. If you want a behaviour closer to what most
browser have, you can set :redirect-strategy
to :lax
in your request to have
automatic redirection work on all methods by transforming the method of the
request to GET
.
Redirect Options:
:trace-redirects
- If true, clj-http will enhance the response object with a
list of redirected URLs with key:
:trace-redirects
. :redirect-strategy
- Sets the redirect strategy for clj-http. Accepts the following:
:none
- Perform no redirects:default
- See https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/impl/client/DefaultRedirectStrategy.html:lax
- See https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/impl/client/LaxRedirectStrategy.html:graceful
- Similar to:default
, but does not throw exceptions when max redirects is reached. This is the redirects behaviour in 2.xnil
- When nil, assumes:default
You may also pass in an instance of RedirectStrategy (in the :redirect-strategy
key) if you want a
behavior that’s not implemented.
Additionally, clj-http will attempt to validate that a redirect host is not invalid, you can disable
this by setting :validate-redirects false
in the request (the default is true)
NOTE: The options :force-redirects
and :follow-redirects
(present in clj-http 2.x are no longer
used). You can use :graceful
to mostly emulate the old redirect behaviour.
clj-http can simplify the maintenance of cookies across requests if it is provided with a cookie store.
(binding [clj-http.core/*cookie-store* (clj-http.cookies/cookie-store)]
(client/post "http://example.com/login" {:form-params {:username "..."
:password "..."}})
(client/get "http://example.com/secured-page")
...)
(The clj-http.cookies/cookie-store
function returns a new empty instance of a
default implementation of org.apache.http.client.CookieStore
.)
This will allow cookies to only be written to the cookie store. Cookies from the cookie-store will not automatically be sent with future requests.
If you would like cookies from the cookie-store to automatically be sent with
each request, specify the cookie-store with the :cookie-store
option:
(let [my-cs (clj-http.cookies/cookie-store)]
(client/post "http://example.com/login" {:form-params {:username "..."
:password "..."}
:cookie-store my-cs})
(client/post "http://example.com/update" {:body my-data
:cookie-store my-cs}))
You can also use the get-cookies
function to retrieve the cookies
from a cookie store:
(def cs (clj-http.cookies/cookie-store))
(client/get "http://google.com" {:cookie-store cs})
(clojure.pprint/pprint (clj-http.cookies/get-cookies cs))
{"NID"
{:domain ".google.com",
:expires #<Date Tue Oct 02 10:12:06 MDT 2012>,
:path "/",
:value
"58=c387....",
:version 0},
"PREF"
{:domain ".google.com",
:expires #<Date Wed Apr 02 10:12:06 MDT 2014>,
:path "/",
:value
"ID=3ba...:FF=0:TM=133...:LM=133...:S=_iRM...",
:version 0}}
Need more docs on this? See the SSL documentation if you want to use your own self-signed certificate.
You can also specify your own keystore/trust-store to be used:
(client/get "https://example.com" {:keystore "/path/to/keystore.ks"
:keystore-type "jks" ; default: jks
:keystore-pass "secretpass"
:trust-store "/path/to/trust-store.ks"
:trust-store-type "jks" ; default jks
:trust-store-pass "trustpass"})
The :keystore/:trust-store
values may be either paths to keystore
files or KeyStore
instances.
The client will throw exceptions on, well, exceptional status codes, meaning all
HTTP responses other than #{200 201 202 203 204 205 206 207 300 301 302 303 304
307}
. clj-http will throw an ex-info exception that can be caught by a regular
(catch Exception e ...)
or in Slingshot’s try+
block:
(client/get "http://example.com/broken")
=> ExceptionInfo clj-http: status 404 clj-http.client/wrap-exceptions/fn--583 (client.clj:41)
;; Or, if you would like the Exception message to contain the entire response:
(client/get "http://example.com/broken" {:throw-entire-message? true})
=> ExceptionInfo clj-http: status 404 {:status 404,
:headers {"server" "nginx/1.0.4",
"x-runtime" "12ms",
"content-encoding" "gzip",
"content-type" "text/html; charset=utf-8",
"date" "Mon, 17 Oct 2011 23:15 :36 GMT",
"cache-control" "no-cache",
"status" "404 Not Found",
"transfer-encoding" "chunked",
"connection" "close"},
:body "...body here..."}
clj-http.client/wrap-exceptions/fn--584 (client.clj:42
;; You can also ignore HTTP-status-code exceptions and handle them yourself:
(client/get "http://example.com/broken" {:throw-exceptions false})
;; Or ignore an unknown host (methods return 'nil' if this is set to
;; true and the host does not exist:
(client/get "http://example.invalid" {:ignore-unknown-host? true})
(spacing added by me to be human readable)
How to use with Slingshot:
; Response map is thrown as exception obj.
; We filter out by status codes
(try+
(client/get "http://example.com/broken")
(catch [:status 403] {:keys [request-time headers body]}
(log/warn "403" request-time headers))
(catch [:status 404] {:keys [request-time headers body]}
(log/warn "NOT Found 404" request-time headers body))
(catch Object _
(log/error (:throwable &throw-context) "unexpected error")
(throw+)))
By default, clj-http will add the {"Accept-Encoding" "gzip, deflate"}
header
to requests, and automatically decompress the resulting gzip or deflate stream
if the Content-Encoding
header is found on the response. If this is undesired,
the {:decompress-body false}
option can be specified:
;; Auto-decompression used: (google requires a user-agent to send gzip data)
(def h {"User-Agent" "Mozilla/5.0 (Windows NT 6.1;) Gecko/20100101 Firefox/13.0.1"})
(def resp (client/get "http://google.com" {:headers h}))
(:orig-content-encoding resp)
=> "gzip" ;; <= google sent response gzipped
;; and without decompression:
(def resp2 (client/get "http://google.com" {:headers h :decompress-body false})
(:orig-content-encoding resp2)
=> nil
If clj-http decompresses something, the “content-encoding” header is removed
from the headers (because the encoding is no longer true). This allows clj-http
to be used as a pass-through proxy with ring. The original content-encoding is
available as :orig-content-encoding
in the response map if auto-decompression
is enabled.
There are four debugging methods you can use:
;; print request info to *out*:
(client/get "http://example.org" {:debug true})
;; print request info to *out*, including request body:
(client/post "http://example.org" {:debug true :debug-body true :body "..."})
;; save the request that was sent in a :request key in the response:
(client/get "http://example.org" {:save-request? true})
;; save the request that was sent in a :request key in the response,
;; including the body content:
(client/get "http://example.org" {:save-request? true :debug-body true})
;; add an HttpResponseInterceptor to the request. This callback
;; is called for each redirects with the following args:
;; ^HttpResponse resp, HttpContext^ ctx
;; this allows low level debugging + access to socket.
;; see http://hc.apache.org/httpcomponents-core-ga/httpcore/apidocs/org/apache/http/HttpResponseInterceptor.html
(client/get "http://example.org" {:response-interceptor (fn [resp ctx] (println ctx))})
To debug what data is being sent on the Socket
when a request is make, clj-http
provides the :capture-socket
option:
(-> (client/post "http://localhost:9200/_search"
{:capture-socket true ;; turn on the capture
:headers {"content-type" "application/json"}
:body "{\"query\": {\"match_all\":{}}}"})
;; Both :raw-socket-str and :raw-socket-bytes are returned with the response
:raw-socket-str
println)
;;; Which outputs something such as:
;; POST /_search HTTP/1.1
;; Connection: close
;; content-type: application/json
;; accept-encoding: gzip, deflate
;; Content-Length: 27
;; Host: localhost:9200
;; User-Agent: Apache-HttpClient/4.5.5 (Java/9.0.1)
;;
;; {"query": {"match_all":{}}}
;; nil
There are currently limitations with :capture-socket
, a custom connection manager or the
:insecure
option cannot be used, and async requests are not yet supported.
Finally, if you want to access the logging that the Apache client does internally, you can set up your dependencies to add the log4j2 libraries and configure the logging for clj-http. In order to do this, you’ll need to add
[org.apache.logging.log4j/log4j-api "2.11.0"]
[org.apache.logging.log4j/log4j-core "2.11.0"]
[org.apache.logging.log4j/log4j-1.2-api "2.11.0"]
To your project.clj
and have a usable log4j2.properties. I have provided one in
resources/log4j2.properties
. Make sure to set:
rootLogger.level = debug
If you want to see debug information (or “trace” for trace logging). When you perform a request you should see something akin to this in the logs:
[2018-03-20T20:36:34,635][DEBUG][o.a.h.c.p.RequestAddCookies] CookieSpec selected: default
[2018-03-20T20:36:34,635][DEBUG][o.a.h.c.p.RequestAuthCache] Auth cache not set in the context
[2018-03-20T20:36:34,635][DEBUG][o.a.h.i.c.BasicHttpClientConnectionManager] Get connection for route {s}->https://example.com:443
[2018-03-20T20:36:34,636][DEBUG][o.a.h.i.c.DefaultManagedHttpClientConnection] http-outgoing-1: set socket timeout to 0
[2018-03-20T20:36:34,636][DEBUG][o.a.h.i.e.MainClientExec ] Opening connection {s}->https://example.com:443
[2018-03-20T20:36:34,644][DEBUG][o.a.h.i.c.DefaultHttpClientConnectionOperator] Connecting to example.com/10.0.0.1:443
[2018-03-20T20:36:34,644][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] Connecting socket to example.com/10.0.0.1:443 with timeout 0
[2018-03-20T20:36:34,692][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] Enabled protocols: [TLSv1, TLSv1.1, TLSv1.2]
[2018-03-20T20:36:34,693][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] Enabled cipher suites:[TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, ... etc ...]
[2018-03-20T20:36:34,693][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] Starting handshake
[2018-03-20T20:36:34,841][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] Secure session established
[2018-03-20T20:36:34,842][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] negotiated protocol: TLSv1.2
[2018-03-20T20:36:34,842][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] negotiated cipher suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
[2018-03-20T20:36:34,843][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] peer principal: CN=example.com
[2018-03-20T20:36:34,843][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] peer alternative names: [example.com, www.example.com]
[2018-03-20T20:36:34,843][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] issuer principal: CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US
[2018-03-20T20:36:34,844][DEBUG][o.a.h.i.c.DefaultHttpClientConnectionOperator] Connection established 192.168.0.29:36792<->10.0.0.1:443
[2018-03-20T20:36:34,844][DEBUG][o.a.h.i.e.MainClientExec ] Executing request POST /post HTTP/1.1
[2018-03-20T20:36:34,844][DEBUG][o.a.h.i.e.MainClientExec ] Target auth state: UNCHALLENGED
[2018-03-20T20:36:34,844][DEBUG][o.a.h.i.e.MainClientExec ] Proxy auth state: UNCHALLENGED
[2018-03-20T20:36:34,845][DEBUG][o.a.h.headers ] http-outgoing-1 >> POST /post HTTP/1.1
[2018-03-20T20:36:34,845][DEBUG][o.a.h.headers ] http-outgoing-1 >> Connection: close
[2018-03-20T20:36:34,845][DEBUG][o.a.h.headers ] http-outgoing-1 >> accept-encoding: gzip, deflate
[2018-03-20T20:36:34,845][DEBUG][o.a.h.headers ] http-outgoing-1 >> Content-Length: 14
[2018-03-20T20:36:34,845][DEBUG][o.a.h.headers ] http-outgoing-1 >> Content-Type: text/plain; charset=UTF-8
[2018-03-20T20:36:34,846][DEBUG][o.a.h.headers ] http-outgoing-1 >> Host: example.com
[2018-03-20T20:36:34,846][DEBUG][o.a.h.headers ] http-outgoing-1 >> User-Agent: Apache-HttpClient/4.5.5 (Java/9.0.1)
[2018-03-20T20:36:34,846][DEBUG][o.a.h.wire ] http-outgoing-1 >> "POST /post HTTP/1.1[\r][\n]"
[2018-03-20T20:36:34,846][DEBUG][o.a.h.wire ] http-outgoing-1 >> "Connection: close[\r][\n]"
[2018-03-20T20:36:34,846][DEBUG][o.a.h.wire ] http-outgoing-1 >> "accept-encoding: gzip, deflate[\r][\n]"
[2018-03-20T20:36:34,847][DEBUG][o.a.h.wire ] http-outgoing-1 >> "Content-Length: 14[\r][\n]"
[2018-03-20T20:36:34,847][DEBUG][o.a.h.wire ] http-outgoing-1 >> "Content-Type: text/plain; charset=UTF-8[\r][\n]"
[2018-03-20T20:36:34,847][DEBUG][o.a.h.wire ] http-outgoing-1 >> "Host: example.com[\r][\n]"
etc etc it will go on forever and be very verbose
This provides both the data sent and received on the wire for debugging purposes.
I’ve also provided an example for changing the log level from clojure in
examples/logging-apache-requests.clj
.
(client/get "http://example.com/protected" {:basic-auth ["user" "pass"]})
(client/get "http://example.com/protected" {:basic-auth "user:pass"})
(client/get "http://example.com/protected" {:digest-auth ["user" "pass"]})
(client/get "http://example.com/protected" {:ntlm-auth ["user" "pass" "host" "domain"]})
(client/get "http://example.com/protected" {:oauth-token "secret-token"})
A more general request
function is also available, which is useful as a
primitive for building higher-level interfaces:
(defn api-action [method path & [opts]]
(client/request
(merge {:method method :url (str "http://example.com/" path)} opts)))
Since 0.9.0, all boolean options can be expressed as either {:debug true}
or
{:debug? true}
, with or without the question mark.
clj-http can use persistent connections to speed up connections if multiple connections are being used:
(with-connection-pool {:timeout 5 :threads 4 :insecure? false :default-per-route 10}
(get "http://example.org/1")
(post "http://example.org/2")
(get "http://example.org/3")
...
(get "http://example.org/999"))
For async request, you can use with-async-connection-pool
(with-async-connection-pool {:timeout 5 :threads 4 :insecure? false :default-per-route 10}
(get "http://example.org/1" {:async? true} resp1 exce1)
(post "http://example.org/2" {:async? true} resp2 exce2)
(get "http://example.org/3" {:async? true} resp3 exce3)
...
(get "http://example.org/999" {:async? true} resp999 exce999))
This is MUCH faster than sequentially performing all requests, because a persistent connection can be used instead creating a new connection for each request.
If you want to start an async request in the respond
callback of an async request and
reuse the pool context, just use reuse-pool
.
(with-async-connection-pool {:timeout 5 :threads 4 :insecure? false :default-per-route 10}
(get "http://example.org/1" {:async? true} resp1 exce1)
(post "http://example.org/2"
{:async? true}
(fn [resp] (get "http://example.org/3"
(reuse-pool {:async? true} resp)
resp3 exce3))
exce2))
There are many advanced options available when creating asynchronous connection pools that can be
configured by passing an :io-config
map in the connection manager parameters. It supports:
:connect-timeout
:interest-op-queued
:io-thread-count
:rcv-buf-size
:select-interval
:shutdown-grace-period
:snd-buf-size
:so-keep-alive
:so-linger
:so-timeout
:tcp-no-delay
See the docstring on with-async-connection-pool
for more information about these options.
If you would prefer to handle managing the connection manager yourself, you can create a connection manager and specify it for each request:
(def cm (clj-http.conn-mgr/make-reusable-conn-manager {:timeout 2 :threads 3}))
(def cm2 (clj-http.conn-mgr/make-reusable-conn-manager {:timeout 10 :threads 1}))
(get "http://example.org/1" {:connection-manager cm2})
(post "http://example.org/2" {:connection-manager cm})
(get "http://example.org/3" {:connection-manager cm2})
;; Don't forget to shut it down when you're done!
(clj-http.conn-mgr/shutdown-manager cm)
(clj-http.conn-mgr/shutdown-manager cm2)
See the docstring on make-reusable-conn-manager
for options and default
values.
In the current version, pooled async request CANNOT specify connection manager.
In some cases, you may want to re-use the same HttpClient
object between requests, either so you
don’t have to build it every time, or because you make some configuration change to the request.
clj-http will return the built HTTP client in :http-client
which you can then specify in
subsequent requests (with :http-client
). Note that in order to reuse the client a connection
manager must be used.
;; Re-use the HttpClient clj-http builds for you:
(let [cm (conn/make-reusable-conn-manager {})
resp (client/get "http://example.com" {:connection-manager cm})
hclient (:http-client resp)]
(client/get "http://example.com/1"
{:connection-manager cm :http-client hclient})
(client/get "http://example.com/2"
{:connection-manager cm :http-client hclient})
(client/get "http://example.com/3"
{:connection-manager cm :http-client hclient}))
;; You can also build your own, using clj-http's helper or manually building it:
(let [cm (conn/make-reusable-conn-manager {})
hclient (core/build-http-client {} cm "https://example.com" false)]
(client/get "http://example.com/1"
{:connection-manager cm :http-client hclient})
(client/get "http://example.com/2"
{:connection-manager cm :http-client hclient})
(client/get "http://example.com/3"
{:connection-manager cm :http-client hclient}))
;; Async http clients may also be created and re-used:
(let [acm (conn/make-reuseable-async-conn-manager {})
ahclient (core/build-async-http-client {} acm "https://example.com" false)]
(client/get "http://example.com/1"
{:connection-manager cm :http-client ahclient}
handle-response handle-failure)
(client/get "http://example.com/2"
{:connection-manager cm :http-client ahclient}
handle-response handle-failure)
(client/get "http://example.com/3"
{:connection-manager cm :http-client ahclient}
handle-response handle-failure))
A proxy can be specified by setting the Java properties: <scheme>.proxyHost
and <scheme>.proxyPort
where <scheme>
is the client scheme used (normally
‘http’ or ‘https’). http.nonProxyHosts
allows you to specify a pattern for
hostnames which do not require proxy routing - this is shared for all schemes.
Additionally, per-request proxies can be specified with the proxy-host
and
proxy-port
options (this overrides http.nonProxyHosts
too):
(client/get "http://example.com" {:proxy-host "127.0.0.1" :proxy-port 8118})
You can also specify the proxy-ignore-hosts
parameter with a list of
hosts where the proxy should be ignored. By default this list is
#{"localhost" "127.0.0.1"}
.
A SOCKS proxy can be used by creating a proxied connection manager with
clj-http.conn-mgr/make-socks-proxied-conn-manager
. Then using that connection
manager in the request.
For example if you wanted to connect to a local socks proxy on port 8081
you
would:
(ns foo.bar
(:require [clj-http.client :as client]
[clj-http.conn-mgr :as conn-mgr]))
(client/get "https://google.com"
{:connection-manager
(conn-mgr/make-socks-proxied-conn-manager "localhost" 8081)})
If your SOCKS connection requires a keystore / trust-store, you can specify that too:
(ns foo.bar
(:require [clj-http.client :as client]
[clj-http.conn-mgr :as conn-mgr]))
(client/get "https://google.com"
{:connection-manager
(conn-mgr/make-socks-proxied-conn-manager "localhost" 8081
{:keystore "/path/to/keystore.ks"
:keystore-type "jks" ; default: jks
:keystore-pass "secretpass"
:trust-store "/path/to/trust-store.ks"
:trust-store-type "jks" ; default jks
:trust-store-pass "trustpass"})})
You can also store the proxied connection manager and reuse it later.
Sometime it is desirable to run a request with some middleware enabled and some
left out, the with-middleware
method provides this functionality:
(with-middleware [#'clj-http.client/wrap-method
#'clj-http.client/wrap-url
#'clj-http.client/wrap-exceptions]
(get "http://example.com")
(post "http://example.com/foo" {:body (.getBytes "foo")}))
To see available middleware, check the clj-http.client/default-middleware
var,
which is a vector of the default middleware that clj-http uses.
clj-http.client/*current-middleware*
is bound to the current list of
middleware during request time.
While clj-http tries to provide the features needed, there are times when it does not provide access
to a parameter that you need. In these cases, you can use a couple of advanced parameters to provide
arbitrary configuration functions to be run on the HttpClientBuilder
by specifying
:http-builder-fns
and :async-http-builder-fns
.
Each of these variables is a sequence of functions of two arguments, the http builder
(HttpClientBuilder
for :http-builder-fns
and HttpAsyncClientBuilder
for
:async-http-builder-fns
) and the request map.
;; A function that takes a builder and disables Apache's cookie management
(defun my-cookie-disabler [^HttpClientBuilder builder
request]
(when (:disable-cookies request)
(.disableCookieManagement builder)))
;; The functions to modify the builder are passed in
(http/post "http://www.example.org" {:http-builder-fns [my-cookie-disabler]
:disable-cookies true})
The functions are run in the order they are passed in (inside a doseq
).
Please send a pull request or open an issue if you have any problems. See CONTRIBUTING.md
for more
information.
If you need to fake clj-http responses (for things like testing and such), check out the clj-http-fake library.
In 2.0.0+ clj-http’s optional dependencies at excluded by default, in order to
use the features you will need to add them to your project.clj
file.
clj-http currently has four optional dependencies, cheshire
, crouton
,
tools.reader
and ring/ring-codec
. Any number of them may be included by
adding them with the clj-http dependency in your project.clj:
;; optional dependencies
[cheshire] ;; for :as :json
[crouton] ;; for :decode-body-headers
[org.clojure/tools.reader] ;; for :as :clojure
[ring/ring-codec] ;; for :as :x-www-form-urlencoded
Prior to 2.0.0, you can exclude the dependencies and clj-http will work without them.
Like clj-http but need something more lightweight without as many external dependencies? Check out clj-http-lite for a project that can be used as a drop-in replacement for clj-http.
This is actually caused by your project attempting to use clj-json and cheshire in the same classloader. You can fix the issue by either not using clj-json (and thus choosing cheshire), or specifying an exclusion for clj-http in your project like this:
(defproject foo "0.1.0-SNAPSHOT"
:dependencies [[org.clojure/clojure "1.3.0"]
[clj-http "0.3.4" :exclusions [cheshire]]])
Note that if you exclude cheshire, json decoding of response bodies and json encoding of form-params cannot happen, you are responsible for your own encoding/decoding.
As of clj-http 0.3.5, you should no longer see this, as Cheshire 3.1.0 and clj-json can now live together without causing problems.
Persistent connections kept alive by the connection manager become stale: the target server shuts down the connection on its end without HttpClient being able to react to that event, while the connection is being idle, thus rendering the connection half-closed or ‘stale’.
This can be solved by using (with-connection-pool) as described in the ‘Using Persistent Connection’ section above.
To run the tests:
$ lein deps $ lein test Run all tests (including integration): $ lein test :all Run tests against all clojure versions $ lein all test $ lein all test :all
With over three million downloads, clj-http is a widely used, battle-tested clojure library. It is also included in other libraries (like database clients) as a low-level http wrapper.
Libraries using clj-http:
Libraries inspired by clj-http:
- aws-sig4
- a pure clojure implementation of AWS v4 signature request signing as middleware
(feel free to open a PR or issue if you’d like to add middleware here)
Released under the MIT License: http://www.opensource.org/licenses/mit-license.php