Hello there!
I started trying Broadway [and BroadwaySqs] Today. Whenever I use SQS, I try using a local instance via ElasticMQ for development purposes via docker (docker run -p 9324:9324 -p 9325:9325 softwaremill/elasticmq-native
).
Is it possible to point to a local ElasticMQ instance using BroadwaySqs? I don't see anything related in the docs.
I tried setting the queue_url parameter to http://localhost:9324/<queue_name>
, but I got the following traceback:
22:14:43.438 [error] GenServer ExAws.Config.AuthCache terminating
** (RuntimeError) Instance Meta Error: {:error, %{reason: :timeout}}
You tried to access the AWS EC2 instance meta, but it could not be reached.
This happens most often when trying to access it from your local computer,
which happens when environment variables are not set correctly prompting
ExAws to fallback to the Instance Meta.
Please check your key config and make sure they're configured correctly:
For Example:
ExAws.Config.new(:s3)
ExAws.Config.new(:dynamodb)
(ex_aws 2.1.7) lib/ex_aws/instance_meta.ex:29: ExAws.InstanceMeta.request/2
(ex_aws 2.1.7) lib/ex_aws/instance_meta.ex:66: ExAws.InstanceMeta.instance_role_credentials/1
(ex_aws 2.1.7) lib/ex_aws/instance_meta.ex:74: ExAws.InstanceMeta.security_credentials/1
(ex_aws 2.1.7) lib/ex_aws/config/auth_cache.ex:83: ExAws.Config.AuthCache.refresh_config/2
(ex_aws 2.1.7) lib/ex_aws/config/auth_cache.ex:44: ExAws.Config.AuthCache.handle_call/3
(stdlib 3.11.2) gen_server.erl:661: :gen_server.try_handle_call/4
(stdlib 3.11.2) gen_server.erl:690: :gen_server.handle_msg/6
(stdlib 3.11.2) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message (from WorldTemp.TempProcessor.Broadway.Producer_0): {:refresh_config, %{access_key_id: [{:system, "AWS_ACCESS_KEY_ID"}, :instance_role], host: "sqs.us-east-1.amazonaws.com", http_client: ExAws.Request.Hackney, json_codec: Jason, normalize_path: true, port: 443, region: "us-east-1", retries: [max_attempts: 10, base_backoff_in_ms: 10, max_backoff_in_ms: 10000], scheme: "https://", secret_access_key: [{:system, "AWS_SECRET_ACCESS_KEY"}, :instance_role]}}
State: ExAws.Config.AuthCache
Client WorldTemp.TempProcessor.Broadway.Producer_0 is alive
(stdlib 3.11.2) gen.erl:167: :gen.do_call/4
(elixir 1.10.2) lib/gen_server.ex:1020: GenServer.call/3
(ex_aws 2.1.7) lib/ex_aws/config.ex:84: ExAws.Config.retrieve_runtime_value/2
(elixir 1.10.2) lib/stream.ex:572: anonymous fn/4 in Stream.map/2
(elixir 1.10.2) lib/enum.ex:3686: Enumerable.List.reduce/3
(elixir 1.10.2) lib/stream.ex:1609: Enumerable.Stream.do_each/4
(elixir 1.10.2) lib/enum.ex:959: Enum.find/3
(ex_aws 2.1.7) lib/ex_aws/config.ex:71: anonymous fn/2 in ExAws.Config.retrieve_runtime_config/1
22:14:43.448 [error] GenServer WorldTemp.TempProcessor.Broadway.Producer_0 terminating
** (stop) exited in: GenServer.call(ExAws.Config.AuthCache, {:refresh_config, %{access_key_id: [{:system, "AWS_ACCESS_KEY_ID"}, :instance_role], host: "sqs.us-east-1.amazonaws.com", http_client: ExAws.Request.Hackney, json_codec: Jason, normalize_path: true, port: 443, region: "us-east-1", retries: [max_attempts: 10, base_backoff_in_ms: 10, max_backoff_in_ms: 10000], scheme: "https://", secret_access_key: [{:system, "AWS_SECRET_ACCESS_KEY"}, :instance_role]}}, 30000)
** (EXIT) an exception was raised:
** (RuntimeError) Instance Meta Error: {:error, %{reason: :timeout}}
I noticed it is assuming the region us-east-1
, so I tried adding the region parameter as part of the producer config (config: [region: "elasticmq"]
):
22:17:36.260 [error] GenServer ExAws.Config.AuthCache terminating
** (RuntimeError) Instance Meta Error: {:error, %{reason: :timeout}}
You tried to access the AWS EC2 instance meta, but it could not be reached.
This happens most often when trying to access it from your local computer,
which happens when environment variables are not set correctly prompting
ExAws to fallback to the Instance Meta.
Please check your key config and make sure they're configured correctly:
For Example:
ExAws.Config.new(:s3)
ExAws.Config.new(:dynamodb)
(ex_aws 2.1.7) lib/ex_aws/instance_meta.ex:29: ExAws.InstanceMeta.request/2
(ex_aws 2.1.7) lib/ex_aws/instance_meta.ex:66: ExAws.InstanceMeta.instance_role_credentials/1
(ex_aws 2.1.7) lib/ex_aws/instance_meta.ex:74: ExAws.InstanceMeta.security_credentials/1
(ex_aws 2.1.7) lib/ex_aws/config/auth_cache.ex:83: ExAws.Config.AuthCache.refresh_config/2
(ex_aws 2.1.7) lib/ex_aws/config/auth_cache.ex:44: ExAws.Config.AuthCache.handle_call/3
(stdlib 3.11.2) gen_server.erl:661: :gen_server.try_handle_call/4
(stdlib 3.11.2) gen_server.erl:690: :gen_server.handle_msg/6
(stdlib 3.11.2) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message (from WorldTemp.TempProcessor.Broadway.Producer_0): {:refresh_config, %{access_key_id: [{:system, "AWS_ACCESS_KEY_ID"}, :instance_role], host: nil, http_client: ExAws.Request.Hackney, json_codec: Jason, normalize_path: true, port: 443, region: "elasticmq", retries: [max_attempts: 10, base_backoff_in_ms: 10, max_backoff_in_ms: 10000], scheme: "https://", secret_access_key: [{:system, "AWS_SECRET_ACCESS_KEY"}, :instance_role]}}
State: ExAws.Config.AuthCache
Client WorldTemp.TempProcessor.Broadway.Producer_0 is alive
(stdlib 3.11.2) gen.erl:167: :gen.do_call/4
(elixir 1.10.2) lib/gen_server.ex:1020: GenServer.call/3
(ex_aws 2.1.7) lib/ex_aws/config.ex:84: ExAws.Config.retrieve_runtime_value/2
(elixir 1.10.2) lib/stream.ex:572: anonymous fn/4 in Stream.map/2
(elixir 1.10.2) lib/enum.ex:3686: Enumerable.List.reduce/3
(elixir 1.10.2) lib/stream.ex:1609: Enumerable.Stream.do_each/4
(elixir 1.10.2) lib/enum.ex:959: Enum.find/3
(ex_aws 2.1.7) lib/ex_aws/config.ex:71: anonymous fn/2 in ExAws.Config.retrieve_runtime_config/1
22:17:36.272 [error] GenServer WorldTemp.TempProcessor.Broadway.Producer_0 terminating
** (stop) exited in: GenServer.call(ExAws.Config.AuthCache, {:refresh_config, %{access_key_id: [{:system, "AWS_ACCESS_KEY_ID"}, :instance_role], host: nil, http_client: ExAws.Request.Hackney, json_codec: Jason, normalize_path: true, port: 443, region: "elasticmq", retries: [max_attempts: 10, base_backoff_in_ms: 10, max_backoff_in_ms: 10000], scheme: "https://", secret_access_key: [{:system, "AWS_SECRET_ACCESS_KEY"}, :instance_role]}}, 30000)
** (EXIT) an exception was raised:
** (RuntimeError) Instance Meta Error: {:error, %{reason: :timeout}}
I see host is now nil, yet I could not go much further. I tried setting scheme to http://
instead of https://
but it had no effect.
Then I tried following the source code, which led me to https://github.com/dashbitco/broadway_sqs/blob/master/lib/broadway_sqs/ex_aws_client.ex#L28
@impl true
def receive_messages(demand, opts) do
receive_messages_opts = build_receive_messages_opts(opts, demand)
opts.queue_url
|> ExAws.SQS.receive_message(receive_messages_opts)
|> ExAws.request(opts.config)
|> wrap_received_messages(opts.ack_ref)
end
Then I checked ExAws.SQS.receive_message/2, which relies on ExAws.SQS.request/3. This might not be only a BroadwaySQS issue because I don't see how would ExAws make the AWS host configurable.
Just so I wrap it up with a question: Is it possible to point BroadwaySQS to a local ElasticMQ instance?
P.S.: I haven't touched any Elixir code in many months, so I am probably missing some important detail here