Configure queue url for multiple AWS accounts about active-elastic-job HOT 24 OPEN

Nope, I think you got everything! Just make sure the region and credentials are defined correctly for each environment and you shouldn't have to adjust anything else. I'm going to go ahead and close this issue but feel free to open it back up if you run into any problems or have any other questions!

from active-elastic-job.

@joeyparis Thanks a lot for your quick response here. I'm just removing the fork from this project and using your version directly. Now I need to test it out and see if I missed anything.

from active-elastic-job.

@joeyparis I'm working on testing setting the endpoint config. I'll get back with you about that shortly.

from active-elastic-job.

@codeguru42 I am definitely open to adding that option if you can't get .endpoint working. My workload is pretty heavy at the moment, but I would accept a PR with testing. Otherwise, it's something I can work on adding a bit down the road!

from active-elastic-job.

I would take a different approach. We use the config.active_job.queue_name_prefix parameter and set it to Rails.env. Then our queue names in SQS are production_report_processing, stage_report_processing, and development_report_processing. In our job file we do queue_as :report_processing and ActiveJob will then prefix it with the result of Rails.env + _ to use the correct queue. Does that make sense? If not I can right out a more thorough example.

from active-elastic-job.

@joeyparis Yes, that makes sense. I'll look at it more this week to see if it works for us. A previous project owner forked a bunch of OSS gems to customize them and I'm trying to use the original gems as much as possible. I don't particularly want to maintain forks of all our dependencies.

from active-elastic-job.

@joeyparis Question: are all your SQS queues in the same account? We have prod, staging, and development in separate AWS accounts so the SQS instances for each one have an entirely different URL, not just the queue name. I can make a PR to show you so that we have some concrete code to talk about.

from active-elastic-job.

@codeguru42 I definitely understand not wanting to maintain forks of all your dependencies, that can get out of hand quickly!

All of our queues are in the same account but that shouldn't be an issue if they are in different accounts. In application.rb as part of your config you can do this:

Rails.application.configure do
  # {...}
  config.active_elastic_job.aws_credentials = Aws::Credentials.new(ENV['AWS_ACCESS_KEY_ID'], ENV['AWS_SECRET_ACCESS_KEY'])
 # {...}
end

Then just make sure each environment has the appropriate access key ID and secret key variables and it should make no difference.

At that point, you could even skip out on using the prefix and use the same name in each version then just update the credentials based on the environment version. Although I would personally be against that because it increases the odds of accidentally believing you're in one environment when you're really in another.

from active-elastic-job.

I don't have any control over the way each environment is set up. That's under the company's IT department.

To give some more details, I have the queues like this:

https://sqs.<region>.amazonaws.com/<some id number>/<queue name>

I have a job:

class MyJob < ApplicationJob
    queue_as :my_queue
end

In my elasticbeanstalk environment, I call MyJob.perform_later.

If I understand correctly, my_queue will be <queue name> and <some id number> is the account id and can be determined automatically from the EB node.

So I don't really need to do anything to configure the queue_urls in the first place. I'm starting to wonder why we even need to customize your library.

If I'm missing anything, feel free to fill me in. Otherwise, I will probably close this issue in the next couple of days.

from active-elastic-job.

@joeyparis So after using this gem directly rather than our fork, I'm getting:

The job is bound to queue at MY_QUEUE.
Unfortunately a queue with this name does not exist in this
region. Either create an Amazon SQS queue named MY_QUEUE -
you can do this in AWS console, make sure to select region
MY_REGION - or you
select another queue for your jobs.

It looks like active-elastic-job is posting to https://sqs.us-east-2.amazonaws.com/ but it needs to be https://sqs.<region>.amazonaws.com/<account_id>. Any suggestions for solutions?

from active-elastic-job.

@codeguru42 you could try using the Rails.application.config.active_elastic_job.endpoint config option for now, but I'm still trying to figure out why my setup doesn't require it but yours does. So I'll get back to you with more info.

from active-elastic-job.

@codeguru42 are the AWS credentials you are using created under each account for testing, or are they all created under one account and the other accounts grant access to them?

from active-elastic-job.

@codeguru42 From what I understand each environment account is maintained under an aggregate company account. We use credentials from the aggregate account, so they are the same across all the subaccounts.

from active-elastic-job.

@codeguru42 what I believe is happening is the AWS SQS SDK does not require an account ID and will default to the account of the credentials which in your case appears to be the aggregate company account preventing it from finding your queues in the subaccounts with just the queue name. The solution to this is either to add an additional config option for queue_owner_aws_account_id or just update the documentation to mention the existing endpoint config option. Can you confirm if using the endpoint field in your respective Rails stage config files works for you?

from active-elastic-job.

@joeyparis After setting endpoint, I get

bad URI(is not URI?): "{\"my_queue\"=>\"https://sqs.<region>.amazonaws.com/<account_id>/my_queue\"}"

Should endpoint include the full URL to the queue? Or should it only include the domain and account id? Or what?

from active-elastic-job.

oh wait...that's a stringified hash, no wonder it's invalid.

from active-elastic-job.

Was the stringified hash an issue on your end or an unexpected outcome from passing a string to endpoint?

from active-elastic-job.

The stringified hash was an artifact of the code I used to set endpoint. Completely on my end. I'm deploying a change to hopefully fix that.

from active-elastic-job.

@joeyparis I fixed the "bad URI" error, but still no luck with setting endpoint. I've tried setting it to both "https://sqs.<region>.amazonaws.com/<account_id>/my_queue" and "https://sqs.<region>.amazonaws.com/<account_id>". Both of these throws ActiveJob::QueueAdapters::ActiveElasticJobAdapter::NonExistentQueue.

from active-elastic-job.

from active-elastic-job.

@joeyparis That looks like a copy/paste error. I double checked and the region is correctly in the URL AFAICT.

from active-elastic-job.

@joeyparis Thanks for your help today. I just want to let you know that I still haven't found a working solution that sets Rails.application.config.active_elastic_job.endpoint. LMK if you have any other suggestions.

from active-elastic-job.

@joeyparis What are your thoughts about adding a queue_owner_aws_account_id config option as you mentioned earlier?

from active-elastic-job.

@joeyparis I'd be happy to send you a PR. I'll see what I can do over the next week or so.

from active-elastic-job.