AWS-Terraform implementation documentation unclear about scim-examples HOT 35 CLOSED

Hi @murty0,

thank you for your question.

we deploy op-scim bridge using Mac and terraform and our example is not limiting you to running terraform on a Linux instance. Linux is used to execute op-scim bridge binary/service in a form of an AWS instance and terraform helps to manage the deployment.

You can install and run terraform code on a variety of platforms including Mac OSX.
https://www.terraform.io/intro/getting-started/install.html

Once you have terraform installed on your Mac system you can execute terraform code from our AWS-Terraform example and that will help you in deploying op-scim bridge in AWS environment.

Could you please let us know which part of our example confused you and we will try to improve it?

Please let me know if you have any questions.

Thank you very much and happy terraforming.

• Andrey.

from scim-examples.

Hi Andrey

My issue wasnt with Terraform. We already use Terraform here.

My issue was with this part of the documentation (https://github.com/1Password/scim-examples/tree/master/aws-terraform), which is to execute the op-scim bridge binary/service :

from scim-examples.

That is correct, we distribute our op-scim bridge as a Debian like package.
It is native to Debian, Ubuntu and other Debian derived distributions.

What system/OS you want to deploy op-scim bridge on?

from scim-examples.

I am using a Mac, and will be deploying this in AWS.

How can I add this repository manually on a Mac? The above mentioned terminal commands are Linux specific.

Is there a way to avoid the manual process altogether?

from scim-examples.

you use/run those commands in your AWS instances (Linux) not on your local Mac System.
you can use AWS instance user-data to configure/bootstrap your AWS instances, for example: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html

from scim-examples.

those steps are already included in user-data cloud init code in the example: https://github.com/1Password/scim-examples/blob/81f66e808941a9215af3fb27b3f4351c9c8b17ff/aws-terraform/terraform/module_scim_app/data/user_data/02-environment.yml

from scim-examples.

Hi Andrey

Thanks for clarifying.

The documentation isnt clear, and seems to be all over the place. There is no clear step by step guide for setting this up.

Is there an update due soon? One of the reps told me its coming soon, possibly next week which would make this entire process simpler?

from scim-examples.

@murty0 we are working on improving it.

Thank you very much for bringing that topic. We will simplify it.

• Andrey.

from scim-examples.

Hi @ag-andrey05

When I run this command on the EC2 instance to get the repo:

echo "deb https://apt.agilebits.com/op-scim/ stable op-scim" > /etc/apt/sources.list.d/op-scim.list

I keep getting this error:

-bash: /etc/apt/sources.list.d/op-scim.list: Permission denied

I have already added the GPG key for the repo to apt-key.

from scim-examples.

Hi @murty0,

That command requires privileged access.
try it like this:
sudo sh -c "echo 'deb https://apt.agilebits.com/op-scim/ stable op-scim' > /etc/apt/sources.list.d/op-scim.list"

We will update documentation to indicate that.

from scim-examples.

Hi @ag-andrey05

Thanks for that. I ran the above command, and it worked. But when I wanted to proceed to the next steps, the server can't recognise the 'op-cim' command

ubuntu@ip-XXXXXXXXX:~$ sudo apt-get update
Hit:1 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic InRelease
Get:2 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:3 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Hit:4 https://download.docker.com/linux/ubuntu bionic InRelease
Get:5 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [401 kB]
Get:6 http://security.ubuntu.com/ubuntu bionic-security InRelease [83.2 kB]
Get:7 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [560 kB]
Get:8 https://apt.agilebits.com/op-scim stable InRelease [3611 B]
Get:9 https://apt.agilebits.com/op-scim stable/op-scim amd64 Packages [1129 B]
Fetched 1212 kB in 1s (2420 kB/s)
Reading package lists... Done

ubuntu@ip-XXXXXXXXX:~$ sudo apt-get install op-scim
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  op-scim
0 upgraded, 1 newly installed, 0 to remove and 52 not upgraded.
Need to get 1679 kB of archives.
After this operation, 6032 kB of additional disk space will be used.
Get:1 https://apt.agilebits.com/op-scim stable/op-scim amd64 op-scim amd64 0.7 [1679 kB]
Fetched 1679 kB in 0s (4481 kB/s)
Selecting previously unselected package op-scim.
(Reading database ... 56812 files and directories currently installed.)
Preparing to unpack .../archives/op-scim_0.7_amd64.deb ...
Unpacking op-scim (0.7) ...
Setting up op-scim (0.7) ...
Warning: The home dir /var/lib/op-scim you specified already exists.
Adding system user `opscim' (UID 111) ...
Adding new user `opscim' (UID 111) with group `nogroup' ...
The home directory `/var/lib/op-scim' already exists.  Not copying from `/etc/skel'.
adduser: Warning: The home directory `/var/lib/op-scim' does not belong to the user you are currently creating.
Created symlink /etc/systemd/system/multi-user.target.wants/op-scim.service → /lib/systemd/system/op-scim.service.

ubuntu@ip-XXXXXXXXX:~$ op-scim
op-scim: command not found

from scim-examples.

@murty0,

op-scim is installed in /var/lib/op-scim/
you can run that command as
/var/lib/op-scim/op-scim --redis-host={cache address} --redis-port={redis port} --session={/path/to/scimsession}

providing redis cache address and port, and path to the session file.

from scim-examples.

@ag-andrey05

ubuntu@ip-XXXXXXXXX:~$ sudo /var/lib/op-scim/op-scim --session={/path/to/scimsession}
[LOG] [0.7] 2018/10/12 14:18:34 (INFO) using default redis connection (redis:6379)
[LOG] [0.7] 2018/10/12 14:18:34 (INFO) creating redis connection with address redis:6379
2018/10/12 14:18:34 cacheConnWithAddress failed to newRedisConn: newRedisConn failed to Dial: dial tcp: lookup redis on 127.0.0.53:53: server misbehaving

from scim-examples.

@murty0 ,

you need to create op-scim session file first:
https://github.com/1Password/scim-examples/tree/master/aws-terraform#session-file

you can generate the session file and bearer token by running the 1Password SCIM Bridge binary in init mode:

/var/lib/op-scim/op-scim/op-scim init

Note: When prompted for user credentials by the init command, use the 1Password credentials for the provision manager user and not an Owner or Administrator user.

Init mode will guide the administrator through an interactive process which generates the encrypted session file and bearer token. Although strong encryption is used to secure the session file, prevent unauthorized access to it. Never store the session file in the same place as the bearer token. The bearer token will be required to configure the identity provider (Azure, Okta, etc) to authenticate your connection.

from scim-examples.

@ag-andrey05

I already have a scimsession file from before, using the create-session-file.sh script:

#!/bin/bash

mkdir session

docker run -it -v $PWD/session:'/op-scim/session' 1password/scim /op-scim/create-session-docker.sh

cp ./session/scimsession ./scimsession
rm -rf ./session

from scim-examples.

Then you need to specify your redis server/instance address and port.
Redis is not included in the op-scim package.
if not specified using --redis-host={cache address} --redis-port={redis port} by default it tries to connect to redis:6379
You can use Cloud provider redis or install redis locally and specify redis connection details using --redis-host={cache address} --redis-port={redis port}

from scim-examples.

ok so this worked:

sudo /var/lib/op-scim/op-scim --redis-host=127.0.0.1 --redis-port=6379 --session=/var/lib/op-scim/scimsession

BUT before running the above, I had to manually do:

sudo -i
cd /var/lib/op-scim
mkdir .op
mv /path/to/scimsession /var/lib/op-scim/.op
exit
sudo apt install redis-server

from scim-examples.

@murty0, thank you very much for the feedback we'll use it to improve our documentation.

from scim-examples.

@ag-andrey05

Hi Andrey

I am still facing issues with this. I was able to run the terraform commands successfully, and was presented with a URL. In Okta, when I am putting in this URL + the bearer token for the provisioning account, it keeps saying that the credentials are invalid. So that is my first question, that why is it saying that the credentials are invalid.

Another question, why do we need Redis? At what point is Redis (local or in AWS ElasticCache) being used? Do we need Redis even though I am using AWS ALB and was presented with a URL?

Lastly, at what point is the SCIM bridge pointing to my 1Password instance ie sandbox-XYZ.1password.com? I have not added this anywhere, as wasn't prompted to do this at any stage, nor in the Terraform variables.tf.

Some of the answers might be obvious, but I am having real difficulties getting this to work.

from scim-examples.

Hi @murty0,

in order to answer your questions i need to clarify few things first.

1. Did you use AWS secrets manager to store the op-scim session file? Or you handle it differently?
2. What does the AWS ALB Target group show? How many healthy targets/instances? (Go to AWS console -> Load Balancing -> Target groups -> Targets)
3. Could you verify that URL you've got points to ALB as an alias? In other words your URL has to be able to resolve to an IP address of your AWS ALB (load balancer).

Redis is used by op-scim application for caching during operation and you need redis either local running on the instance or AWS redis. Redis is unrelated to AWS ALB, they provide different type of functionality. AWS ALB is required to provide SSL termination (transport security) and load balancing, while redis is used for caching.

When you initialize/create your scimsession file you provide details of your 1Password account that includes your sandbox-XYZ.1password.com 1Password endpoint. When op-scim application is starting it reads that information from the scimsession thus you do not need to include your 1Password account details in the terraform variables/configuration.

Let me know if you have questions.

Andrey.

from scim-examples.

Hi @ag-andrey05

1. I am using AWS Secrets Manager to store the op-scim/scimsession file's contents
2. The AWS ALB Target group shows 1 healthy instance and 1 unhealthy instance
3. The URL i've got points to ALB as an alias

Thanks for the explanation about the endpoint.

I have a call in half an hr with one of the engineers from your side. I will walk through the entire set up with them, to be able to hopefully resolve this issue.

from scim-examples.

So the issue was that I had created the secret via the UI.

This was resolved when I used this CLI command:

aws secretsmanager create-secret --name op-scim/scimsession --secret-binary file:///path/to/scimsession --region <aws_region>

from scim-examples.

@ag-andrey05 Thanks!

from scim-examples.

Hi @murty0,
here is the AWS op-scim diagram for your reference.

op-scim.pdf

Terraform code deploys the following AWS and op-scim resources:

• Application load balancer (ALB), listening on port 443 and providing transport security and load balancing.
• Application instances are managed by Autoscaling group (ASG), ASG monitors instances and automatically adjusts capacity to maintain steady, predictable performance. Capacity is configured in main.tf and instance specific configuration is in variables.tf. Application is listening on port 3002 and Instance Security Group allows incoming traffic only from ALB on application port 3002.
• Instance IAM
• Instance and ALB security groups
• public DNS record and Internal DNS zone.
• VPC, subnets, Route Tables, route53, IGW, NGW and their dependencies.

from scim-examples.

@ag-andrey05 This has been immensely and uncomparably helpful, thank you so much!

from scim-examples.

Hi

We were going to deploy the SCIM bridge in to production today, but at the last minute we have run in to issues again! 

The 1Password documentation and/or the scimsession file is incorrect ie step 3 in this link: https://support.1password.com/scim/

After cloning the repo, when I run the script, I get these prompts:

- Enter your sign-in address (example.1password.com)

• Enter the email address for your provisioning account on xxxx

- Enter the Secret Key for [email protected] on xxxx

- Enter the Password for [email protected] on xxxx

• Then it sends me an MFA Duo Push

And here is where the issue is. It then says that the provisioning account doesnt have the appropriate permissions.

There are two problems here.

Firstly, your documentation (linked above) says that:

This setup process will:

Ask you to sign in to your administrator account
Create the provision managers group and the provision manager account
Set up the provision manager account
Generate your bearer token and session file

This is obviously incorrect, as you can see the prompts I am getting after running the script are for the provisioning account, and not for the admin. 

Secondly, and this is linked to the first issue above, I remember around the 10th of October when I was starting this setup, this script was working as expected, and I had setup the sandbox scim bridge only after I had the scimsession file from the script. And as part of the script, I as an admin had to explicitly give the new provisioning account the appropriate permissions for provisioning and deprovisioning. 

This time the script isnt doing that. So something has changed in the last 3 weeks which is breaking the script.

This is really unfortunate, as since the last time we spoke I had been busy getting everything working on the AWS side, and today I faced this issue just before we had to deploy. 

Can you please help me get to a resolution for this ASAP! 

from scim-examples.

@murty0 Sorry for the issues.

The problem you're encountering is because we released v0.7 of the SCIM bridge recently, which includes a new automated setup script that replaces the manual process you went through last time.

The scim-setup.sh script that you're running is linked to the latest tag of the docker image. Since you ran this command pre-0.7, your local copy of the latest tag is 0.6.7. If you delete that image and re-run the command, it will pull a newer image and you'll get what you see in the documentation

If you run docker images, you'll see a list of the images you have local copies of. If you grab the ID of the 1password/scim:latest image, and then run docker image rm [id], you can then re-run the script to get the latest version.

Sorry for the trouble, I'm going to change the script to point at the specific image version so that this won't happen in the future.

from scim-examples.

Here's a PR that fixes it: #19

from scim-examples.

@agilecohix thanks for getting back to me. It works now!

from scim-examples.

Hi @agilecohix and @ag-andrey05

So we have ran in to another problem during our attempt to deploy this.

Firstly, its important for you to know that we are using our internal terraform modules to create this infrastructure.

We have used the ubuntu AMI as specified in your implementation, but overall here at Intercom we use Amazon Linux AMIs.

So the problem is this:

the instance is not passing the ELB heath check for some reason. to find out what is wrong we need to SSH to the instance.
right now SSH is not open in the security group and therefore we cannot connect to the instance. Also, as this is an ubuntu AMI and not our own Intercom AWS Linux AMI, we cannot connect to the instance using our SSH certificates.

we can:
a) open SSH to the world and pass a custom SSH key that would allow us to connect to the instance
b) use an intercom AMI but this would require a 1password rpm package as Amazon linux is centos based not debian

We are in favour of b), as that way we would use a standard intercom AMI as we do not use Ubuntu for anything else. a) is a hack just to get it up and running.

Can you provide us with an rpm package?

from scim-examples.

@agilecohix
In this case the 504 is probably caused by the load balancer, but we still need a standardised way of accessing the server when its in production, to be able to troubleshoot any errors which can in the future come up. Currently as it stands, its a black box for us, which cant be accessed, unless we hardcode the SSH keys in the user_data template, but as you can imagine, this is not the way to do when deploying in production.

After consulting my team, we have decided we will be changing our approach in this. We will now deploy the SCIM bridge using the docker swarm implementation (https://github.com/1Password/scim-examples/tree/master/docker-swarm), but this will be done within an instance which is using our own AWS Linux AMI (centOS), so that it is pre-baked with all the required settings/certs/configs etc which we require for instances to have when deployed in to production. Once we are certain that the docker swarm SCIM bridge is working, we will then create a new AMI with the SCIM bridge baked into it, and then deploy it in AWS along with an ELB, ASG, Security Groups etc.

Can I please arrange a call with who ever is best suited for the docker swarm implementation, so we can go through the entire setup in our test environment. The documentation for this is quite minimal to say the least, and we need to get up and running with this ASAP.

from scim-examples.

@agilecohix

I tried to push through the docker swarm implementation, but still getting "Incorrect credentials, please verify and try again." in Okta.

I am not sure if my implementation is correct, as in the docker-compose.yml file, I changed this line of code:
entrypoint: ["/op-scim/op-scim", "--letsencrypt-domain={YOUR-DOMAIN-HERE}", "--session=/run/secrets/scimsession"]

I removed the "--letsencrypt-domain={YOUR-DOMAIN-HERE}" part, as we dont use letsencrypt, and instead use AWS KMS. I added my Route53/ACM TLS certified domain name there instead.

Also, I had to take a couple of more steps to even get this far.

-the version in the docker-compose.yml file had to be changed from '3.3' to '3.1', otherwise I kept getting 'unsupported Compose file version: 3.3'.
-to start the swarm, I had to do 'docker swarm init'.
-For docker logs, the command which is in the readme for the docker swarm implementation wasnt working out of the box, and I had to enable the experimental flag in '/etc/docker/daemon.json' (https://github.com/docker/docker-ce/blob/master/components/cli/experimental/README.md). Then, the docker stack deploy -c docker-compose.yml op-scim could only be started after doing 'sudo -i', otherwise it didnt have the appropriate permissions

I am writing down all the steps incase someone else happens to end up here. The documentation should be verbose, and nothing should be left to the imagination, and zero prior knowledge should be assumed for the endusers, because to figure out each unclear step takes so much time!

from scim-examples.

@murty0 Sorry for the trouble you're encountering.

If you're using your own certificate, then you should remove the --letsencrypt-domain flag altogether; if TLS is already being terminated, then you do not want letsencrypt trying to fetch another certificate, it will not work. Try removing that flag and let me know if that helps.

As for your other feedback:

• The incompatible docker-compose version is likely because the latest version of Docker compose in the rpm is likely not the latest. Changing it to 3.1 is just fine as we are not using anything 3.3 specific.
• The documentation does assume that you have a swarm cluster already set up. I will add a blurb that says this, and links to the documentation.
• Your issue with the logs is not something I've encountered before. I'll check this out myself.

from scim-examples.

from scim-examples.

@ag-andrey05 @agilecohix thanks for all your help!

Closing this off now, with some final notes.

The implementation is up and running now. The infrastructure was deployed using Terraform, and the SCIM app was deployed using Docker swarm (docker version 1.13.0) within an AWS Linux (CentOS 6) EC2 instance.

Here is the cloud_init user_data template for the autoscaling group to bake the SCIM app in to the server, for anyone who treads here:

user_data = <<EOF
#!/bin/bash

chkconfig docker on

mkdir -p /etc/docker

#turn on experimental flag for docker, otherwise unable to view docker service logs
echo '{"experimental": true}' > /etc/docker/daemon.json

#restart docker service, and experimental should be on this time
service docker restart

#create docker-compose.yml
mkdir /op-scim
touch /op-scim/docker-compose.yml

/bin/echo "version: '3.1'

services:
    scim:
        image: 1password/scim:v0.7.2
        deploy:
          replicas: 1
          restart_policy:
            condition: on-failure
        networks:
          - op-scim
        ports:
          - "80:8080"
        secrets:
          - scimsession
        entrypoint: ["/op-scim/op-scim", "--port=8080",  "--session=/run/secrets/scimsession"]
    redis:
        image: redis:latest
        deploy:
          replicas: 1
          restart_policy:
            condition: on-failure
        networks:
          - op-scim

networks:
  op-scim:

secrets:
  scimsession:
    external: true" > /op-scim/docker-compose.yml

#create scimsession
scimsession=`aws secretsmanager get-secret-value --secret-id <SECRET-ARN> --region <REGION> --output text --query SecretBinary | base64 --decode`

touch /op-scim/scimsession && chmod u+x /op-scim/scimsession
/bin/echo "$scimsession" > /op-scim/scimsession

#initiate the swarm
docker swarm init

#create docker swarm secret
cat /op-scim/scimsession | docker secret create scimsession -

#deploy the docker service/stack
docker stack deploy -c /op-scim/docker-compose.yml op-scim

#send logs to cloudwatch (optional steps from here on)
touch /op-scim/fetch_scim_logs.sh && chmod u+x /op-scim/fetch_scim_logs.sh

/bin/echo "#!/bin/bash
docker service logs -f op-scim_scim >> /var/log/docker_scim.log" > /op-scim/fetch_scim_logs.sh
bash ./op-scim/fetch_scim_logs.sh &

/bin/echo "[general]
state_file = /var/lib/awslogs/agent-state
use_gzip_http_content_encoding = false
[logstream1]
log_group_name = /opscim/var/log/cloud-init-output
log_stream_name = {instance_id}
datetime_format = %Y-%m-%dT%H:%M:%S%z
time_zone = UTC
file = /var/log/cloud-init-output.log
file_fingerprint_lines = 1
multi_line_start_pattern = ^[^\s]
initial_position = start_of_file
encoding = utf_8
buffer_duration = 5000
batch_count = 1000
batch_size = 32768

[logstream2]
log_group_name = /opscim/var/log/docker_scim
log_stream_name = {instance_id}
datetime_format = %Y-%m-%dT%H:%M:%S%z
time_zone = UTC
file = /var/log/docker_scim.log
file_fingerprint_lines = 1
multi_line_start_pattern = ^[^\s]
initial_position = start_of_file
encoding = utf_8
buffer_duration = 5000
batch_count = 1000
batch_size = 32768" > /etc/awslogs/awslogs.conf

service awslogs restart
EOF
}

from scim-examples.