Is your feature request related to a problem? Please describe.
The Lambda function that's run on AWS requires JSON credentials for a GCP service account, which includes a non-rotatable secret key, and through which has broad access to all gSuite user data.

Describe the solution you'd like
GCP has a feature Workload Identity Federation (AWS instructions), which allows AWS to use OAuth to obtain short-lived credentials for impersonating a service account. Rather than having a JSON file with a secret key, instead you only need a JSON file with non-secret information, and everything else is configured in GCP via an identity pool.

Describe alternatives you've considered
The alternative here is just to use service account credentials, which just requires being a bit more careful.

Additional context
According to the AWS instructions, the primary requirement seems to be that golang.org/x/oauth2 must be version v0.0.0-20210218202405-ba52d332ba99 or later. Beyond that, there just needs to be a way to assign an AWS IAM Role to the executing function so that it has a role it can assert during the credential exchange.

This issue come from a comment in issue #62

Hi there!
This bug appeared again for us; we're using last version (0.0.13).
We're syncing three groups, and one of them have more than 100 users (137 right now).
I deleted the state to force a clean run of the sync app, but it still returns that error (and doesn't recreate the state.json).
Thanks a lot!

Originally posted by @snavarro-factorial in #62 (comment)

Describe the bug
If you have more than 50 users, and the attributes of the users in the SCIM side are different from the IDP side the first time, only the first 50 users are updated, the others no.

To Reproduce
Steps to reproduce the behaviour:

1. sync with you idp more than 50 users
2. after the first sync, remove the state file from AWS S3
3. go to AWS SSO console interface and disable users at the end of the list
4. sync again
5. check that the users you disable maybe still disable even if in the idp are enabe

Expected behavior

when the users in the SCIM side have the same username that the idp email, the rest of the attributes needs to be synced with the idp after the first sycn

Describe the bug

Error: cannot sync groups and their members: error doing the first sync: error reconciling groups: error updating groups in SCIM Provider: scim: error updating groups: aws checkHTTPResponse: error code: 400 Bad Request, body:

{
    "schema": [
        "urn:ietf:params:scim:api:messages:2.0:Error"
    ],
    "detail": "Request is unparsable, syntactically incorrect, or violates schema.",
    "status": "400",
    "exceptionRequestId": "88d9c6f3-566d-42b3-b1ed-183c3fc1efc4",
    "timeStamp": "2022-01-26 16:50:48.135"
}

To Reproduce

Migrating from ssosync fails the first time syncing.

Expected behaviour
is expected to do a transparent sync without any error

Screenshots

Describe the bug
once the state file is generated, the field codeVersion is empty

To Reproduce
run the program and check the state file

Expected behavior
field codeVersion with the same version of the program

Screenshots

Describe the bug
Version: 0.0.6
I have issue with sync the newly created group to AWS SSO. I added the new group and run idpscim and I got error:

WARN[0004] aws checkHTTPResponse: body: {"schema":["urn:ietf:params:scim:api:messages:2.0:Error"],"detail":"TENANT does not exist.","status":"404","exceptionRequestId":"93529112-dfce-4fef-82ed-ddd588a90e6b","timeStamp":"2022-02-11 06:10:54.937"}  method=POST status="404 Not Found"
Error: cannot sync groups and their members: error syncing state: error reconciling groups: error creating groups in SCIM Provider: scim: error creating group: aws CreateGroup: error decoding response body: %!w(<nil>), body:

To Reproduce
Steps to reproduce the behavior:

1. Go to Gsuite portal and add new group
2. Run sync command idpscim
3. See error

✗ ./idpscim
using config file: /idp-scim-sync/build/.idpscim.yaml
INFO[0000] starting sync groups                          codeVersion=main
INFO[0000] getting Identity Provider data                group_filter="[name:aws-sso* email:aws-sso* email:administrators*]"
INFO[0003] getting state data
INFO[0004] syncing from state                            lastsync="2022-02-10T16:27:51+01:00" since=14h43m3.72218s
INFO[0004] provider groups and state groups are different
INFO[0004] reconciling groups                            idp=3 state=2
WARN[0004] creating groups                               quantity=1
WARN[0004] creating group                                group=aws-sso-wa
WARN[0004] aws checkHTTPResponse: body: {"schema":["urn:ietf:params:scim:api:messages:2.0:Error"],"detail":"TENANT does not exist.","status":"404","exceptionRequestId":"93529112-dfce-4fef-82ed-ddd588a90e6b","timeStamp":"2022-02-11 06:10:54.937"}  method=POST status="404 Not Found"
Error: cannot sync groups and their members: error syncing state: error reconciling groups: error creating groups in SCIM Provider: scim: error creating group: aws CreateGroup: error decoding response body: %!w(<nil>), body:
Usage:
  idpscim [flags]

Flags:
  -k, --aws-s3-bucket-key string          AWS S3 Bucket key to store the state (default "state.json")
  -b, --aws-s3-bucket-name string         AWS S3 Bucket name to store the state
  -t, --aws-scim-access-token string      AWS SSO SCIM API Access Token
  -e, --aws-scim-endpoint string          AWS SSO SCIM API Endpoint
  -c, --config-file string                configuration file (default ".idpscim.yaml")
  -d, --debug                             fast way to set the log-level to debug
  -q, --gws-groups-filter strings         GWS Groups query parameter, example: --gws-groups-filter 'name:Admin* email:admin*' --gws-groups-filter 'name:Power* email:power*'
  -s, --gws-service-account-file string   path to Google Workspace service account file (default "credentials.json")
  -u, --gws-user-email string             GWS user email with allowed access to the Google Workspace Service Account
  -h, --help                              help for idpscim
  -f, --log-format string                 set the log format (default "text")
  -l, --log-level string                  set the log level [panic|fatal|error|warn|info|debug|trace] (default "info")
  -m, --sync-method string                Sync method to use [groups] (default "groups")
  -v, --version                           version for idpscim

Error: cannot sync groups and their members: error syncing state: error reconciling groups: error creating groups in SCIM Provider: scim: error creating group: aws CreateGroup: error decoding response body: %!w(<nil>), body:

Expected behavior
The new added group in GSuite should be created in AWS SSO

Additional context
When I remove this new group, everything is working fine and I have synced state in S3

# ./idpscimcli gws groups list
using config file: /idp-scim-sync/build/.idpscim.yaml
INFO[0000] 2 groups found
INFO[0000] [
  {
    "email": "[email protected]",
    "etag": "\"lEblPI0s7w1i1rxkFOkQrCYLrPcC1v0pVpIQVj2CXd8/TVN0qWdZ4r4TJc2wW9-CitlYWJs\"",
    "id": "00ihv6364h7nes5",
    "name": "aws-sso-admins"
  },
  {
    "email": "[email protected]",
    "etag": "\"lEblPI0s7w1i1rxkFOkQrCYLrPcC1v0pVpIQVj2CXd8/WAABaGiJAOrI7AFxy3Iq4nloOLM\"",
    "id": "04iylrwe3iefkh8",
    "name": "aws-sso-powerusers"
  }
]

# ./idpscim
using config file: /idp-scim-sync/build/.idpscim.yaml
INFO[0000] starting sync groups                          codeVersion=main
INFO[0000] getting Identity Provider data                group_filter="[name:aws-sso* email:aws-sso* email:administrators*]"
INFO[0003] getting state data
INFO[0003] syncing from state                            lastsync="2022-02-10T16:27:51+01:00" since=14h50m22.429897s
INFO[0003] provider groups and state groups are the same, nothing to do with groups
INFO[0003] provider users and state users are the same, nothing to do with users
INFO[0003] provider groups-members and state groups-members are the same, nothing to do with groups-members
INFO[0003] storing the new state                         groups=2 lastSync="2022-02-11T07:18:13+01:00" users=5
INFO[0003] sync completed
INFO[0003] sync groups completed                         duration=3.796357958s

but when I check the same groups in AWS SSO still I'm getting the same error:

# ./idpscimcli aws groups list
using config file: /idp-scim-sync/build/.idpscim.yaml
WARN[0000] aws checkHTTPResponse: body: {"schema":["urn:ietf:params:scim:api:messages:2.0:Error"],"detail":"TENANT does not exist.","status":"404","exceptionRequestId":"24c1c32c-4a75-456e-97cb-5cb47f83481d","timeStamp":"2022-02-11 06:20:55.414"}  method=GET status="404 Not Found"
ERRO[0000] error listing groups, error: aws ListGroups: error decoding response body: EOF
Error: aws ListGroups: error decoding response body: EOF
Usage:
  idpscimcli aws groups list [flags]

Aliases:
  list, l

Flags:
  -q, --filter string   AWS SSO SCIM API Filter, example: --filter 'displayName eq "Group Bar" and id eq "12324"'
  -h, --help            help for list

Global Flags:
  -t, --aws-scim-access-token string   AWS SSO SCIM API Access Token
  -e, --aws-scim-endpoint string       AWS SSO SCIM API Endpoint
  -c, --config-file string             configuration file (default ".idpscim.yaml")
  -d, --debug                          enable log debug level
  -f, --log-format string              set the log format (default "text")
  -l, --log-level string               set the log level (default "info")
      --output-format string           output format (json|yaml) (default "json")
      --timeout duration               requests timeout (default 10s)

Error: aws ListGroups: error decoding response body: EOF

Please let me know if I can provide more details information / check something additionally.

Describe the vulnerability

Log entries created from user input
Building log entries from user-controlled sources is vulnerable to insertion of forged log entries by a malicious user.

Tracking issue for:

• https://github.com/slashdevops/idp-scim-sync/security/code-scanning/1

Hello,

We have been investigating this issue. When Google is facing some problems, the list of groups returned is incomplete and the idp-scim-sync deletes the group on the AWS side. In other words, the users are losing access to our AWS infra :(

We think that this is related to the callback functions (see here and here). According to google-api-go-client code, "Pages invokes f for each page of results. A non-nil error returned from f will halt the iteration." As you can see in the body of the callback functions, we always return nil, so we never stop the iteration upon receiving an error, which might cause an inconsistency 🤔

Best,
L.

Describe the bug
Error when trying to sync users and groups. The users import, as do the groups. However, during the add members to groups phase a fatal error is returned.
Version: v0.0.14

To Reproduce
CloudWatch trace logs for a sample request

{
"idp": 2,
"level": "info",
"msg": "reconciling groups members",
"scim": 2,
"time": "2022-09-02T17:05:53Z"
}

{
"level": "warning",
"msg": "joining users to groups",
"quantity": 2,
"time": "2022-09-02T17:05:53Z"
}

{
"body": null,
"level": "trace",
"method": "GET",
"msg": "aws newRequest: request",
"path": "/xxxxxxxxxxxxxxxxx/scim/v2/Users",
"query": "filter=userName+eq+%22first.last%40example.com%22",
"time": "2022-09-02T17:05:53Z",
"url": "https://scim.us-east-1.amazonaws.com/xxxxxxxxxxxxxxxxx/scim/v2/Users?filter=userName+eq+%22first.last%40example.com%22"
}

{
"body": "{"totalResults":1,"itemsPerPage":1,"startIndex":1,"schemas":["urn:ietf:params:scim:api:messages:2.0:ListResponse"],"Resources":[{"id":"xxxxxxxxxxxxxxxxx","externalId":"xxxxxxxxxxxxxxxxx","meta":{"resourceType":"User","created":"2022-09-02T16:56:28Z","lastModified":"2022-09-02T16:56:28Z"},"schemas":["urn:ietf:params:scim:schemas:core:2.0:User"],"userName":"[email protected]","name":{"familyName\”:\”Last\”,\”givenName\”:\”First "},"displayName\”:\”First Last\”,\”active":true,"emails":[{"value":"[email protected]","type":"work","primary":true}]}]}",
"level": "trace",
"msg": "aws GetUserByUserName: response raw body data",
"time": "2022-09-02T17:05:53Z"
}

Error: cannot sync groups and their members: error doing the first sync: error reconciling groups members: error creating groups members in SCIM provider: scim: error getting user by email: aws GetUserByUserName: userName: [email protected], error decoding response body: EOF

Expected behavior
User would be added to group

Additional context
There appears to be an error when parsing the json response data in module:
pkg/aws/scim.go around Line 500

Note: This had been working earlier this week, but started throwing this error. I tried deleting the state file with no luck. (State File does not recreate due to failed run). Deleted the entire CF Stack and recreated with no resolution. I don't see anything wrong with the response JSON so I don't know why it would be throwing the EOF error suddenly unless Google has a problem suddenly.

Amazon just announced new APIs to manage SSO users and groups, I'm hoping to get a first impression if there's any interest in migrating to them?

https://aws.amazon.com/blogs/security/announcing-new-aws-iam-identity-center-apis-to-manage-users-and-groups-at-scale/

https://docs.aws.amazon.com/sdk-for-go/api/service/identitystore/

For the idpscim to be able to connect to the S3 bucket, it seems I need to populate the following environment variables:

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• AWS_SESSION_TOKEN

Would it possible to get the credentials using profile? (e.g. using the AWS_PROFILE env var)

Thank you!

When making the installan (or update) through SAR, secrets are always asked. According e.g. #99 Workload Identity Federation should be supported, but there doesn't seem to be any documentation how to use it with the service itself.

Of course general instructions are e.g. at https://dwdraju.medium.com/calling-google-cloud-services-from-aws-using-iam-roles-without-gcp-service-account-credentials-af918c05384d, but is there some config required as sync lambda side?

Hi there,

I successfully deployed the lambda function using the AWS Serverless Repository and was able to sync all Users and Groups into my AWS account by leaving the IDPSCIM_GWS_GROUPS_FILTER blank.

However, I want to sync only a few Groups and my understanding is the only way to achieve this is filter all Groups except the ones I want to sync using IDPSCIM_GWS_GROUPS_FILTER.

I tried setting this parameter but kept running into this error:
Error: cannot sync groups and their members: error getting groups from the identity provider: idp: error listing groups: googleapi: Error 400: Invalid Input: query, invalid

I have tried the following formats:
[email protected],[email protected],[email protected]
'[email protected]','[email protected]','[email protected]'
'[email protected]','[email protected]','[email protected]'
"[email protected]","[email protected],"[email protected]"
email:[email protected],email:[email protected],email:[email protected]
'email:[email protected]','email:[email protected]','email:[email protected]'

Hey I set up AWS SSO with all the regular actions done on Google Workspace.

But when launching this app from the Serverless Application Repository, I have to enter the file contents for the service account credentails from Google.
This part in particular would be nice to be documented.

Eventhough I set up a service account and downloaded the keys as json. I get an authentication error in the lambda functio then.

Please help

Hi,
I'm sort of new to this, I've been doing the manual provisioning in AWS SSO with google workspace directory for a while. I decied to set up the auto provisioning, went through the steps with AWS Serverless Applications. After the first sync, I can see all the users and groups synced up, but all the groups are empty and and none of the users is in their groups. I only had 2 groups (regular users and admin users), From the logs, i can see it sees the group (AWS-SSO) has 69 users. It tried to create them, but it runs into 409 conflict saying there's duplicate users already. I didn't remove all my original users and groups before turning on auto-provisioning, is that a problem?

{
"idp": 69,
"level": "info",
"msg": "reconciling users",
"scim": 50,
"time": "2022-02-14T17:37:21Z"
}

{
"level": "warning",
"msg": "creating users",
"quantity": 23,
"time": "2022-02-14T17:37:21Z"
}

{
"level": "warning",
"method": "POST",
"msg": "aws checkHTTPResponse: body: {"schema":["urn:ietf:params:scim:api:messages:2.0:Error"],"detail":"Refused to create a new, duplicate resource.","status":"409","exceptionRequestId":"279ec396-441a-49f8-b1b6-a4e04c08348c","timeStamp":"2022-02-14 17:37:21.224"}\n",
"status": "409 Conflict",
"time": "2022-02-14T17:37:21Z"
}

Error: cannot sync groups and their members: error doing the first sync: error reconciling users: error creating users in SCIM Provider: scim: error creating user: aws CreateUser: error decoding response body: EOF

To reproduce the problem:
I created a new group, added a brand new user, still getting the same errors in the log, it created the group and user, but user isn't showing as part of the group members on AWS, and there's still no state file being created.

Hi:

I configured the project to run it in my local environment, now I'm trying to deploy the lambda but I haven't found a way to specify the bucket where my state is.
Is there a way to use a pre-existing bucket?

Thank you!

Describe the bug
even when it is not used, the cloudformation template still having the parameter DisableState

To Reproduce
when is used the AWS Serverless public repository - slashdevops/idp-scim-sync to deploy the lambda the parameter DisableState appears in the template.

Expected behavior
parameter DisableState doesn't have to be there

Screenshots

Hello,

In some cases, the local "State" and the AWS state are inconsistent. We stumbled upon the case where the user was deleted in AWS but not in the local State. This generated an error 404, and the synchronization was not occurring anymore :( Would it be possible to ignore this error, and force the user deletion ?

Best,
L.

Is your feature request related to a problem? Please describe.
The current implementation relies on long lived credentials from google cloud.

Describe the solution you'd like
We could use Workload Identity federation described https://cloud.google.com/iam/docs/workload-identity-federation-with-other-clouds

Describe alternatives you've considered
None

Additional context
None

Describe the bug
When attempting to sync a group which contains another group as a member, the sync fails when trying to retrieve user details for that address.

When running, I get the following error:
Error: cannot sync groups and their members: error getting users from the identity provider: idp: error getting user: 019c6y181bx7798, email: accounts@<domain>, error: google: error getting user accounts@<domain>: googleapi: Error 400: Type not supported: userKey, badRequest

To Reproduce

1. Create a group which contains another group as a member
2. Attempt to sync that group to AWS.

Expected behavior
I would expect to have a configuration option that either expanded the group or ignored it.

Additional context
None

Is your feature request related to a problem? Please describe.
The go1.x lambda runtime is being deprecated

Describe the solution you'd like
The provided.al2 runtime should be used instead

Describe alternatives you've considered
I'm not aware of any better runtimes

Additional context
Email from Amazon:

We are ending support for the Go 1.x runtime in AWS Lambda, in line with Amazon Linux AMI end of maintenance support on December 31, 2023 [1]. Lambda continues to support the Go programming language, using the provided.al2 runtime. Using the provided.al2 runtime offers several benefits over the go1.x runtime, including support for AWS Graviton2 processors and a streamlined implementation with a smaller deployment package and faster function invoke path. For more information, see our blog post [2].
As described in the Lambda runtime support policy [3], end of support for language runtimes in Lambda happens in two stages. Starting December 31, 2023, Lambda will no longer apply security patches and other updates to the Go 1.x runtime used by Lambda functions, and functions using Go 1.x will no longer be eligible for technical support. In addition, you will no longer be able to create new Lambda functions using the Go 1.x runtime. Starting January 30, 2024, you will no longer be able to update existing functions using the Go 1.x runtime.
We recommend that you upgrade your existing Go 1.x functions to the provided.al2 runtime before December 31, 2023.
End of support does not impact function execution. Your functions will continue to run. However, they will be running on an unsupported runtime which is no longer maintained or patched by the AWS Lambda team.

Is your feature request related to a problem? Please describe.
I am trying to run idp-scim-sync in a k8s pod, but it does not work as it expects to have a credentials.json locally instead of an aws secret as when running like aws lambda.

Describe the solution you'd like
I would like for it to use environment variables to read secrets if they are set as it does when running like a aws lambda

Describe alternatives you've considered
I tried faking pod being a lambda with _LAMBDA_SERVER_PORT, but that just hangs
There is also option of creating a secret in k8s and mounting that as a file, but would go against our best practices

Additional context

First of all many thanks for your work!

Describe the bug
Users suspended in Google Workspace are synced to AWS SSO as Enabled

To Reproduce

1. Create user in Google Workspace
2. Suspend the new user
3. Sync users to AWS SSO
4. Observe: Suspended user gets synced as a normal enabled user

Expected behavior
User either (preferred) gets synced as "disabled" (this is how ssosync works) or alternatively does not get synced at all (and gets removed if exists)

Describe the bug
due to limitations in the AWS SSO SCIM API, specifically the patchgroup operation when some groups have more than 100 members

...
* A maximum of 100 membership changes are allowed in a single request.
...

the ACIM API response with:

{"schema":["urn:ietf:params:scim:api:messages:2.0:Error"],"schemas":["urn:ietf:params:scim:api:messages:2.0:Error"],"detail":"Request is unparsable, syntactically incorrect, or violates schema.","status":"400","exceptionRequestId":"f32ad558-5b05-45ea-8135-552a2f254d4f","timeStamp":"2022-04-15 14:09:03.693"}

To Reproduce
Have a group in Google Workspace with more than 100 members

Expected behavior
do the sync without any error

Describe the bug
We previously used this tool successfully. However we had some manual users in AWS SSO that would not have groups updated (from before we enabled SCIM). We removed the users thinking they would sync via SCIM but they did not.
So I removed the state file to trigger a full sync, but we get the error in the title.
While at first I thought it similar to #62 , we do not have any groups with more than 100 users in. The state file is also not created.
I've tried removing and re deploying the lambda to no avail.

Error: cannot sync groups and their members: error doing the first sync: error reconciling groups: error updating groups from SCIM provider: scim: error updating groups: statusCode: 400,  errCode: 400 Bad Request, errMsg: {
    "schema": [
        "urn:ietf:params:scim:api:messages:2.0:Error"
    ],
    "schemas": [
        "urn:ietf:params:scim:api:messages:2.0:Error"
    ],
    "detail": "Request is unparsable, syntactically incorrect, or violates schema.",
    "status": "400",
    "exceptionRequestId": "977d19cb-b1d1-46e7-ba4d-936701dd6d3b",
    "timeStamp": "2023-01-03 11:50:22.856"
}

provider.GetGroupsMembers and scim.GetGroupsMembersBruteForce doesn't add the hashCode when the field "Resources" is empty

Is your feature request related to a problem? Please describe.

Just wrapped up most of the setup - things are working well so far with the manual test... thank you!

Though - I just realized that the current implementation does not support SAML user attributes, and I was really hoping to take advantage of AWS's relatively new (Nov 2020) attribute-based access control ABAC: https://aws.amazon.com/blogs/aws/new-attributes-based-access-control-with-aws-single-sign-on/

Describe the solution you'd like

Modify code to also retrieve any custom attributes that have been mapped in the Google Workspace SAML app, and include those in the state file and diff when determining what to send to the AWS SCIM endpoint.

Additional context
Add any other context or screenshots about the feature request here.

Creating custom attributes in the Workspace user settings: https://admin.google.com/ac/customschema:

Mapping custom attributes in the AWS SSO SAML app: https://admin.google.com/ac/apps/saml/XXXXXXXX/attrmapping

AWS SSO attribute-based access mappings: https://us-west-2.console.aws.amazon.com/singlesignon/identity/home?region=us-west-2#!/settings?tab=abac

Hi team,

I want to sync some another attribute of user in GG Workspace like department to AWS SSO, can you please show me how to do this, is there any needed configuration on idp function?

Thanks,

Describe the bug
Only 200 users of 263 members of a google group are sync'd
All appropriate users are added to Identity Center user base, but adding the members to groups seems to be where the problem lies.

To Reproduce
Sync a group containing more than 200 members. You will see two "batches" of 100 users each are retrieved. Syncing stops after the second batch with no error. If you add a single member to the large group, the behavior is to add the new user if it falls under the 200 limit, and a single user is then removed from the group if they fall over the 200 limit.

Expected behavior
all users in a group are added

Additional context
I have not found any quota limits in either the AWS SCIM, AWS or Google that would prevent this. I also do not know why the "3rd" page of users is not coming in, i.e. nextpage token not present, or hard carded value in the source). Execution traces did show that group members were done in batch size of 100 per request.

Thanks in advance

Describe the bug

A Google Workspace can have multiple Gmail domains, e.g. @foo.com and @bar.com. If a user's email domain is changed after previously synced to AWS SSO via SCIM, then a subsequent SSO sync attempts to create a new user leads to an error.

WARN[0012] creating users                                quantity=3
WARN[0012] aws checkHTTPResponse: body: {"schema":["urn:ietf:params:scim:api:messages:2.0:Error"],"schemas":["urn:ietf:params:scim:api:messages:2.0:Error"],"detail":"Refused to create a new, duplicate resource.","status":"409","exceptionRequestId":"1f0de3fa-8bce-49ed-ae22-8604172fcb80","timeStamp":"2022-03-22 01:39:54.152"}  method=POST status="409 Conflict"
Error: cannot sync groups and their members: error syncing state: error reconciling users: error creating users in SCIM Provider: scim: error creating user: aws CreateUser: error decoding response body: EOF

Error: cannot sync groups and their members: error syncing state: error reconciling users: error creating users in SCIM Provider: scim: error creating user: aws CreateUser: error decoding response body: EOF

Afterward, if I re-run the SSO sync script, a new error appears. This time, the program attempts to duplicate the group (not the user):

INFO[0011] provider groups and state groups are different 
INFO[0011] reconciling groups                            idp=4 state=3
WARN[0011] creating groups                               quantity=1
WARN[0011] creating group                                group="AWS SSO - Shared Sandbox - Full Admin"
WARN[0011] aws checkHTTPResponse: body: {"schema":["urn:ietf:params:scim:api:messages:2.0:Error"],"schemas":["urn:ietf:params:scim:api:messages:2.0:Error"],"detail":"Refused to create a new, duplicate resource.","status":"409","exceptionRequestId":"80922f78-7149-4cf6-bf7c-4748047072dd","timeStamp":"2022-03-22 01:40:31.832"}  method=POST status="409 Conflict"
Error: cannot sync groups and their members: error syncing state: error reconciling groups: error creating groups in SCIM Provider: scim: error creating group: aws CreateGroup: error decoding response body: %!w(<nil>), body: 

This is actually strange, as nothing about the group has changed other than the email of a member. Maybe that's all that's needed?

To Reproduce

Steps to reproduce the behavior:

1. [Set up Google Workspace with two Gmail domains]
2. Create user for Mat, e.g. [email protected]
3. Create any group, e.g. aws-sso-test and add Mat to the group
4. Sync group to AWS SSO
5. Change Mat's gmail / primary email domain from [email protected] to [email protected]
6. Attempt to sync again
7. Error
8. Delete group and user from AWS SSO
9. Run sync again, this time everything worked!

Hey 👋 Firstly thank you for this open source project! I'm looking to move over from ssosync. I am using the CloudFormation template to deploy as I am managing the provision of this solution via the LZA project.

For GWSGroupsFilter, I would like to provide a list of specific groups. I already have Groups set up in my IDP and it doesn't make sense to duplicate them just to have a prefix (e.g. cloud- or aws-) as then I'd be managing members in two places. So instead, I'd want to provide exact group email matches, e.g. [email protected], [email protected]..

I have a couple questions:

1. Is it possible to provide multiple filters via the CloudFormation template? I had a peek through the repo and couldn't see examples of doing this, I see the value passed as a single string to IDPSCIM_GWS_GROUPS_FILTER.
2. How does the solution behave if you have nested groups within Google? e.g. [email protected] has 5 members, 4 of them are Users, and 1 is another group called [email protected], which has other members within it.

Hi, awesome tool, thanks for making this.

Is your feature request related to a problem? Please describe.

The initial run creates the State file at the end of its process - unfortunately if it panics before getting to the end during that first run you'll wind up with no State file inside of s3. Then if you run it again you'll just get duplicate errors.

Unfortunately I have 100+ users in AWS now that failed to get put into groups so unless I wipe all of them it'll never be able to perform a sync in the future as it's got no clue on state so attempting to duplicate users.

Describe the solution you'd like
Some quick ideas I had, which may be completely off:

1. An initial empty state file is created before running the sync - so reverting to empty state would be fine
2. Update state file throughout process and in the event it panics you'd only need to modify the last failed entry
3. Output the state to stdout or preferably a file in the directory

Additional context

I'm guessing this happened because I pulled in 100 users. Only 9 groups, but it panicked right after the very first user was added to a group. I've attached the panic in case you were curious.

`➜ build git:(main) ✗ ./idpscim --config-file .idpscim.yaml
using config file: /Users/mikej/workspace/rxrevu/idp-scim-sync/build/.idpscim.yaml
WARN[0004] no state file found in the state repository, creating this
WARN[0004] syncing from scim service, first time syncing
WARN[0004] reconciling the SCIM data with the Identity Provider data
WARN[0004] creating groups quantity=9
WARN[0004] creating group group=aa-client-success
WARN[0004] creating group group=aa-devs
WARN[0005] creating group group=aa-ops
....
WARN[0006] creating users quantity=100
WARN[0148] joining users to groups quantity=3
WARN[0148] adding member to group email=[email protected] group=aa-client-success
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
github.com/slashdevops/idp-scim-sync/pkg/aws.(*SCIMService).GetUserByUserName(0xc0003dc040, {0x1caf0b0, 0xc0000b8000}, {0xc0005535a0, 0x10000c0006a69a8})
/Users/mikej/workspace/corp/idp-scim-sync/pkg/aws/scim.go:303 +0x99d
github.com/slashdevops/idp-scim-sync/internal/scim.(*Provider).CreateGroupsMembers(0xc0003cea10, {0x1caf0b0, 0xc0000b8000}, 0xc0002e29f0)
/Users/mikej/workspace/corp/idp-scim-sync/internal/scim/scim.go:413 +0x524
github.com/slashdevops/idp-scim-sync/internal/core.reconcilingGroupsMembers({0x1caf0b0, 0xc0000b8000}, {0x1cc4750, 0xc0003cea10}, 0xc0002e29f0, 0xc0002e3260)
/Users/mikej/workspace/corp/idp-scim-sync/internal/core/reconciling.go:161 +0x18d
github.com/slashdevops/idp-scim-sync/internal/core.scimSync({0x1caf0b0, 0xc0000b8000}, {0x1cc4750, 0xc0003cea10}, 0xc000539860, 0xc000555e90, 0xc0004b0720)
/Users/mikej/workspace/corp/idp-scim-sync/internal/core/actions.go:90 +0xb86
github.com/slashdevops/idp-scim-sync/internal/core.(*SyncService).SyncGroupsAndTheirMembers(0xc00047c900, {0x1caf0b0, 0xc0000b8000})
/Users/mikej/workspace/corp/idp-scim-sync/internal/core/sync.go:125 +0x705
github.com/slashdevops/idp-scim-sync/cmd/idpscim/cmd.syncGroups()
/Users/mikej/workspace/corp/idp-scim-sync/cmd/idpscim/cmd/root.go:317 +0x759
github.com/slashdevops/idp-scim-sync/cmd/idpscim/cmd.sync()
/Users/mikej/workspace/corp/idp-scim-sync/cmd/idpscim/cmd/root.go:229 +0xa5
github.com/slashdevops/idp-scim-sync/cmd/idpscim/cmd.glob..func1(0x22acda0, {0x1b3942b, 0x2, 0x2})
/Users/mikej/workspace/corp/idp-scim-sync/cmd/idpscim/cmd/root.go:43 +0x17
github.com/spf13/cobra.(*Command).execute(0x22acda0, {0xc0000b2190, 0x2, 0x2})
/Users/mikej/go/pkg/mod/github.com/spf13/[email protected]/command.go:856 +0x60e
github.com/spf13/cobra.(*Command).ExecuteC(0x22acda0)
/Users/mikej/go/pkg/mod/github.com/spf13/[email protected]/command.go:974 +0x3bc
github.com/spf13/cobra.(*Command).Execute(...)
/Users/mikej/go/pkg/mod/github.com/spf13/[email protected]/command.go:902
github.com/slashdevops/idp-scim-sync/cmd/idpscim/cmd.Execute()
/Users/mikej/workspace/corp/idp-scim-sync/cmd/idpscim/cmd/root.go:53 +0x85
main.main()
`

Describe the bug
Error deleting user that doesn't exit in the SCIM side

Error: cannot sync groups and their members: error syncing state: error reconciling users: error deleting users in SCIM Provider: scim: error deleting user: 9067641b8c-172cc7ee-9e21-4b04-b77a-b76c01bf1929, statusCode: 404,  errCode: 404 Not Found, errMsg: 

{
    "schema": [
        "urn:ietf:params:scim:api:messages:2.0:Error"
    ],
    "schemas": [
        "urn:ietf:params:scim:api:messages:2.0:Error"
    ],
    "detail": "USER does not exist.",
    "status": "404",
    "exceptionRequestId": "15a8c7b4-80bf-45da-8607-80c0a954632b",
    "timeStamp": "2022-04-14 07:58:47.009"
}

To Reproduce
Steps to reproduce the behavior:

Expected behavior
generate a warning log with the message and continue with the execution removing the user from the state

Is your feature request related to a problem? Please describe.
Support nested groups in the identity provider side (Google Workspace)

Describe the solution you'd like
It is possible using the feature of Google Workspace directory API query parameter for Method: members.list includeDerivedMembership

AWS Single Sign-On doesn't support nested groups, but the idea is to have all the users members of the first level groups (root level and the nested groups) of google be added to the groups of AWS SSO

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

Is your feature request related to a problem? Please describe.
I'm always frustrated when Google APIs decide to barf and exclude several groups on AWS SSO side.
Google APIs have failed multiple times in the last 5 weeks. When it does, it consistently returns 502 for several minutes (idp-scim-sync handles it nicely!) and when it returns, for another minute it wildly provide a subset of existent groups, and then it returns back to normal.

When the recovery happens, all AWS SSO groups are re-created, but the permissionSets on AWS SSO reference to previously existent groups, forcing us to remap the group to account with permission all over again. This process takes ~20 minutes via Terraform, since the permissionSet also creates new IAM roles that need to be correlated to EKS roles.

Describe the solution you'd like
Introduce a new flag, preventing created groups to be removed, only to be added or modified.

Describe alternatives you've considered
An alternate solution would be in the case of a large diverging number of groups (when comparing state file groups to returned Google groups), attempt to re-fetch the groups up to 3 times, and then proceed with operation.

Describe the bug
Whenever I try to sync with with a search query I get this error:
Error: cannot sync groups and their members: error doing the first sync: error getting groups from the SCIM service: scim: error listing groups: aws checkHTTPResponse: error code: 401 Unauthorized, body:

To Reproduce
Steps to reproduce the behavior:

1. Create a group called "SysAdmin" in Google Admin.
2. Add some users there.
3. Run ./idpscim [...] -q 'name:SysAdmin* email:sysadmin*' (I've tried too with name=SysAdmin and many others as this doc says https://developers.google.com/admin-sdk/directory/v1/guides/search-groups)
4. That error appears.

Expected behavior
It actually syncs that group.

Additional context
If I run ./idpscim without the query parameter it actually starts reading some emails but then occurs the same error as in issue #49 so I can't find how to sync groups.
Also my Google account has full admin permissions, don't know if I'm missing something on the service account (which has domain-wide delegation permissions applied too and they work, since removing them leads to another error).

Thanks!

Describe the bug
If group that matches given group filter has external (from outside organization) users, sync would fail as it tries to receive information of those user(s), too.

This causes error:

{
  "errorMessage": "cannot sync groups and their members: error getting users from the identity provider: idp: error getting user: 12345, email: [email protected], error: google: error getting user [email protected]: googleapi: Error 403: Not Authorized to access this resource/api, forbidden",
  "errorType": "withStack"
}

To Reproduce
Steps to reproduce the behavior:

1. Add group member from outside organization
2. Ensure that that group is going to be synced
3. Run sync
4. See error

Expected behavior
Synchronization to pass.

It could probably print out some warning about user that wasn't able to sync, but single guest shouldn't prevent whole syncronization.

Hi there,

I'm currently attempting to setup your tool using the Serverless Application Repository Cloudformation stack. I've followed your documentation and setup Automated Provisioning, Google Service Account, enabled Domain-wide delegation and provided the necessary OAuth Scopes mentioned in another Issue. Upon running the tool I'm receiving the following error:

cannot sync groups and their members: error getting groups from the identity provider: idp: error listing groups: googleapi: Error 404: Domain not found., notFound: withStack
null

I'm wondering if this is something you've encountered before, and if so is there some obvious misconfiguration involved? I'm sure I've provided the correct values for the application parameters, and there's not too much more configuration to get wrong!

Thank you :)

I'm getting the following "error decoding response body: EOF" error when log-level=trace in the GetUserByUserName method:

Error: cannot sync groups and their members: error doing the first sync: error reconciling groups members: error creating groups members in SCIM provider: scim: error getting user by email: aws GetUserByUserName: userName: [email protected], error decoding response body: EOF

Source code:

@snavarro-factorial mentioned it in this comment too.

Describe the bug
during the reconciliation of the users when I'm migration from ssosyc to this one fails with:

runtime error: index out of range [0] with length 0: boundsError[{     "path": "github.com/aws/[email protected]/lambda/errors.go",     "line": 39,     "label": "lambdaPanicResponse" },{     "path": "github.com/aws/[email protected]/lambda/function.go",     "line": 36,     "label": "(*Function).Invoke.func1" },{     "path": "runtime/panic.go",     "line": 1038,     "label": "gopanic" },{     "path": "runtime/panic.go",     "line": 90,     "label": "goPanicIndex" },{     "path": "work/idp-scim-sync/idp-scim-sync/pkg/aws/scim.go",     "line": 303,     "label": "(*SCIMService).GetUserByUserName" },{     "path": "work/idp-scim-sync/idp-scim-sync/internal/scim/scim.go",     "line": 413,     "label": "(*Provider).CreateGroupsMembers" },{     "path": "work/idp-scim-sync/idp-scim-sync/internal/core/reconciling.go",     "line": 161,     "label": "reconcilingGroupsMembers" },{     "path": "work/idp-scim-sync/idp-scim-sync/internal/core/actions.go",     "line": 90,     "label": "scimSync" },{     "path": "work/idp-scim-sync/idp-scim-sync/internal/core/sync.go",     "line": 125,     "label": "(*SyncService).SyncGroupsAndTheirMembers" },{     "path": "work/idp-scim-sync/idp-scim-sync/cmd/idpscim/cmd/root.go",     "line": 317,     "label": "syncGroups" },{     "path": "work/idp-scim-sync/idp-scim-sync/cmd/idpscim/cmd/root.go",     "line": 229,     "label": "sync" },{     "path": "work/idp-scim-sync/idp-scim-sync/cmd/idpscim/cmd/root.go",     "line": 43,     "label": "glob..func1" },{     "path": "github.com/spf13/[email protected]/command.go",     "line": 856,     "label": "(*Command).execute" },{     "path": "github.com/spf13/[email protected]/command.go",     "line": 974,     "label": "(*Command).ExecuteC" },{     "path": "github.com/spf13/[email protected]/command.go",     "line": 902,     "label": "(*Command).Execute" },{     "path": "reflect/value.go",     "line": 556,     "label": "Value.call" },{     "path": "reflect/value.go",     "line": 339,     "label": "Value.Call" },{     "path": "github.com/aws/[email protected]/lambda/handler.go",     "line": 124,     "label": "NewHandler.func1" },{     "path": "github.com/aws/[email protected]/lambda/handler.go",     "line": 24,     "label": "lambdaHandler.Invoke" },{     "path": "github.com/aws/[email protected]/lambda/function.go",     "line": 64,     "label": "(*Function).Invoke" },{     "path": "reflect/value.go",     "line": 556,     "label": "Value.call" },{     "path": "reflect/value.go",     "line": 339,     "label": "Value.Call" },{     "path": "net/rpc/server.go",     "line": 377,     "label": "(*service).call" },{     "path": "runtime/asm_amd64.s",     "line": 1581,     "label": "goexit" }] | runtime error: index out of range [0] with length 0: boundsError [{"path":"github.com/aws/[email protected]/lambda/errors.go","line":39,"label":"lambdaPanicResponse"},{"path":"github.com/aws/[email protected]/lambda/function.go","line":36,"label":"(*Function).Invoke.func1"},{"path":"runtime/panic.go","line":1038,"label":"gopanic"},{"path":"runtime/panic.go","line":90,"label":"goPanicIndex"},{"path":"work/idp-scim-sync/idp-scim-sync/pkg/aws/scim.go","line":303,"label":"(*SCIMService).GetUserByUserName"},{"path":"work/idp-scim-sync/idp-scim-sync/internal/scim/scim.go","line":413,"label":"(*Provider).CreateGroupsMembers"},{"path":"work/idp-scim-sync/idp-scim-sync/internal/core/reconciling.go","line":161,"label":"reconcilingGroupsMembers"},{"path":"work/idp-scim-sync/idp-scim-sync/internal/core/actions.go","line":90,"label":"scimSync"},{"path":"work/idp-scim-sync/idp-scim-sync/internal/core/sync.go","line":125,"label":"(*SyncService).SyncGroupsAndTheirMembers"},{"path":"work/idp-scim-sync/idp-scim-sync/cmd/idpscim/cmd/root.go","line":317,"label":"syncGroups"},{"path":"work/idp-scim-sync/idp-scim-sync/cmd/idpscim/cmd/root.go","line":229,"label":"sync"},{"path":"work/idp-scim-sync/idp-scim-sync/cmd/idpscim/cmd/root.go","line":43,"label":"glob..func1"},{"path":"github.com/spf13/[email protected]/command.go","line":856,"label":"(*Command).execute"},{"path":"github.com/spf13/[email protected]/command.go","line":974,"label":"(*Command).ExecuteC"},{"path":"github.com/spf13/[email protected]/command.go","line":902,"label":"(*Command).Execute"},{"path":"reflect/value.go","line":556,"label":"Value.call"},{"path":"reflect/value.go","line":339,"label":"Value.Call"},{"path":"github.com/aws/[email protected]/lambda/handler.go","line":124,"label":"NewHandler.func1"},{"path":"github.com/aws/[email protected]/lambda/handler.go","line":24,"label":"lambdaHandler.Invoke"},{"path":"github.com/aws/[email protected]/lambda/function.go","line":64,"label":"(*Function).Invoke"},{"path":"reflect/value.go","line":556,"label":"Value.call"},{"path":"reflect/value.go","line":339,"label":"Value.Call"},{"path":"net/rpc/server.go","line":377,"label":"(*service).call"},{"path":"runtime/asm_amd64.s","line":1581,"label":"goexit"}]

To Reproduce
First sync under the migration from the ssosyc to this one

Expected behavior
The same groups and users