Question about possibility of monitoring multiple tenant domains about o365beat HOT 14 OPEN

from o365beat.

That's awesome, thanks! I've managed to copy the service and configure for the second tenant with the help of your script. The two instances run alongside each other successfully.

Looking forward to having this feature supported without having to run multiple instances.

from o365beat.

@chris-counteractive thanks for the reply,
The AWS-forked elasticsearch is still under the Apache 2.0 license, the standard Filebeat is now under the new Elastic License, and as such the standard Filebeat will not work with the AWS-forked elasticsearch. There is a Filebeat version (Filebeat -oss) https://www.elastic.co/downloads/beats/filebeat-oss still under the Apache 2.0 license that will work with the AWS-fork, but not all the standard modules are included, I will add screenshot of included modules, hence why we had to turn to o365beat as an alternative in a nutshell the compatible filebeat distro for the AWS-forked elasticsearch don't include the o365 module.

Thanks.

from o365beat.

I hadn't thought of this use-case, thanks for bringing it up, I can see why it'd be helpful. It's not currently supported, but I'll tag this as an enhancement request - until we work through the implications you can of course run multiple instances (you'll have to fiddle with the service registration(s), auto-starting, logging, etc., to make it all play nice, but it sounds like you've worked through that before).

To implement this I'll have to break out the configs for the various tenancies, but it's doable ... I'll target it for the 2.0 release, with any other breaking changes. Thanks for the suggestion!

from o365beat.

@ipninichuck I also have a similar use case except my data can all go into the same index.

May I ask how you configured multiple instances of the beat to pull data from multiple tenants?

from o365beat.

from o365beat.

Hi @ipninichuck ,

I follow the steps in your script manually and I run this:

o365beat -e -c /etc/itmx-o365beat/o365beat.yml -path.config /etc/itmx-o365beat -path.data /var/lib/itmx-o365beat -path.logs /var/log/itmx-o365beat

but I get this ERROR:

2020-10-08T16:04:56.431Z ERROR instance/beat.go:916 Exiting: non-200 status during api request.
newly enabled or newly subscribed feeds can take 12 hours or more to provide data.
confirm audit log searching is enabled for the target tenancy (https://docs.microsoft.com/en-us/microsoft-365/compliance/turn-audit-log-search-on-or-off#turn-on-audit-log-search).
req: &{POST https://manage.office.com/api/v1.0/47d8db9b-4dd3-4ab1-8dea-4892453bf581/activity/feed/subscriptions/start?PublisherIdentifier=47d8db9b-4dd3-4ab1-8dea-4892453bf581&contentType=Audit.AzureActiveDirectory HTTP/1.1 1 1 map[Authorization:[Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6ImtnMkxZczJUMENUaklmajRydDZKSXluZW4zOCIsImtpZCI6ImtnMkxZczJUMENUaklmajRydDZKSXluZW4zOCJ9.eyJhdWQiOiJodHRwczovL21hbmFnZS5vZmZpY2UuY29tIiwiaXNzIjoiaHR0cHM6Ly9zdHMud2luZG93cy5uZXQvNDdkOGRiOWItNGRkMy00YWIxLThkZWEtNDg5MjQ1M2JmNTgxLyIsImlhdCI6MTYwMjE3Mjc5NSwibmJmIjoxNjAyMTcyNzk1LCJleHAiOjE2MDIxNzY2OTUsImFpbyI6IkUyUmdZTGl1TCtZajgyMzE3MEQrRXp2eTVRL3lBUUE9IiwiYXBwaWQiOiJmMjg4YzQ4Ni1hMjg3LTRkZTAtYWVkYy0yODk0M2EwZmI0NzAiLCJhcHBpZGFjciI6IjEiLCJpZHAiOiJodHRwczovL3N0cy53aW5kb3dzLm5ldC80N2Q4ZGI5Yi00ZGQzLTRhYjEtOGRlYS00ODkyNDUzYmY1ODEvIiwib2lkIjoiZjhjYWMxODEtMmI4My00NTE2LWI0ODQtNjExYjY0MTYyNDNhIiwicmgiOiIwLkFBQUFtOXZZUjlOTnNVcU42a2lTUlR2MWdZYkVpUEtIb3VCTnJ0d29sRG9QdEhCSEFBQS4iLCJyb2xlcyI6WyJBY3Rpdml0eUZlZWQuUmVhZERscCIsIlNlcnZpY2VIZWFsdGguUmVhZCJdLCJzdWIiOiJmOGNhYzE4MS0yYjgzLTQ1MTYtYjQ4NC02MTFiNjQxNjI0M2EiLCJ0aWQiOiI0N2Q4ZGI5Yi00ZGQzLTRhYjEtOGRlYS00ODkyNDUzYmY1ODEiLCJ1dGkiOiJfRk9BNVUtSlJVMnBYNXdlQzdKeUFBIiwidmVyIjoiMS4wIn0.bWBoyHzIiLC_g-wdNABq9Y3VfZNWaajqNsiPKinq7tlGjDSFrJUtncN5FhF204gMJyLizww8kZ09URpH7fh2vhvQqbsJj7XYB-aqxSGnsHvcexfWYgH7ENP_w4B-UK2jB6whkg0jmE8XgPwHjZU061kGRFUJhTcAhd9jR9wJMapbISts-SzkTJzyzkfNXocoChHxp_z51q6HkK1zDt2JTMAHCgS-GkioBZWxUKKJUx_ZKIo99nqhdSuF295zV1QweSDxVjn6QATUXCINX8xTrXVCsPWCx2EXusDskshAJWTh9VsZfHArsz0vJBigFNyd5otN9EHNkZd-0vochZ3IQw]] {} 0x13fae20 0 [] false manage.office.com map[] map[] map[] }
res: &{401 Unauthorized 401 HTTP/2.0 2 0 map[Cache-Control:[no-cache] Content-Length:[124] Content-Type:[application/json; charset=utf-8] Date:[Thu, 08 Oct 2020 16:04:56 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-IIS/10.0] Www-Authenticate:[Bearer] X-Aspnet-Version:[4.0.30319] X-Powered-By:[ASP.NET]] 0xc0001fe120 124 [] false false map[] 0xc0004ec700 0xc0000c28f0}
{"error":{"code":"AF10001","message":"The permission set () sent in the request does not include the expected permission."}}
Exiting: non-200 status during api request.
newly enabled or newly subscribed feeds can take 12 hours or more to provide data.
confirm audit log searching is enabled for the target tenancy (https://docs.microsoft.com/en-us/microsoft-365/compliance/turn-audit-log-search-on-or-off#turn-on-audit-log-search).
req: &{POST https://manage.office.com/api/v1.0/47d8db9b-4dd3-4ab1-8dea-4892453bf581/activity/feed/subscriptions/start?PublisherIdentifier=47d8db9b-4dd3-4ab1-8dea-4892453bf581&contentType=Audit.AzureActiveDirectory HTTP/1.1 1 1 map[Authorization:[Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6ImtnMkxZczJUMENUaklmajRydDZKSXluZW4zOCIsImtpZCI6ImtnMkxZczJUMENUaklmajRydDZKSXluZW4zOCJ9.eyJhdWQiOiJodHRwczovL21hbmFnZS5vZmZpY2UuY29tIiwiaXNzIjoiaHR0cHM6Ly9zdHMud2luZG93cy5uZXQvNDdkOGRiOWItNGRkMy00YWIxLThkZWEtNDg5MjQ1M2JmNTgxLyIsImlhdCI6MTYwMjE3Mjc5NSwibmJmIjoxNjAyMTcyNzk1LCJleHAiOjE2MDIxNzY2OTUsImFpbyI6IkUyUmdZTGl1TCtZajgyMzE3MEQrRXp2eTVRL3lBUUE9IiwiYXBwaWQiOiJmMjg4YzQ4Ni1hMjg3LTRkZTAtYWVkYy0yODk0M2EwZmI0NzAiLCJhcHBpZGFjciI6IjEiLCJpZHAiOiJodHRwczovL3N0cy53aW5kb3dzLm5ldC80N2Q4ZGI5Yi00ZGQzLTRhYjEtOGRlYS00ODkyNDUzYmY1ODEvIiwib2lkIjoiZjhjYWMxODEtMmI4My00NTE2LWI0ODQtNjExYjY0MTYyNDNhIiwicmgiOiIwLkFBQUFtOXZZUjlOTnNVcU42a2lTUlR2MWdZYkVpUEtIb3VCTnJ0d29sRG9QdEhCSEFBQS4iLCJyb2xlcyI6WyJBY3Rpdml0eUZlZWQuUmVhZERscCIsIlNlcnZpY2VIZWFsdGguUmVhZCJdLCJzdWIiOiJmOGNhYzE4MS0yYjgzLTQ1MTYtYjQ4NC02MTFiNjQxNjI0M2EiLCJ0aWQiOiI0N2Q4ZGI5Yi00ZGQzLTRhYjEtOGRlYS00ODkyNDUzYmY1ODEiLCJ1dGkiOiJfRk9BNVUtSlJVMnBYNXdlQzdKeUFBIiwidmVyIjoiMS4wIn0.bWBoyHzIiLC_g-wdNABq9Y3VfZNWaajqNsiPKinq7tlGjDSFrJUtncN5FhF204gMJyLizww8kZ09URpH7fh2vhvQqbsJj7XYB-aqxSGnsHvcexfWYgH7ENP_w4B-UK2jB6whkg0jmE8XgPwHjZU061kGRFUJhTcAhd9jR9wJMapbISts-SzkTJzyzkfNXocoChHxp_z51q6HkK1zDt2JTMAHCgS-GkioBZWxUKKJUx_ZKIo99nqhdSuF295zV1QweSDxVjn6QATUXCINX8xTrXVCsPWCx2EXusDskshAJWTh9VsZfHArsz0vJBigFNyd5otN9EHNkZd-0vochZ3IQw]] {} 0x13fae20 0 [] false manage.office.com map[] map[] map[] }
res: &{401 Unauthorized 401 HTTP/2.0 2 0 map[Cache-Control:[no-cache] Content-Length:[124] Content-Type:[application/json; charset=utf-8] Date:[Thu, 08 Oct 2020 16:04:56 GMT] Expires:[-1] Pragma:[no-cache] Server:[Microsoft-IIS/10.0] Www-Authenticate:[Bearer] X-Aspnet-Version:[4.0.30319] X-Powered-By:[ASP.NET]] 0xc0001fe120 124 [] false false map[] 0xc0004ec700 0xc0000c28f0}
{"error":{"code":"AF10001","message":"The permission set () sent in the request does not include the expected permission."}}

I am not very expert at this, can you help me to understand where it stucks?

Thank you,
Sara

from o365beat.

from o365beat.

Hello, From the error message, it looks like o365 is replying that the application is not authorized. Did you complete the previous steps in the beat setup process of registering the app in Active Directory and giving it the needed permissions. I believe they are listed in the documentation on the Github page for the beat.

Hi,

thank you for your reply.

I don't have direct access to o365 management. I have to configure a log collector to receive logs from o365.
The customer gives me all required information that I put in o365beat.yml (tenant, client id, directory id, secret)

So do you think that this is not an error depending on running two instances of o365 beat?

from o365beat.

from o365beat.

Hi,

the client already provide us with tenant id, app id, directory id and secret. So I suppose that they already accomplished to your suggestion. Or am I wrong?

I ran

o365beat -e -c /etc/itmx-o365beat/o365beat.yml -path.config /etc/itmx-o365beat -path.data /var/lib/itmx-o365beat -path.logs /var/log/itmx-o365beat

but as output of

ps -ax | grep o365beat

I can't see two line as result but only one regarding the older istance.

Is that a good sign?

from o365beat.

from o365beat.

Hi @chris-counteractive,

Is the multi tenant support enhancement likely to be added in the near future?

We are currently using the o365 module within filebeat with multi tenant support but as with many others we are looking to move to AWS open distro and the o365 module is currently not included with the oss compatible version of filebeat.

o365beat will fill this gap for us but we do have quite a few tenancies that we currently monitor.

Thanks!

from o365beat.

@Vetpeet thanks for the question! Short answer: we hadn't planned to add any features to o365beat since the "official" filebeat 365 module dropped in 7.7.0. Even though the o365 module is under x-pack, I don't know that there's any restriction in filebeat that requires a paid license for any specific modules. That is, I don't think there's any reason you wouldn't be able to ship to AWS-flavored elasticsearch, right? Does filebeat complain if you're trying to send to an oss-compatible ES instance? I've honestly not tried it.

And even if it did, it's might be a more reliable workaround to use the Elastic-licensed filebeat to dump to a jsonl file that you can re-ship with unencumbered filebeat. Certainly kludgy and a bit wasteful, but workable?

If there's an angle on this that I'm not seeing I'm definitely happy to re-assess and perhaps try to get back to feature-parity, definitely not opposed - it just didn't seem to make much sense when the elastic-sponsored filebeat gets most people where they need to go.

from o365beat.