Command Line Is Too Long about ari HOT 17 OPEN

Running into similar issues when trying to run the script against the tenant with over 700 subscriptions: "The command line is too long" and no resources. When using -debug switch it seems to fail on line 1187 because of null values. Script does work when scoping to a single subscription, but would really need this to work at tenant level. Have also tried used the -HeavyLoad switch, but still get same error.

from ari.

I actually cant even get the individual subscription scope to work either. The script starts running and does make process but it eventually stops making progress and starts consuming 99% on my PC memory (32GB) and then I just have to force close powershell.

from ari.

Hello there @brian-taylor-ahead and @DVPXOMKOS

It's not really a limit issue I believe. Apparently what is happening is that we are hitting a processing limit in the computer due to the large number of resources in the environment.

So I'm guessing the first parts of the script are running without any issue, but then it reaches the processing jobs phase and the computer freezes?

In the past we created the processing part of the script to be as fast as possible, so we split all the resources in chunks of 1000 and create a thread in each chunk for each module of the script.

-HeavyLoad makes these chunks larger (2500), making the script take more time to complete but not freezing due to the large number of threads, but apparently your environment is even larger for these chunks.

So I just updated the script to change the chunks in the HeavyLoad to be 7500. Let's see if that amount address the issue.

If it doesn't solve the issue than we increase the chunks even more.

Can you guys try the version 3.1.17 using the -HeavyLoad parameter and tell me how it goes? If possible run the script also using the -Debug parameter.

Thanks

from ari.

@Claudio-Merola I'm currently testing this to see if it works without freezing my PC. I'll update this comment with an edit if it does work.

However, this still isn't an ideal scenario for me. I'd like to have this run against all 500+ subscriptions in my environment. When I try to do that, and @DVPXOMKOS seems to experience this as well, I get the command too long error. So currently the script will not run at all unless I specify a subscription. I'll run it with a debug and see what results I get and post them here.

from ari.

@brian-taylor-ahead I understand, but currently I need to know where exactly that error is coming from, that is why I also need you to use the -Debug parameter, so we can have a better understanding of what is going on with that error.

from ari.

Hi @Claudio-Merola . I tried the updated script with the 7500 Heavyload, but seems to still fail for me when starting the Jobs Collector.

We have a smaller tenant of around 70 subs and under 10,000 resources and I'm able to run the inventory tool fine there, but can't seem to get it to work in our much larger tenant.

from ari.

Hi @DVPXOMKOS, apparently this is an older version of the script, can you confirm this is the version 3.1.18?

from ari.

@Claudio-Merola I'm attaching the debug output. I couldn't get it to format correctly when pasting it in to this comment section. See the attached txt file.

ARI Debug.txt

from ari.

@brian-taylor-ahead Thanks, apparently the script is failing even before extracting the resources:

DEBUG: Number of Subscriptions Found: 550
DEBUG: Checking report folder: C:\AzureResourceInventory
DEBUG: Invoking Inventory Loop for Resources
The command line is too long.
DEBUG: Invoking Inventory Loop for Network Resources
The command line is too long.

That appears to happen due the large number of subscriptions being send to the az graph query command.

I workaround it by splitting the subscriptions in chunks of 200 (I validated the command line can handle that much)

Can you try the version 3.1.20 that I just pushed to github using the -Debug parameter? And in case the script fails again post the debugging here.

Earlier today I also made improvements in the processing parts of the script, you no longer need to specify the -Heavyload...

Just run the script normally and it will automatically identify large environments and split the load.

from ari.

@Claudio-Merola Thanks a ton for working on this. With the recent update, I was able to get it to run without the "command line too long" error. However, the script still froze up before completing. RAM usage spiked to 24 GB and it looked like it would no longer make any progress so I stopped the script. Attaching the debug. Removed a ton of blank space to make it easier to read.

ARI 3.1.20 debug..txt

I should also note that after you made changes to the -HeavyLoad flag last time, I still was not able to get a successful run. The script did make more progress (failed at 41% instead of 24%) but ultimately still had to kill it after it spiked memory usage to max for multiple hours.

from ari.

@brian-taylor-ahead Cool. I see the script was able to extract the resources this time and split the load in 4 jobs for processing, the way it was configured was to split the load in chunks of 40k resources, perhaps that's too much for powershell to handle in a single process. I changed the script to split the load in chunks of 20k resources now.

I expect it will split it in 8 jobs of 20k, I believe it will consume some CPU but it should not overload the memory this time.

Can you give a try with the version 3.1.22 ?

from ari.

@Claudio-Merola No luck it seems. Looks like the same issue as before, with running jobs not completing and memory spiking to near max. I'm wondering if there's just an issue won my side but I'm running pretty good specs. Ryzen 9 5900x with 32 GB of RAM. Here's the most recent debug:

ARI 3.1.22 debug.txt

from ari.

@brian-taylor-ahead Got it. I have seen Powershell having problems when trying to handle more than 3gb per process. I reduced the number to 10k resources per job and made it run the jobs in batches of 5. That might increase the time for the script to run in large environments but I believe the performance impact should be minimal.

Can you give a try with the version 3.1.24 and send us the result?

from ari.

@Claudio-Merola That was definitely the best run I've had so far but it did still end in the same way. The RAM usage continues to grow in small increments until maxing out. As you can see in the debug, the number of jobs running does go down to 1. The RAM usage peaked at about 8GB while running all 5 jobs, but as the jobs started decreasing, it continued to trend upward until maxing at around 24GB (which puts me at 100% usage once you factor in all my other processes). Are we looking at a memory leak issue? I was able to find this article from some years ago: https://www.jhouseconsulting.com/2017/09/25/addressing-the-powershell-garbage-collection-bug-1825. This article specifically talks about this issue affecting loops. CPU usage did spike during this process (to around 80%) but calmed down as jobs finished. Let me know what you think.

As always, here is the debug file:
ARI 3.1.24 debug.txt

from ari.

@brian-taylor-ahead Each one of those jobs is a different Powershell Process. When you say it was consuming 24GB of RAM, it was one single powershell process?

I don't think it's a memory leak, I believe is more a memory usage issue. Due to the amount of resources. But even so I changed the Garbage Collector according to the article you mentioned, it does make sense, but we are not running the garbage collector in a pipe.

I also reduced the chunk again to 5k now.

Can you try with the version 3.1.26 and tell me how it goes?

I also recommend you to run the script with the following parameters to avoid more memory usage by the script:

-SkipPolicy -SkipAdvisory -SkipDiagram

Thanks

from ari.

@Claudio-Merola SUCCESS! The script still spiked my memory all the way to 24GB or so. However, I just let it run and it didnt freeze this time and did eventually complete after a little over 80 minutes. I watched task manager a little more closely and I did see some weirdness there. There were multiple Powershell processes running. One of the threads was using 10GB of memory by itself. The rest looked fine. That one thread seemed to be using 0% CPU the majority of the time. It would occasionally consume a percentage or two of CPU and the memory would slowly increase during that time, but then the CPU would drop to zero and the memory usage would hold steady until the CPU usage started again, at which point it would slowly climb until CPU usage stopped again.

Ultimately, this did end up succeeding for me but I'm sure there is still something weird going on here. In the future, I may just run this is cloud shell exclusively (in this case, I was unable to get the access I needed for that in a timely manner). If you are up for it though, I'm happy to hop on a call with you and let you watch the script run for a bit so you can see this behavior in action.

As always, here's the debug for this most recent run:
ARI 3.1.26 debug.txt

from ari.

@brian-taylor-ahead, Awesome! Glad to hear we got another step!

But I can see in the log that many errors occurred inside the processes. I'm assuming not all the resources in the environment were correctly processed.

I'm investigating a similar case in another environment and I'm starting to suspect the issue is that you might have resources in the environment with too many details in the graph properties (resources with too many information for the script to process correctly).

Can you run the following query in the "Resource Graph Explorer" at the Azure Portal and see if many resources are returned?

resources | where strlen(properties) > 123000 | summarize count() by ['type'],subscriptionId

By the way, I made some extra improvements in the script. So if you want to test the script again against that environment, can you try the version 3.1.27 ?

from ari.