Error: Could not retrieve architectures from server / Error: We are unable to complete your request at this time about fido HOT 13 CLOSED

I tried to reproduce the request using a simple Python script on a clean IP and it works fine, but Fido from the same IP fails. My guess would be they're either checking for certain headers PowerShell sends (or doesn't send) by default, or the TLS fingerprint.

Edit: I did the wrong request, this isn't the one that's blocked.

from fido.

Just a note to add that the reason why we wanted to use Invoke-WebRequest without -UseBasicParsing was so that the JavaScript code from the page would be invoked, precisely so that, if Microsoft added shenanigans to their sessionId generation or validation, we could still fool them from issuing the initial request (from which we'd retrieve the sessionId) as a regular browser with JavaScript processing and all.

Now, looking at the JavaScript that actually generates the sessionID (from https://c.s-microsoft.com/en-gb/CMSScripts/script.jsx) it looks like there isn't much of anything fancy about it:

    function u() {
        var n = (new Date).getTime();
        return "xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx".replace(/[xy]/g, function(t) {
            var i = (n + Math.random() * 16) % 16 | 0;
            return n = Math.floor(n / 16), (t == "x" ? i : i & 3 | 8).toString(16)
        })
    }

Basically, this is just a standard version 4 UUID (variant 1) where, in case the JS random algorithm was a bit too predictable, Microsoft offset the random generation with the number of milliseconds since Epoch.

So, it doesn't look like the way Microsoft detects if a sessionId is to be allowed or not is based on how the UUID is created, but instead, there's probably some whitelisting of sessionId's, that is being enacted through web requests that are being generated (with the JS UUID) when the landing page is being loaded in the browser.

And once again, because Microsoft forcibly removed the ability to execute web page loading with JavaScript, from recent version of Windows/PowerShell, this may be tricky to work around...

from fido.

How nice of Microsoft of adding this new data in their error message (which is basically what we now get instead of the expected ISO download links):

Some users, entities and locations are banned from using this service. For this reason, leveraging anonymous or location hiding technologies when connecting to this service is not generally allowed.

This sure wasn't part of the bland error message they returned previously.
Why do I get the feeling that the entities Microsoft mentions explicitly refers to Rufus/Fido?...

from fido.

Still no solution for this but I have updated Fido to return the actual error message from the Microsoft servers, so you should now get:

Error: We are unable to complete your request at this time. Some users, entities and locations are banned from using this service. For this reason, leveraging anonymous or location hiding technologies when connecting to this service is not generally allowed. If you believe that you encountered this problem in error, please try again. If the problem persists you may contact "Microsoft Support – Contact Us" page for assistance. Refer to message code 715-123130.

I wonder if trying to contact Microsoft Support to tell them that their latest update sucks would accomplish anything...

from fido.

It seems like the error will appear even on a legitimate request made from a residential IP on the actual website if you send too many requests. They (temporarily?) ban you based on both IP and cookies. A fresh IP in incognito mode works fine.

from fido.

Yep, that's what I mentioned here as well.
I ran into it a few times in the past when altering and testing the script.

Basically, you will get this error if, for any reason, Microsoft decides it doesn't like you, and there can be many different causes as to why it doesn't like you...

from fido.

Thanks @0xallie, but it looks to me like your Python script only does the "search languages for a specific edition" query, which is pretty open and which Fido does't have a problem with, but does not do the "get download links for a specific edition and language" which is where we have the issue.

In short, this will work regardless of whether you use Fido, Perl or anything:
https://www.microsoft.com/en-US/api/controls/contentinclude/html?pageId=a8f8f489-4c7f-463a-9ca6-5cff94d8d041&host=www.microsoft.com&segments=software-download,windows11&query=&action=getskuinformationbyproductedition&sessionId=2d9b2d79-99af-4ca0-81b5-eeb773ceb930&productEditionId=2370&sdVersion=2

But this is what the Microsoft servers refuse to serve us:
https://www.microsoft.com/en-US/api/controls/contentinclude/html?pageId=6e2a1789-ef16-4f27-a296-74ef7ef5d96b&host=www.microsoft.com&segments=software-download,windows11&query=&action=GetProductDownloadLinksBySku&sessionId=2d9b2d79-99af-4ca0-81b5-eeb773ceb930&skuId=14966&language=English International&sdVersion=2

Your Perl script does the action=getskuinformationbyproductedition query. It does not do the action=GetProductDownloadLinksBySku. And the latter is the one that fails...

I do appreciate the help though!

from fido.

Ah yes, sorry, I did notice that only the second request fails but then got mixed up when trying to reproduce the request from Fido. I'll experiment more later.

from fido.

By the way, for those willing to help (and who are running Windows), feel free to open a PowerShell script and issue something like .\Fido.ps1 -win 11 -Verbose. This will provide the details of the URLs being queried, along with the errors returned by Microsoft:

PS D:\Projects\Fido> .\Fido.ps1 -win 11 -Verbose
No release specified (-Rel). Defaulting to '22H2 v1 (Build 22621.525 - 2022.10)'.
No edition specified (-Ed). Defaulting to 'Windows 11 Home/Pro/Edu'.
Querying https://www.microsoft.com/en-US/api/controls/contentinclude/html?pageId=a8f8f489-4c7f-463a-9ca6-5cff94d8d041&host=www.microsoft.com&segments=software-download,windows11&query=&action=getskuinformationbyproductedition&sessionId=2d9b2d79-99af-4ca0-81b5-eeb773ceb930&productEditionId=2370&sdVersion=2
No language specified (-Lang). Defaulting to 'English International'.
Querying https://www.microsoft.com/en-US/api/controls/contentinclude/html?pageId=6e2a1789-ef16-4f27-a296-74ef7ef5d96b&host=www.microsoft.com&segments=software-download,windows11&query=&action=GetProductDownloadLinksBySku&sessionId=2d9b2d79-99af-4ca0-81b5-eeb773ceb930&skuId=14966&language=English International&sdVersion=2
Error: We are unable to complete your request at this time. Some users, entities and locations are banned from using this service. For this reason, leveraging anonymous or location hiding technologies when connecting to this service is not generally allowed. If you believe that you encountered this problem in error, please try again. If the problem persists you may contact "Microsoft Support – Contact Us" page for assistance. Refer to message code 715-123130.

From there you can try to poke the last query and see if you can figure out how to bypass the Microsoft restrictions.
Or you can also go to https://www.microsoft.com/en-US/software-download/windows11, and select Inspect in a smart browser (Firefox, Chrome) while generating the ISO download link. You will see pretty much the same queries as the ones highlighted above. Then you can try to re-issue the last query as POST, which should produce the error, since, by default, it'll be missing whatever "special sauce" Microsoft wants, and from there you can try to fiddle with the query parameters, such as the headers and stuff.

from fido.

Some testing in Firefox sending manually crafted POST requests suggests that Microsoft are now validating the sessionId being used is one that was generated by their server, and not a random UUID, as the one we currently use.

In short, if I run a test crafting manual requests and using the UUID I got when loading the original page, I can get both requests answered successfully. But if I run the same test, and alter the UUID by a single digit, the first query works but the second query fails with the error above.

The thing is, I had originally devised the script to read the sessionId from the Microsoft page, but this came with the downside of not being able to use -UseBasicParsing when issuing Invoke-WebRequest, which means that Internet Explorer had to be installed for that to work. Therefore, when someone suggested that we could just use a random UUID, I jumped on doing so in 5f246fc.

Unfortunately, reading the Microsoft sessionId back again won't be as easy a reverting that commit because, since then, and especially on newer versions of Windows, Microsoft BROKE Invoke-WebRequest without -UseBasicParsing, by removing the IE engine altogether (And you can't use Edge for that. At all!), and has not provided any workaround... So this might be yet another PITA to work around.

And of course, this may not the the only counter-measure that Microsoft has taken against third-party download scripts.

from fido.

Further confirmed that simply browsing to https://www.microsoft.com/en-US/software-download/windows11 (without downloading anything) and using the sessionId generated from the landing page into the Fido script makes the script work. So, yeah, Microsoft definitely added sessionId whitelisting to their ISO retrieval queries.

from fido.

Okay, I believe I have a working workaround now (but of course with the question: How long before Microsoft strengthen things further, since this is the second time we're going through this dance). Basically, the session Id whitelisting appears to be accomplished by poking https://vlscppe.microsoft.com/tags with the session Id, so I've added an extra call that does just that.

From my testing, this seems to do the trick, but of course we'll have to see how well it fares in the long run, because it certainly looks like Microsoft are actively trying to prevent scripts like Fido from interacting with their servers...

from fido.

I got this error as well. Ended up signing in to Microsoft Insiders then the ISO download worked fine.

from fido.