Comments (13)
I tried to reproduce the request using a simple Python script on a clean IP and it works fine, but Fido from the same IP fails. My guess would be they're either checking for certain headers PowerShell sends (or doesn't send) by default, or the TLS fingerprint.
Edit: I did the wrong request, this isn't the one that's blocked.
from fido.
Just a note to add that the reason why we wanted to use Invoke-WebRequest
without -UseBasicParsing
was so that the JavaScript code from the page would be invoked, precisely so that, if Microsoft added shenanigans to their sessionId
generation or validation, we could still fool them from issuing the initial request (from which we'd retrieve the sessionId
) as a regular browser with JavaScript processing and all.
Now, looking at the JavaScript that actually generates the sessionID
(from https://c.s-microsoft.com/en-gb/CMSScripts/script.jsx) it looks like there isn't much of anything fancy about it:
function u() {
var n = (new Date).getTime();
return "xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx".replace(/[xy]/g, function(t) {
var i = (n + Math.random() * 16) % 16 | 0;
return n = Math.floor(n / 16), (t == "x" ? i : i & 3 | 8).toString(16)
})
}
Basically, this is just a standard version 4 UUID (variant 1) where, in case the JS random algorithm was a bit too predictable, Microsoft offset the random generation with the number of milliseconds since Epoch.
So, it doesn't look like the way Microsoft detects if a sessionId
is to be allowed or not is based on how the UUID is created, but instead, there's probably some whitelisting of sessionId
's, that is being enacted through web requests that are being generated (with the JS UUID) when the landing page is being loaded in the browser.
And once again, because Microsoft forcibly removed the ability to execute web page loading with JavaScript, from recent version of Windows/PowerShell, this may be tricky to work around...
from fido.
How nice of Microsoft of adding this new data in their error message (which is basically what we now get instead of the expected ISO download links):
Some users, entities and locations are banned from using this service. For this reason, leveraging anonymous or location hiding technologies when connecting to this service is not generally allowed.
This sure wasn't part of the bland error message they returned previously.
Why do I get the feeling that the entities Microsoft mentions explicitly refers to Rufus/Fido?...
from fido.
Still no solution for this but I have updated Fido to return the actual error message from the Microsoft servers, so you should now get:
Error: We are unable to complete your request at this time. Some users, entities and locations are banned from using this service. For this reason, leveraging anonymous or location hiding technologies when connecting to this service is not generally allowed. If you believe that you encountered this problem in error, please try again. If the problem persists you may contact "Microsoft Support – Contact Us" page for assistance. Refer to message code 715-123130.
I wonder if trying to contact Microsoft Support to tell them that their latest update sucks would accomplish anything...
from fido.
It seems like the error will appear even on a legitimate request made from a residential IP on the actual website if you send too many requests. They (temporarily?) ban you based on both IP and cookies. A fresh IP in incognito mode works fine.
from fido.
Yep, that's what I mentioned here as well.
I ran into it a few times in the past when altering and testing the script.
Basically, you will get this error if, for any reason, Microsoft decides it doesn't like you, and there can be many different causes as to why it doesn't like you...
from fido.
Thanks @0xallie, but it looks to me like your Python script only does the "search languages for a specific edition" query, which is pretty open and which Fido does't have a problem with, but does not do the "get download links for a specific edition and language" which is where we have the issue.
In short, this will work regardless of whether you use Fido, Perl or anything:
https://www.microsoft.com/en-US/api/controls/contentinclude/html?pageId=a8f8f489-4c7f-463a-9ca6-5cff94d8d041&host=www.microsoft.com&segments=software-download,windows11&query=&action=getskuinformationbyproductedition&sessionId=2d9b2d79-99af-4ca0-81b5-eeb773ceb930&productEditionId=2370&sdVersion=2
But this is what the Microsoft servers refuse to serve us:
https://www.microsoft.com/en-US/api/controls/contentinclude/html?pageId=6e2a1789-ef16-4f27-a296-74ef7ef5d96b&host=www.microsoft.com&segments=software-download,windows11&query=&action=GetProductDownloadLinksBySku&sessionId=2d9b2d79-99af-4ca0-81b5-eeb773ceb930&skuId=14966&language=English International&sdVersion=2
Your Perl script does the action=getskuinformationbyproductedition
query. It does not do the action=GetProductDownloadLinksBySku
. And the latter is the one that fails...
I do appreciate the help though!
from fido.
Ah yes, sorry, I did notice that only the second request fails but then got mixed up when trying to reproduce the request from Fido. I'll experiment more later.
from fido.
By the way, for those willing to help (and who are running Windows), feel free to open a PowerShell script and issue something like .\Fido.ps1 -win 11 -Verbose
. This will provide the details of the URLs being queried, along with the errors returned by Microsoft:
PS D:\Projects\Fido> .\Fido.ps1 -win 11 -Verbose
No release specified (-Rel). Defaulting to '22H2 v1 (Build 22621.525 - 2022.10)'.
No edition specified (-Ed). Defaulting to 'Windows 11 Home/Pro/Edu'.
Querying https://www.microsoft.com/en-US/api/controls/contentinclude/html?pageId=a8f8f489-4c7f-463a-9ca6-5cff94d8d041&host=www.microsoft.com&segments=software-download,windows11&query=&action=getskuinformationbyproductedition&sessionId=2d9b2d79-99af-4ca0-81b5-eeb773ceb930&productEditionId=2370&sdVersion=2
No language specified (-Lang). Defaulting to 'English International'.
Querying https://www.microsoft.com/en-US/api/controls/contentinclude/html?pageId=6e2a1789-ef16-4f27-a296-74ef7ef5d96b&host=www.microsoft.com&segments=software-download,windows11&query=&action=GetProductDownloadLinksBySku&sessionId=2d9b2d79-99af-4ca0-81b5-eeb773ceb930&skuId=14966&language=English International&sdVersion=2
Error: We are unable to complete your request at this time. Some users, entities and locations are banned from using this service. For this reason, leveraging anonymous or location hiding technologies when connecting to this service is not generally allowed. If you believe that you encountered this problem in error, please try again. If the problem persists you may contact "Microsoft Support – Contact Us" page for assistance. Refer to message code 715-123130.
From there you can try to poke the last query and see if you can figure out how to bypass the Microsoft restrictions.
Or you can also go to https://www.microsoft.com/en-US/software-download/windows11, and select Inspect in a smart browser (Firefox, Chrome) while generating the ISO download link. You will see pretty much the same queries as the ones highlighted above. Then you can try to re-issue the last query as POST
, which should produce the error, since, by default, it'll be missing whatever "special sauce" Microsoft wants, and from there you can try to fiddle with the query parameters, such as the headers and stuff.
from fido.
Some testing in Firefox sending manually crafted POST
requests suggests that Microsoft are now validating the sessionId
being used is one that was generated by their server, and not a random UUID, as the one we currently use.
In short, if I run a test crafting manual requests and using the UUID I got when loading the original page, I can get both requests answered successfully. But if I run the same test, and alter the UUID by a single digit, the first query works but the second query fails with the error above.
The thing is, I had originally devised the script to read the sessionId
from the Microsoft page, but this came with the downside of not being able to use -UseBasicParsing
when issuing Invoke-WebRequest
, which means that Internet Explorer had to be installed for that to work. Therefore, when someone suggested that we could just use a random UUID, I jumped on doing so in 5f246fc.
Unfortunately, reading the Microsoft sessionId
back again won't be as easy a reverting that commit because, since then, and especially on newer versions of Windows, Microsoft BROKE Invoke-WebRequest
without -UseBasicParsing
, by removing the IE engine altogether (And you can't use Edge for that. At all!), and has not provided any workaround... So this might be yet another PITA to work around.
And of course, this may not the the only counter-measure that Microsoft has taken against third-party download scripts.
from fido.
Further confirmed that simply browsing to https://www.microsoft.com/en-US/software-download/windows11 (without downloading anything) and using the sessionId
generated from the landing page into the Fido script makes the script work. So, yeah, Microsoft definitely added sessionId
whitelisting to their ISO retrieval queries.
from fido.
Okay, I believe I have a working workaround now (but of course with the question: How long before Microsoft strengthen things further, since this is the second time we're going through this dance). Basically, the session Id whitelisting appears to be accomplished by poking https://vlscppe.microsoft.com/tags with the session Id, so I've added an extra call that does just that.
From my testing, this seems to do the trick, but of course we'll have to see how well it fares in the long run, because it certainly looks like Microsoft are actively trying to prevent scripts like Fido from interacting with their servers...
from fido.
I got this error as well. Ended up signing in to Microsoft Insiders then the ISO download worked fine.
from fido.
Related Issues (20)
- Add Windows 11 22H2 v2 HOT 4
- Windows 10 22H2 download is broken HOT 1
- Fido Ported to Unix HOT 4
- Only latest release HOT 1
- Improved README.md - Mention Domain for ISO Download HOT 1
- Update Windows productEditionId for 23H2 HOT 3
- Error: The underlying connection was closed: An unexpected error occurred on a send. HOT 2
- Error while trying to download a Windows 10 ISO HOT 1
- One more reason for someone to avoid mct HOT 7
- Error: Download blocked (from Microsoft?) HOT 1
- Microsoft Error HOT 1
- error: we are unable to complete your request at this time HOT 1
- Windows 11 23H2 v2 released HOT 6
- Not downloading or getting URL HOT 3
- Add support for downloading Media Creation Tool ESDs HOT 1
- Doesn't pull older release than 22h2 HOT 1
- Fido is stuck HOT 2
- Operation timed out HOT 3
- version 1.55 appears broken HOT 1
- Error: Could not parse languages HOT 11
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fido.