microsoft / kiota-http-python Goto Github PK

As a user, when creating a client using default middleware, I should be able to customize middleware behavior by passing custom middleware options.

Follow up to #258
(we should really inspect all middleware handlers)

This means if multiple requests need to retry within the lifetime of the client/request adapter/handler, the first requests will retry successfully but the following ones won't. (same issue as redirect handler)

related microsoft/kiota#598
Specification

Environment

• Python Version: 3.7.4
• kiota-http version: 0.6.3
• OS: W10 x64

Stack trace (if available)

Describe the bug
from msgraph import GraphServiceClient does not work. This gives me a syntax error.

To Reproduce

1. install msgraph-sdk: pip install msgraph-sdk (microsoft-kiota-http is included in it)
2. Create a python file with the import below : from msgraph import GraphServiceClient
3. Run the script

Expected behavior
That there is no error when importing

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

related microsoftgraph/msgraph-sdk-design#106 retry_count attribute needs to be udpated and retry_delay attribute needs to be implemented

To be spec complete this library is missing a Chaos middleware.
https://github.com/microsoftgraph/msgraph-sdk-design/

related to microsoft/kiota#4025
add a 3rd case in the error handling of the request adapter for XXX (must be the 3rd in the order, first sepcific code, then 4XX or 5XX depending on the first number, then XXX)

related microsoft/kiota#1063. Graph default scope and default valid host names are present in the authentication library, which makes it more complex to use it against non-graph but MIP protected APIs.

• check that allowed host validator returns true when no hosts are provided (abstractions)
• change the default scope to be set in the get access token method to scheme://host/.default if no scopes have been provided
• remove the default hosts in the authentication library
• create derived authentication and access providers in microsoft graph core that set the default hosts
• update the documentation

Follow up of microsoft/kiota#1699 for Python.
No generation changes required (hence the issue being created over here). Only the runtime rules need to be implemented/validated. (see the linked PRs for examples).

related microsoft/kiota#2056, add kiota-lang/http-lib-version as a product to the user agent request header to help with telemetry. The implementation should not overwrite existing values, just add to it. If possible read the version from the package information at runtime, otherwise read a constant.

When using a proxy e.g

 proxies = {
     "http://": "http://localhost:8030",
     "https://": "http://localhost:8031",
}
custom_client = httpx.AsyncClient(proxies=proxies) 

The client uses the default httpx.AsyncHTTPTransport instead of the AsycKiotaTransport. This is due to the internal implementation of how to create clients with a proxy and results in our middleware pipeline not applying

follow up to microsoft/kiota#4367
and microsoft/kiota#4190

Implement a unit test with a 304 response, no location header, check the request adapter returns null and doesn't throw.

to maintain compatibility with external tooling we should make sure our attribute respect the latest (stable) semantic conventions. When we initially implemented things, the spec was still in draft. The following attributes need to be renamed in the middleware handlers and in the request adapter:
htttp.status_code => http.response.status_code
http.response_content_length => http.response.body.size
http.request_content_length => http.request.body.size
http.flavor => network.protocol.version
http.response_content_type => http.response.header.content-type
http.request_content_type => http.request.header.content-type
http.method => http.request.method
http.host => url.scheme
http.scheme => server.address
http.uri => url.full
http.uri_template => url.uri_template

link to the spec for more information https://opentelemetry.io/docs/specs/semconv/attributes-registry/http/
related microsoftgraph/msgraph-sdk-design#108

Using the Kiota Python generator as described in microsoft/kiota#1996, after fixing the indentation problems manually, I'm getting exceptions from the Kiota httpx_request_adapter.py when the service returns a non-success response. Here's the traceback:

>>> tests = asyncio.run(client.loadtests().sort_and_filter().get(config))
Headers({'host': '6e856190-1a06-44bd-b3eb-4b51ec508b2f.southcentralus.cnt-prod.loadtesting.azure.com', 'accept-encoding': 'gzip, deflate', 'connection': 'keep-alive', 'user-agent': 'python-httpx/0.23.1', 'accept': 'application/json', 'authorization': '[secure]', 'request_options': '{}'})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/homebrew/Cellar/[email protected]/3.11.0/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.0/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.0/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 650, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/mikekistler/Projects/mikekistler/kiota/client/loadtests/sort_and_filter/sort_and_filter_request_builder.py", line 67, in get
    return await self.request_adapter.send_async(request_info, test_model_resource_list.TestModelResourceList, response_handler, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/kiota_http/httpx_request_adapter.py", line 118, in send_async
    await self.throw_failed_responses(response, error_map)
  File "/opt/homebrew/lib/python3.11/site-packages/kiota_http/httpx_request_adapter.py", line 290, in throw_failed_responses
    in error_map) and not (400 <= status_code < 500 and error_map['4XX']
                                                        ~~~~~~~~~^^^^^^^
TypeError: 'NoneType' object is not subscriptable
>>> 

It shows that the problem is an attempt to subscript error_map which is None.

Here's the msort_and_filter().get() method that Kiota generated:

async def get(self,request_configuration: Optional[SortAndFilterRequestBuilderGetRequestConfiguration] = None, response_handler: Optional[ResponseHandler] = None) -> Optional[test_model_resource_list.TestModelResourceList]:
    request_info = self.create_get_request_information(
        request_configuration
    )
    if not self.request_adapter:
        raise Exception("Http core is null") 
    return await self.request_adapter.send_async(request_info, test_model_resource_list.TestModelResourceList, response_handler, None)

The final None in the send_async call is the error_map.

The responses for this operation in the REST API definition are:

        "responses": {
          "200": {
            "description": "The requested load tests",
            "schema": {
              "$ref": "#/definitions/TestModelResourceList"
            }
          },
          "default": {
            "description": "Load Testing service error response.",
            "schema": {
              "$ref": "#/definitions/ErrorResponseBody"
            },
            "headers": {
              "x-ms-error-code": {
                "description": "The error code for specific error that occurred.",
                "type": "string"
              }
            }
          }
        },

I suspect that the problem here is that Kiota is looking for 400 or 500 class responses and finding none, so passes an None as the error map. But Azure uses default to describe error responses.

I am following the guide on: https://github.com/microsoftgraph/msgraph-sdk-python to build a Python package that can pull all my devices from Azure AD (Entra). I have confirmed good working API credentials and that they have the right permissions:

async def get_AADdevices(): MScredential = ClientSecretCredential( tenant_id=os.environ['MSGraphTenantID'], client_id=os.environ['MSGraphClientID'], client_secret=os.environ['MSGraphClientSecret'] ) scopes = ['https://graph.microsoft.com/.default'] client = GraphServiceClient(credentials=MScredential, scopes=scopes) devices = await client.devices.get()

I am getting the following error when it calls the await client.devices.get():

Executed 'Functions.MachineReconciliation' (Failed, Id=d6a7b205-25fc-4dfa-ad22-3621ed0b4144, Duration=9ms) System.Private.CoreLib: Exception while executing function: Functions.MachineReconciliation. System.Private.CoreLib: Result: Failure Exception: ValueError: not enough values to unpack (expected 2, got 1) Stack: File "C:\Program Files\Microsoft\Azure Functions Core Tools\workers\python\3.10/WINDOWS/X64\azure_functions_worker\dispatcher.py", line 479, in _handle__invocation_request call_result = await self._loop.run_in_executor( File "C:\Users\<user>\AppData\Local\Programs\Python\Python310\lib\concurrent\futures\thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "C:\Program Files\Microsoft\Azure Functions Core Tools\workers\python\3.10/WINDOWS/X64\azure_functions_worker\dispatcher.py", line 752, in _run_sync_func return ExtensionManager.get_sync_invocation_wrapper(context, File "C:\Program Files\Microsoft\Azure Functions Core Tools\workers\python\3.10/WINDOWS/X64\azure_functions_worker\extension.py", line 215, in _raw_invocation_wrapper result = function(**args) File "C:\Users\<path>\function_app.py", line 17, in MachineReconciliation asyncio.run(get_AADdevices()) File "C:\Users\<user>\AppData\Local\Programs\Python\Python310\lib\asyncio\runners.py", line 44, in run return loop.run_until_complete(main) File "C:\Users\<user>\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py", line 649, in run_until_complete return future.result() File "C:\Users\<path>\function_app.py", line 40, in get_AADdevices devices = await client.devices.get() File "C:\Users\<path>\.venv\lib\site-packages\msgraph\generated\devices\devices_request_builder.py", line 70, in get return await self.request_adapter.send_async(request_info, DeviceCollectionResponse, error_mapping) File "C:\Users\<path>\.venv\lib\site-packages\kiota_http\httpx_request_adapter.py", line 162, in send_async parent_span = self.start_tracing_span(request_info, "send_async") File "C:\Users\<path>\.venv\lib\site-packages\kiota_http\httpx_request_adapter.py", line 131, in start_tracing_span uri_template = ParametersNameDecodingHandler.decode_uri_encoded_string( File "C:\Users\<path>\.venv\lib\site-packages\kiota_http\middleware\parameters_name_decoding_handler.py", line 92, in decode_uri_encoded_string name, value = name_value.split('=')

details in microsoft/kiota#2084

Environment

• Python Version: 3.11.5
• kiota-http version: 1.3.0
• OS: Linux (openSuse)

Stack trace (if available)

Describe the bug
The RedirectHandler class in the Microsoft Kiota HTTP Python library is currently experiencing an issue where the original response, containing the new URL, is not closed before performing a redirect. This behaviour results in httpcore async layer running out of streams, reaching a limit of 100 streams.

To Reproduce

1. Execute a request using the RedirectHandler in a scenario that triggers a redirect. I used the ms graph sdk with a drive item content.get () call to download multiple files (over 100).
2. Observe the httpcore async behaviour after a redirect, after 100, the call to content.get() blocks (if using ms graph sdk).

Expected behavior
I think the original response should be closed properly before the redirect is performed to prevent httpcore async from running out of streams.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
To "fix" the issue, I have used the following code, not sure if this is the correct solution.

            if redirect_location and current_options.should_redirect:
                max_redirect -= 1
                if not self.increment(response, max_redirect, history[:]):
                    break
                _redirect_span.set_attribute(REDIRECT_COUNT_KEY, len(history))
                new_request = self._build_redirect_request(request, response)
                request = new_request
                # make sure the response is closed
                await response.aclose()
                continue

            response.history = history
            break

When using RedirectHandler for multiple requests without specifying custom options, we encounter an issue where the max_redirect counter is shared across all requests handled by a single RedirectHandler instance. This is problematic when downloading drive items using the Microsoft Graph API, which respond with a 302 Found redirect to a preauthenticated download URL.

After 5 successive downloads, the handler raises Too many redirects exception due to the shared counter reaching its default limit, effectively preventing further downloads.

It seems like ParametersNameDecodingHandler middleware is a workaround that should only affect query parameter names, without replacing values and adding new parameters.
However, the code below is corrupting all parameter values that contain escaped meaningful URL symbols like +, =, ?, & - the latter may add extra parameter names into the URL

from urllib.parse import unquote
from httpx import URL

original = URL("https://google.com?q=1%2b2") # search for "1+2"
updated = URL(unquote(str(original)))
assert updated ==  URL('https://google.com?q=1+2')  # corrupted, now searches for "1 2" !

original = URL("https://google.com/?q=M%26A")  # search for "M&A"
updated = URL(unquote(str(original)))  # corrupted, now searches for "M", adds separate A parameter !

This was extremely hard to find when it was resulting in SDK API errors with Invalid continuation token <TOKEN> passed where actually valid TOKEN that had + in the middle and also ends with =.

Using the Python Graph SDK, I found this issue. The first request sent through the GraphServiceClient succeeds, but subsequent requests throw an exception:

Traceback (most recent call last):
  File "/home/jasonjoh/repos/msgraph-training-python/demo/graphtutorial/repro.py", line 17, in <module>
    users2 = asyncio.run(client.users().get())
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/home/jasonjoh/.local/lib/python3.10/site-packages/msgraph/generated/users/users_request_builder.py", line 129, in get
    return await self.request_adapter.send_async(request_info, user_collection_response.UserCollectionResponse, response_handler, error_mapping)
  File "/home/jasonjoh/.local/lib/python3.10/site-packages/kiota_http/httpx_request_adapter.py", line 114, in send_async
    response = await self.get_http_response_message(request_info)
  File "/home/jasonjoh/.local/lib/python3.10/site-packages/kiota_http/httpx_request_adapter.py", line 329, in get_http_response_message
    resp = await self._http_client.send(request)
  File "/home/jasonjoh/.local/lib/python3.10/site-packages/httpx/_client.py", line 1609, in send
    raise RuntimeError("Cannot send a request, as the client has been closed.")
RuntimeError: Cannot send a request, as the client has been closed.

This seems to be caused by the following line:

This script can reproduce the problem:

import asyncio
from azure.identity.aio import ClientSecretCredential
from kiota_authentication_azure.azure_identity_authentication_provider import AzureIdentityAuthenticationProvider
from msgraph import GraphRequestAdapter, GraphServiceClient

credential = ClientSecretCredential(
    'TENANT_ID',
    'CLIENT_ID',
    'CLIENT_SECRET')
auth_provider = AzureIdentityAuthenticationProvider(credential)
adapter = GraphRequestAdapter(auth_provider)
client = GraphServiceClient(adapter)

users = asyncio.run(client.users().get())
print('got users')

users2 = asyncio.run(client.users().get())
print('got users again')

https://github.com/microsoftgraph/msgraph-sdk-design/blob/master/middleware/CompressionHandler.md

specification work microsoftgraph/msgraph-sdk-design#102