[BUG] OpenAI Legacy Completion endpoint not exposed about azure-sdk-for-net HOT 5 OPEN

@joakimriedel I suppose one further workaround could be to get the source of the last 1.0.0 SDK, rename all namespaces, and push that to a new NuGet package, in order to have the legacy package available as a separate package.

All of these are hacky, however, and their need is quite disappointing.

from azure-sdk-for-net.

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @jpalvarezl @trrwilson.

from azure-sdk-for-net.

Note that this is intentionally not a feature request, because this breaks existing functionality that should still be supported.

from azure-sdk-for-net.

Thank you, @kescherCode, both for the mention here as well as on openai-dotnet. Although this was an intentional omission of the legacy completions surface on both the new client and this major version increment of the preview library, I hear you loud and clear on the migration block that imposes for /completions endpoint use. Breaking changes are very much expected, but breaking changes with no upgrade path are exceptionally painful.

We'll discuss this strategy in the product team and with OpenAI. In the interim, the prior v1.0.0-beta.17 release will continue to be a supported mechanism for using -instruct with the 2024-04-01-preview service API label even while v2.0.0-beta-* does not (at least yet) support them.

from azure-sdk-for-net.

+1 for legacy completions, it is not just for the -instruct model but fine-tunes using babbage-002/davinci-002. We have an important logprobs classification task using a fine tuned babbage-002 which blocks upgrading to 2.0.0-beta.x right now.

Edit: managed to move the code calling the legacy completions into a separate "LegacyOpenAI" project, but had to do the following since the 2.0.0-beta version is pinned centrally for Nuget for all other projects in the solution;

        <PackageReference Include="Azure.AI.OpenAI" VersionOverride="1.0.0-beta.17">
            <NoWarn>NU1605</NoWarn>
        </PackageReference>

posting the workaround here if it might help anyone else.

Edit2: This doesn't work, I get error

System.TypeLoadException: Could not load type 'Azure.AI.OpenAI.Completions' from assembly 'Azure.AI.OpenAI, Version=2.0.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8'.

Looking to solve this right now.

Edit3: Seems like there is no way to load a lower versioned library in a solution already having a higher versioned library loaded so I had to go deep in the rabbit hole here to solve this. I put the v1.0.0-beta.17 version of Azure.AI.OpenAI.dll into my "LegacyOpenAI" project and renamed it to Azure.AI.OpenAI.Legacy.dll to be able to load it in a new AssemblyLoadContext below using reflection... 🤦‍♂️

public async Task<IDictionary<string, float?>> GetTopLogProbsAsync(
    string prompt, string? userId, int maxTokens, float temperature,
    CancellationToken cancellationToken)
{
    AssemblyLoadContext? loadContext = null;

    try
    {
        loadContext = new AssemblyLoadContext(Guid.NewGuid().ToString(), true);

        var pathToAssembly = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, @"Azure.AI.OpenAI.Legacy.dll");
        Assembly assembly = loadContext.LoadFromAssemblyPath(pathToAssembly);

        var optionsType = assembly.GetType("Azure.AI.OpenAI.CompletionsOptions")
            ?? throw new ApplicationException("Could not get Azure.AI.OpenAI.CompletionsOptions from legacy dll");

        dynamic options = Activator.CreateInstance(optionsType, _deploymentName, new[] { prompt })
            ?? throw new ApplicationException("Could not construct Azure.AI.OpenAI.CompletionsOptions from legacy dll");

        options.DeploymentName = _deploymentName;
        options.User = userId;
        options.LogProbabilityCount = 2;
        options.MaxTokens = maxTokens;
        options.Temperature = temperature;

        var clientType = assembly.GetType("Azure.AI.OpenAI.OpenAIClient")
            ?? throw new ApplicationException("Could not get Azure.AI.OpenAI.OpenAIClient from legacy dll");

        dynamic client = (string.IsNullOrEmpty(openAIKey)
            ? Activator.CreateInstance(clientType, _endpoint, new Azure.Identity.DefaultAzureCredential())
            : Activator.CreateInstance(clientType, _endpoint, new Azure.AzureKeyCredential(_openAIKey)))
            ?? throw new ApplicationException("Could not construct Azure.AI.OpenAI.OpenAIClient from legacy dll");

        var response = await client.GetCompletionsAsync(options, cancellationToken);

        dynamic choice = response.Value.Choices[0];
        dynamic logprobs = choice.LogProbabilityModel.TopLogProbabilities[0];

        return logprobs;
    }
    finally
    {
        // Unload the context.
        loadContext?.Unload();
    }
}

from azure-sdk-for-net.