Cannot stream chat completions from Azure

[johanbaath]

Confirm this is a Node library issue and not an underlying OpenAI API issue

• This is an issue with the Node library

Describe the bug

When streaming chat completions using client.chat.completions.create with AzureOpenAI client and reading with ChatCompletionStreamingRunner.fromReadableStream on the client, the following error occurs:

OpenAIError: Cannot read properties of undefined (reading 'content')

Cause:

The error seems to be caused by choice.delta being undefined at some point during the streaming process, usually at the end of the stream.

Questions:

1. Is this a known issue?
2. Will this library support streaming from Azure, given potential differences in response structure?

To Reproduce

1. Initialize an AzureOpenAI client
2. Using client.chat.completions.create stream a response to the client (a web app)
3. Stream the response using ChatCompletionStreamingRunner.fromReadableStream on the client
4. Observe the error

Code snippets

No response

OS

macOS

Node version

Node v20.14.0

Library version

openai 4.56.0

[deyaaeldeen]

Hi @johanbaath, could you share a small repro code?

[johanbaath]

@deyaaeldeen sure something like this:

Server:

const azureOpenaiClient = new AzureOpenAI({
  endpoint: "...",
  deployment: "...",
  apiVersion: "2024-07-01-preview",
  apiKey: "...",
});

response = await azureOpenaiClient.chat.completions.create({
  model: "",
  messages: [
    {
      role: "user",
      content: "hello!",
    },
  ],
  stream: true,
});

Client:

const response = await ky.post(endpoint, {
  body,
  signal: abortController.signal,
});

const runner = ChatCompletionStreamingRunner.fromReadableStream(response.body);

runner.on("content", (delta) => {
  finalText += delta;
});

await runner.finalChatCompletion();

[deyaaeldeen]

Thanks! Could you also share the name and version of the deployed model and the region it is deployed in?

[johanbaath]

@deyaaeldeen gpt-4o, version: 2024-05-13, region: Sweden Central

[deyaaeldeen]

@johanbaath Thanks for confirming! Is asynchronous filter enabled by any chance? This feature has been known to cause such behavior and a similar issue has been reported elsewhere, for example in openai/openai-python#1677 with more context.

[johanbaath]

@johanbaath Thanks for confirming! Is asynchronous filter enabled by any chance? This feature has been known to cause such behavior and a similar issue has been reported elsewhere, for example in openai/openai-python#1677 with more context.

Yes! I use the async filter, is there anything I can do or is this being worked on? Thank you!

[deyaaeldeen]

We're discussing this behavior internally and I'll post an update as soon as I have one. For now, I suggest handling the absence of the delta property even if the type doesn't say it is possible.

[johanbaath]

We're discussing this behavior internally and I'll post an update as soon as I have one. For now, I suggest handling the absence of the delta property even if the type doesn't say it is possible.

Considering that this library does not properly manage this scenario, and given that Azure OpenAI for TypeScript advises migration to openai-node (as detailed here), are you suggesting that we manually patch the existing library as a temporary solution? Thanks!

[deyaaeldeen]

We didn't come to a decision yet regarding whether a change is necessary in either the service API or the client libraries. I'll keep the issue updated once I know more.

In the meantime, I am merely suggesting to handle this issue for now in your code, for example, if you have a steam of completion chunks, you can do the following:

for await (const chunk of steam) {
  for (const choice of chunk.choices) {
    ...
    const delta = choice.delta;
    if (delta) {
      // process delta
    }
    ...
  }
}

[doubianimehdi]

@deyaaeldeen this solution doesn't seem to work , this is really blocking , my company is really stuck due to this ... it works with any other GUI ... this is a shame :(

[deyaaeldeen]

@doubianimehdi I am sorry that you're experiencing issues in your code. Could you try this complete sample and let us know what issues are you facing exactly?: https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/openai/openai/samples/v2/typescript/src/streamChatCompletions.ts

[Swarnalathaa]

@deyaaeldeen following is the error response we are getting

this is the code block we have

        for await (const chunk of stream) {
          if (chunk?.choices?.[0]?.delta?.reasoning_content) {
            const reasoning_content = chunk?.choices?.[0]?.delta?.reasoning_content || '';
          }
          const delta = chunk?.choices?.[0]?.delta;
          if (delta) {
            const token = chunk?.choices?.[0]?.delta?.content || '';
            if (token) {
              reasoningCompleted = true;
              ...
            }
               ...
            }
          }
        }

[doubianimehdi]

@deyaaeldeen @Swarnalathaa is working with me , so she answered about your request :)

[deyaaeldeen]

@doubianimehdi, @Swarnalathaa thank you for sharing. You need to make sure delta is defined before accessing anything underneath it and in the code you provided, there is only one access to content and it is guarded by optional chaining operator: const token = chunk?.choices?.[0]?.delta?.content || ''; which seems ok to me.