​
Setup

​
Setup the CopilotKit Backend Endpoint:

Currently all you require is an endpoint which serves as a proxy to an OpenAI-like LLM endpoint. See below for examples using NextJS edge functions, Portkey Gatewway, and Flask.

​
Manually specified endpoint: nodeJS (typescript)

Make sure to install the ai and openai packages:

/api/copilotkit/chat/route.ts (NextJS)
import { OpenAIStream, StreamingTextResponse } from "ai";
import OpenAI from "openai";
import { CompletionCreateParamsStreaming } from "openai/resources/chat/completions";

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

export const runtime = "edge";

export async function POST(req: Request): Promise<Response> {
  const forwardedProps = await req.json();

  const response = await openai.chat.completions.create({
    model: "gpt-4",
    ...forwardedProps,
    stream: true,
  } as CompletionCreateParamsStreaming);

  const stream = OpenAIStream(response, {
    experimental_onFunctionCall: async (
      { name, arguments: args },
      createFunctionCallMessages
    ) => {
      return undefined; // returning undefined to avoid sending any messages to the client when a function is called. Temporary, bc currently vercel ai sdk does not support returning both text and function calls -- although the API does support it.
    },
  });

  return new StreamingTextResponse(stream);
}

​
Portkey endpoint for connecting to 20+ LLMs: nodeJS (typescript)

Portkey serves as a unified gateway to access over 20 LLMs, such as OpenAI, Azure OpenAI, Anthropic, Google, and more. It also enhances your app’s reliability with features like load balancing various models or keys, triggering automated fallbacks in case of failures, and more.

Make sure to install the ai and portkey-ai packages:

/api/copilotkit/chat/route.ts (NextJS)
import { OpenAIStream, StreamingTextResponse } from "ai";
import Portkey from "portkey-ai";

const portkey = new Portkey({
  apiKey: process.env.PORTKEY_API_KEY,       // Get from https://app.portkey.ai/
  /*****************************/  
  provider: 'openai',                        // Change to 'azure-openai', 'anthropic', 'google-palm', 'anyscale' etc.
  Authorization: process.env.OPENAI_API_KEY, // Pass the Bearer auth key for the chosen provider
  /*****************************/
});

export const runtime = "edge";

export async function POST(req: Request): Promise<Response> {
  const forwardedProps = await req.json();

  const portkeyResponse = await portkey.chat.completions.create({
    model: "gpt-4",
    ...forwardedProps,
    stream: true,
  });

  const jsonResponseString = JSON.stringify(portkeyResponse);
  
  const response = new Response(jsonResponseString, {
    headers: { 'Content-Type': 'application/json' }
  });

  const stream = OpenAIStream(response, {
    experimental_onFunctionCall: async (
      { name, arguments: args },
      createFunctionCallMessages
    ) => {
      return undefined; // returning undefined to avoid sending any messages to the client when a function is called. Temporary, bc currently vercel ai sdk does not support returning both text and function calls -- although the API does support it.
    },
  });

  return new StreamingTextResponse(stream);
}

Check out the Portkey docs to learn how to connect other LLMs and use more Portkey feautres like caching, tagging, tracing, and more.

​
Manually specified endpoint: flask (python)

Make sure to install the Flask and openai packages.

from flask import Flask, request, Response
from flask import Response
import openai

app = Flask(__name__)

# Assuming the API key is set in environment variables
client = openai.OpenAI(
    api_key="" # add your api key here
)

def generate_stream(**forwarded_props):
    """
    Generator function to stream the chat completion response.
    """
    # Ensure streaming is enabled
    forwarded_props['stream'] = True
    forwarded_props['model'] = "gpt-4-1106-preview"

    response_stream = client.chat.completions.create(**forwarded_props)

    for response in response_stream:
        text = response.choices[0].delta.content or ""
        if text:
            yield text


@app.route('/copilotkit_endpoint', methods=['POST'])
def copilotkit_endpoint():
    """
    Endpoint to handle chat requests and stream responses.
    """

    # Extracting all properties from the request JSON
    forwarded_props = request.get_json(force=True)

    # Generate and return the streaming response
    return Response(
        generate_stream(**forwarded_props),
        content_type='text/plain; charset=utf-8',
        headers={
            'Access-Control-Allow-Origin': '*',
        }
    )
DevelopmentMarkdown Syntax