Prompt recipe - controlling request rate?

Options
TrevorHall
TrevorHall Registered Posts: 15 ✭✭✭✭

Is it possible to control the rate at which a prompt recipe issues requests? I ran my first one yesterday and very quickly hit the TPM rate cap on our OpenAI model deployment.

Best Answer

  • AdrienL
    AdrienL Dataiker, Alpha Tester Posts: 196 Dataiker
    Answer ✓
    Options
    Hi,
    On the connection, there are some settings to control the rate of queries. There is no direct limit for TPM, but reducing parallelism will usually fix it.

Answers

Setup Info
    Tags
      Help me…