Bug With Credits Usage
My agents were on ChatGpt 4o, but credits were used like I was on ChatGpt 4 Turbo. So it wasted 2000 of my credits on 5 interactions. Can you take a look on it?
33 Replies
Hey, can you share a screenshot here?
Of the logs from the transcripts
can you open up the debug mode please
in settings on the top
how many interactions is this from?
screenshot is from one
can i see the chart though?
also, does this use the knowledge base?
2k credits is crazy high
yes it does use knowledge base
hm, can you DM me your email and I'm going to take a deeper look in a few hours when im done meetings
okay
thanks
@Chidi can you do a deeper dive here? I'm unsure how these interactions could spend so many tokens
Hi @Medoni !
I took a look at your "AI asistent za izračunavanje komponenata i cena u Stefmark Shop-u" agent, and I didn't notice anything out of the ordinary in the debug messages.
The highest resource consumption was when your agent was using GPT-4 (a relatively expensive LLM). You can convert tokens to credits following this pricing table: https://docs.voiceflow.com/docs/credits-pricing-table#/
yeah 17000 tokens is 8.5 credits per message
hi @Chidi , problem was that GPT 4 Turbo used 2,022 credits, it is not about GPT-4.
On just few messages
was there an infinite loop? what language was it? @Medoni
Serbian language, no loop at all.
Use 4o or 40-mini
Yeah I am using it, but I just tested Turbo version for like 5 conversitions, and it used 2k credits, which is really crazy. If it was using only 8.5 credits per message it would be impossible to use 2k credits
Yeah, short term I have a plan to help and have DM'd you, then we have a new logging system launching soon which I think you can use to understand more granularly where this is coming from
Yeah, I had this issue too when I was testing my model. Just turns out the token to credit conversion wasn’t very clear (to me). After several tests I know this to be the math (not sure if this is what voiceflow intended since you said 8.5 credits but this is how it’s actually charging):
It’s basically 18000 tokens / 1000 = 18 x LLM multiplier (in this case gpt-4) which is 7. Then 18x7 = 126 credits. So that one agent response consumed 126 credits.
The same response using gpt-4-turbo (2.5 multiplier) would be 45 credits.
I believe VF defaults to Claude when I first started so unknowingly I threw an 8k token prompt in there and I was burning at like a rate of 7-8 credits per agent response unknowingly. I burned 2000 credits testing/debugging that day 😭 but now I know better.
Claude was so good though. Having a tough replicating on the more cost effective models. You get what you pay for 🤷♂️
Ah, yeah glad you figured it out. We have credits per message in debug rn as well
I set agent on chat gpt 4 , it is usinc Claude 3.5 ?
so chat gpt 4 is using more credits than gpt 4 turbo?
if all interactions were on gpt 4 turbo ( but they weren't) it would be impossible to waste this many credits according to this calculator
Yeah, you can check the pricing table here: https://docs.voiceflow.com/docs/credits-pricing-table
credit per unit (1k tokens)
Gpt 4 = 7
Gpt 4 turbo = 2.5
Gpt 4o = 1.25 (my recommendation, use this.)
thank you
Yeah, you probably did a lot of testing when it was set to Claude (it happened to me too) before changing to OpenAI model. Gpt 4o is more cost effective and it’s “better” than 4 and 4 turbo in almost all benchmarks.
Gpt 4o-mini is also pretty capable and credit per unit is only 0.04. If you want to play safe, stay with 4o. If you want to test, try 4o mini to see how it does. My concern is the language, I have zero experience on how these models work in other languages
yeah, I tested it and 4o is really better
yes gpt-4 uses far more credits