Bug With Credits Usage
My agents were on ChatGpt 4o, but credits were used like I was on ChatGpt 4 Turbo. So it wasted 2000 of my credits on 5 interactions. Can you take a look on it?
BHey, can you share a screenshot here?
BOf the logs from the transcripts
M
M
Bcan you open up the debug mode please
Bin settings on the top
M
Bhow many interactions is this from?
Mscreenshot is from one
Bcan i see the chart though?
Balso, does this use the knowledge base?
B2k credits is crazy high
Myes it does use knowledge base
Bhm, can you DM me your email and I'm going to take a deeper look in a few hours when im done meetings
Mokay
Bthanks
B@Chidi can you do a deeper dive here? I'm unsure how these interactions could spend so many tokens
CHi @Medoni !
I took a look at your "AI asistent za izračunavanje komponenata i cena u Stefmark Shop-u" agent, and I didn't notice anything out of the ordinary in the debug messages.
The highest resource consumption was when your agent was using GPT-4 (a relatively expensive LLM). You can convert tokens to credits following this pricing table: https://docs.voiceflow.com/docs/credits-pricing-table#/
I took a look at your "AI asistent za izračunavanje komponenata i cena u Stefmark Shop-u" agent, and I didn't notice anything out of the ordinary in the debug messages.
The highest resource consumption was when your agent was using GPT-4 (a relatively expensive LLM). You can convert tokens to credits following this pricing table: https://docs.voiceflow.com/docs/credits-pricing-table#/
Byeah 17000 tokens is 8.5 credits per message
Mhi @Chidi , problem was that GPT 4 Turbo used 2,022 credits, it is not about GPT-4.
MOn just few messages
Bwas there an infinite loop? what language was it? @Medoni
MSerbian language, no loop at all.
WUse 4o or 40-mini
MYeah I am using it, but I just tested Turbo version for like 5 conversitions, and it used 2k credits, which is really crazy. If it was using only 8.5 credits per message it would be impossible to use 2k credits
BYeah, short term I have a plan to help and have DM'd you, then we have a new logging system launching soon which I think you can use to understand more granularly where this is coming from
CYeah, I had this issue too when I was testing my model. Just turns out the token to credit conversion wasn’t very clear (to me). After several tests I know this to be the math (not sure if this is what voiceflow intended since you said 8.5 credits but this is how it’s actually charging):
It’s basically 18000 tokens / 1000 = 18 x LLM multiplier (in this case gpt-4) which is 7. Then 18x7 = 126 credits. So that one agent response consumed 126 credits.
The same response using gpt-4-turbo (2.5 multiplier) would be 45 credits.
I believe VF defaults to Claude when I first started so unknowingly I threw an 8k token prompt in there and I was burning at like a rate of 7-8 credits per agent response unknowingly. I burned 2000 credits testing/debugging that day
but now I know better.
Claude was so good though. Having a tough replicating on the more cost effective models. You get what you pay for
It’s basically 18000 tokens / 1000 = 18 x LLM multiplier (in this case gpt-4) which is 7. Then 18x7 = 126 credits. So that one agent response consumed 126 credits.
The same response using gpt-4-turbo (2.5 multiplier) would be 45 credits.
I believe VF defaults to Claude when I first started so unknowingly I threw an 8k token prompt in there and I was burning at like a rate of 7-8 credits per agent response unknowingly. I burned 2000 credits testing/debugging that day
but now I know better. Claude was so good though. Having a tough replicating on the more cost effective models. You get what you pay for
BAh, yeah glad you figured it out. We have credits per message in debug rn as well
MI set agent on chat gpt 4 , it is usinc Claude 3.5 ?
Mso chat gpt 4 is using more credits than gpt 4 turbo?
M
M
Mif all interactions were on gpt 4 turbo ( but they weren't) it would be impossible to waste this many credits according to this calculator
CYeah, you can check the pricing table here: https://docs.voiceflow.com/docs/credits-pricing-table
credit per unit (1k tokens)
Gpt 4 = 7
Gpt 4 turbo = 2.5
Gpt 4o = 1.25 (my recommendation, use this.)
credit per unit (1k tokens)
Gpt 4 = 7
Gpt 4 turbo = 2.5
Gpt 4o = 1.25 (my recommendation, use this.)
Mthank you
CYeah, you probably did a lot of testing when it was set to Claude (it happened to me too) before changing to OpenAI model. Gpt 4o is more cost effective and it’s “better” than 4 and 4 turbo in almost all benchmarks.
Gpt 4o-mini is also pretty capable and credit per unit is only 0.04. If you want to play safe, stay with 4o. If you want to test, try 4o mini to see how it does. My concern is the language, I have zero experience on how these models work in other languages
Gpt 4o-mini is also pretty capable and credit per unit is only 0.04. If you want to play safe, stay with 4o. If you want to test, try 4o mini to see how it does. My concern is the language, I have zero experience on how these models work in other languages
Myeah, I tested it and 4o is really better
Byes gpt-4 uses far more credits