Counting tokens in VoiceFlow
I checked the FAQ and searched Discord, but have not found a way to count the number of tokens used during an API call. It is a variable somewhere because when I run the agent in debug mode the Voiceflow chat box responds with how many tokens were used. Can anyone point me to a resource that explains how to use the token_total variable.
4 Replies
In Voiceflow, tokens are used to facilitate interactions with AI models. The number of tokens used depends on the AI model and the complexity of the interaction. For example, more powerful AI models like GPT-4 use more tokens compared to GPT-3.5.
Tokens are used by both the input (what is being sent to the AI model) and output (what is being received from the AI model). The 'token' slider on the AI steps controls the output message, but the input tokens depend on the amount of information sent to the AI model as part of your prompt.
Different features in Voiceflow use different amounts of tokens:
- Knowledge Base: A question and answer will use anywhere from 800-1500 tokens depending on the size of the question and answer.
- Response AI Step (With Memory): Uses 150-2000 tokens, as it saves the last 3 conversation turns for context.
- Response AI Step (Without Memory): Uses roughly 150-300 tokens.
For more detailed information on how tokens are calculated and used, you can refer to the following documentation:
- OpenAI Token Documentation
- Anthropic Token Documentation
These resources will provide you with a comprehensive understanding of token usage and management in Voiceflow.
I understand the idea of tokens, however, I need to count tokens or use the {token_total} variable. I cannot locate information on how to count tokens.
You don’t use tokens when you make an API call.
Steve, thank you for the response, I appreciate it. I am new and likely using the wrong term(s) or didn't explain what I am doing well.
I have set the AI and gotten a response from the AI, I saved that response as a variable {ai_response}. I have been trying to use an API POST to sent {ai_response} to a PHP script so it can count the characters in {ai_response}, divide that number by 4, and use that as an estimate to calculate how many tokens were used to generate the {ai_response}. I started down this path and then realized when I run an agent in Voiceflow that it tells me the number of tokens that are used with each AI input, output, and token_total. I am looking for the best way to dertermin how many total tokens are used and write that to a database so I can track customer data usage. Do you know if it is possible to access the total_tokens variable in Voiceflow? Or, is using an API to get an estimate of tokens used my best option. Sorry if I am asking a simple question, I am still learning, and thank you so much for any guidance you can give.