other-emerald•2mo ago
Agent button tool issues
My agent step using 4o-mini doesn't use the button tool when it's supposed to, but it works with other llm models, why?
31 Replies
Things to check when 4o-mini doesn’t pick the Button tool
1️⃣ Button tool must be enabled in the Agent configuration
In the Agent step open Tools > “Use special chat tools” and be sure Buttons is toggled on. (ref: “Agents can now automatically use buttons, cards, and carousels…”)
2️⃣ The channel you’re testing matters
Buttons, cards and carousels are ignored on phone calls. If you are calling the bot or using the voice test tool the model will always answer with plain text, no matter which model you pick. (same source – see note in the banner)
3️⃣ Model skill / prompt clarity
There’s no documented hard limit on GPT-4o-mini, but smaller models are less consistent at following the tool-selection instructions. Giving the agent an explicit example inside the Instructions (e.g. “If you want the user to pick an option, respond with the Button tool and list the options instead of plain text”) usually fixes it.
4️⃣ Fallback: use a Button step
If you need a guarantee that the UI element shows, add a dedicated Button Step instead of relying on the agent to decide. The Button step always renders its buttons regardless of the underlying LLM.
Try the checklist above in a web-chat test first; once it works there, switch back to your preferred channel.
Docs: https://docs.voiceflow.com
In the Agent step open Tools > “Use special chat tools” and be sure Buttons is toggled on. (ref: “Agents can now automatically use buttons, cards, and carousels…”)
2️⃣ The channel you’re testing matters
Buttons, cards and carousels are ignored on phone calls. If you are calling the bot or using the voice test tool the model will always answer with plain text, no matter which model you pick. (same source – see note in the banner)
3️⃣ Model skill / prompt clarity
There’s no documented hard limit on GPT-4o-mini, but smaller models are less consistent at following the tool-selection instructions. Giving the agent an explicit example inside the Instructions (e.g. “If you want the user to pick an option, respond with the Button tool and list the options instead of plain text”) usually fixes it.
4️⃣ Fallback: use a Button step
If you need a guarantee that the UI element shows, add a dedicated Button Step instead of relying on the agent to decide. The Button step always renders its buttons regardless of the underlying LLM.
Try the checklist above in a web-chat test first; once it works there, switch back to your preferred channel.
Docs: https://docs.voiceflow.com
@Tico | Voiceflow Assistant This doesn't resolve the issue, i tried all of the above except step 4 which doesn't apply and it's still persisting.
@NiKo | Voiceflow @KimLooo @Daniel @jacklyn
Hey guys, sorry to bother you, just wanted to know if you have any knowledge of a current issue with dynamic buttons/carousels with agents, as many of us in the community are facing the same problem. Thanks in advance
Can you share your prompt?
Some llms need more help than others 😅
Yeah we've heard this a few times and unfortunately it seems to be a model limitation 🙁 Have you tried switching over to claude?
chatgpt 4o is able to generate carousels but not buttons, while claude 4.0 sonnet is able to generate buttons and not carousels 😦 it's unfortunate, as carousels/buttons is one of the best features for a good UX
I don't think it's the problem, because even when I ask explicitely to have buttons openai models are unable to display any, so they default to bullet lists. when I ask anthropic models to do the same it works 100% of the time, but when a carousel is to be showed it fails and the chat hangs.
I even tried a prompt that's just something alone the lines of "You are a helpful assistant [...]. You always start by saying "Hello" and then displaying 4 buttons : A, B, C and D." and it's not working
@Dali I have an agent that generates carousels every time, i’m using Claude 4. In your prompt are providing examples of the carousel or button JavaScript schema?
Super interesting Steve thanks for the input. I’m not, if you could provide the schéma sample you’re using that’d be amazing
@Dali, sure this is what i use for carousels:
I tell the llm to use this schema strictly for the carousels.
Hey @Steve B. | IntelleSync.ai , it's still not working.
I will share you my prompt if you want to reproduce the issue. I dedicated it to troubleshooting the carousel, it's a simple prompt :
{weather_data} is a simple json with weather info. If you want to try the behaviour, here is the prototype : https://creator.voiceflow.com/prototype/688b44e7c4336541b3310dfb It starts with a welcome message, but the carousel is never generated.
{weather_data} is a simple json with weather info. If you want to try the behaviour, here is the prototype : https://creator.voiceflow.com/prototype/688b44e7c4336541b3310dfb It starts with a welcome message, but the carousel is never generated.
Did you map the placholders with your schema from the {weather_data} array?
ie image_url, description_text etc
Yes I did, and now i'm getting the carousel but once in a while. It's a very unreliable feature, especially the fact that changing from one model to another completely disturbs the behaviour
Claude 4 works, but Claude 3.7 doesnt, and Chatgpt4-o doesn't neither...
for teams wanting to experiment with different LLMs, especially more cost-effective ones, it's a very unreliable solution.
Can you paste your weather array here?
I hardcoded the values within the schema to eliminate the problem of mapping the placeholders
can you share the response from your weather data api?
{
"weather": [
{
"country": "USA",
"city": "New York",
"temperature_celsius": 27,
"condition": "Sunny"
},
{
"country": "France",
"city": "Paris",
"temperature_celsius": 22,
"condition": "Cloudy"
},
{
"country": "Japan",
"city": "Tokyo",
"temperature_celsius": 30,
"condition": "Partly Cloudy"
},
{
"country": "Brazil",
"city": "Rio de Janeiro",
"temperaturecelsius": 28,
"condition": "Rainy"
}
]
}
I even tried this :
"payload": {
"layout": "Carousel",
"cards": [
{
"imageUrl": "https://upload.wikimedia.org/wikipedia/commons/thumb/c/c1/Google%22G%22logo.svg/768px-Google%22G%22_logo.svg.png",
"title": "Test Card",
"description": {
"text": "Hello, this is a test card."
},
"buttons": [
{
"name": "Click me",
"request": {
"type": "action",
"payload": {
"label": "Click me",
"actions": [
{
"type": "open_url",
"payload": {
"url": "https://www.google.com"
}
}
]
}
}
}
]
}
]
}
Which is your template with hardcoded values and it's not working
Experiencing the same with GPT models being absolutely unreliable with buttons whatever I try. My solution now is using Haiku 3.5 as the model cost is somewhat acceptable. For some reason only Claude models work reliably with buttons for me. Gemini only provides buttons without an agent message (also when I explicitly state it should write the message first). GPT doesn't really do anything correctly in this regard. Claude is super reliable but sonnet/opus are too expensive for public production use.
Does anyone have any luck with lighter models so far?
@Dali I tested it 10 times, it works for me. Take a look at my prompt. Please, let me know if it works for you.
@Dali, That what i meant about mapping. In this part of @Andrew's prompt:
Thank you very much, I’ll be testing it this afternoon and will let you know.
I understand the importance of explicitly mapping the fields to help the LLM.
It’s strange that cards and buttons have a 100% rate of success without having to map everything, opposite to the carousel.
@Steve B. | IntelleSync.ai @Andrew Thank you guys, I have been able to figure out the issue. It wasn't the structure of my prompt or my json, that worked perfectly-- the problem is the length of the json.
As soon as you 5 elements, the carousel stops working. At 4, everything is okay, and I haven't had an issue.
Is this a known limitation ?
@Dali check please carousel llm description, maybe you have some limits there. If you don't have it means it's an issue
I tried with your prompt, and with the same weather data but expanded with three additional countries. It doesn't work
So it's an issue
The LLM description explicitely says 10 cards, but it stops at 4
Ok, found the issue. Increasing the max token resolved the issue. If anyone wondering. Thanks
@Dali i was going to suggest this, glad it's all working now 🙂
Hey folks, we just released some updates on the Agent step docs regarding button, cards and carousel generations! Feel free to take a look:
https://docs.voiceflow.com/docs/agents#/agent-generated-components 👀
Great stuff @Anirudh
Hi guys, thanks a lot for the time you spent helping and the additional doc, it helps a lot.
Do you know if it's currently possible for a card button's action generated by an agent to register a choice and continue the conversation ?
E.g : I have a chatbot that will suggest 3 products as cards, with a "Choose" button. If the user clicks choose, it will continue the conversation and remember that the product was chosen, to then ask for more info (e.g when they want it delivered, the color, etc.)
With traditional custom action dynamic carousels you can capture last_event payloads which could help you do this, although I haven't tried it with agent generated carousels.
Yeah I experimented with traditional carousels it works fine, but with agents, when the agent generates an action payload of button_action : [], clicking the button makes the conversation load infinitely and it doesnt work anymore.