Make Your Agent Listen: Tactics for Obedience

One of the primary frustrations I’ve had while developing agents is the lack of obedience from LLMs, particularly when it came to tool calling. I would expose many tools to the agent with what I thought were clear, technical, descriptions, yet upon executing them it would frequently fail to do what I wanted.

For example, we wanted our video generation agent (called Pamba) to check whether the user had provided enough information such that composing the creative concept for a video could begin. We supplied it with a tool called checkRequirements() thinking it would naturally get called at the beginning of the conversation prior to composeCreative(). Despite clear instructions, in practice this almost never happened, and the issue became worse as more tools were added.

Initially I thought the cause of the LLM failing to listen might be an inherent intelligence limitation, but to my pleasant surprise this was not the case, instead, it was my failure to understand the way it holds attention. How we interact with the agent seems to matter just as much as what information we give it when trying to make precise tool calls.

I decided to share the tactics that I've learned since I haven't had any success finding concrete advice on this topic online or through ChatGPT at the time when I needed it most. I hope this helps. 

Tactic 1: Include Tool Parameters that Are Unused, but Serve as Reminders

Passing in a parameter like userExpressedIntentToOverlayVideo below forces the model to become aware of a condition it may otherwise ignore. That awareness can influence downstream behavior, like helping the model decide what tool to call next. 

In our particular case we were struggling to get the model to invoke a tool called overlayVideo() after generateVideo() even when the user expressed the intent to do both together. By supplying this parameter into the generateVideo() tool we reminded the LLM of the user's intent to call this second tool afterwards.

In case passing in the parameter still isn't a sufficient reminder you can also consider returning the value of that parameter in the tool response like I did above (along with whatever the main result of the tool was).

Tactic 2: Return Tool Responses with Explicit Stop Signals

Often the LLM behaves too autonomously, failing to understand when to bring the result of a tool back to the user for confirmation or feedback before proceeding onto the next action. What I've found to work particularly well for solving this is explicitly stating that it should do so, inside of the tool response. I transform the tool response by prepending to it something to the effect of "Do not call any more tools. Return the following to the user: ..." 

Tactic 3: Encode Step Numbers in Tool Descriptions with Mandatory or Optional Tags

In some instances we want our agent to execute through a particular workflow, involving a concrete set of steps. Starting the tool description with something like the following has worked exceptionally well compared to everything else that I've tried.

Tactic 4: Forget System Prompts, Retrieve Capabilities via Tool Calls

LLMs often ignore system prompts once tool calling is enabled. I’m not sure if it’s a bug or just a quirk of how attention works but either way, you shouldn’t count on global context sticking.

What I’ve found helpful instead is to provide a dedicated tool that returns this context explicitly. For example:

Tactic 5: Enforce Execution Order via Parameter Dependencies

Sometimes the easiest way to control tool sequencing is to build in hard dependencies.

Let’s say you want the LLM to call checkRequirements() before it calls composeCreative(). Rather than relying on step numbers or prompt nudges, you can make that dependency structural:

Now it can’t proceed unless it’s already completed the prerequisite (unless it hallucinates).

Tactic 6: Guard Tool Execution with Sanity Check Parameters

Sometimes the agent calls a tool when it's clearly not ready. Rather than letting it proceed incorrectly, you can use boolean sanity checks to bounce it back.

One approach I’ve used goes something like this:

Tactic 7: Embed Conditional Thinking in the Response

Sometimes the model needs a nudge to treat a condition as meaningful. One tactic I've found helpful is explicitly having the model output the condition as a variable or line of text before continuing with the rest of the response.

For example, if you're generating a script for a film and some part of it is contingent on whether a dog is present in the image, instruct the model to include something like the following in its response:

doesImageIncludeDog = true/false

By writing the condition out explicitly, it forces the model to internalize it before producing the dependent content. Surprisingly, even in one-shot contexts, this kind of scaffolding reliably improves output quality. The model essentially "sees" its own reasoning and adjusts accordingly.

You can strip the line from the final user-facing response if needed, but keep it in for the agent's own planning.

These tactics aren't going to fix every edge case. Agent obedience remains a moving target, and what works today may become obsolete as models improve their ability to retain context, reason across tools, and follow implicit logic.

That said, in our experience, these patterns solve about 80% of the tool-calling issues we encounter. They help nudge the model toward the right behavior without relying on vague system prompts or blind hope.

As the field matures, we’ll no doubt discover better methods and likely discard some of these. But for now, they’re solid bumpers for keeping your agent on track. If you’ve struggled with similar issues, I hope this helped shorten your learning curve.