Today, January 23, 2025, OpenAI released their Operator functionality. This was particularly exciting for me because I’ve spent the past few months immersed in the world of AI-driven UI automation. As part of my work with the team at Outshift by Cisco, we’ve been researching agentic UI automation and its potential for Cisco use cases. I wanted to share some thoughts on OpenAI’s release and how it relates to what we’ve been exploring.

Agentic: The word of 2025

We were promised a future filled with HAL9000-level robots and Jarvis-like assistants who would revolutionize our lives. Instead, most of us are just asking ChatGPT how to word an email or clarify whether it’s "affect" or "effect."

I think this gap between expectation and reality comes down to how users interact with AI and how these tools integrate into our workflows. The allure of ChatGPT’s ability to provide instant answers has reshaped expectations for applications, platforms, and social media. Yet, as we’ve boxed ourselves into the confines of a chat window, we’ve failed to imagine how AI might move beyond these boundaries. OpenAI’s Operator release validates what I’ve been feeling: the agentic revolution is evolving, and the battleground for its future is the user interface.

Challenges of API-driven agents

API-driven agents, while powerful, function as black-box systems. For simple tasks—creating GitHub repositories, sending emails—they’re fine. But when it comes to mission-critical, complex business use cases, trust, accountability, and oversight are essential. Right now, API agents just aren’t there.

Take Alexa, for example. It was marketed as a tool to simplify shopping on Amazon, but my guess is that it’s rarely used for that. Why? Because people want control over their decisions, and API-driven agents often take that control away. They make for impressive demos, but they rarely translate into practical, real-world solutions.

Human-in-the-loop: The foundation for agentic UX

What makes OpenAI’s approach groundbreaking is their focus on human-in-the-loop design. By anchoring the process in the UI, they enable humans to intervene and take control at any point in the agent’s workflow.

This approach addresses a critical challenge explored in the paper Do LLMs Know When Not to Answer? Investigating Abstention Abilities of Large Language Models. The paper introduces the concept of Abstention Ability (AA)—an AI’s capacity to recognize and communicate its limitations. Even advanced models like GPT-4 often struggle with abstention, particularly in complex reasoning or niche knowledge areas, leading to unreliable outputs. This reinforces the need for human intervention as a safeguard in high-stakes applications.

Internally, we’ve observed similar challenges, such as agents filling out form fields incorrectly due to a lack of confidence or context. While it’s sometimes helpful for models to infer what you might want, this behavior can lead to unintended consequences and frustrating user experiences. The paper provides actionable strategies to mitigate these risks, such as employing "Strict Prompting" and "Chain-of-Thought (CoT)" techniques to improve abstention and decision-making. By integrating these techniques with a UI-first approach, systems can better identify when human intervention is necessary and ensure safer, more reliable outcomes.

Automating the browser

There are two primary approaches to automating browser and desktop applications:

  • Screenshot-Based Methods: OpenAI’s approach involves capturing the UI visually and taking action based on this data.
  • Textual Parsing Methods: Strategies like those from AgentOccam and Jina.ai focus on parsing HTML to isolate elements like buttons and dropdowns, simplifying the DOM into a format that’s easy for LLMs to process.

Both approaches have their strengths and weaknesses. Screenshot-based automation is powerful but struggles with variability in styling, layout, and design, which can introduce a lot of visual noise and make it harder for agents to identify interactable elements consistently. Additionally, this method requires significant compute resources to process visual data and translate it into actionable insights.

On the other hand, text-based methods allow agents to process entire page structures at once, eliminating the need for simulated scrolling and potentially reducing the number of actions required. This can be particularly advantageous for applications that demand speed and efficiency. However, parsing large amounts of text comes with its own challenges, such as hitting token limits or losing critical context when simplifying complex DOM structures.

Designing for humans and AIs together

The future of agentic-focused UI design raises an important tension: How do we design interfaces that are clear for both humans and AIs?

I’m reminded of a PR comment where a WSL dev declined a suggestion to improve documentation readability because "the tables don't translate well" to an AI context. It received over 900 dislikes, highlighting a growing frustration with prioritizing AI comprehension at the expense of human experience.

We need to create UIs that are interpretable by AIs without compromising human experience. Accessibility principles and semantic HTML provide a good starting point, but balancing these priorities will be critical as agentic systems begin exploring the web alongside us.

I recently came across a demo that perfectly captured the promise of agentic AI. A creator showed how they can leverage existing Gemini LLMs, voice-to-text input, and an API interface for Blender to create a variety of scenes using only his voice. This kind of experimentation highlights the beauty of agentic AI—empowering users to work directly within their tools rather than around them. It’s not about AI taking over; it’s about collaboration, where humans retain agency and creative control.

What makes the UI agentic automation strategy so powerful is its role as the bridge between the agentic and non-agentic worlds. By embedding agents directly into existing workflows and environments, it enables AI to function seamlessly in spaces that weren’t explicitly designed for automation. This approach also opens the floodgates for AI agents to work out-of-the-box on a wide range of use cases that would otherwise require significant time and effort to build from scratch.

A personal reflection

As I look ahead, I’m struck by how much potential this space holds. I absolutely love the web that we've created and I hope that we can continue to build on it. We can welcome AI into our lives, but we must do so in a way that enhances our experiences rather than detracts from them. As API-driven agents become more prevalent, I hope we can continue to utilize the browser as our primary interface for human-computer-AI interaction.