OpenAI Operator AI agent beats Claude’s Computer Use, but it’s not perfect

Operator is OpenAI’s new AI agent for automating browser tasks, powered by a model called CUA (Computer-Using Agent), while Claude’s Computer Use from Anthropic uses a version of Claude 3.5 Sonnet to operate both in browsers and on desktop apps. 

Each is designed to “see” the screen via screenshots, press buttons, type text, and complete other real-world tasks normally done by people.

Also read: OpenAI launches Operator: How will this AI agent impact the industry?

So in this article, I will walk you through the key differences between OpenAI’s Operator and Claude’s Computer Use to let you know what fits the best for your workload.

But before we jump to any details, here is the brief overview of the comparison: 

Feature	OpenAI Operator	Claude Computer Use
Browser Task Performance	Superior (87% on WebVoyager)	Moderate (56% on WebVoyager)
Desktop App Interaction	Not yet available	Available
Error Handling	Self-corrects; hands control back	Experimental; prone to errors
Accessibility	Limited to Pro users	Broader beta availability
API Independence	Operates without APIs	Operates without APIs
Cost	$200 per month	Free beta access

How does Operator work

Operator is powered by OpenAI’s Computer-Using Agent (CUA), Operator leverages GPT-4o’s multimodal capabilities. It interprets graphical user interfaces (GUIs) using screenshots and interacts with them via virtual mouse and keyboard inputs. This enables it to perform tasks such as filling out forms, booking tickets, and shopping online without relying on APIs.

On the other hand, Claude Computer Use is a part of Anthropic’s Claude 3.5 Sonnet, this feature also uses screenshots to analyse GUIs and performs actions like clicking, typing, and navigating menus. It extends beyond browser-based tasks to desktop applications, offering more extensive functionality.

Performance Benchmarks

Here comes the most interesting part as we believe the benchmarks are one of the most important factors to prove the overall usability of a model. We have divided it into 3 parts:

OSWorld Benchmark (General Computer Tasks):

Operator (CUA) scored 38.1%, significantly outperforming Claude’s best score of 22%. Humans typically score around 72.4% on this benchmark. While Claude Computer Use lags behind Operator in this benchmark, it is still competitive in the broader AI landscape.

WebVoyager Benchmark (Browser-Specific Tasks):

In this benchmark, Operator (CUA) achieved a leading score of 87%, showcasing its strength in browser-based automation. Claude Computer Use scored 56%, indicating moderate performance in browser-specific tasks.

Also read: AI agents explained: Why OpenAI, Google and Microsoft are building smarter AI agents

For comparison, Google DeepMind’s Mariner scored 83.5%, placing it between Operator and Claude.

Specialised Benchmarks (Agentic Tasks):

Claude 3.5 Sonnet has shown strong results on benchmarks like SWE-Bench Verified (49%) and TAU-Bench for tool use tasks (69.2% in retail and 46% in airline domains). These benchmarks focus on coding and tool interaction, areas where Claude excels.

Task Scope

Operator is currently limited to browser-based tasks such as booking tickets, ordering groceries, and filling out forms. It cannot yet interact with desktop applications but plans to expand its capabilities through API integrations in the future.

On the other hand, Claude Computer Use offers broader functionality by interacting with desktop apps in addition to web browsers. This makes it suitable for automating workflows across different software platforms, such as managing spreadsheets or editing documents.

Error Handling

Both systems are experimental and prone to errors.

Also read: What is ChatGPT Tasks: Automating productivity, one reminder at a time

Operator has self-correction mechanisms for minor mistakes but hands control back to the user when encountering challenges like CAPTCHAs or login requirements.

Claude has been reported as slower and more error-prone, sometimes failing at basic actions such as scrolling or zooming.

Accessibility

After benchmarks, we believe the price and availability are the most important factors. Even though Operator seems to be an overall superior choice to Computer Use, the accessibility part changes the perspective all at once.

Operator is available exclusively to ChatGPT Pro users in the United States at $200 per month. OpenAI plans to extend access to Plus, Team, and Enterprise users in the future.

Claude Computer Use, on the other hand, is available in beta through Anthropic’s platform, with free access for some users. It has also been integrated into tools such as Canva for testing purposes.

Conclusion

OpenAI’s Operator currently excels in browser-based automation tasks due to its higher benchmark scores and robust reasoning capabilities. However, Anthropic’s Claude Computer Use offers greater versatility by extending its functionality to desktop applications. 

Also the $200 price tag and availability is what makes Operator fall behind. 

Both systems are still in early development stages, with significant room for improvement in speed, reliability, and autonomy. Depending on specific needs — whether focused on web automation or general computer interaction — users may prefer one over the other.

Also read: OpenAI in 2024: ChatGPT Search, Sora AI video, and all the big wins