ChatGPT Agent isn’t magic: Here’s what it can (and can’t) do
I tried it. I liked parts of it. It’s flawed.
Yes, this is a big release, but it isn’t GPT-5. We’re not there yet. ChatGPT Agent combines and synthesizes its entire feature set for a genuinely impressive multimodal experience. It’s the most advanced agent we’ve seen so far. But it’s nowhere near artificial general intelligence.
Think of it as Operator 2.0 with more bells and whistles. It whips out a virtual laptop to build slides, shop online, or run market research. You can even ask it to do all of those tasks in sequential order with one good power prompt.
I watched OpenAI’s Agent intro video nine hours after Altman and team announced its release. That’s considered behind schedule these days, so I jumped right on in and threw everything at it from basic booking tasks to downloading gated content.
Here’s what worked. And what fell flat.
ChatGPT Agent can make reservations. Kind of.
I started by asking Agent to book a table at Figmor, a local breakfast diner in Buffalo.
It failed, but that was mostly on me. ChatGPT Agent was quick to find that Figmor doesn’t let you book reservations online. The screenshot below shows Agent identifying the issue:
Agent couldn’t pull booking info from Yelp either, eventually concluding that the only way to book was to call. I asked Agent to call Figmor for me. See below:
Either Agent can’t make phone calls or didn’t feel like trying, so I moved on.
Next up? VICE
VICE is another popular restaurant in Buffalo with an online reservation system. This one should have been easy, but Agent still struggled.
What would take me one minute took Agent more than eight. Here’s what went wrong:
- It tried scrolling while already at the bottom of the booking form.
- It fiddled with checkboxes instead of filling in the required phone number field.
- It took 4 tries to select the right booking time from the drop-down.
- It said it was experiencing connectivity issues after pages were fully loaded.
In the end? I got my reservation, or would have, if I’d given Agent my phone number. Here’s the final screen:
ChatGPT Agent did fine with the info it had, but only asked for the missing piece after trying to work around it for eight minutes. The end result was slow, brittle execution.
This was a letdown, especially for one of Agent’s headline features. I thought maybe booking (or speed) wasn’t its strong suit.
That is until I threw a quick marketing task at it.
Booking a demo with SE Ranking? Easy!
When your booking process is straightforward, Agent doesn’t stumble at all. See for yourself:
See how it only took 37 seconds? That’s a clear difference from the other two tests.
This got me thinking: if tech giants like OpenAI and Google say AI agents are shaping the future of search, then maybe calling Agent slow misses the point. And even if it’s slow on average, that doesn’t matter long-term because semi-autonomous tools like Agent will keep getting faster and smarter with every iteration.
And if you’re like SE Ranking and optimize your booking page for AI agents, 8+ minutes turns into 30 seconds. So is ChatGPT Agent actually slow to book? Or are local businesses behind schedule with soon-to-be outdated UIs?
This line of questioning inspired my next test.
Navigating a gated form: Hubspot’s 2025 State of Marketing Report
If ChatGPT Agent could complete a marketing task for a user, like booking a demo with SE Ranking, running a lead generation test was the next logical step. That’s because AI agents could now download, process, and synthesize data from gated content, making them synthetic leads. Forbes even called them AI customers.
This implies you can convert them.
So I thought: “If this test works, it means agencies need to start thinking about optimizing their lead generation process for AI agents.”
Then I stopped theorizing and gave myself the green light to run the test.
The prompt:
I was stunned. Agent was so close! It didn’t even need me to step in and take over to download Hubspot’s 2025 State of Marketing report. See for yourself:
The image above is proof that Agent found the page, clicked on the ‘Get the trends’ button, and filled out the form all on its own. Agent also successfully navigated the document to prepare a report summary. This was after it clicked submit and downloaded the file, without my help.
That’s two-for-two when sites are agent-friendly. Throw the agent at a clean, intuitive interface and it will do what you ask—and fast. Which brings me back to my original question: Is this really a matter of capability? Or are most businesses just not ready for agents?
The next phase of this test added a new layer of complexity to my question.
It’s not just about readiness. It’s about access.
Agent-ready or not, most businesses probably won’t feel the impact—yet. As long as Google keeps the reins tight on AI, Agent’s power is theoretical. Not even access to a virtual laptop gives Agent full autonomy.
Near the end of this test, Agent encountered a security bottleneck. It successfully downloaded Hubspot’s report and wrote a summary about it. But when Agent tried emailing the summary to me? Cue the Google restrictions.
The image above shows Google blocking Agent entry to my email account for using an unsupported browser. System-level privacy on Google’s servers stopped it dead in its tracks
So even though Agent is semi-autonomous, it’s still fenced in. Agent runs on a browser Google blocks from accessing email. That’s no accident. It’s a caged feature limited by the very system it runs on.
Which might not be a bad thing. There are many good reasons why Altman and team haven’t taken the training wheels off this tech just yet.
But not every platform shuts ChatGPT Agent out
Anastasia Kotsiubynska, our SEO team lead, prompted Agent to access the platform through her account. The screen recording below shows Agent successfully logging in—
—though only after asking Anastasia to take over. Agent couldn’t input her credentials on its own, so it asked her to step in. This hesitation suggests Agent recognized the boundary between permission and trespass.
But once inside, Agent completed the task in under two minutes. No scrolling hiccups. No connection issues. No brittle logic. Just execution.
Let’s pull Anastasia’s prompt to evaluate Agent’s process:
Simple enough—for a human. Here’s how Agent handled finding keyword suggestions for the query ‘seo tools’:
- It waited for the dashboard to load →
- It navigated to the Keyword Suggestions tool →
- It entered ‘seo tools’ into the search field and clicked analyze →
- It checked for keyword suggestions, related keywords, and other sections →
- It opened the ‘View detailed report’ for ‘similar keywords’ to gather the first 10 keywords →
- It reviewed the list of keyword suggestions to confirm the top 10 →
- It examined the rows to check for duplicates or missing entries →
- It gathered the top 10 SEO suggestions for ‘seo tools’, excluding the base term.
That process is as close to painless as an AI agent can muster. It even recognized that ‘seo tools’ and ‘seo tool’ were duplicates, and removed one to clean up the final result.
Agent’s process for the next two queries was just as smooth sailing.
So what does this mean for agencies?
Agent just moved through the entire funnel, from awareness to login to in-platform action. That’s not task automation. That’s a synthetic customer.
So what happens when more agents start navigating the web the same way?
- Suddenly, forms aren’t just for people anymore.
- CTAs aren’t just calls to action. They become instructions for machines.
- UX friction that breaks the agent’s flow could flag processes as confusing for edge-case users (on slower connections, with assistive tech, etc).
The big tell? These tests show that the only thing standing between a future where optimizing your site for agents matters and the present is browser access restrictions.
Want to get ahead of the curve? Here’s how to prepare your lead gen for a future where agents run free.
But if you’re after the practical, present-day takeaway, here’s what ChatGPT Agent means for your business right now.
ChatGPT Agent isn’t just a tool. It’s a stress-test for your funnel.
Let’s be clear: my tests didn’t uncover broken UIs—at least not for most human users. VICE’s booking form was straightforward for me. I made a reservation in 30 seconds. Agent flailed for eight minutes. That’s a current-AI limitation, not a UX failure.
And yet, when the conditions were right, Agent did a pretty solid job. This could mean that businesses optimizing for edge cases and accessibility may unintentionally be preparing for an agent-driven future.
And yes, Agent’s performance is inconsistent. I ran four tests. Anastasia brought a fifth into the mix. Agent only fully succeeded twice. But the trajectory is clear. AI agents are the best they’ve ever been and the worst they’ll ever be. So now’s the time to treat them like part of the funnel.
Ask yourself: When AI agents work at scale, will my website be ready for them?
