Repository navigation
Fail to Reproduce WebArena Results with GenericAgent-GPT-4o #249
Description
Dear AgentLab Authors,
Thank you for the great work! I'm trying to reproduce the WebArena Results with GenericAgent-GPT-4o. In particular, I used the following code. Everything should just follow AgentLab's default. However the number I got is 25 which is significantly lower than 31.4 as shown on the BrowserGym Leaderboard. Do you have any suggestions for the reproduction? Any code available to reproduce the performance ~31?
Thanks again for you great contribution to the community!
from agentlab.agents.generic_agent import AGENT_4o
from agentlab.experiments.study import make_study
from agentlab.experiments.study import Study
study = make_study(
benchmark="webarena",
agent_args=[AGENT_4o],
comment="repo 4o agent",
)
study.run(n_jobs=5)
Activity
recursix commentedon Oct 2, 2025
I'm sorry for not getting back to you sooner, and thank you for reporting this.
Could you give us some more details about your experiments? I.e., which os, which version of packages? Info should be in the experiment directory.
We recently realized that when running on a laptop, e.g., MacBook, a retina display is activated, doubling the DPI of the playwright and causing some actions to fail, e.g.,
click. This can cause a significant drop in performance.In a later version, we may run everything in a container.