The `generate-data.py` script does not generat a `nightly.json` file

[suhaibmujahid]

This seems to cause mozilla/bugbug#5575

[suhaibmujahid]

I see an error when generating crashes for beta, which occurs before nightly:

stab-crashes/generate-data.py

Line 341 in 06d72fb

for channel in ["release", "beta", "nightly"]:

The error crash script, so it does not continue generating the nightly.json file:

https://community-tc.services.mozilla.com/tasks/fkiIH-4ASi6a3bw8RN246w/runs/0/logs/public/logs/live.log

Getting top-200 startup crashes for the 'release' channel
Getting version information from Socorro... ✔
Getting top signatures from Socorro... ✔
Getting trends for top signatures from Socorro... ✔
Getting bugs linked to the top signatures from Bugzilla....... ✔
Resolving duplicate bugs to the bugs they've been duplicated to... ✔



Getting top-200 crashes for the 'beta' channel
Getting version information from Socorro... ✔
Getting top signatures from Socorro... ✔
Getting trends for top signatures from Socorro... ✔
Getting bugs linked to the top signatures from Bugzilla....... ✔
Traceback (most recent call last):
  File "/stab-crashes/generate-data.py", line 352, in <module>
    stats = get(
            ^^^^
  File "/stab-crashes/generate-data.py", line 272, in get
    k: v for k, v in Bugzilla.follow_dup(duplicate_ids).items() if v is not None
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/libmozdata/bugzilla.py", line 291, in follow_dup
    bz.wait_bugs()
  File "/usr/local/lib/python3.12/site-packages/libmozdata/bugzilla.py", line 191, in wait_bugs
    r.result()
  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 710, in send
    r = dispatch_hook("response", hooks, r, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/hooks.py", line 30, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/libmozdata/bugzilla.py", line 564, in __bugs_cb
    for bug in res.json()["bugs"]:
               ~~~~~~~~~~^^^^^^^^
KeyError: 'bugs'

[suhaibmujahid]

Looking at the update history, this seems to be an intermittent failure. Looking at the history here: https://github.com/mozilla/stab-crashes/commits/14f5deb348a88b56b6762d69e8bc4bd8ab060a9d/nightly.json?since=2025-12-30&until=2026-01-07

• Dec 30: Removed
• Jan 4: added
• Jan 5: Updated
• Jan 6: Removed

[marco-c]

I wonder if it is some sort of rate limiting from Bugzilla.

[calixteman]

I wonder if it is some sort of rate limiting from Bugzilla.

Could it be because of fastly ?

[suhaibmujahid]

@dklawren do we have Fastly in front of https://bugzilla.mozilla.org/rest?

[dklawren]

@dklawren do we have Fastly in front of https://bugzilla.mozilla.org/rest?

Fastly is in front of all BMO so including /rest. It could be rate limiting either through Fastly or BMO's builtin rate limiting or both. If we know the user agent and/or IP address of the script we can add it to the allow list in Fastly and also I can add to BMO too.