Bug 2019457 - Refactor CI/CD test cases to make it easier to incorporate automated tests from extensions into the normal test framework#2553
Open
dklawren wants to merge 6 commits intomozilla-bteam:masterfrom
Conversation
Collaborator
dklawren
commented
Feb 26, 2026
dklawren
commented
- Move all of the helper libraries from qa/t/lib to Bugzilla/QA
- Move the config file to conf/
- Move the generate_test_data.pl to scripts/
- Update all of the current scripts to use the new Bugzilla/QA location
- Update the Github actions config to use the new way of calling all of the tests
…ated automated tests from extensions into the normal test framework
There was a problem hiding this comment.
Pull request overview
Refactors the CI/QA test harness to use new Bugzilla::QA::* and Bugzilla::Test::* helper modules, relocates QA configuration/scripts, and updates GitHub Actions to run tests using the new invocation patterns.
Changes:
- Updated test files to import helpers from
Bugzilla::QA::*/Bugzilla::Test::*and adjusted@INC(use lib) accordingly. - Moved/updated QA config and data generation script paths (e.g., config under
conf/, generator underscripts/). - Modified Docker/GitHub Actions test commands to pass explicit prove args/patterns.
Reviewed changes
Copilot reviewed 123 out of 124 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| t/critic.t | Switches from Support::Files to Bugzilla::Test::Files and updates use lib. |
| t/bmo/passwords.t | Uses Bugzilla::QA::Util and updates use lib. |
| t/bmo/oauth2-client.t | Uses Bugzilla::QA::Util and updates use lib. |
| t/bmo/bounced-emails.t | Uses Bugzilla::QA::Util and updates use lib. |
| t/012throwables.t | Updates helper namespaces; changes error-usage scanning logic. |
| t/011pod.t | Switches to Bugzilla::Test::Files and updates test count. |
| t/010dependencies.t | Switches to Bugzilla::Test::Files and updates module extraction loop. |
| t/009bugwords.t | Switches to Bugzilla::Test::{Files,Templates} and updates include paths. |
| t/008filter.t | Switches to Bugzilla::Test::Templates and updates include paths. |
| t/007util.t | Switches to Bugzilla::Test::Files and updates use lib. |
| t/006spellcheck.t | Switches to Bugzilla::Test::Files and updates test list. |
| t/005whitespace.t | Switches to Bugzilla::Test::{Files,Templates} and updates test list. |
| t/004template.t | Switches to Bugzilla::Test::Templates and updates use lib. |
| t/002goodperl.t | Switches to Bugzilla::Test::Files and updates test list. |
| t/001compile.t | Switches to Bugzilla::Test::Files and uses Bugzilla::Test::Systemexec for compile checks. |
| scripts/generate_test_data.pl | Simplifies @INC setup; reads config from conf/selenium_test.conf by default. |
| scripts/entrypoint.pl | Updates how tests are invoked and where test data generation runs from. |
| qa/t/webservice_user_offer_account_by_email.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/webservice_user_login_logout.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/webservice_user_get.t | Uses Bugzilla::QA::{Util,Tests} and updates use lib. |
| qa/t/webservice_user_create.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/webservice_product_get.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/webservice_product_create.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/webservice_product.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/webservice_jsonp.t | Uses Bugzilla::QA::Util and updates client callsite. |
| qa/t/webservice_group_create.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/webservice_bugzilla.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/webservice_bug_update_see_also.t | Uses Bugzilla::QA::{Util,Tests} and updates use lib. |
| qa/t/webservice_bug_update.t | Uses Bugzilla::QA::{Util,Tests} and updates use lib. |
| qa/t/webservice_bug_search.t | Uses Bugzilla::QA::{Util,Tests} and updates use lib. |
| qa/t/webservice_bug_legal_values.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/webservice_bug_history.t | Uses Bugzilla::QA::{Util,Tests} and updates use lib. |
| qa/t/webservice_bug_get_bugs.t | Uses Bugzilla::QA::{Util,Tests} and updates use lib. |
| qa/t/webservice_bug_get.t | Uses Bugzilla::QA::{Util,Tests} and updates use lib. |
| qa/t/webservice_bug_fields.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/webservice_bug_create.t | Uses Bugzilla::QA::{Util,Tests} and updates use lib. |
| qa/t/webservice_bug_comments.t | Updates isa checks to Bugzilla::QA::RPC::* and imports Bugzilla::QA::*. |
| qa/t/webservice_bug_attachments.t | Updates generator file path and isa checks to Bugzilla::QA::RPC::*. |
| qa/t/webservice_bug_add_comment.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/webservice_bug_add_attachment.t | Updates isa checks to Bugzilla::QA::RPC::* and imports Bugzilla::QA::Util. |
| qa/t/rest_triage_owners.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/rest_see_also.t | Uses Bugzilla::QA::{Util,Tests} and updates use lib. |
| qa/t/rest_search_api.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/rest_reminders.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/rest_relationship_trees.t | Uses Bugzilla::QA::{Util,Tests} and updates use lib. |
| qa/t/rest_pulsebot.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/rest_last_change_time_non_bot.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/rest_lando_uplift.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/rest_group_get.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/rest_github_push_comment.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/rest_github_pull_request.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/rest_components.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/rest_classification.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/rest_bugzilla.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/rest_bug.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/archived/test_email_preferences.t | Uses Bugzilla::QA::Util. |
| qa/t/4_test_webhooks.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/4_test_votes.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/4_test_user_privs.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/4_test_user_preferences.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/4_test_user_matching.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/4_test_user_groups.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/4_test_time_summary.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/4_test_target_milestones.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/4_test_sudo_sessions.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/4_test_status_whiteboard.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/4_test_show_all_products.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/4_test_shared_searches.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/3_test_see_also.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/3_test_security.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/3_test_search.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/3_test_saved_searches.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/3_test_sanity_check.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/3_test_require_login.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/3_test_reminders.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/3_test_rate_limit.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/3_test_qa_contact.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/3_test_private_attachments.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/3_test_new_release.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/3_test_net_attachments.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/3_test_component_graveyard.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/2_test_milestones.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/2_test_login_totp.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/2_test_login_duo.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/2_test_login.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/2_test_lockout.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/2_test_keywords.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/2_test_groups.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/2_test_flags2.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/2_test_flags.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/2_test_enter_new_bug.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/2_test_edit_products_properties.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/2_test_dependencies.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/2_test_default_groups.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/1_test_duo_requirement.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/1_test_custom_fields_admin.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/1_test_custom_fields.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/1_test_create_user_accounts.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/1_test_cookie_consent.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/1_test_config.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/1_test_classifications.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/1_test_choose_priority.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/1_test_bug_edit.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/1_test_bmo_retire_values.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/1_test_bmo_enter_new_bug.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/1_test_bmo_autolinkification.t | Uses Bugzilla::QA::Util and updates use lib. |
| qa/t/1_test_attention.t | Uses Bugzilla::QA::Util and updates use lib. |
| external_test_api.pl | Updates hardcoded selenium config path to /app/conf/selenium_test.conf. |
| extensions/JiraWebhookSync/t/bmo/jira_webhook_sync.t | Uses Bugzilla::QA::Util and updates use lib. |
| extensions/BMO/t/bmo/recorded_future.t | Uses Bugzilla::QA::Util and updates use lib. |
| extensions/BMO/t/bmo/bmo_etl.t | Uses Bugzilla::QA::Util and updates use lib. |
| docker-compose.test.yml | Updates BZ_QA_CONF_FILE to point at /app/conf/selenium_test.conf. |
| conf/selenium_test.conf | Updates attachment fixture path to /app/conf/patch.diff. |
| conf/patch.diff | Adds patch fixture under conf/. |
| Bugzilla/Test/Templates.pm | Renames package to Bugzilla::Test::Templates and updates helper imports. |
| Bugzilla/Test/Systemexec.pm | Renames package to Bugzilla::Test::Systemexec. |
| Bugzilla/Test/Files.pm | Renames package to Bugzilla::Test::Files. |
| Bugzilla/QA/Util.pm | Renames package to Bugzilla::QA::Util, updates config path default, updates RPC requires, and adjusts Selenium helper behavior. |
| Bugzilla/QA/Tests.pm | Renames package to Bugzilla::QA::Tests. |
| Bugzilla/QA/RPC/XMLRPC.pm | Renames package to Bugzilla::QA::RPC::XMLRPC and updates base class. |
| Bugzilla/QA/RPC/JSONRPC.pm | Renames package(s) to Bugzilla::QA::RPC::* and updates class references. |
| Bugzilla/QA/RPC.pm | Renames package to Bugzilla::QA::RPC and updates imports and isa checks. |
| Bugzilla/QA/REST.pm | Renames QA REST client module (but currently has a package naming issue). |
| .github/workflows/ci.yml | Updates CI to pass explicit prove args/patterns for sanity/webservice/selenium/bmo tests. |
Comments suppressed due to low confidence (2)
Bugzilla/QA/REST.pm:12
Bugzilla/QA/REST.pmdeclarespackage Bugzillas::QA::REST;(extra “s”), but the rest of the file referencesBugzilla::QA::REST(EXPORT var, bless). This mismatch will prevent the module from being loaded/used correctly. Rename the package toBugzilla::QA::REST(and ensure all internal package-qualified variables match).
Bugzilla/QA/Util.pm:332go_to_bugremoved the previous delay that was explicitly documented as avoiding a race (“Sometimes we try to click edit bug before it is ready…”). There isn’t an alternative wait here before clickingmode-btn-readonly, so this may reintroduce intermittent Selenium failures. Please add an explicit wait for the edit button to be present/clickable (preferred) or restore the short delay.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
dklawren
changed the title
…ated automated tests from extensions into the normal test framework
* upstream/qa-refactor: (48 commits) Fixed issue with null glob Bug 2019457 - Refactor CI/CD test cases to make it easier to incorporated automated tests from extensions into the normal test framework Bug 2029522 - Modernize CI workflow: docker compose v2 and actions/checkout@v4 Bug 2028222 - Pasting multi-line text after selecting multi-line text does not overwrite, but applies markup for link [skip ci] Bug 2012634 - "Phabricator Revisions" table overflows on X axis on mobile Bug 2023761 - [GITHUB] Allow use of individual api keys for pull requests and push comments instead of single share secret Bumped version to 20260407.1 Bug 2029523 - Selenium tests flake with element click intercepted errors in CI Bug 1934846 - Show both banners if a bug is marked as security sensitive and moco confidential Bug 2029520 - ReadTheDocs webhook has stopped triggering builds since Feb 17 (additional fixes needed) Bug 2029520 - ReadTheDocs webhook has stopped triggering builds since Feb 17 Bug 2029518 - Clean up outdated content and fix errors in README Bug 2029517 - Update CONTRIBUTING.md with current conventions and fix stale links Bug 2028704 - [PHAB-BOT] The feed query has a bug where the last feed id is not being updated properly when a single story fails due to database rollback Bumped version to 20260325.1 Bug 2026225 - PhabBugz: Increase number of feed story retries from 5 to 100 to fix times when the gap may be more than 5 Bug 2024216 - Emoji comment reaction tooltips not visible on old closed bugs when adding reactions is disabled Bumped version to 20260323.3 Bumped version to 20260323.2 Bumped version to 20260323.1 ...
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.