Skip to content

Fix comma-separated selected_fields in BigQuery to SQL transfers - #71775

Open
FrankYang0529 wants to merge 1 commit into
apache:mainfrom
FrankYang0529:bigquery-to-sql-normalize-selected-fields
Open

Fix comma-separated selected_fields in BigQuery to SQL transfers#71775
FrankYang0529 wants to merge 1 commit into
apache:mainfrom
FrankYang0529:bigquery-to-sql-normalize-selected-fields

Conversation

@FrankYang0529

@FrankYang0529 FrankYang0529 commented Aug 18, 2026

Copy link
Copy Markdown
Member

Why

  • The docstring says selected_fields is a "List of fields to return (comma-separated)". The operator passes that value to two places: BigQueryHook.list_rows() splits it on commas, but insert_rows(target_fields=...) does not.

  • common.sql builds the column list with ", ".join(map(escape_word, target_fields)), so a string is iterated character by character. With selected_fields="emp_name,salary":

    -- before
    INSERT INTO employees (e, m, p, _, n, a, m, e, ",", s, a, l, a, r, y) VALUES (%s, %s)
    -- after
    INSERT INTO employees (emp_name, salary) VALUES (%s, %s)

How

  • BigQueryToSqlBaseOperator.__init__ normalises a comma-separated string into list[str], so BigQueryToPostgres / BigQueryToMySql / BigQueryToMsSql all get a real column list on both the BigQuery read and the SQL write side.

Verification

  • uv run --project providers/google pytest providers/google/tests/unit/google/cloud/transfers/test_bigquery_to_postgres.py providers/google/tests/unit/google/cloud/transfers/test_bigquery_to_mysql.py -v

Was generative AI tooling used to co-author this PR?
  • Yes - Claude Code

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Signed-off-by: PoAn Yang <payang@apache.org>
@boring-cyborg boring-cyborg Bot added area:providers provider:google Google (including GCP) related issues labels Aug 18, 2026
@FrankYang0529
FrankYang0529 marked this pull request as ready for review August 19, 2026 07:16
@FrankYang0529
FrankYang0529 requested a review from shahar1 as a code owner August 19, 2026 07:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant