[inchang-ing]

Fixes #8889.

Problem

_scan_number in sqlglot/tokenizer_core.py consumed every _ it met while NUMBERS_CAN_BE_UNDERSCORE_SEPARATED is set (duckdb, clickhouse), so a numeric literal kept going past the point where DuckDB ends it. DuckDB only reads _ as a digit separator between two digits; anywhere else the number ends and the rest starts the implicit alias.

Before, with read="duckdb":

input	sqlglot (before)	DuckDB
SELECT 1__2	SELECT 12	SELECT 1 AS __2
SELECT 1._5	SELECT 1.5	SELECT 1. AS _5
SELECT 1_	SELECT 1	SELECT 1 AS _
SELECT 1e_5	SELECT 1e5	SELECT 1 AS e_5

Fix

Two conditions in _scan_number:

• the underscore branch now requires a digit on both sides (_char and the char after _), instead of accepting any underscore;
• an exponent marker is not consumed when it is directly followed by _, because an exponent has to start with a digit. So 1e_5 ends the number at 1 rather than leaving a dangling 1e token.

Numbers that use the separator normally are unchanged: 1_000 -> 1000, 1_0_0 -> 100.

Verification

Added test_digit_separators to tests/dialects/test_duckdb.py covering the four cases above plus the three well-formed separator forms.

• With the tokenizer change reverted and the new test kept: test_digit_separators fails on SELECT 1__2 ('SELECT 12' != 'SELECT 1 AS __2').
• With the change: tests.dialects.test_duckdb 64 tests OK.
• Full suite SKIP_INTEGRATION=1 python -m unittest: 1568 tests, OK (60 skipped), no new failures.
• ruff check / ruff format --check on both touched files: formatted, and no new findings on the changed lines.

Note

ClickHouse uses the same dialect flag, so it inherits the same rule. I verified the behaviour against DuckDB only, and did not check ClickHouse's own rule for these inputs.

[treysp]

This introduces a ClickHouse regression:

-- Original (returns 7)
SELECT 1e_5foo
FROM (SELECT 7 AS `1e_5foo`)

-- Transpiled (returns 1)
SELECT 1 AS e_5foo
FROM (SELECT 7 AS "1e_5foo")

The new early break bypasses the identifiers_can_start_with_digit handling below, splitting the identifier into a number and alias. Could we preserve that setting and add a Clickhouse regression test?

[inchang-ing]

Good catch — that setting was the one path I missed. Fixed in 68a8d03.

The exponent-marker break now skips dialects where identifiers_can_start_with_digit is set, so that token falls through to the identifier branch again and stays whole:

-- before the fix
SELECT 1 AS e_5foo FROM (SELECT 7 AS "1e_5foo")
-- now
SELECT 1e_5foo FROM (SELECT 7 AS "1e_5foo")

Added test_number_like_identifiers in tests/dialects/test_clickhouse.py covering that statement (plus SELECT 1e_5 and SELECT 1_000, to keep the separator rule covered for ClickHouse). With the guard removed the new case fails with exactly the split above; with it in place the file passes and the full suite is 1465 passed / 22795 subtests.

One consequence worth flagging in the same dialect: SELECT 1e_5 now tokenizes as the identifier 1e_5 rather than as the number 1e5 it used to collapse to, because _ there no longer counts as a separator. That follows from the same setting rather than from the DuckDB rule, but say the word if you'd rather keep the old reading for that shape and I'll special-case it.

[georgesittas]

Thanks!