Skip to content

chore: Use native impl of elt function #5040

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kazantsev-maksim wants to merge 77 commits into
base: main
Choose a base branch
from
Open

chore: Use native impl of elt function #5040

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
77 commits
Select commit Hold shift + click to select a range
768b3e9
impl map_from_entries
Dec 14, 2025
c68c342
Revert "impl map_from_entries"
Dec 16, 2025
d887555
Merge branch 'apache:main' into main
kazantsev-maksim Dec 16, 2025
231aa90
Merge branch 'apache:main' into main
kazantsev-maksim Dec 17, 2025
9500bbb
Merge branch 'apache:main' into main
kazantsev-maksim Dec 24, 2025
9577481
Merge branch 'apache:main' into main
kazantsev-maksim Dec 28, 2025
3791557
Merge branch 'apache:main' into main
kazantsev-maksim Jan 2, 2026
7c2f082
Merge branch 'apache:main' into main
kazantsev-maksim Jan 3, 2026
609a605
Merge branch 'apache:main' into main
kazantsev-maksim Jan 6, 2026
a151b2c
Merge branch 'apache:main' into main
kazantsev-maksim Jan 7, 2026
ad3e7f5
Merge branch 'apache:main' into main
kazantsev-maksim Jan 10, 2026
ea92e4b
Merge branch 'apache:main' into main
kazantsev-maksim Jan 14, 2026
8dfeca3
Merge branch 'apache:main' into main
kazantsev-maksim Jan 17, 2026
559741e
Merge branch 'apache:main' into main
kazantsev-maksim Jan 20, 2026
ebda14e
Merge branch 'apache:main' into main
kazantsev-maksim Jan 21, 2026
408152e
Merge branch 'apache:main' into main
kazantsev-maksim Jan 23, 2026
d7857b2
Merge branch 'apache:main' into main
kazantsev-maksim Jan 24, 2026
aef41be
Merge branch 'apache:main' into main
kazantsev-maksim Jan 29, 2026
5ac1c58
Merge branch 'apache:main' into main
kazantsev-maksim Jan 30, 2026
9ae8e23
Merge branch 'apache:main' into main
kazantsev-maksim Feb 1, 2026
5ca3888
Merge branch 'apache:main' into main
kazantsev-maksim Feb 4, 2026
160a817
Merge branch 'apache:main' into main
kazantsev-maksim Feb 5, 2026
88fc313
Merge branch 'apache:main' into main
kazantsev-maksim Feb 7, 2026
e14c180
Merge branch 'apache:main' into main
kazantsev-maksim Feb 13, 2026
610a885
Merge branch 'apache:main' into main
kazantsev-maksim Feb 20, 2026
f8acb2c
Merge branch 'apache:main' into main
kazantsev-maksim Feb 21, 2026
ec94897
Merge branch 'apache:main' into main
kazantsev-maksim Feb 26, 2026
43405e4
Merge branch 'apache:main' into main
kazantsev-maksim Feb 27, 2026
47b4915
Merge branch 'apache:main' into main
kazantsev-maksim Mar 1, 2026
26e2682
Merge branch 'apache:main' into main
kazantsev-maksim Mar 3, 2026
6cb5f07
Merge branch 'apache:main' into main
kazantsev-maksim Mar 4, 2026
ec194fb
Merge branch 'apache:main' into main
kazantsev-maksim Mar 31, 2026
256fccb
Merge branch 'apache:main' into main
kazantsev-maksim Apr 3, 2026
912c8f9
Merge branch 'apache:main' into main
kazantsev-maksim Apr 3, 2026
561a664
Merge branch 'apache:main' into main
kazantsev-maksim Apr 8, 2026
d926ef4
Merge branch 'apache:main' into main
kazantsev-maksim Apr 11, 2026
671412c
Merge branch 'apache:main' into main
kazantsev-maksim Apr 17, 2026
c9f52d1
Merge branch 'apache:main' into main
kazantsev-maksim Apr 22, 2026
67f72d9
Merge branch 'apache:main' into main
kazantsev-maksim Apr 23, 2026
314e594
Merge branch 'apache:main' into main
kazantsev-maksim Apr 24, 2026
ac8292f
Merge branch 'apache:main' into main
kazantsev-maksim May 1, 2026
c9c140e
Merge branch 'apache:main' into main
kazantsev-maksim May 7, 2026
decca58
Merge branch 'apache:main' into main
kazantsev-maksim May 13, 2026
0919b33
Merge branch 'apache:main' into main
kazantsev-maksim May 16, 2026
7495e21
Merge branch 'apache:main' into main
kazantsev-maksim May 19, 2026
0a37a60
Merge branch 'apache:main' into main
kazantsev-maksim May 21, 2026
abbba84
Merge branch 'apache:main' into main
kazantsev-maksim May 25, 2026
6020560
Merge branch 'apache:main' into main
kazantsev-maksim May 28, 2026
e2bdfb1
Merge branch 'apache:main' into main
kazantsev-maksim May 31, 2026
3edfc33
Merge branch 'apache:main' into main
kazantsev-maksim Jun 3, 2026
a39e860
Merge branch 'apache:main' into main
kazantsev-maksim Jun 4, 2026
e88dd7b
Merge branch 'apache:main' into main
kazantsev-maksim Jun 5, 2026
3e29d37
Merge branch 'apache:main' into main
kazantsev-maksim Jun 7, 2026
4068359
Merge branch 'apache:main' into main
kazantsev-maksim Jun 12, 2026
a3cb8de
Merge branch 'apache:main' into main
kazantsev-maksim Jun 13, 2026
b33726f
Merge branch 'apache:main' into main
kazantsev-maksim Jun 21, 2026
698f7a1
Merge branch 'apache:main' into main
kazantsev-maksim Jun 22, 2026
18162a6
Merge branch 'apache:main' into main
kazantsev-maksim Jun 23, 2026
6f6eb6f
Merge branch 'apache:main' into main
kazantsev-maksim Jul 1, 2026
c21a42e
Merge branch 'apache:main' into main
kazantsev-maksim Jul 2, 2026
618ae48
Merge branch 'apache:main' into main
kazantsev-maksim Jul 3, 2026
4d068e3
Merge branch 'apache:main' into main
kazantsev-maksim Jul 3, 2026
a2f519f
Merge branch 'apache:main' into main
kazantsev-maksim Jul 4, 2026
36e13d5
Merge branch 'apache:main' into main
kazantsev-maksim Jul 7, 2026
2c8ae52
Merge branch 'apache:main' into main
kazantsev-maksim Jul 8, 2026
593f7b6
Merge branch 'apache:main' into main
kazantsev-maksim Jul 11, 2026
b1d3a1a
Merge branch 'apache:main' into main
kazantsev-maksim Jul 14, 2026
e2de8c0
Merge branch 'apache:main' into main
kazantsev-maksim Jul 26, 2026
c3fdf28
chore: use native impl of elt function
Jul 26, 2026
31cde1d
chore: use native impl of elt function
Jul 26, 2026
e6fd376
Merge branch 'apache:main' into main
kazantsev-maksim Aug 1, 2026
96e42aa
fix PR issues
Aug 1, 2026
c1831c5
fix PR issues
Aug 1, 2026
8180acb
docs
Aug 1, 2026
11528e3
Merge branch 'apache:main' into main
kazantsev-maksim Aug 1, 2026
136349a
Merge remote-tracking branch 'origin/main' into native_elt_func
Aug 1, 2026
32dbee2
docs
Aug 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
136 changes: 68 additions & 68 deletions docs/source/user-guide/latest/expressions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -546,74 +546,74 @@ expression-level). The `outer` variants are wired but marked `Incompatible`; the

## string_funcs

| Function | Status | Implementation | Notes |
| --- | --- | --- | --- |
| `ascii` | ✅ | Native | |
| `base64` | ✅ | Native | |
| `bit_length` | ✅ | Native | |
| `btrim` | ✅ | — | |
| `char` | ✅ | Native | |
| `char_length` | ✅ | Native | |
| `character_length` | ✅ | Native | |
| `chr` | ✅ | Native | |
| `collate` | 🔜 | — | Spark collation (umbrella [#2190](https://github.com/apache/datafusion-comet/issues/2190)) |
| `collation` | ✅ | — | Constant-folded to a literal (Spark 4.0+) |
| `concat_ws` | ✅ | Native | |
| `contains` | ✅ | — | |
| `decode` | ✅ | — | |
| `elt` | ✅ | Codegen dispatch | |
| `encode` | 🔜 | — | Lowers to `StaticInvoke(encode)` (not allowlisted); falls back |
| `endswith` | ✅ | — | |
| `find_in_set` | ✅ | Codegen dispatch | |
| `format_number` | ✅ | Codegen dispatch | |
| `format_string` | ✅ | Codegen dispatch | |
| `initcap` | ✅ | Hybrid | |
| `instr` | ✅ | Native | |
| `lcase` | ✅ | Hybrid | |
| `left` | ✅ | Native | |
| `len` | ✅ | Native | |
| `length` | ✅ | Native | |
| `levenshtein` | ✅ | Native | |
| `locate` | ✅ | Codegen dispatch | |
| `lower` | ✅ | Hybrid | |
| `lpad` | ✅ | — | |
| `ltrim` | ✅ | Native | |
| `luhn_check` | ✅ | — | Native via `StaticInvoke` (tests: luhn_check.sql) |
| `mask` | ✅ | — | Routed through the JVM codegen dispatcher |
| `octet_length` | ✅ | Native | |
| `overlay` | ✅ | Codegen dispatch | |
| `position` | ✅ | Codegen dispatch | |
| `printf` | ✅ | Codegen dispatch | |
| `regexp_count` | ✅ | — | Runs natively (rewrites to `size(regexp_extract_all(...))`) |
| `regexp_extract` | ✅ | Native | |
| `regexp_extract_all` | ✅ | Native | |
| `regexp_instr` | ✅ | Codegen dispatch | Routed through the JVM codegen dispatcher |
| `regexp_replace` | ✅ | Hybrid | |
| `regexp_substr` | ✅ | — | Runs natively (rewrites to `nullif(regexp_extract(...), '')`) |
| `repeat` | ✅ | Native | |
| `replace` | ✅ | Hybrid | |
| `right` | ✅ | Native | |
| `rpad` | ✅ | — | |
| `rtrim` | ✅ | Native | |
| `soundex` | ✅ | Native | |
| `space` | ✅ | Native | |
| `split` | ✅ | Hybrid | |
| `split_part` | ✅ | — | Spark 4.0+ |
| `startswith` | ✅ | — | |
| `substr` | ✅ | Native | |
| `substring` | ✅ | Native | |
| `substring_index` | ✅ | Native | |
| `to_binary` | ✅ | — | Hex form accelerated; other formats fall back |
| `to_char` | ✅ | Codegen dispatch | |
| `to_number` | ✅ | Codegen dispatch | |
| `to_varchar` | ✅ | Codegen dispatch | |
| `translate` | ✅ | Native | DataFusion's `translate` iterates over Unicode graphemes (Spark uses code points) and substitutes U+0000 instead of treating it as a deletion sentinel, so the native path is opt-in via allowIncompatible |
| `trim` | ✅ | Native | |
| `try_to_binary` | ✅ | — | Runs natively (rewrites to `try_eval(to_binary(...))`) |
| `try_to_number` | ✅ | Codegen dispatch | Routed through the JVM codegen dispatcher |
| `ucase` | ✅ | Hybrid | |
| `unbase64` | ✅ | Codegen dispatch | |
| `upper` | ✅ | Hybrid | |
| Function | Status | Implementation | Notes |
| --- | --- |------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `ascii` | ✅ | Native | |
| `base64` | ✅ | Native | |
| `bit_length` | ✅ | Native | |
| `btrim` | ✅ | — | |
| `char` | ✅ | Native | |
| `char_length` | ✅ | Native | |
| `character_length` | ✅ | Native | |
| `chr` | ✅ | Native | |
| `collate` | 🔜 | — | Spark collation (umbrella [#2190](https://github.com/apache/datafusion-comet/issues/2190)) |
| `collation` | ✅ | — | Constant-folded to a literal (Spark 4.0+) |
| `concat_ws` | ✅ | Native | |
| `contains` | ✅ | — | |
| `decode` | ✅ | — | |
| `elt` | ✅ | Hybrid | Codegen dispatch when spark.sql.ansi.enabled=true |
| `encode` | 🔜 | — | Lowers to `StaticInvoke(encode)` (not allowlisted); falls back |
| `endswith` | ✅ | — | |
| `find_in_set` | ✅ | Codegen dispatch | |
| `format_number` | ✅ | Codegen dispatch | |
| `format_string` | ✅ | Codegen dispatch | |
| `initcap` | ✅ | Hybrid | |
| `instr` | ✅ | Native | |
| `lcase` | ✅ | Hybrid | |
| `left` | ✅ | Native | |
| `len` | ✅ | Native | |
| `length` | ✅ | Native | |
| `levenshtein` | ✅ | Native | |
| `locate` | ✅ | Codegen dispatch | |
| `lower` | ✅ | Hybrid | |
| `lpad` | ✅ | — | |
| `ltrim` | ✅ | Native | |
| `luhn_check` | ✅ | — | Native via `StaticInvoke` (tests: luhn_check.sql) |
| `mask` | ✅ | — | Routed through the JVM codegen dispatcher |
| `octet_length` | ✅ | Native | |
| `overlay` | ✅ | Codegen dispatch | |
| `position` | ✅ | Codegen dispatch | |
| `printf` | ✅ | Codegen dispatch | |
| `regexp_count` | ✅ | — | Runs natively (rewrites to `size(regexp_extract_all(...))`) |
| `regexp_extract` | ✅ | Native | |
| `regexp_extract_all` | ✅ | Native | |
| `regexp_instr` | ✅ | Codegen dispatch | Routed through the JVM codegen dispatcher |
| `regexp_replace` | ✅ | Hybrid | |
| `regexp_substr` | ✅ | — | Runs natively (rewrites to `nullif(regexp_extract(...), '')`) |
| `repeat` | ✅ | Native | |
| `replace` | ✅ | Hybrid | |
| `right` | ✅ | Native | |
| `rpad` | ✅ | — | |
| `rtrim` | ✅ | Native | |
| `soundex` | ✅ | Native | |
| `space` | ✅ | Native | |
| `split` | ✅ | Hybrid | |
| `split_part` | ✅ | — | Spark 4.0+ |
| `startswith` | ✅ | — | |
| `substr` | ✅ | Native | |
| `substring` | ✅ | Native | |
| `substring_index` | ✅ | Native | |
| `to_binary` | ✅ | — | Hex form accelerated; other formats fall back |
| `to_char` | ✅ | Codegen dispatch | |
| `to_number` | ✅ | Codegen dispatch | |
| `to_varchar` | ✅ | Codegen dispatch | |
| `translate` | ✅ | Native | DataFusion's `translate` iterates over Unicode graphemes (Spark uses code points) and substitutes U+0000 instead of treating it as a deletion sentinel, so the native path is opt-in via allowIncompatible |
| `trim` | ✅ | Native | |
| `try_to_binary` | ✅ | — | Runs natively (rewrites to `try_eval(to_binary(...))`) |
| `try_to_number` | ✅ | Codegen dispatch | Routed through the JVM codegen dispatcher |
| `ucase` | ✅ | Hybrid | |
| `unbase64` | ✅ | Codegen dispatch | |
| `upper` | ✅ | Hybrid | |

---

Expand Down
Loading