Feature Summary
Opening a new thread since I can't reopen the old discussion here #4074
My question is: how can I write a function in a 2-port UDF to access input data at the same time?
For example, if data from port 0 is df0, and data from port 1 is df1, how should I design my code so a function f(df0, df1) could be executed properly?
Attached here is an example in my use case, where port 0 is a df of 19.5k lines and ~200 columns (say df0, while data from port 1 is smaller with 63 lines (say df1). AI bot suggested that we put a parameter with a fix number of lines matching with n lines in df0 to check if df0 has finished loading, which in general cases would not work.
This is other things I have tried (in pseudocode):
- Attempt 1:
if port == 0:
df0 <- df0
else if port == 1:
df1 <- df1
#a function that does something with both df0 and df1
function_to_called(df0, df1)
This will not work because df1 will finish loading before df0, and function_to_called will throw an error that df0 is not found.
2) Attempt 2:
Same logic as above, but I forced port 1 data to depend on port 0 so port 0 finishes first (as recommended in #4074). Now function_to_called will throw an error that df1 is not found, because it's called before df1 is loaded.
The attached screenshot is in R, but the question stands the same for Python-based UDF.
Proposed Solution or Design
I expect a way to check if data has finished loading.
Affected Area
No response
Feature Summary
Opening a new thread since I can't reopen the old discussion here #4074
My question is: how can I write a function in a 2-port UDF to access input data at the same time?
For example, if data from
port 0isdf0, and data fromport 1isdf1, how should I design my code so a functionf(df0, df1)could be executed properly?Attached here is an example in my use case, where
port 0is a df of 19.5k lines and ~200 columns (saydf0, while data fromport 1is smaller with 63 lines (saydf1). AI bot suggested that we put a parameter with a fix number of lines matching with n lines indf0to check ifdf0has finished loading, which in general cases would not work.This is other things I have tried (in pseudocode):
This will not work because
df1will finish loading beforedf0, andfunction_to_calledwill throw an error thatdf0is not found.2) Attempt 2:
Same logic as above, but I forced
port 1data to depend onport 0soport 0finishes first (as recommended in #4074). Nowfunction_to_calledwill throw an error thatdf1is not found, because it's called beforedf1is loaded.The attached screenshot is in R, but the question stands the same for Python-based UDF.
Proposed Solution or Design
I expect a way to check if data has finished loading.
Affected Area
No response