testing: adding catalog api and large table fetch benchmark testcase#1561
testing: adding catalog api and large table fetch benchmark testcase#1561Anshu6250 wants to merge 6 commits into
Conversation
Anshu6250
changed the title
| ASSERT_EQ(Connect(connection_string, conn), SQL_SUCCESS) << "Failed to connect to the database."; | ||
|
|
||
| // Generate 90 Lakh (9,000,000) rows by cross-joining two arrays of 3,000 | ||
| std::string query = R"( |
There was a problem hiding this comment.
don't do this, utilise the existing tables like kirltest tables for huge data and all_data_types I think exists for all data types
Anshu6250
force-pushed
the
catalog-api-benchmarks
branch
4 times, most recently
from
15288a7 to
f36e919
Compare
Anshu6250
changed the title
| case SQL_FLOAT: | ||
| case SQL_DOUBLE: col_ptr->data_type = SQL_C_DOUBLE; break; | ||
| case SQL_BIT: col_ptr->data_type = SQL_C_BIT; break; | ||
| default: col_ptr->data_type = SQL_C_CHAR; break; |
There was a problem hiding this comment.
have cases for other data types as well like date, time, numeric, range
| DescribeCol(conn, col_ptr, i); | ||
|
|
||
| switch (col_ptr->data_type) { | ||
| case SQL_BIGINT: |
There was a problem hiding this comment.
same here have cases for other data types as well like date, time, numeric, range
| TEST(DataFetchPerformance, Benchmark_PowerBI_Mimic_NewTimestampTable) { | ||
| auto conn = std::make_shared<ODBCHandles>(); | ||
|
|
||
| std::string connection_string = kDefaultConnectionString + ";AllowHtapiForLargeResults=1;HTAPI_ActivationThreshold=0;"; |
There was a problem hiding this comment.
make all test cases parametrized and run with htapi and without
| DescribeCol(conn, col_ptr, i); | ||
|
|
||
| switch (col_ptr->data_type) { | ||
| case SQL_BIGINT: |
There was a problem hiding this comment.
instead of doing this , use SqlToCdataTypes(col_ptr) instead , as here we are not doing conversion simply returning as string, maybe can have a separate test case to bind with SQL_C_CHAR only
Anshu6250
force-pushed
the
catalog-api-benchmarks
branch
2 times, most recently
from
a21251d to
c73f564
Compare
|
|
@Anshu6250 In this PR, create the cmake target only. We can make the CI changes in the next one. |
| break; | ||
| case SQL_VARCHAR: | ||
| case SQL_CHAR: | ||
| case SQL_NUMERIC: |
There was a problem hiding this comment.
why are we returning SQL_NUMERIC and SQL_DECIMAL as SQL_C_CHAR ?
There was a problem hiding this comment.
We map them to SQL_C_CHAR to prevent precision loss, as BigQuery's exact-precision types support up to 38-76 digits of precision. Standard C++ double types max out at 15-17 digits, so fetching them as strings is the safest way to extract the data without introducing silent truncation or rounding errors
Anshu6250
force-pushed
the
catalog-api-benchmarks
branch
from
ffd8387 to
5de256e
Compare
Anshu6250
force-pushed
the
catalog-api-benchmarks
branch
from
5de256e to
8b5644d
Compare
Anshu6250
force-pushed
the
catalog-api-benchmarks
branch
from
8b5644d to
a94193b
Compare
4 participants
This PR groups all catalog and data fetch benchmarks into a single file (catalog_performance_example.cc) and introduces a custom performance_test CMake target. Now, we can isolate and execute only the performance tests by running
cmake --build build --target performance_test.MacOS and Windows Pipeline: link