feat: add multi-version Flink module structure (1.17/1.18/1.19/1.20)#17
Open
fightBoxing wants to merge 5 commits into
Open
feat: add multi-version Flink module structure (1.17/1.18/1.19/1.20)#17fightBoxing wants to merge 5 commits into
fightBoxing wants to merge 5 commits into
Conversation
This commit introduces a new namespace abstraction layer for Lance catalog integration with Flink. Key components added: - AbstractLanceNamespaceAdapter: Interface defining namespace operations - LanceNamespaceAdapter: Implementation with direct Lance Namespace SDK API calls - LanceNamespaceConfig: Type-safe configuration management with ImplType enum - BaseLanceNamespaceCatalog: Base catalog implementation for Flink integration - LanceNamespaceAdapterITCase: Comprehensive integration tests (23 test cases) - MockLanceNamespace: Mock implementation for standalone testing Features: - Direct API calls to Lance Namespace SDK (no reflection) - Support for both directory and REST namespace implementations - Complete CRUD operations for namespaces and tables - Full metadata management - Production-ready error handling and resource management
- Add BaseLanceNamespaceCatalog with full Flink Catalog API support - Add LanceCatalogFactory for creating namespace adapters - Refactor LanceNamespaceAdapter with improved CRUD operations - Update LanceNamespaceConfig with extra_level and parent support
…e backend - Remove MockLanceNamespace as lance-core natively supports DirectoryNamespace and RestNamespace - Update LanceNamespaceAdapterITCase to use real DirectoryNamespace backend for testing - Add nested RestNamespaceTests enabled via LANCE_REST_URI environment variable - DirectoryNamespace is used for storage tests, RestNamespace for API tests when available
…raction layer - Remove AbstractLanceNamespaceAdapter interface - Let BaseLanceNamespaceCatalog use LanceNamespace API directly - Simplify LanceNamespaceAdapter to only handle connection init and resource management - Translate all Chinese comments to English - Update tests accordingly
- Convert to multi-module Maven project with version-specific modules - Each module (lance-flink-1.17/1.18/1.19/1.20) compiles shared src/ with its Flink version - Use build-helper-maven-plugin to reference shared source and resources - All modules compile with Java 8 target (consistent with Flink official builds) - Add maven-enforcer-plugin for Maven/JDK version and dependency checks - Add spotless-maven-plugin for code formatting and license header checks - Remove unused TableSchema import in LanceDynamicTableSource - Remove duplicate SPI files from version modules (already referenced via build-helper) - Remove unnecessary lance-flink-base module (code already in shared src/) - Remove JDK-based Maven profiles (all modules now build uniformly)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Convert the project to a multi-module Maven structure supporting Flink versions 1.17, 1.18, 1.19, and 1.20.
Changes
Multi-Module Structure
lance-flink-1.17,lance-flink-1.18,lance-flink-1.19,lance-flink-1.20src/directory with its corresponding Flink versionbuild-helper-maven-pluginto reference shared source and resource directoriesBuild Configuration
maven-enforcer-pluginfor Maven/JDK version and dependency convergence checksspotless-maven-pluginfor code formatting and Apache license header checksCode Cleanup
TableSchemaimport inLanceDynamicTableSourcelance-flink-basemodule (code already in sharedsrc/)How to Build