Skip to content

chore: upgrade etl and adopt memory-based batching #48

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
psteinroe wants to merge 1 commit into
base: main
Choose a base branch
from
Open

chore: upgrade etl and adopt memory-based batching #48

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 100 additions & 3 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 3 additions & 2 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,10 +52,11 @@ tokio-postgres = { git = "https://github.com/MaterializeInc/rust-postgres", defa
tokio-rustls = { version = "0.26.2", default-features = false }
tracing = { version = "0.1.41", default-features = false }
tracing-subscriber = { version = "0.3", default-features = false, features = ["fmt", "env-filter"] }
sysinfo = { version = "0.38.4", default-features = false, features = ["system"] }
x509-cert = { version = "0.2.2", default-features = false }

etl = { git = "https://github.com/supabase/etl", rev = "3122cb1c8df9649e9caaa4ee9d89e1611df29baf" }
etl-postgres = { git = "https://github.com/supabase/etl", rev = "3122cb1c8df9649e9caaa4ee9d89e1611df29baf" }
etl = { git = "https://github.com/supabase/etl", rev = "b47d296cca6ea7bcb5262b627e5c93ef81f8acc2" }
etl-postgres = { git = "https://github.com/supabase/etl", rev = "b47d296cca6ea7bcb5262b627e5c93ef81f8acc2" }
uuid = { version = "1.19.0", default-features = false, features = ["v4"] }

# Optional sink dependencies.
Expand Down
4 changes: 2 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,8 +68,8 @@ stream:
tls:
enabled: false
batch:
max_size: 1000
max_fill_secs: 5
memory_budget_ratio: 0.2
max_fill_ms: 5000

sink:
type: kafka
Expand Down
4 changes: 2 additions & 2 deletions docs/getting-started.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,8 @@ stream:
tls:
enabled: false
batch:
max_size: 1000
max_fill_secs: 5
memory_budget_ratio: 0.2
max_fill_ms: 5000

sink:
type: webhook
Expand Down
4 changes: 2 additions & 2 deletions docs/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,8 +68,8 @@ stream:
tls:
enabled: false
batch:
max_size: 1000
max_fill_secs: 5
memory_budget_ratio: 0.2
max_fill_ms: 5000

sink:
type: kafka
Expand Down
18 changes: 9 additions & 9 deletions docs/reference/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,8 @@ stream:
interval_secs: 10
retries: 3
batch:
max_size: 1000
max_fill_secs: 5
memory_budget_ratio: 0.2
max_fill_ms: 5000

sink:
type: kafka
Expand Down Expand Up @@ -148,25 +148,25 @@ Number of failed probes before the connection is considered dead.

Controls how events are grouped before delivery.

#### `max_size`
#### `memory_budget_ratio`

| | |
|--|--|
| Type | integer |
| Type | float |
| Required | No |
| Default | 1000 |
| Default | 0.2 |

Maximum events per batch. Larger batches improve throughput but increase latency.
Fraction of total process memory reserved for batched stream payloads. Higher values allow larger batches but increase memory usage.

#### `max_fill_secs`
#### `max_fill_ms`

| | |
|--|--|
| Type | integer |
| Required | No |
| Default | 5 |
| Default | 10000 |

Maximum time to fill a batch in seconds. Lower values reduce latency but may result in smaller batches.
Maximum time to fill a batch in milliseconds. Lower values reduce latency but may result in smaller batches.

## Sink Configuration

Expand Down
4 changes: 2 additions & 2 deletions docs/sinks/elasticsearch.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,8 +77,8 @@ stream:
tls:
enabled: false
batch:
max_size: 1000
max_fill_secs: 5
memory_budget_ratio: 0.2
max_fill_ms: 5000

sink:
type: elasticsearch
Expand Down
4 changes: 2 additions & 2 deletions docs/sinks/gcp-pubsub.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,8 +77,8 @@ stream:
tls:
enabled: false
batch:
max_size: 1000
max_fill_secs: 5
memory_budget_ratio: 0.2
max_fill_ms: 5000

sink:
type: gcp-pubsub
Expand Down
4 changes: 2 additions & 2 deletions docs/sinks/kafka.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,8 +71,8 @@ stream:
tls:
enabled: false
batch:
max_size: 1000
max_fill_secs: 5
memory_budget_ratio: 0.2
max_fill_ms: 5000

sink:
type: kafka
Expand Down
4 changes: 2 additions & 2 deletions docs/sinks/kinesis.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,8 +87,8 @@ stream:
tls:
enabled: false
batch:
max_size: 1000
max_fill_secs: 5
memory_budget_ratio: 0.2
max_fill_ms: 5000

sink:
type: kinesis
Expand Down
4 changes: 2 additions & 2 deletions docs/sinks/meilisearch.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,8 +89,8 @@ stream:
tls:
enabled: false
batch:
max_size: 1000
max_fill_secs: 5
memory_budget_ratio: 0.2
max_fill_ms: 5000

sink:
type: meilisearch
Expand Down
4 changes: 2 additions & 2 deletions docs/sinks/nats.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,8 +62,8 @@ stream:
tls:
enabled: false
batch:
max_size: 1000
max_fill_secs: 5
memory_budget_ratio: 0.2
max_fill_ms: 5000

sink:
type: nats
Expand Down
4 changes: 2 additions & 2 deletions docs/sinks/rabbitmq.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,8 +76,8 @@ stream:
tls:
enabled: false
batch:
max_size: 1000
max_fill_secs: 5
memory_budget_ratio: 0.2
max_fill_ms: 5000

sink:
type: rabbitmq
Expand Down
4 changes: 2 additions & 2 deletions docs/sinks/redis-streams.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,8 +75,8 @@ stream:
tls:
enabled: false
batch:
max_size: 1000
max_fill_secs: 5
memory_budget_ratio: 0.2
max_fill_ms: 5000

sink:
type: redis-streams
Expand Down
4 changes: 2 additions & 2 deletions docs/sinks/redis-strings.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,8 +73,8 @@ stream:
tls:
enabled: false
batch:
max_size: 1000
max_fill_secs: 5
memory_budget_ratio: 0.2
max_fill_ms: 5000

sink:
type: redis-strings
Expand Down
4 changes: 2 additions & 2 deletions docs/sinks/sns.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,8 +83,8 @@ stream:
tls:
enabled: false
batch:
max_size: 1000
max_fill_secs: 5
memory_budget_ratio: 0.2
max_fill_ms: 5000

sink:
type: sns
Expand Down
Loading
Loading