Skip to content

Emit metrics related to binlog processing#52

Open
forge33 wants to merge 1 commit into
masterfrom
emit_binlog_event_metrics
Open

Emit metrics related to binlog processing#52
forge33 wants to merge 1 commit into
masterfrom
emit_binlog_event_metrics

Conversation

@forge33

@forge33 forge33 commented Jul 6, 2026

Copy link
Copy Markdown

Related issue: https://github.com/Shopify/schema-migrations/issues/5838

Description

This PR adds metrics around binlog processing so we can see what gh-ost is reading, forwarding, and filtering while streaming row events. Metrics follow similar patterns as backfilter's binlog tailer/reader.

Metric Type Tags When
binlog_events count type:processed, table:<name> Every rows event read from the binlog
binlog_rows count type:processed, table:<name>, binlog_event_type:insert|update|delete Same
binlog_events count type:filtered, table:<name> Rows event on a table that is not the migration table or changelog table
binlog_rows count type:filtered, table:<name>, binlog_event_type:... Same
binlog_events count type:consumed, table:<name>, binlog_event_type:... Rows event on the original table or _ghc changelog table
binlog_rows count type:consumed, table:<name>, binlog_event_type:... Same
binlog_rows_in_event histogram table:<name> Row count per consumed rows event
transaction_num_row_events histogram End of transaction: consumed row events on relevant tables (flush on next GTID, or on XID in file-position mode)
transaction_num_rows histogram End of transaction: logical rows in those consumed events
gtid_event_transaction_length_bytes histogram Each GTID event (TransactionLength, MySQL 8.0.2+)
unfiltered_commit_group_size gauge When LastCommitted changes (SequenceNumber - LastCommitted)
binlog_streamer_blocked_on_out_channel_ms histogram Time blocked sending a consumed rows event to the events channel

In case this PR introduced Go code changes:

  • contributed code is using same conventions as original code
  • script/cibuild returns with no formatting errors, build errors or unit test errors.

@forge33 forge33 requested review from coding-chimp and grodowski July 6, 2026 18:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant