Syncing with anchors

[stefan-balke]

Hi there,

I am currently syncing with anchors and have here and there errors.
This thread will be a collection of "problems" I encountered.
Maybe we can use this then as a basis to improve the codebase.

• __check_anchor_pairs could be run for all anchors in advance, at the moment it makes a test for each pair, i.e., you sync runs for a while and breaks then...often near the end :-) (it is called before but somehow the loop runs until it breaks...numba involved?)
• same function if a1 > f_len1 / feature_rate or a2 > f_len2 / feature_rate: break in two cases, also the error message should not say audio files since both sequences can come from "anything" (e.g., score)
• explanation of leaving out the (0, 0), (-1, -1) could use some makeover, also it says audio here

To be continued while I work with it.

I guess a running example with anchor points could help.

Steps could be:

• DTW
• DTW + Start-End-Times
• DTW + Anchors
• DTW + Anchors + linear_inp_ind (basically degenerating to region-wise linear sync)

Best
Stefan

[yiitozer]

@stefan-balke Thanks for bringing these up, I am going to check this point today:

__check_anchor_pairs could be run for all anchors in advance, at the moment it makes a test for each pair, i.e., you sync runs for a while and breaks then...often near the end :-) (it is called before but somehow the loop runs until it breaks...numba involved?)

All your other suggestions sound feasible to me.

[yiitozer]

@stefan-balke , @Curucail, @meinardmueller

Regarding the point:

DTW + Start-End-Times

Over the years, several users have reported that manually cropping silences at the beginning and end of audio files is quite tedious. To address this, I was considering to add an optional silence detection step as part of the processing pipeline, e.g., librosa.effects.split or pydub.detect_silence. This could be exposed as an optional argument (e.g., strip_silence=True), so existing behavior is preserved by default.
What are your thoughts?

[stefan-balke]

Silence detection has been a problem since the beginning of MIR.
One could add it as an optional parameter, however, I personally would not rely on this.
Furthermore, especially in syncing, silence is not the only issue, more problematic are things like applause.
However, when you have a "well-behaved" studio recording, things might just work.

[yiitozer]

I see your point, I think we can leave this out of the library then. Since the boundary conditions are too varied (applause, noise, etc.) to handle reliably in the general case, and users with well-behaved studio recordings can easily implement a workaround on their end.

[yiitozer]

@stefan-balke Thanks for bringing these up, I am going to check this point today:

__check_anchor_pairs could be run for all anchors in advance, at the moment it makes a test for each pair, i.e., you sync runs for a while and breaks then...often near the end :-) (it is called before but somehow the loop runs until it breaks...numba involved?)

All your other suggestions sound feasible to me.

@stefan-balke I looked into the __check_anchor_pairs issue today but wasn't able to reproduce the error on my end. Do you think that this is related to numba, I didn't quite understand it.