NUTCH-2522 Bidirectional URL exemption filter

[okedoki]

No description provided.

[sebastian-nagel]

Clean PR with correct code format and documentation! Thanks, @okedoki!

Afaics, the implementation does the following:

1. take the lowercased host part of both from and to URL
2. match all regex rules defined in the rules files and remove the matched part
3. finally, if from and to host are equal return true => URL is accepted ("exempted" from ignore external host exclusion)

Is this correct?

Wouldn't be a different rule file format more suitable?

• the leading +/- is not used
• don't know whether this makes sense, but could also define the replace string, ev. including references to captured groups, cf. the file format used by urlnormalizer-regex

[sebastian-nagel]

Configuration files should be added as *.template. And are "instantiated" (copied) during the first compilation. Users than can modify the content without conflicts and undesired overwrites.

[sebastian-nagel]

Description does not fit the following line/rule.

[sebastian-nagel]

The description does not match the implementation.

[sebastian-nagel]

Why the rule starts with +-? The dot is not escaped, would also apply to wwwfinder catching wwwf.

[okedoki]

@sebastian-nagel
Thanks for the suggestion to use urlnormalizer-regex. I rewrote the plugin based on this approach( now it makes sense to refactor urlnormalizer-regex and this plugin to use the same code base).

The usage is correct, at this moment we apply the same regex for both input and output url and see if they match each other.

In the future it can be improved with two separated regex for input and output.