Skip to content

Add URL filtering#72

Open
Voamorim wants to merge 3 commits intomainfrom
urls_filter
Open

Add URL filtering#72
Voamorim wants to merge 3 commits intomainfrom
urls_filter

Conversation

@Voamorim
Copy link

No description provided.

Copy link
Member

@cunha cunha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Main points:

  • Tests don't match expected behavior; we need to define a number of domain and path levels beyond which we start "anonymization"
  • Need tests to cover URL query parameters
  • Need tests to cover special domains and CC TLDs

Copy link
Member

@cunha cunha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Forgot to mention CC TLDs in the previous review.

Copy link
Member

@cunha cunha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another round of comments. I think we should focus on comparing the URL identity keys (after we "erase" subdomains, path levels, and query parameter values). The filter is just taking one URL for each distinct identity key.

Copy link
Member

@cunha cunha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting a few more additional testes around ccTLDs.

Comment on lines +272 to +285
def test_regular_cctld(self, filter_instance):
"""Test URL with regular ccTLD (not special) identity key."""
url = "http://example.uk/page"
result = filter_instance.anonymize_url(url)
# Regular ccTLDs use 3-level domain key (domain + cc tld)
expected = "http://example.uk/page"
assert result == expected

def test_special_cctld(self, filter_instance):
"""Test URL with special ccTLD identity key."""
url = "http://a.b.c.d.example.co/page"
result = filter_instance.anonymize_url(url)
expected = "http://1w.1w.1w.d.example.co/page"
assert result == expected
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These tests need extending. We should test whether actual ccTLDs get handled correctly.

We should give

c.example.co.uk/page1
b.c.example.co.uk/page1
d.c.example.co.uk/page1
a.b.c.example.co.uk/page1

to anonymize_url and check that it returns

c.example.co.uk/page1
1w.c.example.co.uk/page1
1w.c.example.co.uk/page1
1w.1w.c.example.co.uk/page1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants