Skip to content

fix(StratfordUponAvonCouncil): Add postcode lookup via council API#2057

Open
InertiaUK wants to merge 2 commits into
robbrad:masterfrom
InertiaUK:fix-stratford-postcode-lookup
Open

fix(StratfordUponAvonCouncil): Add postcode lookup via council API#2057
InertiaUK wants to merge 2 commits into
robbrad:masterfrom
InertiaUK:fix-stratford-postcode-lookup

Conversation

@InertiaUK
Copy link
Copy Markdown
Contributor

@InertiaUK InertiaUK commented May 12, 2026

Summary

  • API-resolved UPRNs don't match council's system — calendar POST returns search form instead of results
  • Added postcode search via api.stratford.gov.uk/v1/addresses/postcode/ to resolve correct UPRN before POSTing to calendar
  • Address matching: UPRN → house number → first result

Test

  • Postcode: CV36 4AB
  • UPRN: 100070212698

Summary by CodeRabbit

  • Improvements
    • Enhanced address resolution for Stratford-upon-Avon collections: supports postcode, property number (PAON) and UPRN lookup for more accurate matches.
    • More reliable parsing of collection calendars, extracting bin types and upcoming dates consistently.
  • Bug Fixes
    • Fails fast with clearer validation when a collection schedule or matching address cannot be found.

Review Change Stack

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 12, 2026

Warning

Review limit reached

@InertiaUK, we couldn't start this review because you've used your available PR reviews for now.

Your plan currently allows 2 reviews/hour. Refill in 25 minutes and 52 seconds.

Your organization has run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After more review capacity refills, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than trial, open-source, and free plans. In all cases, review capacity refills continuously over time.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 75184ad2-23d0-40a0-8b0d-c1a1c9ef3182

📥 Commits

Reviewing files that changed from the base of the PR and between 858b8dd and d832621.

📒 Files selected for processing (1)
  • uk_bin_collection/uk_bin_collection/councils/StratfordUponAvonCouncil.py
📝 Walkthrough

Walkthrough

The parser now resolves a UPRN via Stratford’s address API using postcode/paon/uprn, POSTs the resolved UPRN to the calendar endpoint, and parses the returned HTML table to compute the earliest upcoming collection date per bin, failing fast on missing data.

Changes

Waste Collection Data Flow

Layer / File(s) Summary
Inputs, imports, session and address lookup
uk_bin_collection/uk_bin_collection/councils/StratfordUponAvonCouncil.py
Imports normalized; parse_data reads and validates postcode/paon/uprn, constructs address and calendar endpoints, creates a requests.Session with browser headers, and calls the Stratford address API.
Resolve UPRN from API results
uk_bin_collection/uk_bin_collection/councils/StratfordUponAvonCouncil.py
Selects UPRN by exact uprn match if provided; otherwise matches paon against addressLine1/fullAddress (case-insensitive prefix, then substring), falling back to the first returned address.
POST resolved UPRN and parse calendar
uk_bin_collection/uk_bin_collection/councils/StratfordUponAvonCouncil.py
POSTs frmUPRN to the calendar endpoint, verifies response, ensures table.table exists, extracts bin headers, computes earliest upcoming date per bin (ignoring “Not Collected”), formats with date_format, and raises ValueError when table or bins are missing.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

Suggested reviewers

  • dp247

Poem

🐰 I hop through code where postcodes play,
I fetch the UPRN to find the way,
POST and parse the calendar's art,
Dates aligned — each bin takes heart. ✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately describes the main change: adding postcode lookup via the council API to fix UPRN resolution issues in StratfordUponAvonCouncil.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@codecov
Copy link
Copy Markdown

codecov Bot commented May 12, 2026

❌ 1 Tests Failed:

Tests completed Failed Passed Skipped
1 1 0 0
View the top 1 failed test(s) by shortest run time
uk_bin_collection.tests.step_defs.test_validate_council::test_scenario_outline[StratfordUponAvonCouncil]
Stack Traces | 0.281s run time
fixturefunc = <function scrape_step at 0x7fb442c4f6a0>
request = <FixtureRequest for <Function test_scenario_outline[StratfordUponAvonCouncil]>>
kwargs = {'context': <test_validate_council.Context object at 0x7fb443b865d0>, 'headless_mode': 'True', 'local_browser': 'False', 'selenium_url': 'http://localhost:4444'}

    def call_fixture_func(
        fixturefunc: _FixtureFunc[FixtureValue], request: FixtureRequest, kwargs
    ) -> FixtureValue:
        if is_generator(fixturefunc):
            fixturefunc = cast(
                Callable[..., Generator[FixtureValue, None, None]], fixturefunc
            )
            generator = fixturefunc(**kwargs)
            try:
                fixture_result = next(generator)
            except StopIteration:
                raise ValueError(f"{request.fixturename} did not yield a value") from None
            finalizer = functools.partial(_teardown_yield_fixture, fixturefunc, generator)
            request.addfinalizer(finalizer)
        else:
            fixturefunc = cast(Callable[..., FixtureValue], fixturefunc)
>           fixture_result = fixturefunc(**kwargs)

../../../..../pypoetry/virtualenvs/uk-bin-collection-EwS6Gn8s-py3.12/lib/python3.12.../site-packages/_pytest/fixtures.py:898: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../tests/step_defs/test_validate_council.py:101: in scrape_step
    context.parse_result = CollectData.run()
uk_bin_collection/uk_bin_collection/collect_data.py:101: in run
    return self.client_code(
uk_bin_collection/uk_bin_collection/collect_data.py:121: in client_code
    return get_bin_data_class.template_method(address_url, **kwargs)
uk_bin_collection/uk_bin_collection/get_bin_data.py:61: in template_method
    bin_data_dict = self.get_and_parse_data(this_url, **kwargs)
uk_bin_collection/uk_bin_collection/get_bin_data.py:84: in get_and_parse_data
    bin_data_dict = self.parse_data("", url=address_url, **kwargs)
.../uk_bin_collection/councils/StratfordUponAvonCouncil.py:13: in parse_data
    check_postcode(user_postcode)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

postcode = None

    def check_postcode(postcode: str):
        """
        Checks a postcode exists and validates UK formatting against a RegEx string
            :param postcode: Postcode to parse
        """
        postcode_api_url = "https://api.postcodes.io/postcodes/"
        postcode_api_response = requests.get(f"{postcode_api_url}{postcode}")
    
        if postcode_api_response.status_code != 200:
            val_error = json.loads(postcode_api_response.text)
>           raise ValueError(
                f"Exception: {val_error['error']} Status: {val_error['status']}"
            )
E           ValueError: Exception: Invalid postcode Status: 404

uk_bin_collection/uk_bin_collection/common.py:46: ValueError

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Nitpick comments (1)
uk_bin_collection/uk_bin_collection/councils/StratfordUponAvonCouncil.py (1)

31-32: ⚡ Quick win

Add timeout to requests to prevent indefinite hangs.

Both s.get() and s.post() (line 76) lack a timeout parameter. If the server is unresponsive, the request could block indefinitely.

⏱️ Proposed fix to add timeouts
-        r1 = s.get(f"{api_base}/{postcode_stripped}", params=params, verify=False)
+        r1 = s.get(f"{api_base}/{postcode_stripped}", params=params, verify=False, timeout=30)

And similarly for the POST:

-        r2 = s.post(calendar_url, data=payload, verify=False)
+        r2 = s.post(calendar_url, data=payload, verify=False, timeout=30)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@uk_bin_collection/uk_bin_collection/councils/StratfordUponAvonCouncil.py`
around lines 31 - 32, Requests in StratfordUponAvonCouncil.py use s.get(...)
(assigned to r1) and s.post(...) without timeouts; update both calls to include
a sensible timeout (e.g., timeout=10) so they don't hang indefinitely, i.e., add
the timeout keyword to the s.get(...) call that creates r1 and to the
s.post(...) call later (preserving params, data, and verify settings) and
consider centralizing the value as a constant at the top of the module if reuse
is desired.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@uk_bin_collection/uk_bin_collection/councils/StratfordUponAvonCouncil.py`:
- Around line 108-114: The loop over column_headers and collection_info can
silently misalign when their lengths differ; update the zip call in the loop
that iterates over column_headers, collection_info (where next_collection_dates
and date_obj are used) to use zip(..., strict=True) so a mismatch raises
immediately, or if mismatches may be legitimate, wrap that loop in a try/except
to catch the ValueError, log a warning including the offending column_headers
and collection_info, and skip or handle the malformed row so
next_collection_dates is not silently corrupted.
- Around line 10-13: The code calls check_postcode(user_postcode) and later uses
user_postcode.replace(...) which will raise AttributeError when user_postcode is
None; update StratfordUponAvonCouncil logic to explicitly validate user_postcode
before using it: ensure user_postcode is not None (e.g., if not user_postcode:
raise ValueError or return early) or guard the call to check_postcode so you
only call check_postcode and subsequent user_postcode.replace(...) when
user_postcode is provided; refer to the variables user_postcode and the function
check_postcode to locate and change the flow.
- Around line 42-64: The code silently falls back to the first address when a
user-supplied UPRN (user_uprn) doesn't match any address in addresses; change
this to fail explicitly: after the loop that attempts to match user_uprn (the
block that sets selected_uprn when str(addr.get("uprn")) == str(user_uprn)), if
user_uprn was provided and selected_uprn is still unset, raise a clear exception
(or return an explicit error) indicating "UPRN not found" instead of assigning
addresses[0]["uprn"]; keep the existing PAON matching logic unchanged for cases
where user_uprn is not provided.

---

Nitpick comments:
In `@uk_bin_collection/uk_bin_collection/councils/StratfordUponAvonCouncil.py`:
- Around line 31-32: Requests in StratfordUponAvonCouncil.py use s.get(...)
(assigned to r1) and s.post(...) without timeouts; update both calls to include
a sensible timeout (e.g., timeout=10) so they don't hang indefinitely, i.e., add
the timeout keyword to the s.get(...) call that creates r1 and to the
s.post(...) call later (preserving params, data, and verify settings) and
consider centralizing the value as a constant at the top of the module if reuse
is desired.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: fbe20f84-2c9b-4043-9e49-8ed8b635bf35

📥 Commits

Reviewing files that changed from the base of the PR and between 8ecf878 and fe928be.

📒 Files selected for processing (1)
  • uk_bin_collection/uk_bin_collection/councils/StratfordUponAvonCouncil.py

Comment thread uk_bin_collection/uk_bin_collection/councils/StratfordUponAvonCouncil.py Outdated
@InertiaUK InertiaUK force-pushed the fix-stratford-postcode-lookup branch from fe928be to 858b8dd Compare May 12, 2026 15:24
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@uk_bin_collection/uk_bin_collection/councils/StratfordUponAvonCouncil.py`:
- Line 31: The GET calls use verify=False (seen in the s.get call assigned to r1
and the second similar call), which will emit InsecureRequestWarning; either
re-enable SSL verification or, if you must keep verify=False, suppress warnings
once by disabling urllib3.exceptions.InsecureRequestWarning before performing
these requests (or set the session to ignore warnings), and ensure suppression
is applied where r1 is created and the second API call around line 76 so users
are not spammed with warnings.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: c99fb70a-1f0f-4878-8a7e-db02d1e1670a

📥 Commits

Reviewing files that changed from the base of the PR and between fe928be and 858b8dd.

📒 Files selected for processing (1)
  • uk_bin_collection/uk_bin_collection/councils/StratfordUponAvonCouncil.py

Comment thread uk_bin_collection/uk_bin_collection/councils/StratfordUponAvonCouncil.py Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant