JSONEncoder's tolist() branch hangs and exhausts memory when a mock reaches a response, instead of raising TypeError

[Macavirus]

This is a follow-up to #7546, which was closed as an unactionable edge case. I hit this while running some tests, memory exploded and my machine ground to a halt.

  import json
  from unittest.mock import Mock
  from rest_framework.utils.encoders import JSONEncoder
  
  json.dumps(Mock(), cls=JSONEncoder)  # never returns, memory grows until OOM

The stdlib json encoder raises TypeError here. DRF's default() sees tolist, calls it, gets another Mock, and repeats forever.

Happens here: https://github.com/encode/django-rest-framework/blob/main/rest_framework/utils/encoders.py#L61-L63

[IlyaKhalafi]

There is a useful distinction from an ordinary circular reference here. An unrestricted Mock invents tolist, and its default return value is another mock. Repeating that conversion creates new objects, so JSON's circular-reference check does not catch the chain.

For the test that triggered this, configure the mocked call to return the actual response-shaped data rather than letting its default mock reach the renderer.

from unittest.mock import Mock

client = Mock()
client.fetch.return_value = {"items": [{"id": 1}]}

# Pass client.fetch() through the same response code as the real result.

If you want to test rejection of an unsupported object, Mock(spec_set=object) raises TypeError with this encoder instead of inventing tolist. The same check passes for MagicMock(spec_set=object).

I checked both the encoder source at 14998d0 and DRF 3.18.3 with Django 6.1.2. With an instrumented encoder that stops after eight conversions, the original Mock() reaches eight distinct objects through the real default() path. The restricted mock raises immediately, and a configured dict return value serializes normally. NumPy array and scalar controls still work too. Both Mock and MagicMock showed that eight-object chain. I did not run the unrestricted reproducer to OOM or establish how that unbounded test would terminate.

This avoids the repeated conversion in these test cases, but does not change DRF's behavior for an unrestricted mock. Any library change would need to preserve genuine tolist() support while rejecting this case.