Skip to content

Commit a7f1426

Browse files
owenpearsonclaude
andauthored
test: derive the type, encoding and presence unit specs
Covers encoding/message_encoding.md, encoding/msgpack_interop.md, presence/rest_presence.md, and the types specs for messages, mutable messages, options, presence messages, tokens, paginated results and errors. The msgpack interop tests read fixtures from ably-common, which the pinned submodule predates, so they are gated on the fixture file being present. Ten tests carry the deviation mark, covering the presence get filters, PresenceMessage#size, href and cause on a deserialised error, the presence attributes a ProtocolMessage would supply, and the endpoint left unset. Three assert what protocol.md contradicts and carry the spec_error mark instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 3175058 commit a7f1426

14 files changed

Lines changed: 3175 additions & 1 deletion

‎test/uts/deviations.md‎

Lines changed: 68 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,15 @@ it pass, and it is marked `@spec_error` — a skip gated on `RUN_DEVIATIONS`, th
2929
stays green, a real regression still shows, and the failure is one environment variable
3030
away. Each is filed upstream, in the issues named below.
3131

32+
The tests gated this way:
33+
34+
| Test | Spec error |
35+
|---|---|
36+
37+
| `test_rsp4_history_pagination` | RSP4 - wire action 4 asserted to be LEAVE |
38+
| `test_tp3_presence_to_json` | TP3 - an outgoing action asserted as the string `"enter"` |
39+
| `test_tp3_null_attributes_omitted` | TP3 - the same outgoing string assertion |
40+
3241
Where instead only a specification's *fixture*, *setup* or *label* is at fault, the
3342
assertion it carries still stands. Those tests keep the corrected fixture (or the
3443
corrected label in a comment), pass, and carry a `# UTS SPEC ERROR:` comment at the
@@ -64,6 +73,44 @@ references in these entries are against `ably/specification@d9a04ca`.
6473
returns a one-element array, which is what `time.md` itself uses and what RSC16
6574
describes. Any SDK that indexes the array raises.
6675

76+
### Spec points are mislabelled across four specs
77+
78+
The assertions are sound; the points they are filed under are not.
79+
80+
| Spec | Filed as | `features.md` says |
81+
|---|---|---|
82+
| `error_types.md` | TI2 = `statusCode`, TI3 = `message`, TI5 = `cause` | TI1 carries every attribute. TI2 is "server errors inherit from ErrorInfo", TI3 the ably-common submodule, TI5 help URLs in log entries. Only TI4 (`href`) is right |
83+
| `token_types.md` | TD1 = token … TD5 = clientId | Every TD label is one short: TD1 is the type, TD2 = token … TD6 = clientId. TE2 and TE4 are swapped, "TE6 - nonce" names the point given to `fromJson`, and there is no TK6 at all |
84+
| `authorize.md` | RSA10e, RSA10g, RSA10h, RSA10i | RSA10e is features RSA10g, RSA10g is RSA10f, RSA10h is RSA10j, and RSA10i maps to nothing |
85+
| `idempotency.md` | RSL1k2 = id format, RSL1k3 = unique base | RSL1k1 is the id format, RSL1k2 client-supplied ids, RSL1k3 mixed batches. RSL1k4 and RSL1k5 are listed with no tests |
86+
87+
The cost is coverage: TI2, TI3 and TI5 *as specified* are untested by the suite.
88+
89+
### Fixtures that cannot hold their stated values
90+
91+
- `RSP5/decode-cipher-channel-7`: the ciphertext is 32 bytes, an IV plus one AES-CBC
92+
block, but the stated plaintext `{"secret":"data"}` is 17 bytes and pads to two
93+
blocks. Decrypting it yields `{"example":{"jso` — a truncated copy of a longer
94+
fixture.
95+
- `RSL6b/unrecognized-encoding-preserved-0`: the payload `"encrypted-data-here"` is
96+
not decodable base64 (17 alphabet characters), yet the test asserts base64 is
97+
decoded and the result is bytes.
98+
- `RSL4/encoding-fixtures-ably-common-0` loads `encoding.json` from ably-common. No
99+
such file exists; RSL6a1 names `messages-encoding.json`, which has a different
100+
schema and runs in the decode direction.
101+
102+
### Presence actions are written as strings
103+
104+
Presence wire fixtures across `rest_presence.md` and `presence_message_types.md`
105+
write `"action": "enter"`. `protocol.md` encodes the action as the enum ordinal, and
106+
the suite's own TP2 asserts ordinals. `presence-to-json-2` is the sharpest case: it
107+
asserts an *outgoing* body carries `"enter"`, which would be invalid on the wire.
108+
109+
`RSP4/history-pagination-1` asserts action `4` is `leave`, contradicting
110+
`RSP4a/history-returns-paginated-1` and `RSP5/presence-action-mapping-8` in the same
111+
file, and the protocol, which fix LEAVE at 3 and UPDATE at 4. The closing note on
112+
`RSP_Action_1` ("3 = leave (some SDKs use 4)") invites the confusion and should go.
113+
67114
### Fallback tests contradict each other and the retry rules
68115

69116
- `RSC15f/successful-fallback-cached-0` queues a response for one named fallback and
@@ -81,13 +128,15 @@ describes. Any SDK that indexes the array raises.
81128

82129
| Spec | Fault |
83130
|---|---|
84-
131+
| `options_types.md` | The `TO/endpoint-affects-host-0` "Expected Rest Host" column uses pre-REC1 hostnames (`rest.ably.io`, `test-rest.ably.io`). No assertion reads the column, so the derived test is unaffected |
85132
| `channels_collection.md` | Header claims RSN3b and RSN3c; neither has a test |
86133
| `stats.md` | Fixture nests counts under `all`, which `Stats.from_dict` never reads |
87134
| `rest_client.md` | `RSC17` has two byte-identical tests; header lists RSC7 and RSC7b with no tests |
88135
| `rest_client.md` | `RSC18` requires constructor-time failure; RSA1/RSC18 only say "any attempt to use" |
89136
| `request.md` | `version` is written as an integer but lands in a header |
90137
| `fallback.md` | REC3a, REC3b and REC3 drive a Realtime client but sit in `rest/unit` |
138+
| `message_encoding.md`, `msgpack_interop.md`, `annotations.md` | Six sections carry no Test ID; ids were inferred by sibling convention |
139+
| `publish.md`, `rest_presence.md`, `message_encoding.md`, `history.md`, `idempotency.md` | All point at `/Users/paddy/data/worknew/dev/dart-experiments/...` for the mock contract |
91140

92141
## Failing Tests
93142

@@ -99,7 +148,18 @@ the mark is the only change needed once the SDK behaviour lands.
99148
| Spec points | Missing | Tests |
100149
|---|---|---|
101150

151+
| RSP3a2, RSP3a3 | `clientId` and `connectionId` filters on `RestPresence#get`. `Presence.get` takes only `limit`, while `Presence.history` does take its documented params | 3 |
152+
| TP5 | `size` on `PresenceMessage`. The related `maxMessageSize` gap is adapted rather than gated, below; `features.md` TM6 has no UTS test | 1 |
153+
102154
| RSC2, RSC3, RSC4, TO3b, TO3c, TO3c2 | `log_handler` as a client option, and any use of `log_level` — it is stored on `Options` and read by nothing | 4 |
155+
| TI4, TI1/TI5 | `href` anywhere in the SDK, and `cause` when deserialising. `AblyException.from_dict` and `raise_for_response` read only `message`, `statusCode` and `code`, so both fields are dropped from server errors | 2 |
156+
| TP3a, TP3d, TP3g | Presence attributes defaulted from the encapsulating ProtocolMessage. There is no ProtocolMessage type; `ably/realtime/channel.py:751-761` passes the presence array through without context. Matters for synthesized-leave detection and `memberKey` | 3 |
157+
158+
### Options
159+
160+
| Spec points | Behaviour |
161+
|---|---|
162+
| TO3 | `endpoint` is left unset when none is given, per `TO/endpoint-affects-host-0`. `Options.__init__` resolves the default eagerly to the REC1a routing policy id `main`, so the attribute never reads back as null. Only the default case is gated; the two that name an endpoint are derived and pass |
103163

104164
### Requests
105165

@@ -119,8 +179,15 @@ comment above. These run, so they guard against regression.
119179
| RSC18 | The constructor rejects basic auth over HTTP | Construction succeeds; 40103 is raised from `make_request` when a request needing Basic Auth is attempted, and no request goes out. RSA1/RSC18 say only "any attempt to use" | Compliant; the UTS is stricter than its source |
120180
| REC1b1, REC1c1 | Code 40000, or a message containing "invalid" or "conflict" | 400/40106 with a specific message. The features spec mandates no code | Cosmetic |
121181

182+
| HP6 | `errorCode` is a number | The raw header string, `'40101'` | Open bug, trivial |
183+
| HP8 | `headers` is a map | A list of `(name, value)` pairs, so the lookup the spec describes is impossible without converting, and case-insensitivity is lost | Open bug; changing the return type is breaking |
122184
| RSC19e | An error indicated idiomatically | `httpx.ConnectError` / `ReadTimeout` reach the caller unwrapped, because `AblyRest.request` carries no `@catch_all` unlike `time()` and `stats()`. The messages do name the failure | Borderline; defensible under RSC19e |
123185
| RSC15a | Six hosts tried | Three. `Options.__get_hosts` truncates to `http_max_retry_count`, which TO3l5 sanctions | Intentional |
186+
| TI | `ErrorInfo` equality by attributes | No `__eq__`, so errors compare by identity. Python exceptions conventionally do, and the requirement appears nowhere in `features.md`. Adding `__eq__` without `__hash__` would make `AblyException` unhashable and break any caller that puts one in a set | Intentional |
187+
| TD5, RSA16a | `capability` is stringified JSON | A `Capability` object, a public convenience type used throughout `auth`. Narrowing the return type to `str` would break every caller that indexes or mutates it, so it is reserved for a future major | Intentional; a breaking change to align |
188+
189+
| TO3l8 | `maxMessageSize` is a client option, default 65536 | Rejected by `Options.__init__`. `ably/realtime/channel.py:422` reads it with `getattr(..., 65536)`, so the default holds but cannot be configured, nor overridden by `connectionDetails` (CD2c) | Open bug |
190+
| TO3l1, TO3l5 | `httpRequestTimeout` and `httpMaxRetryCount` carry their defaults on the options object | Left unset; the effective defaults are applied downstream by `Http` and by `Options.__get_hosts`. The spec's values are milliseconds, while ably-python's `http_request_timeout` is seconds | Intentional |
124191

125192
## Mock Infrastructure Limitations
126193

‎test/uts/rest/unit/encoding/__init__.py‎

Whitespace-only changes.

0 commit comments

Comments
 (0)