You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit a7f1426
Browse filesBrowse the repository at this point in the historyBrowse files
test: derive the type, encoding and presence unit specs
Covers encoding/message_encoding.md, encoding/msgpack_interop.md,
presence/rest_presence.md, and the types specs for messages, mutable
messages, options, presence messages, tokens, paginated results and errors.
The msgpack interop tests read fixtures from ably-common, which the pinned
submodule predates, so they are gated on the fixture file being present.
Ten tests carry the deviation mark, covering the presence get filters,
PresenceMessage#size, href and cause on a deserialised error, the presence
attributes a ProtocolMessage would supply, and the endpoint left unset.
Three assert what protocol.md contradicts and carry the spec_error mark
instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: test/uts/deviations.md
+68-1Lines changed: 68 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -29,6 +29,15 @@ it pass, and it is marked `@spec_error` — a skip gated on `RUN_DEVIATIONS`, th
29
29
stays green, a real regression still shows, and the failure is one environment variable
30
30
away. Each is filed upstream, in the issues named below.
31
31
32
+
The tests gated this way:
33
+
34
+
| Test | Spec error |
35
+
|---|---|
36
+
37
+
|`test_rsp4_history_pagination`| RSP4 - wire action 4 asserted to be LEAVE |
38
+
|`test_tp3_presence_to_json`| TP3 - an outgoing action asserted as the string `"enter"`|
39
+
|`test_tp3_null_attributes_omitted`| TP3 - the same outgoing string assertion |
40
+
32
41
Where instead only a specification's *fixture*, *setup* or *label* is at fault, the
33
42
assertion it carries still stands. Those tests keep the corrected fixture (or the
34
43
corrected label in a comment), pass, and carry a `# UTS SPEC ERROR:` comment at the
@@ -64,6 +73,44 @@ references in these entries are against `ably/specification@d9a04ca`.
64
73
returns a one-element array, which is what `time.md` itself uses and what RSC16
65
74
describes. Any SDK that indexes the array raises.
66
75
76
+
### Spec points are mislabelled across four specs
77
+
78
+
The assertions are sound; the points they are filed under are not.
79
+
80
+
| Spec | Filed as |`features.md` says |
81
+
|---|---|---|
82
+
|`error_types.md`| TI2 = `statusCode`, TI3 = `message`, TI5 = `cause`| TI1 carries every attribute. TI2 is "server errors inherit from ErrorInfo", TI3 the ably-common submodule, TI5 help URLs in log entries. Only TI4 (`href`) is right |
83
+
|`token_types.md`| TD1 = token … TD5 = clientId | Every TD label is one short: TD1 is the type, TD2 = token … TD6 = clientId. TE2 and TE4 are swapped, "TE6 - nonce" names the point given to `fromJson`, and there is no TK6 at all |
84
+
|`authorize.md`| RSA10e, RSA10g, RSA10h, RSA10i | RSA10e is features RSA10g, RSA10g is RSA10f, RSA10h is RSA10j, and RSA10i maps to nothing |
85
+
|`idempotency.md`| RSL1k2 = id format, RSL1k3 = unique base | RSL1k1 is the id format, RSL1k2 client-supplied ids, RSL1k3 mixed batches. RSL1k4 and RSL1k5 are listed with no tests |
86
+
87
+
The cost is coverage: TI2, TI3 and TI5 *as specified* are untested by the suite.
88
+
89
+
### Fixtures that cannot hold their stated values
90
+
91
+
-`RSP5/decode-cipher-channel-7`: the ciphertext is 32 bytes, an IV plus one AES-CBC
92
+
block, but the stated plaintext `{"secret":"data"}` is 17 bytes and pads to two
93
+
blocks. Decrypting it yields `{"example":{"jso` — a truncated copy of a longer
94
+
fixture.
95
+
-`RSL6b/unrecognized-encoding-preserved-0`: the payload `"encrypted-data-here"` is
96
+
not decodable base64 (17 alphabet characters), yet the test asserts base64 is
97
+
decoded and the result is bytes.
98
+
-`RSL4/encoding-fixtures-ably-common-0` loads `encoding.json` from ably-common. No
99
+
such file exists; RSL6a1 names `messages-encoding.json`, which has a different
100
+
schema and runs in the decode direction.
101
+
102
+
### Presence actions are written as strings
103
+
104
+
Presence wire fixtures across `rest_presence.md` and `presence_message_types.md`
105
+
write `"action": "enter"`. `protocol.md` encodes the action as the enum ordinal, and
106
+
the suite's own TP2 asserts ordinals. `presence-to-json-2` is the sharpest case: it
107
+
asserts an *outgoing* body carries `"enter"`, which would be invalid on the wire.
108
+
109
+
`RSP4/history-pagination-1` asserts action `4` is `leave`, contradicting
110
+
`RSP4a/history-returns-paginated-1` and `RSP5/presence-action-mapping-8` in the same
111
+
file, and the protocol, which fix LEAVE at 3 and UPDATE at 4. The closing note on
112
+
`RSP_Action_1` ("3 = leave (some SDKs use 4)") invites the confusion and should go.
113
+
67
114
### Fallback tests contradict each other and the retry rules
68
115
69
116
-`RSC15f/successful-fallback-cached-0` queues a response for one named fallback and
@@ -81,13 +128,15 @@ describes. Any SDK that indexes the array raises.
81
128
82
129
| Spec | Fault |
83
130
|---|---|
84
-
131
+
|`options_types.md`| The `TO/endpoint-affects-host-0` "Expected Rest Host" column uses pre-REC1 hostnames (`rest.ably.io`, `test-rest.ably.io`). No assertion reads the column, so the derived test is unaffected |
85
132
|`channels_collection.md`| Header claims RSN3b and RSN3c; neither has a test |
86
133
|`stats.md`| Fixture nests counts under `all`, which `Stats.from_dict` never reads |
87
134
|`rest_client.md`|`RSC17` has two byte-identical tests; header lists RSC7 and RSC7b with no tests |
88
135
|`rest_client.md`|`RSC18` requires constructor-time failure; RSA1/RSC18 only say "any attempt to use" |
89
136
|`request.md`|`version` is written as an integer but lands in a header |
90
137
|`fallback.md`| REC3a, REC3b and REC3 drive a Realtime client but sit in `rest/unit`|
138
+
|`message_encoding.md`, `msgpack_interop.md`, `annotations.md`| Six sections carry no Test ID; ids were inferred by sibling convention |
139
+
|`publish.md`, `rest_presence.md`, `message_encoding.md`, `history.md`, `idempotency.md`| All point at `/Users/paddy/data/worknew/dev/dart-experiments/...` for the mock contract |
91
140
92
141
## Failing Tests
93
142
@@ -99,7 +148,18 @@ the mark is the only change needed once the SDK behaviour lands.
99
148
| Spec points | Missing | Tests |
100
149
|---|---|---|
101
150
151
+
| RSP3a2, RSP3a3 |`clientId` and `connectionId` filters on `RestPresence#get`. `Presence.get` takes only `limit`, while `Presence.history` does take its documented params | 3 |
152
+
| TP5 |`size` on `PresenceMessage`. The related `maxMessageSize` gap is adapted rather than gated, below; `features.md` TM6 has no UTS test | 1 |
153
+
102
154
| RSC2, RSC3, RSC4, TO3b, TO3c, TO3c2 |`log_handler` as a client option, and any use of `log_level` — it is stored on `Options` and read by nothing | 4 |
155
+
| TI4, TI1/TI5 |`href` anywhere in the SDK, and `cause` when deserialising. `AblyException.from_dict` and `raise_for_response` read only `message`, `statusCode` and `code`, so both fields are dropped from server errors | 2 |
156
+
| TP3a, TP3d, TP3g | Presence attributes defaulted from the encapsulating ProtocolMessage. There is no ProtocolMessage type; `ably/realtime/channel.py:751-761` passes the presence array through without context. Matters for synthesized-leave detection and `memberKey`| 3 |
157
+
158
+
### Options
159
+
160
+
| Spec points | Behaviour |
161
+
|---|---|
162
+
| TO3 |`endpoint` is left unset when none is given, per `TO/endpoint-affects-host-0`. `Options.__init__` resolves the default eagerly to the REC1a routing policy id `main`, so the attribute never reads back as null. Only the default case is gated; the two that name an endpoint are derived and pass |
103
163
104
164
### Requests
105
165
@@ -119,8 +179,15 @@ comment above. These run, so they guard against regression.
119
179
| RSC18 | The constructor rejects basic auth over HTTP | Construction succeeds; 40103 is raised from `make_request` when a request needing Basic Auth is attempted, and no request goes out. RSA1/RSC18 say only "any attempt to use" | Compliant; the UTS is stricter than its source |
120
180
| REC1b1, REC1c1 | Code 40000, or a message containing "invalid" or "conflict" | 400/40106 with a specific message. The features spec mandates no code | Cosmetic |
121
181
182
+
| HP6 |`errorCode` is a number | The raw header string, `'40101'`| Open bug, trivial |
183
+
| HP8 |`headers` is a map | A list of `(name, value)` pairs, so the lookup the spec describes is impossible without converting, and case-insensitivity is lost | Open bug; changing the return type is breaking |
122
184
| RSC19e | An error indicated idiomatically |`httpx.ConnectError` / `ReadTimeout` reach the caller unwrapped, because `AblyRest.request` carries no `@catch_all` unlike `time()` and `stats()`. The messages do name the failure | Borderline; defensible under RSC19e |
123
185
| RSC15a | Six hosts tried | Three. `Options.__get_hosts` truncates to `http_max_retry_count`, which TO3l5 sanctions | Intentional |
186
+
| TI |`ErrorInfo` equality by attributes | No `__eq__`, so errors compare by identity. Python exceptions conventionally do, and the requirement appears nowhere in `features.md`. Adding `__eq__` without `__hash__` would make `AblyException` unhashable and break any caller that puts one in a set | Intentional |
187
+
| TD5, RSA16a |`capability` is stringified JSON | A `Capability` object, a public convenience type used throughout `auth`. Narrowing the return type to `str` would break every caller that indexes or mutates it, so it is reserved for a future major | Intentional; a breaking change to align |
188
+
189
+
| TO3l8 |`maxMessageSize` is a client option, default 65536 | Rejected by `Options.__init__`. `ably/realtime/channel.py:422` reads it with `getattr(..., 65536)`, so the default holds but cannot be configured, nor overridden by `connectionDetails` (CD2c) | Open bug |
190
+
| TO3l1, TO3l5 |`httpRequestTimeout` and `httpMaxRetryCount` carry their defaults on the options object | Left unset; the effective defaults are applied downstream by `Http` and by `Options.__get_hosts`. The spec's values are milliseconds, while ably-python's `http_request_timeout` is seconds | Intentional |
0 commit comments