Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions deploy/helm/aiperf-operator/templates/crd-aiperfjob.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -330,6 +330,17 @@ spec:
routing. Values support environment variable substitution.
additionalProperties:
type: string
urlHeaders:
type: array
description: 'Extra HTTP headers for individual targets: entry ``i`` applies to requests sent to ``urls[i]``
and may be ``{}``. Merged over ``headers``; on a name clash (case-insensitive) the per-URL value
wins. Use for routing hints such as ``X-data-parallel-rank``. Credential-bearing header names are
rejected; put credentials in ``headers`` or ``api_key``. Example: [{X-data-parallel-rank: ''0''},
{X-data-parallel-rank: ''1''}]'
items:
type: object
additionalProperties:
type: string
extra:
type: object
description: 'Additional fields to include in request body. Merged into every request. Common fields:
Expand Down
11 changes: 11 additions & 0 deletions deploy/helm/aiperf-operator/templates/crd-aiperfsweep.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -469,6 +469,17 @@ spec:
routing. Values support environment variable substitution.
additionalProperties:
type: string
urlHeaders:
type: array
description: 'Extra HTTP headers for individual targets: entry ``i`` applies to requests sent to ``urls[i]``
and may be ``{}``. Merged over ``headers``; on a name clash (case-insensitive) the per-URL value
wins. Use for routing hints such as ``X-data-parallel-rank``. Credential-bearing header names are
rejected; put credentials in ``headers`` or ``api_key``. Example: [{X-data-parallel-rank: ''0''},
{X-data-parallel-rank: ''1''}]'
items:
type: object
additionalProperties:
type: string
extra:
type: object
description: 'Additional fields to include in request body. Merged into every request. Common fields:
Expand Down
16 changes: 16 additions & 0 deletions docs/cli-options.md
Original file line number Diff line number Diff line change
Expand Up @@ -506,6 +506,10 @@ Strategy for selecting URLs when multiple `--url` values are provided. 'round_ro
<br/>_Choices: [`round_robin`]_
<br/>_Default: `round_robin`_

#### `--url-header` `<list>`

HTTP header to send only to one `--url` target, as `INDEX:Header:Value`, where `INDEX` is the 0-based position of the URL in the `--url` list. Can be specified multiple times. Global `--header` values still apply to every target; on a name clash (case-insensitive) the per-URL value wins. Repeated identical URLs stay distinct targets, so `--url http://localhost:8000 --url http://localhost:8000 --url-header 0:X-data-parallel-rank:0 --url-header 1:X-data-parallel-rank:1` pins each round-robin target to a vLLM data-parallel rank. Values are sent as given (never coerced). Credential-bearing header names are rejected here; pass credentials with `--header` or `--api-key`.

#### `--request-timeout-seconds` `<float>`

Maximum time in seconds to wait for each HTTP request to complete, including connection establishment, request transmission, and response receipt. Applies to both streaming and non-streaming requests. Requests exceeding this timeout are cancelled and recorded as failures.
Expand Down Expand Up @@ -2187,6 +2191,10 @@ Strategy for selecting URLs when multiple `--url` values are provided. 'round_ro
<br/>_Choices: [`round_robin`]_
<br/>_Default: `round_robin`_

#### `--url-header` `<list>`

HTTP header to send only to one `--url` target, as `INDEX:Header:Value`, where `INDEX` is the 0-based position of the URL in the `--url` list. Can be specified multiple times. Global `--header` values still apply to every target; on a name clash (case-insensitive) the per-URL value wins. Repeated identical URLs stay distinct targets, so `--url http://localhost:8000 --url http://localhost:8000 --url-header 0:X-data-parallel-rank:0 --url-header 1:X-data-parallel-rank:1` pins each round-robin target to a vLLM data-parallel rank. Values are sent as given (never coerced). Credential-bearing header names are rejected here; pass credentials with `--header` or `--api-key`.

#### `--request-timeout-seconds` `<float>`

Maximum time in seconds to wait for each HTTP request to complete, including connection establishment, request transmission, and response receipt. Applies to both streaming and non-streaming requests. Requests exceeding this timeout are cancelled and recorded as failures.
Expand Down Expand Up @@ -3863,6 +3871,10 @@ Strategy for selecting URLs when multiple `--url` values are provided. 'round_ro
<br/>_Choices: [`round_robin`]_
<br/>_Default: `round_robin`_

#### `--url-header` `<list>`

HTTP header to send only to one `--url` target, as `INDEX:Header:Value`, where `INDEX` is the 0-based position of the URL in the `--url` list. Can be specified multiple times. Global `--header` values still apply to every target; on a name clash (case-insensitive) the per-URL value wins. Repeated identical URLs stay distinct targets, so `--url http://localhost:8000 --url http://localhost:8000 --url-header 0:X-data-parallel-rank:0 --url-header 1:X-data-parallel-rank:1` pins each round-robin target to a vLLM data-parallel rank. Values are sent as given (never coerced). Credential-bearing header names are rejected here; pass credentials with `--header` or `--api-key`.

#### `--request-timeout-seconds` `<float>`

Maximum time in seconds to wait for each HTTP request to complete, including connection establishment, request transmission, and response receipt. Applies to both streaming and non-streaming requests. Requests exceeding this timeout are cancelled and recorded as failures.
Expand Down Expand Up @@ -6046,6 +6058,10 @@ Strategy for selecting URLs when multiple `--url` values are provided. 'round_ro
<br/>_Choices: [`round_robin`]_
<br/>_Default: `round_robin`_

#### `--url-header` `<list>`

HTTP header to send only to one `--url` target, as `INDEX:Header:Value`, where `INDEX` is the 0-based position of the URL in the `--url` list. Can be specified multiple times. Global `--header` values still apply to every target; on a name clash (case-insensitive) the per-URL value wins. Repeated identical URLs stay distinct targets, so `--url http://localhost:8000 --url http://localhost:8000 --url-header 0:X-data-parallel-rank:0 --url-header 1:X-data-parallel-rank:1` pins each round-robin target to a vLLM data-parallel rank. Values are sent as given (never coerced). Credential-bearing header names are rejected here; pass credentials with `--header` or `--api-key`.

#### `--request-timeout-seconds` `<float>`

Maximum time in seconds to wait for each HTTP request to complete, including connection establishment, request transmission, and response receipt. Applies to both streaming and non-streaming requests. Requests exceeding this timeout are cancelled and recorded as failures.
Expand Down
37 changes: 37 additions & 0 deletions docs/tutorials/multi-url-load-balancing.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,12 +100,49 @@ aiperf profile --model llama \
--request-count 100
```

## Per-URL Headers

`--header` sends the same headers to every URL. To send a header to only one target, use `--url-header INDEX:Header:Value`, where `INDEX` is the 0-based position of the URL in the `--url` list. Repeated identical URLs stay distinct round-robin targets, so this pins each target to a vLLM data-parallel rank through vLLM's `X-data-parallel-rank` header, without the DP load balancer in the measurement:

```bash
# Server started with: vllm serve llama --data-parallel-size 2
aiperf profile --model llama \
--url http://localhost:8000 \
--url http://localhost:8000 \
--url-header 0:X-data-parallel-rank:0 \
--url-header 1:X-data-parallel-rank:1 \
--concurrency 16 \
--request-count 200
```

In a config file, `endpoint.url_headers` is a list aligned with `endpoint.urls`; use `{}` for a URL without extra headers:

```yaml
benchmark:
endpoint:
urls: [http://localhost:8000, http://localhost:8000, http://localhost:8000]
headers:
X-Team: perf
url_headers:
- {X-data-parallel-rank: "0"}
- {}
- {X-data-parallel-rank: "2"}
```

- Global headers still apply to every URL. A per-URL header replaces a global header with the same name (compared case-insensitively).
- A session keeps its URL for every turn, so all turns of a multi-turn conversation carry the same per-URL headers.
- Values are sent as given; unlike `--header`, they are never coerced to numbers or booleans.
- An index past the last `--url` is a configuration error.
- Credential-bearing header names (for example `Authorization` or names containing `token` or `api-key`) are rejected: per-URL headers are not redacted from artifacts or transported as secrets. Pass credentials with `--header` or `--api-key`.
- Per-URL headers apply to inference requests. Readiness probes and control hooks (`--reset-kv-cache`, `--server-profiler`) keep using the global headers.

## CLI Options

| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `--url` | list | localhost:8000 | One or more endpoint URLs; multiple URLs enable load balancing |
| `--url-strategy` | enum | round_robin | Strategy for distributing requests across multiple URLs |
| `--url-header` | list | none | `INDEX:Header:Value` header sent only to the URL at that 0-based position; repeatable |

## Behavior Notes

Expand Down
26 changes: 26 additions & 0 deletions src/aiperf/common/models/model_endpoint_info.py
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,12 @@ def _redact_headers(self, value: list[tuple[str, str]]) -> list[tuple[str, str]]

return redact_header_tuples(value)

url_headers: list[dict[str, str]] = Field(
default=[],
description="Extra headers for individual URLs: entry i applies to "
"base_urls[i]. Merged over headers for requests sent to that URL.",
)

api_key: str | None = Field(
default=EndpointDefaults.API_KEY,
description="API key to use for the endpoint.",
Expand Down Expand Up @@ -218,6 +224,25 @@ def get_url(self, index: int | None = None) -> str:
return self.base_urls[0]
return self.base_urls[index % len(self.base_urls)]

def get_headers(self, index: int | None = None) -> dict[str, str]:
"""Get the custom headers for the URL at ``index``.

Global ``headers`` apply to every URL; the URL's ``url_headers`` are laid
over them, replacing a global header of the same name regardless of
casing so the request never carries both. ``index`` wraps around like
:meth:`get_url`.
"""
headers = dict(self.headers or ())
position = 0 if index is None else index % len(self.base_urls)
overrides = (
self.url_headers[position] if position < len(self.url_headers) else None
)
if overrides:
replaced = {name.lower() for name in overrides}
headers = {k: v for k, v in headers.items() if k.lower() not in replaced}
headers.update(overrides)
return headers


class ModelEndpointInfo(AIPerfBaseModel):
"""Information about a model endpoint."""
Expand Down Expand Up @@ -264,6 +289,7 @@ def from_config(cls, cfg: BenchmarkConfig) -> ModelEndpointInfo:
allow_empty_content=ep.allow_empty_content,
base_urls=list(ep.urls),
headers=list((getattr(ep, "headers", {}) or {}).items()),
url_headers=list(getattr(ep, "url_headers", []) or []),
extra=list((getattr(ep, "extra", {}) or {}).items()),
timeout=ep.timeout,
api_key=ep.api_key,
Expand Down
49 changes: 48 additions & 1 deletion src/aiperf/config/endpoint.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@

from pydantic import (
AfterValidator,
BeforeValidator,
ConfigDict,
Field,
PrivateAttr,
Expand All @@ -37,7 +38,11 @@
parse_enabled_or_config,
require_relative_path,
)
from aiperf.config.loader.parsing import normalize_http_urls
from aiperf.config.loader.parsing import (
normalize_http_urls,
parse_url_headers_for_urls,
url_index_out_of_range_message,
)
from aiperf.config.sagemaker import SageMakerConfig
from aiperf.plugin.enums import (
EndpointType,
Expand Down Expand Up @@ -546,6 +551,21 @@ def _redact_headers(self, value: dict[str, str]) -> dict[str, str]:

return redact_headers(value) or {}

url_headers: Annotated[
list[dict[str, str]],
Field(
default_factory=list,
description="Extra HTTP headers for individual targets: entry ``i`` "
"applies to requests sent to ``urls[i]`` and may be ``{}``. Merged "
"over ``headers``; on a name clash (case-insensitive) the per-URL "
"value wins. Use for routing hints such as ``X-data-parallel-rank``. "
"Credential-bearing header names are rejected; put credentials in "
"``headers`` or ``api_key``. "
"Example: [{X-data-parallel-rank: '0'}, {X-data-parallel-rank: '1'}]",
),
BeforeValidator(parse_url_headers_for_urls),
]

extra: Annotated[
dict[str, Any],
Field(
Expand Down Expand Up @@ -831,6 +851,33 @@ def _validate_endpoint_boundaries(self) -> Self:
raise ValueError("endpoint.path must start with a leading slash")
return self

@model_validator(mode="after")
def _validate_url_headers(self) -> Self:
"""Bind per-URL headers to existing URLs and keep credentials out of them.

Credential-bearing headers are redacted from serialized configs and
rehydrated out of band only for ``headers`` (``AIPERF_INJECTED_HEADERS``),
so a secret here would reach worker processes as ``<redacted>``.
"""
from aiperf.common.redact import is_sensitive_header_name

if len(self.url_headers) > len(self.urls):
raise ValueError(
url_index_out_of_range_message(
len(self.url_headers) - 1, len(self.urls)
)
)
for index, headers in enumerate(self.url_headers):
sensitive = sorted(n for n in headers if is_sensitive_header_name(n))
if sensitive:
raise ValueError(
f"url_headers: {', '.join(sensitive)} for URL {index} looks "
"credential-bearing. Per-URL headers are not redacted or "
"transported as secrets; pass credentials with --header or "
"--api-key instead."
)
return self

@model_validator(mode="after")
def _validate_template_required(self) -> Self:
if self.type == EndpointType.TEMPLATE and self.template is None:
Expand Down
1 change: 1 addition & 0 deletions src/aiperf/config/flags/_converter_endpoint.py
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ def _endpoint_template_fallback(endpoint: dict[str, Any]) -> None:
# Map (CLIConfig endpoint field name) -> (AIPerfConfig endpoint key).
_ENDPOINT_FIELD_MAP: dict[str, str] = {
"url_selection_strategy": "url_strategy",
"url_headers": "url_headers",
"endpoint_type": "type",
"streaming": "streaming",
"allow_empty_content": "allow_empty_content",
Expand Down
1 change: 1 addition & 0 deletions src/aiperf/config/flags/_section_fields.py
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@
"transport",
"endpoint_type",
"url_selection_strategy",
"url_headers",
"per_chunk_usage",
"urls",
"use_legacy_max_tokens",
Expand Down
21 changes: 21 additions & 0 deletions src/aiperf/config/flags/cli_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,7 @@
parse_str_or_dict_as_tuple_list,
parse_str_or_list,
parse_str_or_list_of_positive_values,
parse_url_headers,
raise_rejected_input_file,
reject_moved_custom_dataset_type,
require_turn_mean_at_least_one,
Expand Down Expand Up @@ -235,6 +236,26 @@ class CLIConfig(BaseConfig):
),
] = EndpointDefaults.URL_STRATEGY

url_headers: Annotated[
Any,
Field(
description="HTTP header to send only to one `--url` target, as `INDEX:Header:Value`, where `INDEX` "
"is the 0-based position of the URL in the `--url` list. Can be specified multiple times. "
"Global `--header` values still apply to every target; on a name clash (case-insensitive) the "
"per-URL value wins. Repeated identical URLs stay distinct targets, so "
"`--url http://localhost:8000 --url http://localhost:8000 --url-header 0:X-data-parallel-rank:0 "
"--url-header 1:X-data-parallel-rank:1` pins each round-robin target to a vLLM data-parallel rank. "
"Values are sent as given (never coerced). Credential-bearing header names are rejected here; "
"pass credentials with `--header` or `--api-key`.",
),
BeforeValidator(parse_url_headers),
CLIParameter(
name=("--url-header",),
consume_multiple=True,
group=Groups.ENDPOINT,
),
] = []

timeout_seconds: Annotated[
float,
Field(
Expand Down
Loading
Loading