[Critical] Load balancing endpoints have a payload limit of ~1MB

[deanflaviano]

The connection is killed by runpod for all request above ~1MB.

Ping @muhsinking @DeJayDev

Here's how to reproduce it:

#!/usr/bin/env python3
"""
FastAPI test server - JSON with base64 encoded image
"""

import asyncio
import base64
import logging
from fastapi import FastAPI, HTTPException
from fastapi.responses import JSONResponse
from pydantic import BaseModel
import uvicorn

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

app = FastAPI()


class ImageRequest(BaseModel):
    """Request model with base64 encoded image"""
    image: str
    prompt: str = "test"
    max_resolution: str = "4k"
    convert_to_bw: bool = True


@app.post("/predict")
async def predict(request: ImageRequest):
    """
    Accept JSON with base64 image, wait 5 seconds, return base64 image
    """
    try:
        # Decode base64 image
        try:
            image_bytes = base64.b64decode(request.image)
        except Exception as e:
            raise HTTPException(status_code=400, detail=f"Invalid base64 image data: {str(e)}")
        
        if len(image_bytes) == 0:
            raise HTTPException(status_code=400, detail="Image data is empty")
        
        logger.info(f"Received image: {len(image_bytes)} bytes, prompt: {request.prompt}")
        
        # Wait 5 seconds to simulate processing
        await asyncio.sleep(5)
        
        # Encode image back to base64
        image_b64 = base64.b64encode(image_bytes).decode('utf-8')
        
        return JSONResponse(
            content={
                "image": image_b64,
                "success": True
            },
            status_code=200
        )
        
    except HTTPException:
        raise
    except Exception as e:
        logger.error(f"Error processing request: {str(e)}")
        raise HTTPException(status_code=500, detail=f"Server error: {str(e)}")


@app.get("/ping")
async def health_check():
    """Health check endpoint"""
    return {"status": "healthy"}


if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=8000)

Dockerfile

# V1.0.3
FROM python:3.11-slim

WORKDIR /app

# Install dependencies
RUN pip install --no-cache-dir \
    fastapi \
    uvicorn[standard] \
    python-multipart

# Copy all test scripts
COPY app.py .

# Default command - rename the script you want to run as app.py
# Or override with: docker run <image> python3 <script_name>.py
CMD ["python3", "app.py"]

Working: (301 KB)

curl -X POST "endpoint_url" \
    -H 'Authorization: Bearer Token' \
    -H 'Content-Type: application/json' \
    -d @small_request.json

Not-working: (3,4 MB)

curl -X POST "endpoint_url" \
    -H 'Authorization: Bearer Token' \
    -H 'Content-Type: application/json' \
    -d @request.json

request.json
small_request.json

[deanflaviano]

Also, your loadbalancer-example:v1.0 fails when the it reaches 2060013 bytes (2MB). [502: Bad gateway | Cloudflare]

Iteration 192: Size = 1910013 bytes, Status = 200, Time = 0.75s
Iteration 193: Size = 1920013 bytes, Status = 200, Time = 1.99s
Iteration 194: Size = 1930013 bytes, Status = 200, Time = 0.81s
Iteration 195: Size = 1940013 bytes, Status = 200, Time = 0.57s
Iteration 196: Size = 1950013 bytes, Status = 200, Time = 0.58s
Iteration 197: Size = 1960013 bytes, Status = 200, Time = 0.57s
Iteration 198: Size = 1970013 bytes, Status = 200, Time = 0.66s
Iteration 199: Size = 1980013 bytes, Status = 200, Time = 0.59s
Iteration 200: Size = 1990013 bytes, Status = 200, Time = 0.68s
Iteration 201: Size = 2000013 bytes, Status = 200, Time = 0.60s
Iteration 202: Size = 2010013 bytes, Status = 200, Time = 0.66s
Iteration 203: Size = 2020013 bytes, Status = 200, Time = 0.58s
Iteration 204: Size = 2030013 bytes, Status = 200, Time = 0.58s
Iteration 205: Size = 2040013 bytes, Status = 200, Time = 0.59s
Iteration 206: Size = 2050013 bytes, Status = 200, Time = 0.58s
Iteration 207: Size = 2060013 bytes, Status = 502, Time = 0.47s

============================================================
REQUEST FAILED!
============================================================
Final prompt size: 2060013 bytes
Request time: 0.47 seconds
Status code: 502
Response: <!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]>    <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]>    <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>


<title>END-POINT.api.runpod.ai | 502: Bad gateway</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="

Here's a test script to test your loadbalancer-example:v1.0:

import requests
import random
import string
import time
import sys

def generate_random_chars(length):
    """Generate random characters"""
    return ''.join(random.choices(string.ascii_letters + string.digits + string.punctuation + ' ', k=length))

def main():
    url = "https://YOUR_ENDPOINT.api.runpod.ai/generate"
    headers = {
        'Authorization': 'Bearer YOUR_API_KEY_HERE',
        'Content-Type': 'application/json'
    }
    
    prompt = "Hello, world!"
    iteration = 0
    
    print(f"Starting with prompt size: {len(prompt.encode('utf-8'))} bytes")
    print("-" * 60)
    
    while True:
        iteration += 1
        
        # Prepare the request payload
        payload = {"prompt": prompt}
        
        # Measure request time
        start_time = time.time()
        
        try:
            response = requests.post(url, headers=headers, json=payload, timeout=60)
            request_time = time.time() - start_time
            
            print(f"Iteration {iteration}: Size = {len(prompt.encode('utf-8'))} bytes, "
                  f"Status = {response.status_code}, Time = {request_time:.2f}s")
            
            # Check if request failed
            if response.status_code != 200:
                print("\n" + "=" * 60)
                print("REQUEST FAILED!")
                print("=" * 60)
                print(f"Final prompt size: {len(prompt.encode('utf-8'))} bytes")
                print(f"Request time: {request_time:.2f} seconds")
                print(f"Status code: {response.status_code}")
                print(f"Response: {response.text}")  # response
                break
                
        except requests.exceptions.RequestException as e:
            request_time = time.time() - start_time
            print("\n" + "=" * 60)
            print("REQUEST EXCEPTION!")
            print("=" * 60)
            print(f"Final prompt size: {len(prompt.encode('utf-8'))} bytes")
            print(f"Request time: {request_time:.2f} seconds")
            print(f"Error: {str(e)}")
            break
        
        # Add 10000 random characters for next iteration
        prompt += generate_random_chars(10000)

if __name__ == "__main__":
    main()

To use it, replace:

YOUR_ENDPOINT with your RunPod endpoint subdomain
YOUR_API_KEY_HERE with your bearer token