Multimodal: Image Analysis (Vision)

Multimodal models accept images as input in addition to text. This enables tasks like image descriptions, chart analysis, OCR-like text recognition, or visual question answering, all through the same /chat/completions endpoint you also use for plain text.

Technically, an image is embedded by passing a message’s content field as an array. Each element in it is either a text part ({"type": "text", ...}) or an image part ({"type": "image_url", ...}). The model receives both together and can reference the text as well as the image in its response.

Content Array Format

Instead of a string, content contains a list of parts. A typical example looks like this:

[
  {"type": "text", "text": "What does this image show?"},
  {"type": "image_url", "image_url": {"url": "https://example.com/chart.png"}}
]

The image_url.url field may contain a publicly reachable HTTPS URL or, depending on the client SDK, a base64 data URL (e.g. data:image/png;base64,...). For base64 URLs, ensure the correct MIME-type prefix.

curl

curl -X POST https://ai.noris.de/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "model": "vllm/release/gemma-4-31b-it",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "text", "text": "Briefly describe what this image shows."},
          {"type": "image_url", "image_url": {"url": "https://example.com/image.png"}}
        ]
      }
    ]
  }'

The response follows the same structure as plain-text requests, the generated text is under choices[0].message.content.

Python

The openai SDK supports the content array natively:

from openai import OpenAI

client = OpenAI(base_url="https://ai.noris.de/v1", api_key="YOUR_API_KEY")

response = client.chat.completions.create(
    model="vllm/release/gemma-4-31b-it",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Briefly describe what this image shows."},
                {"type": "image_url", "image_url": {"url": "https://example.com/image.png"}}
            ]
        }
    ]
)
print(response.choices[0].message.content)

Supported Models

Not all models on the platform support image input. As of today, the following mapping applies:

Model	Vision-capable	Note
Gemma 4 31B (IT)	Yes	Multimodal, suited for general image analysis.
Qwen 3.6 27B	Yes	Multimodal, strong in code/diagram analysis.
GPT-OSS 120B	No	Text-only reasoning model.
GLM 5.2	No	Text model with very long context.

If an image is sent to a non-multimodal model, the API returns an error. Before integrating, verify that the chosen model supports vision, or use Gemma 4 31B or Qwen 3.6 27B directly. See Models for details.