Skip to navigation
Multimodal

Multimodal

Beyond text, HyperFlowSI lets you send images, PDFs, audio and video to models that understand them, and generate images and video from a prompt. Everything goes through the same API and the same key.

Inputs

All inputs go to POST /v1/chat/completions. Instead of a string, a message’s content is a list of parts, and each part has a type:

InputPart typePage
Imageimage_urlImage inputs
PDFfilePDF inputs
Audioinput_audioAudio inputs
Videovideo_urlVideo inputs

You can mix text with several inputs, of one kind or several, in one message. How many a request can carry depends on the model.

{
  "role": "user",
  "content": [
    { "type": "text", "text": "Compare the chart with the report." },
    { "type": "image_url", "image_url": { "url": "https://example.com/chart.png" } },
    { "type": "file", "file": { "filename": "report.pdf", "file_data": "data:application/pdf;base64,..." } }
  ]
}

Outputs

OutputEndpointPage
ImagesPOST /v1/images/generations, POST /v1/images/editsImage generation
VideoPOST /v1/videosVideo generation

Chat models answer in text. Audio output, text-to-speech and speech-to-text aren’t available.

URL or base64

InputURLBase64
ImageYesYes, as a data:image/...;base64, URL
PDFDepends on the modelYes, as a data:application/pdf;base64, URL
AudioNoYes, raw base64 with a format
VideoDepends on the model’s hostYes, as a data:video/...;base64, URL

For public files, a URL keeps the request small and skips encoding. Use base64 for local or private files.

A request body can be at most 25 MB, base64 included (base64 adds about a third to a file’s size). A larger request is refused with 413; send a link or a smaller file.

Which models take what

Each model’s page on Models shows the inputs it takes in its Modalities row. The same list is in modalities.input from the model’s catalog entry, with the / in the id written as %2F. Among its other fields:

curl https://ai.hyperflow.fun/public/models/openai%2Fgpt-6-luna
{ "id": "openai/gpt-6-luna", "modalities": { "input": ["text", "image"], "output": ["text"] } }

The input names are text, image, file (PDF), audio and video. Every chat model takes text. Images, PDFs, audio and video depend on the model. HyperFlowSI checks each part against what the model takes before anything is held: a part the model can’t read is refused with a 400 that names the model and the content, for example:

{ "error": { "message": "openai/gpt-6-luna does not take input_audio content", "type": "invalid_request_error", "code": "400" } }

Only the part types in the table above are accepted. Nothing is charged for a refused request.

Billing

The model turns every input into input tokens, and you pay for them at its input price. How many tokens an image, a PDF page, a second of audio or a second of video takes depends on the model, so check usage.prompt_tokens on a first request. Image generation is priced per image or by tokens, and video per second; see their pages.

Next: Image inputs: send pictures to a chat model.