Multimodal Processing
Multimodal Processing
dspy-go represents mixed text and binary inputs with core.ContentBlock.
pkg/llms converts those blocks to llm-go’s provider-neutral message parts,
and llm-go owns the provider wire format.
Gemini, Anthropic, OpenAI, OpenAI Codex, and compatible endpoints can expose
vision through the adapter. Capability varies by model, so check
llm.Capabilities() for core.CapabilityVision or
core.CapabilityMultimodal before relying on image input.
Image Analysis with a Module
Declare image inputs with core.NewImageField and provide values with
core.NewImageBlock:
package main
import (
"context"
"fmt"
"os"
"github.com/XiaoConstantine/dspy-go/pkg/core"
"github.com/XiaoConstantine/dspy-go/pkg/llms"
"github.com/XiaoConstantine/dspy-go/pkg/modules"
)
func main() {
llm, err := llms.NewGeminiLLM("", core.ModelGoogleGeminiFlash)
if err != nil {
panic(err)
}
signature := core.NewSignature(
[]core.InputField{
{Field: core.NewImageField("image",
core.WithDescription("The image to analyze"))},
{Field: core.NewTextField("question",
core.WithDescription("Question about the image"))},
},
[]core.OutputField{
{Field: core.NewTextField("answer",
core.WithDescription("Answer based on the image"))},
},
).WithInstruction("Analyze the image and answer the question.")
predictor := modules.NewPredict(signature).WithStructuredOutput()
predictor.SetLLM(llm)
imageData, err := os.ReadFile("image.jpg")
if err != nil {
panic(err)
}
result, err := predictor.Process(context.Background(), map[string]any{
"image": core.NewImageBlock(imageData, "image/jpeg"),
"question": "What objects are visible?",
})
if err != nil {
panic(err)
}
fmt.Println(result["answer"])
}Predict detects non-text fields in the signature and calls
GenerateWithContent rather than flattening the image into a text prompt.
Direct Content Generation
Use the content API when you do not need signature formatting or parsed module outputs:
content := []core.ContentBlock{
core.NewTextBlock("Describe this image in detail."),
core.NewImageBlock(imageData, "image/jpeg"),
}
response, err := llm.GenerateWithContent(ctx, content)
if err != nil {
return err
}
fmt.Println(response.Content)The MIME type must match the supplied bytes. Common image types include
image/jpeg, image/png, image/gif, and image/webp; actual support is
provider- and model-specific.
Streaming Multimodal Content
content := []core.ContentBlock{
core.NewTextBlock("Describe the scene."),
core.NewImageBlock(imageData, "image/jpeg"),
}
stream, err := llm.StreamGenerateWithContent(ctx, content)
if err != nil {
return err
}
defer stream.Cancel()
for chunk := range stream.ChunkChannel {
if chunk.Error != nil {
return chunk.Error
}
fmt.Print(chunk.Content)
}Call Cancel if the consumer stops before the channel closes. Context
cancellation also propagates to the provider stream.
Multiple Images
Content blocks preserve order, so a prompt can refer to images by position:
content := []core.ContentBlock{
core.NewTextBlock("Compare the first image with the second."),
core.NewImageBlock(before, "image/jpeg"),
core.NewImageBlock(after, "image/jpeg"),
core.NewTextBlock("List the important differences."),
}
response, err := llm.GenerateWithContent(ctx, content)For a module, declare one core.NewImageField per image and pass a
core.NewImageBlock for each corresponding input.
Audio Blocks
dspy-go can represent audio with core.NewAudioBlock(data, mimeType). Before
using it, check for core.CapabilityAudio; not every llm-go provider or model
accepts audio input.
Model Selection
Use current core.ModelID constants rather than retired model strings. For
example:
- Gemini:
core.ModelGoogleGeminiFlashorcore.ModelGoogleGeminiPro - OpenAI: a vision-capable model such as
core.ModelOpenAIGPT4o - Anthropic: one of the currently exported Claude family constants
Provider model availability changes independently of dspy-go. Confirm that the selected model supports the content type and size you send.
Practical Guidance
- Resize unnecessarily large images before sending them to reduce latency and provider cost.
- Preserve the original MIME type after image conversion.
- Use clear questions and output field descriptions; vision output is still model generation, not deterministic OCR.
- Keep conversation history in your application or agent transcript. Reusing a
Predictmodule does not by itself create a persistent multimodal chat.
Complete Example
The
examples/multimodal
program covers image analysis, vision Q&A, repeated image interactions,
streaming, and multiple-image input using Gemini:
export GEMINI_API_KEY="your-api-key"
go run ./examples/multimodal/...