How to classify, describe and analyze images using GPT-4o vision

On this page
You can use GPT-4o's vision capabilities through the regular OpenAI Chat
Completions API: put an image (as a URL or base64 data) into the user
message, describe the task in the system message, and GPT-4o returns a
label, a description or an analysis. A low-resolution image costs about
$0.000213 with gpt-4o-2024-08-06.
At the time of writing, GPT-4o is one of the best multimodal models on the market. Multimodal means it works with several input formats: text, images and audio.
This post covers the vision part of GPT-4o:
- How to send images to GPT-4o with the OpenAI API
- How to label and categorize images without training an image classification model first
- How to describe an image
- How to run open-ended image analysis, with a little data analytics on top
Update (November 2026): OpenAI has deprecated GPT-4o, and the
gpt-4o-2024-08-06 snapshot used below shuts down on January 14, 2027 (see
the deprecations page).
The code works with newer vision models, but image token costs differ per
model, and for GPT-4o the auto detail mode now uses the same sizing as
high. Check OpenAI's vision
guide before you
reuse the cost figures below.
How to use the GPT-4o vision capabilities
Using GPT-4o vision is straightforward and works much like the well-known chat completion capabilities.
As with chat completion, you create a messages array containing an
optional system message and a user message. You can also add "assistant"
messages as part of the chat history or as part of a few-shot prompting
strategy. The content
field of the user message can then contain image data as well as text.
The image needs to be provided as a downloadable URL or as base64-encoded image data. Here are both options in Python.
Using GPT-4o vision with image url
To pass an image by link, upload it to a public web host and add the URL to the message array as shown below.
Using GPT-4o vision with base64-encoded image data
If the image is stored locally, read the file and encode it in base64. You
still use the image_url field, but this time with base64 image data.
Note: Make sure to change the jpeg modifier in the example below
to your actual file type.
Using GPT-4o vision with multiple images
GPT-4o can also handle several images per prompt. Add another object of
type image_url to your messages. You can even mix base64 and URL
images.
Note: Make sure to have a recent version of the OpenAI python package to follow this example. Version 1.40.0 or above is sufficient.
Then send the messages with the OpenAI SDKs or the OpenAI API. Here is the rest of the Python example:
So the whole flow is:
- Create the messages array as shown above.
- Send the image to GPT-4o with the OpenAI API or SDKs, together with a prompt of your choosing.
- Get the results back, shaped by your prompt.
Image detail modes: low, high and auto
You can send images of any resolution and format, and GPT-4o (plus the image pipeline in front of it) will make the best of them. For cost and latency control, OpenAI offers two image detail modes, or actually three:
low: The low-res mode. The image is automatically downsampled to 512x512 pixels and costs the equivalent of 85 tokens. This mode is fast and cheap.high: High resolution mode also downsamples the image to 512x512 pixels first and shows the LLM this low-res version, so it gets an overview. Then the image is scaled to fit within 2048x2048 pixels with its shortest side at most 768 pixels, and the model gets detailed crops of that version, at 512x512 pixels each. Each crop costs 170 tokens, so this mode takes significantly longer thanlow.auto: The default. GPT-4o looks at the size of the original image and pickshighorlowaccordingly.
Note: If you need very detailed analyses of your images, you might
opt for the high res option. Keep in mind, though, that the details
reach the LLM as 512x512 pixel crops of your original image. If the
details you care about are larger than 512x512 pixels, they will not be
presented correctly. In that case, either downsample the image manually
so that the details are small enough, or use the low res mode.
To set the mode, add a detail key to the image_url object in your
messages array:
Image classification with GPT-4o
Image classification used to be a time-consuming process. You had to select an image classification model and build a training dataset of at least 100 or so images, labeled with the exact categories you want to classify later. Then you decided on a training strategy, trained the model, validated the result and hosted it. Only then could you use the classifier.
With models like GPT-4o, this process is more or less obsolete. They are very good at Zero-Shot classification: you don't need to train them for a specific task, and they can predict classes they never saw during training.
For image classification, this means you send an image to the model, ask it to pick one of the categories you provide, and you'll most likely get a very good result without any model training.
We found that this works best when you briefly describe the
classification task in the system message and also list the categories
there. The user message then contains only the image.
Note: For some use cases you still want your own image classifier, namely very specific niches. If you want to classify different types of mushrooms, for example, you'd better train your own specialized model. GPT-4o and similar models correctly identify more general things, like different animal types (rather than breeds of one animal).
Here's an example:
Describing images using GPT-4o vision
Another common task is having GPT-4o describe an image. The most important
part is to be precise about which aspect of the image you want described,
because an image can be described from many angles. Again, the system
message is the best place to define what exactly you want.
A typical use case: you store images in a database and use the descriptions to search for them.
The example output was:
If the description is too detailed, not detailed enough, or in the wrong style, you just change the system prompt. That's the powerful part: you have a full-blown LLM with all its prompt understanding, and it can now also read and understand images. You can even use few-shot prompting or similar techniques.
Light data analytics using GPT-4o vision
So far, GPT-4o has understood images, created descriptions and labels, and followed simple prompts that define what the response should look like.
Now for something more elaborate. Many thousands of data analysts have built beautiful dashboards with potentially very helpful insights. Yet time and again it turns out that not a single soul looks at these poor, beautiful dashboards. (I can make fun of that, as I was one of those data analysts.)
If most people are too busy to look at analytics results, what if we make AI look at our dashboards? (Yes, I'm aware this sounds a little desperate.)
The strategy is simple: take a screenshot of a dashboard, send it to GPT-4o and ask specific questions about it. One real-world application is an open-ended alerting system: the LLM analyzes a dashboard with data over time and notifies you only when something out of the ordinary happens.
Here, we test whether GPT-4o makes a good dashboard analyst by asking it for a written summary of this dashboard:
Web analytics dashboard with referrers, pageview funnel and daily pageviews
-
Install the
pytest-playwrightpackage, which provides Playwright for python. Playwright navigates to web pages and, among other things, takes screenshots of them. -
Automatically create a screenshot of your dashboard.
This example assumes the dashboard is publicly available. If your dashboard needs authentication, have a look at the playwright documentation.
-
Use the screenshot for GPT-4o vision inference:
This was the response from the LLM:
That's a remarkably good result, given that we simply threw a dashboard at the model and asked for a summary. I'm especially impressed by how well the model read the charts and found the correct dates (for example the spike of 400 pageviews on July 18th).
If you want GPT-4o to read documents rather than dashboards, we cover that in parsing PDF, Word and Excel documents with GPT-4o. The same idea works with an open-source model: we used one to extract text and tables from complex PDF pages rendered as images.
Costs and pricing of GPT-4o vision for image prompts
How much does GPT-4o vision cost? We'd say it's quite cheap. The following
prices are for the GPT-4o variant gpt-4o-2024-08-06, which was new when we
wrote this:
- Any image in
lowresolution mode: 0.02 cents ($0.000213) or $2 per 10,000 images. - 1920 x 1080 in
highresolution mode: 0.28 cents ($0.002763) or $2.76 per 1,000 images.
Important note: At the time of writing, the gpt-4o-2024-08-06 model was the cheapest model of the GPT-4o series for vision. Even the otherwise very cost-efficient gpt-4o-mini model is about twice as expensive, with worse output quality. So stick with this model, at least for now.
For more ways to keep model spend down, see our strategies to reduce LLM costs.
Conclusion
GPT-4o vision handles image classification, description and analysis without specialized training or a complex setup. You build a messages array with an image URL or base64 data, put the task and any categories in the system message, and choose a detail mode to balance cost and precision.
In our tests, it classified and described a sample image, and it turned a web analytics dashboard into a management summary with the correct peak date. At around $2 per 10,000 low-res images, it's cheap enough for most use cases. Analyzing visual data such as dashboards is the part we find most intriguing.
As a next step, take one of the examples above and try different prompts and detail modes on your own images to find what works best for your use case.
Further reading
- How to fine-tune GPT-4o mini (for free until September 2024)
- Chat with your confluence using LLMs
- How to fine-tune your very own LLM
- Saving costs with LLM routing: the right model for the right task
Interested in building high-quality AI agent systems?
We prepared a comprehensive guide based on cutting-edge research for how to build robust, reliable AI agent systems that actually work in production. This guide covers:
- Understanding the 14 systematic failure modes in multi-agent systems
- Evidence-based best practices for agent design
- Structured communication protocols and verification mechanisms



