How to classify, describe and analyze images using GPT-4o vision

Andreas Nigg
Andreas Nigg · Co-Founder, Pondhouse Data
Published Updated 8 min read
How to classify, describe and analyze images using GPT-4o vision
On this page

You can use GPT-4o's vision capabilities through the regular OpenAI Chat Completions API: put an image (as a URL or base64 data) into the user message, describe the task in the system message, and GPT-4o returns a label, a description or an analysis. A low-resolution image costs about $0.000213 with gpt-4o-2024-08-06.

At the time of writing, GPT-4o is one of the best multimodal models on the market. Multimodal means it works with several input formats: text, images and audio.

This post covers the vision part of GPT-4o:

  • How to send images to GPT-4o with the OpenAI API
  • How to label and categorize images without training an image classification model first
  • How to describe an image
  • How to run open-ended image analysis, with a little data analytics on top

Update (November 2026): OpenAI has deprecated GPT-4o, and the gpt-4o-2024-08-06 snapshot used below shuts down on January 14, 2027 (see the deprecations page). The code works with newer vision models, but image token costs differ per model, and for GPT-4o the auto detail mode now uses the same sizing as high. Check OpenAI's vision guide before you reuse the cost figures below.

How to use the GPT-4o vision capabilities

Using GPT-4o vision is straightforward and works much like the well-known chat completion capabilities.

As with chat completion, you create a messages array containing an optional system message and a user message. You can also add "assistant" messages as part of the chat history or as part of a few-shot prompting strategy. The content field of the user message can then contain image data as well as text.

The image needs to be provided as a downloadable URL or as base64-encoded image data. Here are both options in Python.

Using GPT-4o vision with image url

To pass an image by link, upload it to a public web host and add the URL to the message array as shown below.

1 messages=[
2 {
3 "role": "system",
4 "content": "You are an image classification system. Classify the following images to be either 'cat' or 'dog', in the following format {'type': 'cat/dog'}"
5 },
6 {
7 "role": "user",
8 "content": [
9 {"type": "text", "text": "Classify this image."},
10 {
11 "type": "image_url",
12 "image_url": {
13 "url": "https://upload.wikimedia.org/wikipedia/commons/9/99/Brooks_Chase_Ranger_of_Jolly_Dogs_Jack_Russell.jpg",
14 },
15 },
16 ],
17 }
18 ]

Using GPT-4o vision with base64-encoded image data

If the image is stored locally, read the file and encode it in base64. You still use the image_url field, but this time with base64 image data.

Note: Make sure to change the jpeg modifier in the example below to your actual file type.

1 import base64
2
3 with open(my_image, "rb") as image_file:
4 base64_image = base64.b64encode(image_file.read()).decode('utf-8')
5
6 messages=[
7 {
8 "role": "system",
9 "content": "You are an image classification system. Classify the following images to be either 'cat' or 'dog', in the following format {'type': 'cat/dog'}"
10 },
11 {
12 "role": "user",
13 "content": [
14 {"type": "text", "text": "Classify this image."},
15 {
16 "type": "image_url",
17 "image_url": {
18 "url": f"data:image/jpeg;base64,{base64_image}"
19 },
20 },
21 ],
22 }
23 ]

Using GPT-4o vision with multiple images

GPT-4o can also handle several images per prompt. Add another object of type image_url to your messages. You can even mix base64 and URL images.

Note: Make sure to have a recent version of the OpenAI python package to follow this example. Version 1.40.0 or above is sufficient.

1 with open(my_image, "rb") as image_file:
2 base64_image = base64.b64encode(image_file.read()).decode('utf-8')
3
4 messages=[
5 {
6 "role": "system",
7 "content": "You are an image classification system. Classify the following images to be either 'cat' or 'dog', in the following format {'type': 'cat/dog'}"
8 },
9 {
10 "role": "user",
11 "content": [
12 {"type": "text", "text": "Classify this image."},
13 {
14 "type": "image_url",
15 "image_url": {
16 "url": f"data:image/jpeg;base64,{base64_image}"
17 },
18 },
19 ],
20 },
21 {
22 "role": "user",
23 "content": [
24 {"type": "text", "text": "Classify this image."},
25 {
26 "type": "image_url",
27 "image_url": {
28 "url": "https://upload.wikimedia.org/wikipedia/commons/9/99/Brooks_Chase_Ranger_of_Jolly_Dogs_Jack_Russell.jpg"
29 },
30 },
31 ],
32 }
33 ]

Then send the messages with the OpenAI SDKs or the OpenAI API. Here is the rest of the Python example:

1from openai import OpenAI
2
3client = OpenAI(api_key="your api key")
4# message = ... see the above examples ...
5response = client.chat.completions.create(
6 model="gpt-4o-2024-08-06",
7 messages=messages,
8 max_tokens=300,
9)
10print(response.choices[0].message.content)
11#{'type': 'dog'}

So the whole flow is:

  1. Create the messages array as shown above.
  2. Send the image to GPT-4o with the OpenAI API or SDKs, together with a prompt of your choosing.
  3. Get the results back, shaped by your prompt.

Image detail modes: low, high and auto

You can send images of any resolution and format, and GPT-4o (plus the image pipeline in front of it) will make the best of them. For cost and latency control, OpenAI offers two image detail modes, or actually three:

  • low: The low-res mode. The image is automatically downsampled to 512x512 pixels and costs the equivalent of 85 tokens. This mode is fast and cheap.
  • high: High resolution mode also downsamples the image to 512x512 pixels first and shows the LLM this low-res version, so it gets an overview. Then the image is scaled to fit within 2048x2048 pixels with its shortest side at most 768 pixels, and the model gets detailed crops of that version, at 512x512 pixels each. Each crop costs 170 tokens, so this mode takes significantly longer than low.
  • auto: The default. GPT-4o looks at the size of the original image and picks high or low accordingly.

Note: If you need very detailed analyses of your images, you might opt for the high res option. Keep in mind, though, that the details reach the LLM as 512x512 pixel crops of your original image. If the details you care about are larger than 512x512 pixels, they will not be presented correctly. In that case, either downsample the image manually so that the details are small enough, or use the low res mode.

To set the mode, add a detail key to the image_url object in your messages array:

1"image_url": {
2 "url": "your-url-here",
3 "detail": "high"
4},

Image classification with GPT-4o

Image classification used to be a time-consuming process. You had to select an image classification model and build a training dataset of at least 100 or so images, labeled with the exact categories you want to classify later. Then you decided on a training strategy, trained the model, validated the result and hosted it. Only then could you use the classifier.

With models like GPT-4o, this process is more or less obsolete. They are very good at Zero-Shot classification: you don't need to train them for a specific task, and they can predict classes they never saw during training.

For image classification, this means you send an image to the model, ask it to pick one of the categories you provide, and you'll most likely get a very good result without any model training.

We found that this works best when you briefly describe the classification task in the system message and also list the categories there. The user message then contains only the image.

Note: For some use cases you still want your own image classifier, namely very specific niches. If you want to classify different types of mushrooms, for example, you'd better train your own specialized model. GPT-4o and similar models correctly identify more general things, like different animal types (rather than breeds of one animal).

Here's an example:

1from openai import OpenAI
2
3client = OpenAI(api_key="your api key")
4messages=[
5 {
6 "role": "system",
7 "content": """You are an image classification system, asked to classify animals.
8 Classify the following images to be one of the following categories: ['dog', 'cat', 'cow', 'cangaroo', 'other'].
9 In the following format {'type': '<type of animal>'}"""
10 },
11 {
12 "role": "user",
13 "content": [
14 {"type": "text", "text": "Classify this image."},
15 {
16 "type": "image_url",
17 "image_url": {
18 "url": "https://upload.wikimedia.org/wikipedia/commons/9/99/Brooks_Chase_Ranger_of_Jolly_Dogs_Jack_Russell.jpg",
19 },
20 },
21 ],
22 }
23]
24response = client.chat.completions.create(
25 model="gpt-4o-2024-08-06",
26 messages=messages,
27 max_tokens=300,
28)
29print(response.choices[0].message.content)

Describing images using GPT-4o vision

Another common task is having GPT-4o describe an image. The most important part is to be precise about which aspect of the image you want described, because an image can be described from many angles. Again, the system message is the best place to define what exactly you want.

A typical use case: you store images in a database and use the descriptions to search for them.

1from openai import OpenAI
2
3client = OpenAI(api_key="your api key")
4messages=[
5 {
6 "role": "system",
7 "content": """You are a system to describe images. The images should be stored in a database and your descriptions are used to search and find these images.
8 Therefore, describe the main content of the image which might be relevant for search/find operations.
9 Just provide the description of the image, no other text.
10 Example: House with beautiful garden in small town, sunny wheather, crowd of people in front.
11 """
12 },
13 {
14 "role": "user",
15 "content": [
16 {"type": "text", "text": "Describe this image."},
17 {
18 "type": "image_url",
19 "image_url": {
20 "url": "https://upload.wikimedia.org/wikipedia/commons/9/99/Brooks_Chase_Ranger_of_Jolly_Dogs_Jack_Russell.jpg",
21 },
22 },
23 ],
24 }
25]
26response = client.chat.completions.create(
27 model="gpt-4o-2024-08-06",
28 messages=messages,
29 max_tokens=300,
30)
31print(response.choices[0].message.content)

The example output was:

1A small, alert dog standing on green grass, with a predominantly white coat
2and a brown patch on its head and a black spot on its side. The dog has a short
3tail and a slightly raised head, looking toward the distance, with a
4background of stacked wooden logs.

If the description is too detailed, not detailed enough, or in the wrong style, you just change the system prompt. That's the powerful part: you have a full-blown LLM with all its prompt understanding, and it can now also read and understand images. You can even use few-shot prompting or similar techniques.

Light data analytics using GPT-4o vision

So far, GPT-4o has understood images, created descriptions and labels, and followed simple prompts that define what the response should look like.

Now for something more elaborate. Many thousands of data analysts have built beautiful dashboards with potentially very helpful insights. Yet time and again it turns out that not a single soul looks at these poor, beautiful dashboards. (I can make fun of that, as I was one of those data analysts.)

If most people are too busy to look at analytics results, what if we make AI look at our dashboards? (Yes, I'm aware this sounds a little desperate.)

The strategy is simple: take a screenshot of a dashboard, send it to GPT-4o and ask specific questions about it. One real-world application is an open-ended alerting system: the LLM analyzes a dashboard with data over time and notifies you only when something out of the ordinary happens.

Here, we test whether GPT-4o makes a good dashboard analyst by asking it for a written summary of this dashboard:

Web analytics dashboard with referrers, pageview funnel and daily pageviews

  1. Install the pytest-playwright package, which provides Playwright for python. Playwright navigates to web pages and, among other things, takes screenshots of them.

  2. Automatically create a screenshot of your dashboard.

    This example assumes the dashboard is publicly available. If your dashboard needs authentication, have a look at the playwright documentation.

    1import asyncio
    2from playwright.async_api import async_playwright
    3import base64
    4
    5from openai import OpenAI
    6client = OpenAI(api_key="your api key")
    7
    8async with async_playwright() as p:
    9 browser = await p.chromium.launch()
    10 page = await browser.new_page()
    11 await page.goto('https://your-dashboard-url')
    12 await page.screenshot(path=f'dashboard_gpt4o.png')
    13 await browser.close()
  3. Use the screenshot for GPT-4o vision inference:

    1
    2with open("dashboard_gpt4o.png", "rb") as image_file:
    3 base64_image = base64.b64encode(image_file.read()).decode('utf-8')
    4
    5messages=[
    6 {
    7 "role": "system",
    8 "content": """You are a data analyst. You get an image of a dashboard.
    9 Create a written report based on this dashboard.
    10 The report should be short and concise and should contain the most important metrics.
    11 The report is aimed at high-level management, so make sure it only covers the most prevalent facts.
    12 Focus on actionable insights.
    13 """
    14 },
    15 {
    16 "role": "user",
    17 "content": [
    18 {"type": "text", "text": "Create a written summary of this dashboard."},
    19 {
    20 "type": "image_url",
    21 "image_url": {
    22 "url": f"data:image/png;base64,{base64_image}",
    23 "detail": "high"
    24 },
    25 },
    26 ],
    27 }
    28]
    29response = client.chat.completions.create(
    30 model="gpt-4o",
    31 messages=messages,
    32 max_tokens=1024,
    33)
    34print(response.choices[0].message.content)

This was the response from the LLM:

1### Dashboard Summary Report
2
3#### Referring Domains (Last 14 Days)
4
5- **Top Referring Domain:** www.google.com is the leading referrer with approximately 1,100+ referrals.
6- **Direct Visits:** The second highest source, with around 400 direct visits.
7- **Other Key Referrers:** Medium.com, linkedin.com, and pondhouse-data.com also contribute notably to the referral traffic, with each driving between 100 and 200 referrals.
8
9#### Pageview Funnel (Last 7 Days)
10
11- **First Page View:** 1,149 users (100%) completed the first page view.
12- **Second Page View Drop-off:** 5.3% or 61 users continued to the second page view, indicating a 94.69% drop-off (1,088 users).
13- **Third Page View Drop-off:** From the second page, only 1.9% or 22 users progressed to the third page view, with another substantial drop-off of 63.93% (39 users).
14
15#### Pageview Count by Event's Path Name (Last 30 Days)
16
17- **Daily Variation:** Pageview counts exhibit significant daily fluctuations, generally ranging between 50 to 300 pageviews.
18- **Peak Traffic Day:** Notably, July 18, 2024, saw a peak in pageviews, surpassing 400 views, indicating a potential event or campaign spike.
19
20### Actionable Insights
21
221. **Enhance Engagement Post First Page:** The significant drop-offs after the first page view indicate potential issues in user journey or content engagement beyond the first page.
23 Further analysis and improvement of the user experience after the first page could help retain users.
242. **Focus on Key Referrers:** Investment in strengthening relationships, collaborations or advertising with top referrers, particularly Google, can help maintain or enhance traffic.
25
263. **Investigate High-traffic Days:** The spike in traffic on July 18, 2024, warrants closer inspection to understand what drove the increase. Replicating successful strategies from that day could boost future traffic.
27
284. **Continual Monitoring and Testing:** Regular monitoring is essential to understand trends better and identify smaller opportunities and issues in user traffic and engagement patterns.

That's a remarkably good result, given that we simply threw a dashboard at the model and asked for a summary. I'm especially impressed by how well the model read the charts and found the correct dates (for example the spike of 400 pageviews on July 18th).

If you want GPT-4o to read documents rather than dashboards, we cover that in parsing PDF, Word and Excel documents with GPT-4o. The same idea works with an open-source model: we used one to extract text and tables from complex PDF pages rendered as images.

Costs and pricing of GPT-4o vision for image prompts

How much does GPT-4o vision cost? We'd say it's quite cheap. The following prices are for the GPT-4o variant gpt-4o-2024-08-06, which was new when we wrote this:

  • Any image in low resolution mode: 0.02 cents ($0.000213) or $2 per 10,000 images.
  • 1920 x 1080 in high resolution mode: 0.28 cents ($0.002763) or $2.76 per 1,000 images.

Important note: At the time of writing, the gpt-4o-2024-08-06 model was the cheapest model of the GPT-4o series for vision. Even the otherwise very cost-efficient gpt-4o-mini model is about twice as expensive, with worse output quality. So stick with this model, at least for now.

For more ways to keep model spend down, see our strategies to reduce LLM costs.

Conclusion

GPT-4o vision handles image classification, description and analysis without specialized training or a complex setup. You build a messages array with an image URL or base64 data, put the task and any categories in the system message, and choose a detail mode to balance cost and precision.

In our tests, it classified and described a sample image, and it turned a web analytics dashboard into a management summary with the correct peak date. At around $2 per 10,000 low-res images, it's cheap enough for most use cases. Analyzing visual data such as dashboards is the part we find most intriguing.

As a next step, take one of the examples above and try different prompts and detail modes on your own images to find what works best for your use case.

Further reading

Interested in building high-quality AI agent systems?

We prepared a comprehensive guide based on cutting-edge research for how to build robust, reliable AI agent systems that actually work in production. This guide covers:

  • Understanding the 14 systematic failure modes in multi-agent systems
  • Evidence-based best practices for agent design
  • Structured communication protocols and verification mechanisms
Andreas Nigg

About the author

Andreas Nigg

Co-Founder, Pondhouse Data

Andreas has spent more than eight years building data applications. At Pondhouse Data he designs and ships AI systems and integrates them into complex, real-world environments.