Documentation menu

Image to Text

Caption images and ask questions about them in PHP with local vision-language models such as SmolVLM and BLIP. CLI and PHP examples and options.

Describe an image, write a caption or ask a question about what is in a picture, with a local transformers image-to-text model.

Setup#

Terminal 
vendor/bin/loves-ai setup image-to-text
vendor/bin/loves-ai pull HuggingFaceTB/SmolVLM-256M-Instruct

Supported models#

Both kinds work:

  • Vision-language models (task image-text-to-text), e.g. HuggingFaceTB/SmolVLM-256M-Instruct or HuggingFaceTB/SmolVLM-500M-Instruct. They answer a question about the image; without one, they describe it.
  • Captioning models (task image-to-text), e.g. Salesforce/blip-image-captioning-base or nlpconnect/vit-gpt2-image-captioning. They write a short caption; a prompt is the start of the caption, e.g. "a photography of".

The runner rejects models it cannot load before starting, with an explanation: text-only models, GGUF files, ONNX-only repositories and models that need their own Python code (such as Florence-2 or Moondream).

From the command line#

Terminal 
vendor/bin/loves-ai image-to-text HuggingFaceTB/SmolVLM-256M-Instruct photo.jpg "What are the animals doing?"
Output 
👀 Studying your picture with HuggingFaceTB/SmolVLM-256M-Instruct… Grab a warm drink while it finds the right words ☕
If you wish to see all logs, re-run the command with the "--debug" option.
🎉 HuggingFaceTB/SmolVLM-256M-Instruct says:
The animals are sleeping.

Options#

Option Meaning
--max-new-tokens=N Maximum length of the text in tokens (default: 256)
--temperature=T Randomness: 0 always picks the likeliest words (default: the model's own)
--seed=N Random seed, for reproducible text
--device=DEVICE cpu, cuda, mps… (default: the best available)
--log-file=PATH Append the runner's output to this file
--debug Show the runner's output, and the text as it is written

Defaults come from config/image-to-text.php (log_file).

From PHP#

PHP 
use PhpLovesAi\Runner\ImageToText;

// Finds the runner and the pulled model in the project's .local directory by itself.
$imageToText = new ImageToText();

$description = $imageToText->generate('HuggingFaceTB/SmolVLM-256M-Instruct', storage_path('app/photo.jpg'));

$answer = $imageToText->generate(
    model: 'HuggingFaceTB/SmolVLM-256M-Instruct',
    imagePath: storage_path('app/photo.jpg'),
    prompt: 'Is there any text in this image? Write it out.',
    maxNewTokens: 100,
    temperature: 0.0,
);

Errors#

  • ImageNotFoundException when the image file does not exist
  • BinaryNotInstalledException when setup image-to-text has not been run
  • ModelNotFoundException when the model was not pulled yet
  • UnsupportedModelException when the model cannot read images
  • RunFailedException, with the runner's error output, when generation fails, e.g. because the file is not an image