TL：DR;直到有更好的模型卡（虽然https://huggingface.co/shauray/Llava-Llama-2-7B-hf似乎有一个有意义的用法，但我不知道from transformers import LlavaProcessor, LlavaForCausalLM从何而来，见https://huggingface.co/shauray/Llava-Llama-2-7B-hf/discussions/1），S似乎无处不在。所以我受够了，继续使用replicate.com，见https://replicate.com/yorickvp/llava-13b/api

import replicate
output = replicate.run(
    "yorickvp/llava-13b:2facb4a474a0462c15041b78b1ad70952ea46b5ec6ad29583c0b29dbd4249591",
    input=dict(
        image=open("eiffel_tower_336.jpg", "rb"),
        prompt="Describe what is on the photo in great detail, be very verbose"
    )
)
# The yorickvp/llava-13b model can stream output as it's running.
# The predict method returns an iterator, and you can iterate over that output.
for item in output:
    # https://replicate.com/yorickvp/llava-13b/versions/2facb4a474a0462c15041b78b1ad70952ea46b5ec6ad29583c0b29dbd4249591/api#output-schema
    print(item, end="")

字符串
就这么简单。
详细调试：LLaVA GitHub repo应该包含CLI或GUI如何与后端交互。至于CLI：https://github.com/haotian-liu/LLaVA/blob/main/llava/serve/cli.py
1.我们可以看到有一个基于模型类型的conv_mode。在我们的例子中，它是Llama 2 https://github.com/haotian-liu/LLaVA/blob/6bfd90754621c0277672d1418336a31d976c4ec3/llava/serve/cli.py#L34C5-L35C36

if 'llama-2' in model_name.lower():
    conv_mode = "llava_llama_2"

型
这意味着https://github.com/haotian-liu/LLaVA/blob/f47c16e4aeac6d4d61259800ca9cd33b26824113/llava/conversation.py#L277

conv_llava_llama_2 = Conversation(
    system="You are a helpful language and vision assistant. "
           "You are able to understand the visual content that the user provides, "
           "and assist the user with a variety of tasks using natural language.",
    roles=("USER", "ASSISTANT"),
    version="llama_v2",
    messages=(),
    offset=0,
    sep_style=SeparatorStyle.LLAMA_2,
    sep="<s>",
    sep2="</s>",
)

型
我们可以在这里看到分隔符和分隔符样式。LLAMA_2SeparatorStyle：

elif self.sep_style == SeparatorStyle.LLAMA_2:
            wrap_sys = lambda msg: f"<<SYS>>\n{msg}\n<</SYS>>\n\n"
            wrap_inst = lambda msg: f"[INST] {msg} [/INST]"

型
1.图片处理：https://github.com/haotian-liu/LLaVA/blob/f47c16e4aeac6d4d61259800ca9cd33b26824113/llava/serve/cli.py#L54

image = load_image(args.image_file)
    # Similar operation in model_worker.py
    image_tensor = process_images([image], image_processor, args)
    if type(image_tensor) is list:
        image_tensor = [image.to(model.device, dtype=torch.float16) for image in image_tensor]
    else:
        image_tensor = image_tensor.to(model.device, dtype=torch.float16)

型
load_image简单地加载二进制图像数据，而不管它的格式（jpg，png等）。然后process_images将图像转换为Tensor：https://github.com/haotian-liu/LLaVA/blob/f47c16e4aeac6d4d61259800ca9cd33b26824113/llava/mm_utils.py#L28C5-L28C19在转换之前，它将图像规则化为带有填充的方形。为了获得最佳效果，我已经提供了一个方形336 x 336 px或224 x 224 px图像。然后这个Tensor化的图像被插入到提示符中，看起来像是用图像分隔符标记前缀：https://github.com/haotian-liu/LLaVA/blob/f47c16e4aeac6d4d61259800ca9cd33b26824113/llava/serve/cli.py#L75

if model.config.mm_use_im_start_end:
    inp = DEFAULT_IM_START_TOKEN + DEFAULT_IMAGE_TOKEN + DEFAULT_IM_END_TOKEN + '\n' + inp
else:
    inp = DEFAULT_IMAGE_TOKEN + '\n' + inp
conv.append_message(conv.roles[0], inp)

型
对于实际的推断，image_tensor作为参数传递给model。尽管在调用API时，model不存在于客户端。
1.让我们看看GRadio前端如何调用后端：https://github.com/haotian-liu/LLaVA/blob/main/llava/serve/gradio_web_server.py这是有效负载assmebled：https://github.com/haotian-liu/LLaVA/blob/f47c16e4aeac6d4d61259800ca9cd33b26824113/llava/serve/gradio_web_server.py#L224

pload = {
    "model": model_name,
    "prompt": prompt,
    "temperature": float(temperature),
    "top_p": float(top_p),
    "max_new_tokens": min(int(max_new_tokens), 1536),
    "stop": state.sep if state.sep_style in [SeparatorStyle.SINGLE, SeparatorStyle.MPT] else state.sep2,
    "images": f'List of {len(state.get_images())} images: {all_image_hash}',
}

型
然后images被替换为：pload['images'] = state.get_images()。state在对话对象中，我们已经看过了：https://github.com/haotian-liu/LLaVA/blob/f47c16e4aeac6d4d61259800ca9cd33b26824113/llava/conversation.py#L109它对图像进行了加密，看起来也像base64对PNG格式的图像进行了编码。
我厌倦了尝试组合，我一直得到

ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from primary with message "{
  "code": 400,
  "type": "InternalServerException",
  "message": "\u0027llava\u0027"
}

型

LLaVa现在在Transformers库中得到了原生支持，现在应该更容易部署了。
电话：https://huggingface.co/docs/transformers/main/model_doc/llava
集线器上的检查点：https://huggingface.co/llava-hf