added vllm examples

README.md

@@ -114,6 +114,110 @@ print(f'User: {question}\nAssistant: {response}')
 
 ```
 
+### Inference with vLLM
+h2ovl-mississippi models are also supported by vllm [v0.6.4](https://github.com/vllm-project/vllm/releases/tag/v0.6.4) and later version.
+
+First install vllm 
+```bash
+pip install vllm
+```
+
+### Offline inference
+```python
+from vllm import LLM, SamplingParams
+from transformers import AutoTokenizer
+from PIL import Image
+
+question = "Describe this image in detail"
+image = Image.open("assets/a_cat.png")
+model_name = "h2oai/h2ovl-mississippi-2b"
+
+
+llm = LLM(
+    model=model_name,
+)
+
+tokenizer = AutoTokenizer.from_pretrained(model_name,
+                                            trust_remote_code=True)
+
+messages = [{'role': 'user', 'content': f"<image>\n{question}"}]
+prompt = tokenizer.apply_chat_template(messages,
+                                        tokenize=False,
+                                        add_generation_prompt=True)
+
+# Stop tokens for H2OVL-Mississippi
+# https://huggingface.co/h2oai/h2ovl-mississippi-2b
+stop_token_ids = [tokenizer.eos_token_id]
+
+sampling_params = SamplingParams(n=1,
+                                 temperature=0.8, 
+                                 top_p=0.8,
+                                 seed=777, # Seed for reprodicibility
+                                 max_tokens=1024,
+                                 stop_token_ids=stop_token_ids)
+
+# Single prompt inference
+outputs = llm.generate({
+    "prompt": prompt,
+    "multi_modal_data": {"image": image},
+},
+sampling_params=sampling_params)
+
+# look at the output
+for o in outputs:
+    generated_text = o.outputs[0].text
+    print(generated_text)
+
+```
+Pleaes see more examples at https://docs.vllm.ai/en/latest/models/vlm.html#offline-inference 
+
+
+
+### Online inference with OpenAI-Compatible Vision API
+Run the following command to start the vLLM server with the h2ovl-mississippi-2b model:
+```bash
+vllm serve h2oai/h2ovl-mississippi-2b --dtype auto --api-key token-abc123
+```
+
+```python
+from openai import OpenAI
+client = OpenAI(
+    base_url="http://0.0.0.0:8000/v1",
+    api_key="token-abc123",
+)
+
+# check the model name
+model_name = client.models.list().data[0].id
+print(model_name)
+
+# use chat completion api
+response = client.chat.completions.create(
+    model=model_name,
+    messages=[{
+        'role':
+        'user',
+        'content': [{
+            'type': 'text',
+            'text': 'describe this image in detail',
+        }, {
+            'type': 'image_url',
+            'image_url': {
+                'url':
+                # an image example from https://galaxyofai.com/opencv-with-python-full-tutorial-for-data-science/
+                # this is a cat
+                'https://galaxyofai.com/wp-content/uploads/2023/04/image-42.png',
+            },
+        }],
+    }],
+    temperature=0.8,
+    top_p=0.8)
+print(response)
+
+
+```
+Please see more examples at https://docs.vllm.ai/en/latest/models/vlm.html#online-inference
+
+
 
 ## Prompt Engineering for JSON Extraction