LLMs: Ollama, vLLM, Hugging Face, LangChain, and Open WebUI

$ ollama serve --help
Start ollama

Usage:
  ollama serve [flags]

Aliases:
  serve, start

Flags:
  -h, --help   help for serve

Environment Variables:
      OLLAMA_DEBUG               Show additional debug information (e.g. OLLAMA_DEBUG=1)
      OLLAMA_HOST                IP Address for the ollama server (default 127.0.0.1:11434)
      OLLAMA_KEEP_ALIVE          The duration that models stay loaded in memory (default "5m")
      OLLAMA_MAX_LOADED_MODELS   Maximum number of loaded models (default 1)
      OLLAMA_MAX_QUEUE           Maximum number of queued requests
      OLLAMA_MODELS              The path to the models directory
      OLLAMA_NUM_PARALLEL        Maximum number of parallel requests (default 1)
      OLLAMA_NOPRUNE             Do not prune model blobs on startup
      OLLAMA_ORIGINS             A comma separated list of allowed origins
      OLLAMA_TMPDIR              Location for temporary files
      OLLAMA_FLASH_ATTENTION     Enabled flash attention
      OLLAMA_LLM_LIBRARY         Set LLM library to bypass autodetection
      OLLAMA_MAX_VRAM            Maximum VRAM

//  ensure that the model checkpoint is prepared.
$ ollama list
NAME                    ID              SIZE    MODIFIED
phi3:mini               64c1188f2485    2.4 GB  17 minutes ago

curl http://localhost:11434/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"messages":[{"role":"user","content":"Say this is a test"}],"model":"phi3:mini"}'

pip install openai

from openai import OpenAI
client = OpenAI(
    base_url='http://localhost:11434/v1/',
    api_key='ollama',  # required but ignored
)
chat_completion = client.chat.completions.create(
    messages=[
        {
            'role': 'user',
            'content': 'Say this is a test',
        }
    ],
    model='phi3:mini',
)

# The official .NET library for the OpenAI API
dotnet add package OpenAI --prerelease

using OpenAI.Chat;

ChatClient client = new(
    model: "phi3:mini",
    credential: "EMPTY_OPENAI_API_KEY",
    options: new OpenAI.OpenAIClientOptions { Endpoint = new Uri("http://localhost:11434/v1/") });

ChatCompletion completion = client.CompleteChat("Say 'this is a test.'");

Console.WriteLine($"[ASSISTANT]: {completion}");

pip install vllm

$ pip list | egrep 'vllm|transformers'
transformers                      4.41.2
vllm                              0.5.0
vllm-flash-attn                   2.5.9

$ python -m vllm.entrypoints.openai.api_server --help
vLLM OpenAI-Compatible RESTful API server.

options:
  --host HOST           host name
  --port PORT           port number
  --api-key API_KEY     If provided, the server will require this key to be presented in the header.
  --model MODEL         Name or path of the huggingface model to use.
  --max-model-len MAX_MODEL_LEN
                        Model context length. If unspecified, will be automatically derived from the model config.
  --gpu-memory-utilization GPU_MEMORY_UTILIZATION
                        The fraction of GPU memory to be used for the model executor, which can range from 0 to 1. For example, a value of 0.5 would imply 50% GPU memory utilization. If unspecified, will use
                        the default value of 0.9.
  --served-model-name SERVED_MODEL_NAME [SERVED_MODEL_NAME ...]
                        The model name(s) used in the API. If multiple names are provided, the server will respond to any of the provided names. The model name in the model field of a response will be the
                        first name in this list. If not specified, the model name will be the same as the `--model` argument. Noted that this name(s)will also be used in `model_name` tag content of
                        prometheus metrics, if multiple names provided, metricstag will take the first one.

# Start an OpenAI-compatible API service
python -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2-0.5B-Instruct

pip install -U "huggingface_hub[cli]"

# If saw connection to https://huggingface.co/ failed, uncomment the following line:
# ENV HF_ENDPOINT=https://hf-mirror.com

huggingface-cli download sentence-transformers/all-MiniLM-L6-v2