AIGatewayTargetModelDatabricksConfig
objectDatabricks-specific configuration for a model.
The number of dimensions for embedding outputs.
The maximum number of tokens to generate in the response.
Cost per input token for billing and cost tracking.
Cost per output token for billing and cost tracking.
Controls randomness in the model output. Higher values produce more varied responses.
Limits the number of highest-probability tokens considered during generation.
Nucleus sampling probability mass. Tokens with cumulative probability up to top_p are considered.
The upstream URL for the model endpoint.
Allowed values:databricks
The Databricks workspace instance ID.
AIGatewayTargetModelGeminiConfig
objectGoogle Gemini-specific configuration for a model.
The number of dimensions for embedding outputs.
The maximum number of tokens to generate in the response.
Cost per input token for billing and cost tracking.
Cost per output token for billing and cost tracking.
Controls randomness in the model output. Higher values produce more varied responses.
Limits the number of highest-probability tokens considered during generation.
Nucleus sampling probability mass. Tokens with cumulative probability up to top_p are considered.
The upstream URL for the model endpoint.
Allowed values:gemini
The Google Cloud location ID for the model endpoint.
The custom API endpoint for the Gemini model.
AIGatewayTargetModelHuggingfaceConfig
objectHugging Face-specific configuration for a model.
The number of dimensions for embedding outputs.
The maximum number of tokens to generate in the response.
Cost per input token for billing and cost tracking.
Cost per output token for billing and cost tracking.
Controls randomness in the model output. Higher values produce more varied responses.
Limits the number of highest-probability tokens considered during generation.
Nucleus sampling probability mass. Tokens with cumulative probability up to top_p are considered.
The upstream URL for the model endpoint.
Allowed values:huggingface
Whether to use the Hugging Face inference cache.
Default:false
Whether to wait for the model to load if it is not ready.
Default:false
AIGatewayTargetModelLlama2Config
objectLlama2-specific configuration for a model.
The number of dimensions for embedding outputs.
The maximum number of tokens to generate in the response.
Cost per input token for billing and cost tracking.
Cost per output token for billing and cost tracking.
Controls randomness in the model output. Higher values produce more varied responses.
Limits the number of highest-probability tokens considered during generation.
Nucleus sampling probability mass. Tokens with cumulative probability up to top_p are considered.
The upstream URL for the model endpoint.
Allowed values:llama2
The request format to use when communicating with the Llama2 model.
Allowed values:ollamaopenairaw
AIGatewayTargetModelMistralConfig
objectMistral-specific configuration for a model.
The number of dimensions for embedding outputs.
The maximum number of tokens to generate in the response.
Cost per input token for billing and cost tracking.
Cost per output token for billing and cost tracking.
Controls randomness in the model output. Higher values produce more varied responses.
Limits the number of highest-probability tokens considered during generation.
Nucleus sampling probability mass. Tokens with cumulative probability up to top_p are considered.
The upstream URL for the model endpoint.
Allowed values:mistral
The request format to use when communicating with the Mistral model.
Allowed values:ollamaopenai