diff options
| author | Nate Sesti <sestinj@gmail.com> | 2023-09-12 01:23:05 -0700 |
|---|---|---|
| committer | Nate Sesti <sestinj@gmail.com> | 2023-09-12 01:23:05 -0700 |
| commit | 2197552db697d36f3f4d8107d4c41bafbca5f621 (patch) | |
| tree | 56c3de3f3902ee54b2df42b6ce2b904d95935a28 /docs | |
| parent | db3277bdc207cc8233ccc4c1d44a5c8e50785769 (diff) | |
| download | sncontinue-2197552db697d36f3f4d8107d4c41bafbca5f621.tar.gz sncontinue-2197552db697d36f3f4d8107d4c41bafbca5f621.tar.bz2 sncontinue-2197552db697d36f3f4d8107d4c41bafbca5f621.zip | |
docs: :memo: update to models docs
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/docs/customization/models.md | 42 | ||||
| -rw-r--r-- | docs/docs/reference/Models/openai.md | 3 |
2 files changed, 14 insertions, 31 deletions
diff --git a/docs/docs/customization/models.md b/docs/docs/customization/models.md index 93ea2a57..d9d1aa2b 100644 --- a/docs/docs/customization/models.md +++ b/docs/docs/customization/models.md @@ -4,22 +4,22 @@ Continue makes it easy to swap out different LLM providers. Once you've added an Commercial Models -- [MaybeProxyOpenAI](#adding-an-openai-api-key) (default) - Use gpt-4 or gpt-3.5-turbo free with our API key, or with your API key. gpt-4 is probably the most capable model of all options. -- [OpenAI](#azure-openai-service) - Use any OpenAI model with your own key. Can also change the base URL if you have a server that uses the OpenAI API format, including using the Azure OpenAI service, LocalAI, etc. -- [AnthropicLLM](#claude-2) - Use claude-2 with your Anthropic API key. Claude 2 is also highly capable, and has a 100,000 token context window. +- [MaybeProxyOpenAI](../reference/Models/maybe_proxy_openai.md) (default) - Use gpt-4 or gpt-3.5-turbo free with our API key, or with your API key. gpt-4 is probably the most capable model of all options. +- [OpenAI](../reference/Models/openai.md) - Use any OpenAI model with your own key. Can also change the base URL if you have a server that uses the OpenAI API format, including using the Azure OpenAI service, LocalAI, etc. +- [AnthropicLLM](../reference/Models/anthropic.md) - Use claude-2 with your Anthropic API key. Claude 2 is also highly capable, and has a 100,000 token context window. Local Models -- [Ollama](#run-llama-2-locally-with-ollama) - If you have a Mac, Ollama is the simplest way to run open-source models like Code Llama. -- [OpenAI](#local-models-with-openai-compatible-server) - If you have access to an OpenAI-compatible server (e.g. llama-cpp-python, LocalAI, FastChat, TextGenWebUI, etc.), you can use the `OpenAI` class and just change the base URL. -- [GGML](#local-models-with-ggml) - An alternative way to connect to OpenAI-compatible servers. Will use `aiohttp` directly instead of the `openai` Python package. -- [LlamaCpp](#llamacpp) - Build llama.cpp from source and use its built-in API server. +- [Ollama](../reference/Models/ollama.md) - If you have a Mac, Ollama is the simplest way to run open-source models like Code Llama. +- [OpenAI](../reference/Models/openai.md) - If you have access to an OpenAI-compatible server (e.g. llama-cpp-python, LocalAI, FastChat, TextGenWebUI, etc.), you can use the `OpenAI` class and just change the base URL. +- [GGML](../reference/Models/ggml.md) - An alternative way to connect to OpenAI-compatible servers. Will use `aiohttp` directly instead of the `openai` Python package. +- [LlamaCpp](../reference/Models/llamacpp.md) - Build llama.cpp from source and use its built-in API server. Open-Source Models (not local) -- [TogetherLLM](#together) - Use any model from the [Together Models list](https://docs.together.ai/docs/models-inference) with your Together API key. -- [ReplicateLLM](#replicate) - Use any open-source model from the [Replicate Streaming List](https://replicate.com/collections/streaming-language-models) with your Replicate API key. -- [HuggingFaceInferenceAPI](#huggingface) - Use any open-source model from the [Hugging Face Inference API](https://huggingface.co/inference-api) with your Hugging Face token. +- [TogetherLLM](../reference/Models/together.md) - Use any model from the [Together Models list](https://docs.together.ai/docs/models-inference) with your Together API key. +- [ReplicateLLM](../reference/Models/replicate.md) - Use any open-source model from the [Replicate Streaming List](https://replicate.com/collections/streaming-language-models) with your Replicate API key. +- [HuggingFaceInferenceAPI](../reference/Models/hf_inference_api.md) - Use any open-source model from the [Hugging Face Inference API](https://huggingface.co/inference-api) with your Hugging Face token. ## Change the default LLM @@ -41,29 +41,11 @@ The `default` and `medium` properties are different _model roles_. This allows d Below, we describe the `LLM` classes available in the Continue core library, and how they can be used. -## Adding an OpenAI API key - -## claude-2 - -## Run Llama-2 locally with Ollama - -## Local models with OpenAI-compatible server - -## Local models with ggml - -## Llama.cpp - -## Together - -## Replicate - -## Hugging Face - ## Self-hosting an open-source model -If you want to self-host on Colab, RunPod, HuggingFace, Haven, or another hosting provider you will need to wire up a new LLM class. It only needs to implement 3 primary methods: `stream_complete`, `complete`, and `stream_chat`, and you can see examples in `continuedev/src/continuedev/libs/llm`. +If you want to self-host on Colab, RunPod, HuggingFace, Haven, or another hosting provider, you will need to wire up a new LLM class. It only needs to implement 3 primary methods: `stream_complete`, `complete`, and `stream_chat`, and you can see examples in `continuedev/src/continuedev/libs/llm`. -If by chance the provider has the exact same API interface as OpenAI, the `GGML` class will work for you out of the box, after changing the endpoint at the top of the file. +If by chance the provider has the exact same API interface as OpenAI, the `OpenAI` class will work for you out of the box, after changing only the `api_base` parameter. ## Azure OpenAI Service diff --git a/docs/docs/reference/Models/openai.md b/docs/docs/reference/Models/openai.md index 0ade1a8f..4eb4906f 100644 --- a/docs/docs/reference/Models/openai.md +++ b/docs/docs/reference/Models/openai.md @@ -23,6 +23,7 @@ config = ContinueConfig( Options for serving models locally with an OpenAI-compatible server include: +- [LM Studio](https://lmstudio.ai/) - [text-gen-webui](https://github.com/oobabooga/text-generation-webui/tree/main/extensions/openai#setup--installation) - [FastChat](https://github.com/lm-sys/FastChat/blob/main/docs/openai_api.md) - [LocalAI](https://localai.io/basics/getting_started/) @@ -36,4 +37,4 @@ Options for serving models locally with an OpenAI-compatible server include: ### Inherited Properties -<ClassPropertyRef name='model' details='{"title": "Model", "description": "The name of the model to be used (e.g. gpt-4, codellama)", "type": "string"}' required={true} default=""/><ClassPropertyRef name='api_key' details='{"title": "Api Key", "description": "OpenAI API key", "type": "string"}' required={true} default=""/><ClassPropertyRef name='title' details='{"title": "Title", "description": "A title that will identify this model in the model selection dropdown", "type": "string"}' required={false} default=""/><ClassPropertyRef name='system_message' details='{"title": "System Message", "description": "A system message that will always be followed by the LLM", "type": "string"}' required={false} default=""/><ClassPropertyRef name='context_length' details='{"title": "Context Length", "description": "The maximum context length of the LLM in tokens, as counted by count_tokens.", "default": 2048, "type": "integer"}' required={false} default="2048"/><ClassPropertyRef name='unique_id' details='{"title": "Unique Id", "description": "The unique ID of the user.", "type": "string"}' required={false} default=""/><ClassPropertyRef name='timeout' details='{"title": "Timeout", "description": "Set the timeout for each request to the LLM. If you are running a local LLM that takes a while to respond, you might want to set this to avoid timeouts.", "default": 300, "type": "integer"}' required={false} default="300"/><ClassPropertyRef name='prompt_templates' details='{"title": "Prompt Templates", "description": "A dictionary of prompt templates that can be used to customize the behavior of the LLM in certain situations. For example, set the \"edit\" key in order to change the prompt that is used for the /edit slash command. Each value in the dictionary is a string templated in mustache syntax, and filled in at runtime with the variables specific to the situation. See the documentation for more information.", "default": {}, "type": "object"}' required={false} default="{}"/>
\ No newline at end of file +<ClassPropertyRef name='model' details='{"title": "Model", "description": "The name of the model to be used (e.g. gpt-4, codellama)", "type": "string"}' required={true} default=""/><ClassPropertyRef name='api_key' details='{"title": "Api Key", "description": "OpenAI API key", "type": "string"}' required={true} default=""/><ClassPropertyRef name='title' details='{"title": "Title", "description": "A title that will identify this model in the model selection dropdown", "type": "string"}' required={false} default=""/><ClassPropertyRef name='system_message' details='{"title": "System Message", "description": "A system message that will always be followed by the LLM", "type": "string"}' required={false} default=""/><ClassPropertyRef name='context_length' details='{"title": "Context Length", "description": "The maximum context length of the LLM in tokens, as counted by count_tokens.", "default": 2048, "type": "integer"}' required={false} default="2048"/><ClassPropertyRef name='unique_id' details='{"title": "Unique Id", "description": "The unique ID of the user.", "type": "string"}' required={false} default=""/><ClassPropertyRef name='timeout' details='{"title": "Timeout", "description": "Set the timeout for each request to the LLM. If you are running a local LLM that takes a while to respond, you might want to set this to avoid timeouts.", "default": 300, "type": "integer"}' required={false} default="300"/><ClassPropertyRef name='prompt_templates' details='{"title": "Prompt Templates", "description": "A dictionary of prompt templates that can be used to customize the behavior of the LLM in certain situations. For example, set the \"edit\" key in order to change the prompt that is used for the /edit slash command. Each value in the dictionary is a string templated in mustache syntax, and filled in at runtime with the variables specific to the situation. See the documentation for more information.", "default": {}, "type": "object"}' required={false} default="{}"/> |