Skip to content

quickstart.mdx

Latest commit

 

History

History
170 lines (142 loc) · 4.03 KB

quickstart.mdx

File metadata and controls

170 lines (142 loc) · 4.03 KB
Raw
title Quickstart
description Cortex Quickstart.
slug quickstart

import Tabs from "@theme/Tabs"; import TabItem from "@theme/TabItem";

Local Installation

Cortex has a Local Installer with all of the required dependencies, so that once you've downloaded it, no internet connection is required during the installation process.

Start a Cortex Server

This command starts the Cortex's' API server at localhost:39281. 

cortex start
```sh cortex.exe start ```

Pull Models

This command allows users to download a model from these Model Hubs:

It displays available quantizations, recommends a default and downloads the desired quantization.

The following two options will show you all of the available models under those names. Cortex will first search on its own hub for models like `llama3.3`, and in huggingface for hyper specific ones like `bartowski/Meta-Llama-3.1-8B-Instruct-GGU`. ```sh cortex pull llama3.3 ``` or, 
cortex pull bartowski/Meta-Llama-3.1-8B-Instruct-GGUF
```sh cortex pull llama3.3 ``` ```sh cortex.exe pull bartowski/Meta-Llama-3.1-8B-Instruct-GGUF ```

Run a Model

This command downloads the default gguf model (if not available in your file system) from the Cortex Hub, starts the model, and chat with the model.

```sh cortex run llama3.3 ``` ```sh cortex.exe run llama3.3 ```

:::info All model files are stored in the ~/cortex/models folder. :::

Using the Model

API

curl http://localhost:39281/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
  "model": "llama3.1:8b-gguf",
  "messages": [
    {
      "role": "user",
      "content": "Hello"
    },
  ],
  "stream": true,
  "max_tokens": 1,
  "stop": [
      null
  ],
  "frequency_penalty": 1,
  "presence_penalty": 1,
  "temperature": 1,
  "top_p": 1
}'

Refer to our API documentation for more details.

Show the System State

This command displays the running model and the hardware system status (RAM, Engine, VRAM, Uptime).

```sh cortex ps ``` ```sh cortex.exe ps ```

Stop a Model

This command stops the running model.

```sh cortex models stop llama3.3 ``` ```sh cortex.exe models stop llama3.3 ```

Stop a Cortex Server

This command stops the Cortex.cpp API server at localhost:39281 or whichever other port you used to start cortex.

```sh cortex stop ``` ```sh cortex.exe stop ```

What's Next?

Now that Cortex is set up, you can continue on to any of the following sections:

  • Adjust the folder path and configuration using the .cortexrc file.
  • Explore the Cortex's data folder to understand how data gets stored.
  • Learn about the structure of the model.yaml file in Cortex.