NVIDIA · sumitkbh · Nov 9, 2023 · Nov 9, 2023
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -0,0 +1,11 @@
+repos:
+  - repo: https://github.com/Lucas-C/pre-commit-hooks
+    rev: v1.4.2
+    hooks:
+      - id: insert-license
+        files: ^RetrievalAugmentedGeneration/
+        exclude: ^RetrievalAugmentedGeneration/llm-inference-server/conversion_scripts/|^RetrievalAugmentedGeneration/llm-inference-server/ensemble_models
+        types: [python]
+        args:
+          - --license-filepath
+          - RetrievalAugmentedGeneration/LICENSE.md
diff --git a/RetrievalAugmentedGeneration/.gitattributes b/RetrievalAugmentedGeneration/.gitattributes
@@ -0,0 +1 @@
+notebooks/dataset.zip filter=lfs diff=lfs merge=lfs -text
diff --git a/RetrievalAugmentedGeneration/.gitignore b/RetrievalAugmentedGeneration/.gitignore
@@ -0,0 +1,25 @@
+# Python Exclusions
+.venv
+__pycache__
+
+# Sphinx Exclusions
+_build
+
+# Helm Exclusions
+**/charts/*.tgz
+
+# project temp files
+deploy/*.log
+deploy/*.txt
+**/my.*
+**/my-*
+
+# Next JS Exclusions
+**/.next
+frontend/frontend_js/out
+frontend-sdxl/frontend_js/out
+**/node_modules
+
+# Docker Compose exclusions
+volumes/
+uploaded_files/
diff --git a/RetrievalAugmentedGeneration/Dockerfile b/RetrievalAugmentedGeneration/Dockerfile
@@ -0,0 +1,11 @@
+ARG BASE_IMAGE_URL=nvcr.io/nvidia/pytorch
+ARG BASE_IMAGE_TAG=23.08-py3
+
+
+FROM ${BASE_IMAGE_URL}:${BASE_IMAGE_TAG}
+COPY chain_server /opt/chain_server
+RUN --mount=type=bind,source=requirements.txt,target=/opt/requirements.txt \
+    python3 -m pip install --no-cache-dir -r /opt/requirements.txt
+
+WORKDIR /opt
+ENTRYPOINT ["uvicorn", "chain_server.server:app"]
diff --git a/RetrievalAugmentedGeneration/Dockerfile.notebooks b/RetrievalAugmentedGeneration/Dockerfile.notebooks
@@ -0,0 +1,28 @@
+# Use a base image with Python
+FROM python:3.10-slim
+
+# Set working directory
+WORKDIR /app
+
+#COPY notebooks
+COPY notebooks/*.ipynb .
+
+RUN mkdir -p /app/imgs
+
+COPY notebooks/dataset.zip .
+
+COPY notebooks/imgs/* imgs/
+
+COPY chain_server/trt_llm.py .
+
+COPY notebooks/requirements.txt .
+#Run pip dependencies
+RUN pip3 install -r requirements.txt
+
+RUN apt-get update && apt-get install -y unzip wget git libgl1-mesa-glx libglib2.0-0
+
+# Expose port 8888 for JupyterLab
+EXPOSE 8888
+
+# Start JupyterLab when the container runs
+CMD ["jupyter", "lab", "--allow-root", "--ip=0.0.0.0","--NotebookApp.token=''", "--port=8888"]
diff --git a/RetrievalAugmentedGeneration/README.md b/RetrievalAugmentedGeneration/README.md
@@ -0,0 +1,156 @@
+# Retrieval Augmented Generation
+
+## Project Details
+**Project Goal**: An external reference for a chatbot to question answer off public press releases & tech blogs. Performs document ingestion & Q&A interface using best open models in any cloud or customer datacenter, leverages the power of GPU-accelerated Milvus for efficient vector storage and retrieval, along with TRT-LLM, to achieve lightning-fast inference speeds with custom LangChain LLM wrapper.
+
+## Components
+- **LLM**: Llama2 -- 7b, 13b, and 70b all supported. 13b and 70b generate good responses. Wanted best open-source model available at the time of creation.
+- **LLM Backend**: TRT-LLM for speed.
+- **Vector DB**: Milvus because it's GPU accelerated.
+- **Embedding Model**: e5-large-v2 since it appeared to be one of the best embedding model available at the moment.
+- **Framework(s)**: LangChain and LlamaIndex.
+
+This reference workflow uses a variety of components and services to customize and deploy the RAG based chatbot. The following diagram illustrates how they work together. Refer to the [detailed architecture guide](./docs/architecture.md) to understand more about these components and how they are tied together.
+
+
+![Diagram](./../RetrievalAugmentedGeneration/images/image3.jpg)
+
+
+# Getting Started
+This section covers step by step guide to setup and try out this example workflow.
+
+## Prerequisites
+Before proceeding with this guide, make sure you meet the following prerequisites:
+
+- You should have at least one NVIDIA GPU. For this guide, we used an A100 data center GPU.
+
+    - NVIDIA driver version 535 or newer. To check the driver version run: ``nvidia-smi --query-gpu=driver_version --format=csv,noheader``.
+    - If you are running multiple GPUs they must all be set to the same mode (ie Compute vs. Display). You can check compute mode for each GPU using
+    ``nvidia-smi -q -d compute``
+
+### Setup the following
+
+- Docker and Docker-Compose are essential. Please follow the [installation instructions](https://docs.docker.com/engine/install/ubuntu/).
+
+        Note:
+            Please do **not** use Docker that is packaged with Ubuntu as the newer version of Docker is required for proper Docker Compose support.
+
+            Make sure your user account is able to execute Docker commands.
+
+
+- NVIDIA Container Toolkit is also required. Refer to the [installation instructions](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html).
+
+
+- NGC Account and API Key
+
+    - Please refer to [instructions](https://docs.nvidia.com/ngc/gpu-cloud/ngc-overview/index.html) to create account and generate NGC API key.
+    - Docker login to `nvcr.io` using the following command:
+      ```
+        docker login nvcr.io
+      ```
+
+- You can download Llama2 Chat Model Weights from [Meta](https://ai.meta.com/resources/models-and-libraries/llama-downloads/) or [HuggingFace](https://huggingface.co/meta-llama/Llama-2-13b-chat-hf/).
+
+    **Note for checkpoint downloaded using Meta**:
+
+        When downloading model weights from Meta, you can follow the instructions up to the point of downloading the models using ``download.sh``. There is no need to deploy the model using the steps mentioned in the repository. We will use Triton to deploy the model.
+
+        Meta will download two additional files, namely tokenizer.model and tokenizer_checklist.chk, outside of the model checkpoint directory. Ensure that you copy these files into the same directory as the model checkpoint directory.
+
+
+    **Note**:
+
+        In this workflow, we will be leveraging a Llama2 (13B parameters) chat model, which requires 50 GB of GPU memory.  If you prefer to leverage 7B parameter model, this will require 38GB memory. The 70B parameter model initially requires 240GB memory.
+        IMPORTANT:  For this initial version of the workflow, an A100 GPU is supported.
+
+
+## Install Guide
+###  Step 1: Move to deploy directory
+    cd deploy
+
+###  Step 2: Set Environment Variables
+
+Modify ``compose.env`` in the ``deploy`` directory to set your environment variables. The following variables are required.
+
+    # full path to the local copy of the model weights
+    export MODEL_DIRECTORY="$HOME/src/Llama-2-13b-chat-hf"
+
+    # the architecture of the model. eg: llama
+    export MODEL_ARCHITECTURE="llama"
+
+    # the name of the model being used - only for displaying on frontend
+    export MODEL_NAME="llama-2-13b-chat"
+
+    # [OPTIONAL] the config file for chain server
+    APP_CONFIG_FILE=/dev/null
+
+
+### Step 3: Build and Start Containers
+- Pull lfs files. This will pull large files from repository.
+    ```
+        git lfs pull
+    ```
+- Run the following command to build containers.
+    ```
+        source compose.env; docker compose build
+    ```
+
+- Run the following command to start containers.
+    ```
+        source compose.env; docker compose up -d
+    ```
+    > ⚠️ **NOTE**: It will take a few minutes for the containers to come up and may take up to 5 minutes for the Triton server to be ready. Adding the `-d` flag will have the services run in the background. ⚠️
+
+- Run ``docker ps -a``. When the containers are ready the output should look similar to the image below.
+    ![Docker Output](./images/docker-output.png "Docker Output Image")
+
+### Step 4: Experiment with RAG in JupyterLab
+
+This AI Workflow includes Jupyter notebooks which allow you to experiment with RAG.
+
+- Using a web browser, type in the following URL to open Jupyter
+
+    ``http://host-ip:8888``
+
+- Locate the LLM Streaming Client notebook ``01-llm-streaming-client.ipynb`` which demonstrates how to stream responses from the LLM.
+
+- Proceed with the next 4 notebooks:
+
+    - [Document Question-Answering with LangChain](../notebooks/02_langchain_simple.ipynb)
+
+    - [Document Question-Answering with LlamaIndex](../notebooks/03_llama_index_simple.ipynb)
+
+    - [Advanced Document Question-Answering with LlamaIndex](../notebooks/04_llamaindex_hier_node_parser.ipynb)
+
+    - [Interact with REST FastAPI Server](../notebooks/05_dataloader.ipynb)
+
+### Step 5: Run the Sample Web Application
+A sample chatbot web application is provided in the workflow. Requests to the chat system are wrapped in FastAPI calls.
+
+- Open the web application at ``http://host-ip:8090``.
+
+- Type in the following question without using a knowledge base: "How many cores are on the Nvidia Grace superchip?"
+
+    **Note:** the chatbot mentions the chip doesn't exist.
+
+- To use a knowledge base:
+
+    - Click the **Knowledge Base** tab and upload the file [dataset.zip](./RetrievalAugmentedGeneration/notebook/dataset.zip).
+
+- Return to **Converse** tab and check **[X] Use knowledge base**.
+
+- Retype the question:  "How many cores are on the Nvidia Grace superchip?"
+
+
+# Learn More
+1. [Architecture Guide](./docs/architecture.md): Detailed explanation of different components and how they are tried up together.
+2. Component Guides: Component specific features are enlisted in these sections.
+   1. [Chain Server](./docs/chat_server.md)
+   2. [NeMo Framework Inference Server](./docs/llm_inference_server.md)
+   3. [Jupyter Server](./docs/jupyter_server.md)
+   4. [Sample frontend](./docs/frontend.md)
+3. [Configuration Guide](./docs/configuration.md): This guide covers different configurations available for this workflow.
+4. [Support Matrix](./docs/support_matrix.md): This covers GPU, CPU, Memory and Storage requirements for deploying this workflow.
+
+# Known Issues
+- Uploading a file with size more than 10 MB may fail due to preset timeouts during the ingestion process.
diff --git a/RetrievalAugmentedGeneration/chain_server/__init__.py b/RetrievalAugmentedGeneration/chain_server/__init__.py
@@ -0,0 +1,16 @@
+# SPDX-FileCopyrightText: Copyright (c) 2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""A microservice for hosting Langchain Chains."""
diff --git a/RetrievalAugmentedGeneration/chain_server/chains.py b/RetrievalAugmentedGeneration/chain_server/chains.py
@@ -0,0 +1,90 @@
+# SPDX-FileCopyrightText: Copyright (c) 2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""LLM Chains for executing Retrival Augmented Generation."""
+import base64
+from pathlib import Path
+from typing import Generator
+
+from llama_index import Prompt, download_loader
+from llama_index.node_parser import SimpleNodeParser
+from llama_index.query_engine import RetrieverQueryEngine
+from llama_index.response.schema import StreamingResponse
+
+from chain_server.utils import (
+    LimitRetrievedNodesLength,
+    get_config,
+    get_doc_retriever,
+    get_llm,
+    get_text_splitter,
+    get_vector_index,
+    is_base64_encoded,
+    set_service_context,
+)
+
+
+def llm_chain(
+    context: str, question: str, num_tokens: int
+) -> Generator[str, None, None]:
+    """Execute a simple LLM chain using the components defined above."""
+    set_service_context()
+    prompt = get_config().prompts.chat_template.format(
+        context_str=context, query_str=question
+    )
+    response = get_llm().stream_complete(prompt, tokens=num_tokens)
+    gen_response = (resp.delta for resp in response)
+    return gen_response
+
+
+def rag_chain(prompt: str, num_tokens: int) -> Generator[str, None, None]:
+    """Execute a Retrieval Augmented Generation chain using the components defined above."""
+    set_service_context()
+    get_llm().llm.tokens = num_tokens  # type: ignore
+    retriever = get_doc_retriever(num_nodes=4)
+    qa_template = Prompt(get_config().prompts.rag_template)
+    query_engine = RetrieverQueryEngine.from_args(
+        retriever,
+        text_qa_template=qa_template,
+        node_postprocessors=[LimitRetrievedNodesLength()],
+        streaming=True,
+    )
+    response = query_engine.query(prompt)
+
+    # Properly handle an empty response
+    if isinstance(response, StreamingResponse):
+        return response.response_gen
+    return StreamingResponse(iter([])).response_gen  # type: ignore
+
+
+def ingest_docs(data_dir: str, filename: str) -> None:
+    """Ingest documents to the VectorDB."""
+    unstruct_reader = download_loader("UnstructuredReader")
+    loader = unstruct_reader()
+    documents = loader.load_data(file=Path(data_dir), split_documents=False)
+
+    encoded_filename = filename[:-4]
+    if not is_base64_encoded(encoded_filename):
+        encoded_filename = base64.b64encode(encoded_filename.encode("utf-8")).decode(
+            "utf-8"
+        )
+
+    for document in documents:
+        document.metadata = {"filename": encoded_filename}
+
+    index = get_vector_index()
+    text_splitter = get_text_splitter()
+    node_parser = SimpleNodeParser.from_defaults(text_splitter=text_splitter)
+    nodes = node_parser.get_nodes_from_documents(documents)
+    index.insert_nodes(nodes)
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		notebooks/dataset.zip filter=lfs diff=lfs merge=lfs -text