For deep integration into the AI toolbox, the AIMS 5.0 AI Toolbox utils can be used. For installing the utils, please check the GitHub page. The AI Toolbox Utils helps to create Python notebooks which can be deployed without modification, enabling the developers to create AI tools in notebooks and deploy them as Docker based services.
The script can be used to deploy notebooks into proof-of-concept services. At this point, only REST server deployment is supported directly.
The tool can be used by running
python3 -m aitoolbox
To deploy a tool as a service, use
python3 -m aitoolbox deploy <tool directory> -o <deploy dir>
Note, that the deploy dir is removed and replaced with the files required for deployment. The exact way of deployment is described in the documentation of the tools.
To get familiar with the AI tools, clone the demo tool with
cd ~/aitools
git clone https://github.com/hollosigergely/demo-tool.git
Check the directory structure of the demo tool, which is the prefered way of organize the directory structure of a AI Tool.
├── assets
│ └── aitoolbox-impl.png
├── config.ai
├── .deploy
│ ├── docker-compose.yml
│ └── service
│ ├── Dockerfile
│ └── requirements.txt
├── doc
│ └── BP.md
├── query.ipynb
├── README.md
└── test
└── example.http
Each tool has a config.ai metadata file, describing the most important information about the tool itself in YAML format, consisting of the name and description of the tool and deploy information.
Tools shall have a README.md file which introduces the tool itself, describing the goal of the tool and the basic usage information.
Tools typicall provides a couple of ipynb Python nootebook files, however, tools do not need to be implemented in Python, they can use any (e.g. compiled) languages without Jupyter support. Here, query.ipynb implements the algorithm itself (which is a simple one), but complicated algorithms can consists of multiple notebook files, and also learning algorithms can have training notebooks.
The .deploy directory contains everything which is required for deploying the tool as a service. assets contains any file which is required (e.g. small data, binary files, etc.). Huge files shall be referenced instead of version checked.
The doc directory holds the detailed documentation of the tool. One of the is the BP.md which describes the experiences gathered while developing the tool itself.
The test directory holds basic tests or example inputs for the service.
The file holds the main information about the tool. The demo tool has for example:
tool:
name: "AI Toolbox Demo Service"
descr: "This is an example demo service for the AIMS AI Toolbox"
version: 1.0.0
deploy:
mode: "rest_builtin"
deploy_dir: ".deploy"
nb_path: "query.ipynb"
These are the mandatory parts. At this moment, only rest_builtin deployment is supported. The deployment stub directory (deploy_dir) and the default service notebook (nb_path) is defined.
To ease development and evaluation of AI Tools, the AI Toolbox encourages the developers to use Jupyter notebooks to present their solution to AI problems. Notebooks has a couple of advantages:
Please feel free to include various notebooks in your tool, however, we recommend to include two basic notebook for AI tools:
To edit notebooks, it is preferred to use Visual Code editor, since it supports notebook editing and also execution of cells in different kernels.
Please note, that notebooks support only Python and R code execution.
For example notebook, see the query.ipynb in the demo tool.
Properly formatted notebooks can be readily deployed into REST or other services using AI Toolbox utils. For an AI Tool, the deployable notebook is defined in the config.ai file by setting deploy.nb_path.
Deployable notebooks has some further requirements. Deployable notebooks consists of three different sections, separated by markdown cells with specific content. The two special markdown cells are:
# Setup header content# Service header contentThe three different sections are:
The deployed service is infered from the deployable notebook using the AI Toolbox Utils:
python -m aitoolbox gensrc <tool_dir> -o <out_dir>
Also, the deploy command of the AI Toolbox Util creates the source beyond copying the deploy stub files. The generation of the source code is as follows:
setup.pyservice.pyAll the cells are included, except:
#!skipThe setup script is executed before starting the REST server. At each request, the service script is called with the appropriate arguments. For handling different contexts (e.g. notebook context and service context), developers are encouraged to use the AI Toolbox Support Library, which provide seamless integration of notebooks with REST services.
TODO: Sequence diagram
TODO
Notebooks can be executed on remote hardware, e.g. using a remote machine with advanced GPUs. In this case, it is preferred to use a remote Jupyter instance. To create remote Jupyter instance, setup the notebook on the remote machine using e.g. pip:
pip install notebook
For a sustainable solution, use dockerized solution. The Dockerfile bellow creates an Docker image e.g. based on the nVidia CUDA 11.8 image:
FROM nvidia/cuda:11.8.0-runtime-ubuntu22.04
WORKDIR /root
RUN apt update && \
apt -y install python3-pip \
&& rm -rf /var/lib/apt/lists/*
ENV TZ=Europe/Budapest \
DEBIAN_FRONTEND=noninteractive
RUN pip3 install notebook
CMD ["/usr/local/bin/jupyter", "notebook", "--allow-root","--ip","0.0.0.0","--port","8888"]
EXPOSE 8888
To use the image, create the Dockerfile above, and in the same directory, run
docker build -t cuda_jupyter .
After building the image, run the container with GPU support as
docker run --rm --gpus all -p 8888:8888 cuda_jupyter
The remote Jupyter session can be accessed rmeotely on the host port 8888. Note: using GPUs in Docker images requires the nVidia Container Toolkit to be installed among with the proper driver for the video card!
TODO: Example images for Visual Code
TODO