The Notebook Pipeline consists of two stages - Build and Host. They are separate Pachyderm pipelines but are linked together to form a cohesive process. The user is able to identify and track progress (and logs) throughout both the Build and Host stages of the Notebook Pipeline. Status is shown via
kaos notebook list, while logs are available via
kaos notebook build-logs.
The Notebook Pipeline requires nothing to deploy an empty hosted notebook but can be optionally supplied with a valid source bundle and data bundle.
The source bundle is responsible for supplying the code and environment for deploying a notebook. Its nature should be treated as ephemeral and dynamic since versioning is handled with kaos. In other words, a user does not need to adapt chaotic naming conventions (i.e.
The source bundle requires, at minimum, the following basic structure.
$ tree notebooknotebook└── example├── Dockerfile└── model└── requirements.txt
The data bundle is responsible for submitting local data to the hosted notebook.
The sample local data bundle for the mnist model is shown below.
$ tree mnistmnist└── data└── features├── test│ └── test_mini.csv├── training│ └── training_mini.csv└── validation└── validation_mini.csv
Specific resources can be attached to any hosted notebook with the following options.
Float defining the desired compute (in cores or time)
String defining the desired memory (only valid with SI suffixes)
Integer defining the desired graphical processing (in cores)