diff --git a/src/sandbox/res/matrix-ci.png b/src/sandbox/res/matrix-ci.png new file mode 100644 index 0000000..81c0c70 Binary files /dev/null and b/src/sandbox/res/matrix-ci.png differ diff --git a/src/sandbox/training.md b/src/sandbox/training.md index 39f1692..fa1e427 100644 --- a/src/sandbox/training.md +++ b/src/sandbox/training.md @@ -58,7 +58,8 @@ steps: ``` See the official [documentation](https://woodpecker-ci.org/docs/usage/workflow-syntax) for the syntax. -Generally, the pipeline is based on different steps, and in each step, another container environment can be chosen. In the example above, first an official tensorflow container with python 3 is used to run the training python script. In the second step, the model gets compressed and pushed on the temp. sandbox storage. +Generally, the pipeline is based on different steps, and in each step, another container environment can be chosen. In the example above, first an official tensorflow container with python 3 is used to run the training python script. In most cases you can find predefined containers at [Dockerhub](https://hub.docker.com/) or GPU supported containers at [NVIDIA](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch). If needed, custom images can be created and stored internally (on the Sandbox Git package repository) or any other public available container repository. In the second step, the model gets compressed and pushed on the temp. sandbox storage. + 3. Commit and push 4. See current state of the pipelines at the [overview site](https://ci.sandbox.iuk.hdm-stuttgart.de/repos) @@ -95,6 +96,12 @@ which returns a json with the download url of your uploaded file. - Choose a proper way to output some reasonable logs during your training, so it wont spam the logs too heavily - training exists after 60 minutes: increase maximum duration in the ci repository settings + +## Advanced Parameters (Matrix Workflos) +The woodpecker cli yaml defintion files support [matrix workflows](https://woodpecker-ci.org/docs/usage/matrix-workflows), such that multiple pipeline runs are executed with all combinations of the predefined variables. +See the [test-ci](https://git.sandbox.iuk.hdm-stuttgart.de/grosse/test-ci/src/branch/matrix) matrix branch as an example to define multiple pipeline runs with different epochs and optimizers. In the CI it is showed with different label for each parameter: +![repos](./res/matrix-ci.png) + ## Useful Links - [Sandbox GIT](https://git.sandbox.iuk.hdm-stuttgart.de/) - [Sandbox CI](https://ci.sandbox.iuk.hdm-stuttgart.de) @@ -103,4 +110,5 @@ which returns a json with the download url of your uploaded file. - [PyTorch](https://pytorch.org/docs/stable/index.html) - [TensorFlow](https://www.tensorflow.org/versions/r2.15/api_docs/python/tf) - [NVIDIA PyTorch Container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch) -- [NVIDIA Tensorflow Container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tensorflow) \ No newline at end of file +- [NVIDIA Tensorflow Container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tensorflow) +- [Dockerhub](https://hub.docker.com/) \ No newline at end of file