grosse
/
sandbox-docs-public
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

added matrix
ci/woodpecker/push/woodpecker Pipeline was successful Details

This commit is contained in:
Malte Grosse 2024-06-14 13:13:47 +02:00
parent 9cec953db2
commit 7aec926276
2 changed files with 10 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

BIN
src/sandbox/res/matrix-ci.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 501 KiB

10
src/sandbox/training.md
View File

@ -58,7 +58,8 @@ steps:
``` ```
See the official [documentation](https://woodpecker-ci.org/docs/usage/workflow-syntax) for the syntax. See the official [documentation](https://woodpecker-ci.org/docs/usage/workflow-syntax) for the syntax.
Generally, the pipeline is based on different steps, and in each step, another container environment can be chosen. In the example above, first an official tensorflow container with python 3 is used to run the training python script. In the second step, the model gets compressed and pushed on the temp. sandbox storage. Generally, the pipeline is based on different steps, and in each step, another container environment can be chosen. In the example above, first an official tensorflow container with python 3 is used to run the training python script. In most cases you can find predefined containers at [Dockerhub](https://hub.docker.com/) or GPU supported containers at [NVIDIA](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch). If needed, custom images can be created and stored internally (on the Sandbox Git package repository) or any other public available container repository. In the second step, the model gets compressed and pushed on the temp. sandbox storage.
3. Commit and push 3. Commit and push
4. See current state of the pipelines at the [overview site](https://ci.sandbox.iuk.hdm-stuttgart.de/repos) 4. See current state of the pipelines at the [overview site](https://ci.sandbox.iuk.hdm-stuttgart.de/repos)
@ -95,6 +96,12 @@ which returns a json with the download url of your uploaded file.
- Choose a proper way to output some reasonable logs during your training, so it wont spam the logs too heavily - Choose a proper way to output some reasonable logs during your training, so it wont spam the logs too heavily
- training exists after 60 minutes: increase maximum duration in the ci repository settings - training exists after 60 minutes: increase maximum duration in the ci repository settings
## Advanced Parameters (Matrix Workflos)
The woodpecker cli yaml defintion files support [matrix workflows](https://woodpecker-ci.org/docs/usage/matrix-workflows), such that multiple pipeline runs are executed with all combinations of the predefined variables.
See the [test-ci](https://git.sandbox.iuk.hdm-stuttgart.de/grosse/test-ci/src/branch/matrix) matrix branch as an example to define multiple pipeline runs with different epochs and optimizers. In the CI it is showed with different label for each parameter:
![repos](./res/matrix-ci.png)
## Useful Links ## Useful Links
- [Sandbox GIT](https://git.sandbox.iuk.hdm-stuttgart.de/) - [Sandbox GIT](https://git.sandbox.iuk.hdm-stuttgart.de/)
- [Sandbox CI](https://ci.sandbox.iuk.hdm-stuttgart.de) - [Sandbox CI](https://ci.sandbox.iuk.hdm-stuttgart.de)
@ -104,3 +111,4 @@ which returns a json with the download url of your uploaded file.
- [TensorFlow](https://www.tensorflow.org/versions/r2.15/api_docs/python/tf) - [TensorFlow](https://www.tensorflow.org/versions/r2.15/api_docs/python/tf)
- [NVIDIA PyTorch Container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch) - [NVIDIA PyTorch Container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch)
- [NVIDIA Tensorflow Container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tensorflow) - [NVIDIA Tensorflow Container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tensorflow)
- [Dockerhub](https://hub.docker.com/)