105 lines
6.8 KiB
Markdown
105 lines
6.8 KiB
Markdown
# Development Environment
|
|
|
|
|
|
|
|
## Introduction
|
|
The **Sandbox Development Environment** enables students and lecturers to create and work with interactive case studies. It also provides a development environment for researchers and advanced programmers.
|
|
|
|
The Sandbox is available at: [https://sandbox.iuk.hdm-stuttgart.de](https://sandbox.iuk.hdm-stuttgart.de)
|
|
|
|
Please carefully handle the resources and follow the fair use principle and as well as the German laws.
|
|
## Getting Started
|
|
|
|
Within the sandbox, the different disciplines of the IKID project can provide tasks to be worked on by the respective student groups. Both text-based tasks and programmatic tasks can be processed. For example, Markdown files can be created for editing textual tasks.
|
|
Currently, inside the technical lectures students (in the role User) get in touch with prorgramming languages for the first time. Therefore, the Sandbox platform was created, in which experiments and Usecases with Python of all User levels can be carried out.
|
|
|
|
|
|
|
|
1. **Sign in**, use your **HdM Credentials** at [Sandbox](https://sandbox.iuk.hdm-stuttgart.de/)
|
|
2. Select the environment you want to start (three options, GPU enviroments are limited to 10 instances in total)
|
|
1. **Datascience environment**
|
|
2. **GPU PyTorch environment** (PyTorch library with GPU support, choose only if you realy need the graphic card, otherwise you block resources from those who need them)
|
|
3. **GPU TF environment** (Tensorflow library with GPU support, choose only if you realy need the graphic card, otherwise you block resources from those who need them)
|
|
3. Create or upload a .ipynb file:
|
|
1. **create a empty .ipynb file:**
|
|
|
|
2. **upload a existing .ipynb file:**
|
|
|
|
4. **open** the file from the filebrowser & start working!
|
|
.
|
|
5. **After you finished your work dont forget to shutdown your server!** Therefore, you should shutdown your server to release server resources. **Select File, Hub Control Panel**
|
|
|
|
7. Select **Stop Server**
|
|
|
|
1. Select the arrow in the upper right corner to **Logout**
|
|
|
|
|
|
|
|
## Technical Overview
|
|
|
|
### Environments
|
|
The Sandbox provides multiple scientific environments. If any additional packages or libraries are needed, check if they are available on [PyPi](https://pypi.org/) and use pip to install it. If a problem occurs, please open an issue on our [GIT](https://git.sandbox.iuk.hdm-stuttgart.de/).
|
|
|
|
|
|
#### Datascience environment
|
|
* Available Data Science image is based on [Data Science Container](https://git.sandbox.iuk.hdm-stuttgart.de/grosse/-/packages/container/jupyterlab-datascience/latest)
|
|
* most common data analysis library's included for Python are available.
|
|
|
|
Please open an [issue](https://git.sandbox.iuk.hdm-stuttgart.de/grosse/jupyterlab-datascience/issues/new) if any additional packages are needed or issues occurred.
|
|
|
|
|
|
#### GPU PyTorch environment
|
|
* Available GPU image is based on [Datascience GPU Container](https://git.sandbox.iuk.hdm-stuttgart.de/grosse/-/packages/container/jupyterlab-datascience-gpu/latest)
|
|
* support added for the NVIDIA GPU A100 computations based on python library PyTorch.
|
|
|
|
Please open an [issue](https://git.sandbox.iuk.hdm-stuttgart.de/grosse/jupyterlab-datascience-gpu/issues/new) if any additional packages are needed or issues occurred.
|
|
|
|
#### GPU TF environment
|
|
* Available GPU image is based on [Datascience GPU Container](https://git.sandbox.iuk.hdm-stuttgart.de/grosse/-/packages/container/jupyterlab-datascience-gpu/latest)
|
|
* support added for the NVIDIA GPU A100 computations based on python library Tensorflow.
|
|
|
|
Please open an [issue](https://git.sandbox.iuk.hdm-stuttgart.de/grosse/jupyterlab-datascience-gpu/issues/new) if any additional packages are needed or issues occurred.
|
|
|
|
### Resources
|
|
Each instance has following resource limits:
|
|
- maximum 2 physical CPUs, guranteed 0.5 CPUs
|
|
- maximum 10GB DDR memory, guranteed 1GB
|
|
- maximum of 1GB HDD
|
|
|
|
For GPU enabled environments, 40GB shared (time sliced) GPU memory is availble. For additional information, please see the [official nvidia documentation](https://docs.nvidia.com/datacenter/tesla/mig-user-guide/index.html).
|
|
|
|
### Limitations
|
|
Please carefully follow the terms and conditions at [Sandbox](https://sandbox.iuk.hdm-stuttgart.de) website. The HDD free space can not be extended. The Sandbox should only be used for short running GPU tasks. For longer running trainings, please use the [Training Environment](sandbox/training.md). After GPU usage please stop your kernel/instance or free the blocked GPU resources manually. After 30 minutes of inactivity, the instanced will automatically removed. The HDD space is persistent, but will be deleted after 6 months.
|
|
|
|
- **Important** Only for **GPU TF environment** use following memory limitation:
|
|
Tensorflow allocates in the beginning of each session all available resources. With the following chunk of code the memory limitation is set to 6024 MB GPU memory. If you dont limit the memory, all other users dont have any GPU support.
|
|
|
|
```
|
|
gpus = tf.config.experimental.list_physical_devices('GPU')
|
|
if gpus:
|
|
|
|
try:
|
|
tf.config.experimental.set_virtual_device_configuration(gpus[0], [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=6024)])
|
|
|
|
except RuntimeError as e:
|
|
print(e)
|
|
```
|
|
At the end of your file you have to clean your session that the GPU resource is released.
|
|
|
|
```
|
|
tf.keras.backend.clear_session()
|
|
```
|
|
|
|
## Troubleshooting
|
|
- when using javascript based plugins, it can happened that they either not load or run buggy: make sure you use chrome. Other browsers are often not supported by third party plugins
|
|
- Notebooks/Server/Dev Environment is going down: To avoid blocking resources, each environment has a certain timeout (frontend inactivity), so idling notebooks get culled to free resources for other user.
|
|
- my favourite xyz python packages is missing: use conda/pip (or inside a notebook: !pip install) to add additional packages
|
|
- all my data is gone after the semester break: all persistent storages get recycled of every(!) user each semester break. Please backup your data locally if needed
|
|
- package conflicts: common issue is to install a unspecific library version, please specify or upgrade all dependencies manually.
|
|
|
|
|
|
## Useful Links
|
|
|
|
- [Jupyter Documentation](https://docs.jupyter.org/en/latest/)
|
|
- [pip](https://pip.pypa.io/en/stable/user_guide/)
|
|
- [python](https://docs.python.org/3.11/) |