Compare commits

..

No commits in common. "2eedd31b2a90b612cc327f4abce8d525a2daa491" and "4499f2bbfad3ef0d22d10f5bbe6241a3b7bdc30b" have entirely different histories.

22 changed files with 41 additions and 220 deletions

View File

@ -13,6 +13,3 @@ pipeline:
repo: git.sandbox.iuk.hdm-stuttgart.de/grosse/sandbox-docs-public repo: git.sandbox.iuk.hdm-stuttgart.de/grosse/sandbox-docs-public
dockerfile: Dockerfile dockerfile: Dockerfile
tags: latest tags: latest
branches:
exclude: cspecht

View File

@ -8,14 +8,19 @@
- [Hardware](architecture/hardware.md) - [Hardware](architecture/hardware.md)
- [Software](architecture/software.md) - [Software](architecture/software.md)
# Playground # (Basic Usage) Sandbox
- [Overview](sandbox/overview.md) - [Overview](sandbox/overview.md)
- [Development Environment](sandbox/dev_env.md) - [Services](sandbox/services.md)
- [Getting Started](sandbox/dev_env#getting-started.md) - [Getting Started](sandbox/services.md)
- [Environments](sandbox/dev_env#environments.md) - [Environments](sandbox/services.md)
- [Resources](sandbox/dev_env#resources.md) - [Resources](sandbox/services.md)
- [Limitations](sandbox/dev_env#limitations.md) - [Limitations](sandbox/services.md)
- [Use Cases](sandbox/use_cases.md) - [Use Cases](sandbox/services.md)
- [Data Pool](sandbox/data_pool.md)
- [Training Environment](sandbox/training.md) <!---
- [Data Generator](sandbox/data_generator.md) environments
-> feedback issues
# (Advanced Usage) Sandbox
--!>

View File

@ -1,21 +1,3 @@
# Hardware # Hardware
![Sandbox Architecture](res/dell-server.jpeg)
## Sandbox Server Configuration
* PowerEdge R7525 Motherboard, 2HE
* 2x AMD 7543 2.8GHz,32C/64T,256M (Milan EPYC)
* 8 x 64GB RDIMM, 3200MT/s, Dual Rank
* 2 x 1.6TB SSD SAS ISE Mix Use 12Gbps 512e 2.5in
* 6 x 480GB SSD SATA Mix Use 6Gbps 512 2.5in
* 8 x 2.4TB 10K RPM SAS ISE 12Gbps 512e 2.5in
* 1 x Intel E810-XXV Dual Port 10/25GbE SFP28, OCP NIC 3.0
* 1 x NVIDIA Ampere A100, PCIe, 300W, 80GB Passive
## NAS
Additional 20TB backup storage is located at IKARUS NAS.
siehe praesi

View File

@ -1,36 +1,3 @@
# Architecture Overview # Architecture Overview
## Conceptual Design mixed use->praesi
![Sandbox Architecture](res/sandbox-architecture.png)
## Development Environment
An interactive, browser-based, development playground provides beginners and advanced users an isolated and safe AI playground.
## Training Environment
For long running processes (building/training) a userfriendly pipeline is available.
## Use Cases
The sandbox offers the possibility to provide interactive use cases and demo scripts for online courses.
## Data Generator
In order to analyse training behaviour of maschine learning models, synthetic data can be generator according to the user requirements. (tbd)
## Data Pool
The Data Pool allows the user to exchange data with other sandbox users, advanced users, and scientific users. Therefore the users can use the storage for their assignments (notebook files), data (training) and other sandbox driven data.
## Sandbox Use Cases
The following scenarios can be derived for the system: on the one hand, the system is to be used for teaching and on the other hand, the system is to be used for training.
Therefore, a mixed-use architecture is provided, which allows the system to be used for these two different requirements, hence following role concept has been designed:
### Roles
**User**
* Teaching/Use Cases in the area of ethics, business and law.
* Assignments Editing
* Provide interactive examples / demos and course material
**Advanced User**
* Teaching/Use Cases in the field of IT
* Tasks Editing
* Provide/develop (own) examples/demos.
**Scientific User**
* Data generation using synthetic data generator (tbd)
* GPU usage
* version control (GIT)
* CI/CD for long running tasks
* Storage

Binary file not shown.

Before

Width:  |  Height:  |  Size: 118 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 70 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 19 KiB

View File

@ -1,10 +1,3 @@
# Software # Software
## Software Defined Architecture
![SDA](res/vms.png)
The server is separated into three virtual machines, according to their responsibilities, and is highly extendable regarding new external hardware.
- Master VM: Management environment for software orchestration
- Node 1 VM: Application node for software foundation like databases, sandbox controller, ci/cd controller
- Node 2 VM: Worker node for ci/cd and sandbox playground
//container verteilte system

View File

@ -1,4 +0,0 @@
# Data Generator
Currently under heavy development, coming soon....stayed tuned!

View File

@ -1,44 +0,0 @@
# Data Pool
In the following section we describe how to store data on the Sandbox. There are three different ways to do achieve: Inside the Sandbox, headless file upload on Object storage and Git LFS.
## Inside the Sandbox
To store the data inside the Sandbox, you just have to drag & drop or click on the upload button to save the file to your running instance. You can also create folders and new Notebooks. Please see the free space limitations [here](sandbox/dev_env#resources.md).
![file upload](res/sandbox_upload_file_selector.png "File Upload")
## Headless file storage
To use the headless object storage, you can upload a file via REST-Interface or curl. The json response message provides you the destination url.
**Upload Example**
```python
import requests
url = "https://share.storage.sandbox.iuk.hdm-stuttgart.de/upload"
filename = "ds_salaries.csv"
files = {'fileUpload': (filename, open(filename, 'rb'),'text/csv')}
r = requests.post(url, files=files)
```
**Example JSON response**
```json
{
"PublicUrl": "https://storage.sandbox.iuk.hdm-stuttgart.de/upload/a1236b2b-49bf-4047-a536-20dab15b7777/untitled.txt",
"Size": 11,
"Expiration": "2023-10-04T00:00:00Z"
}
```
**Import Example**
```python
import pandas as pd
url = "https://storage.sandbox.iuk.hdm-stuttgart.de/upload/ec6c1c9c-ea9b-47ff-97cf-f92d870e8fb9/ds_salaries.csv"
df = pd.read_csv(url)
```
## Git LFS
The following solution we highly recomment only for users which are familiar with git command line tools! Git Large File Storage (LFS). An open source Git extension for versioning large files. Git LFS replaces large files (audio, sample, datasets, videos) by a text pointer inside git. The files get stored on our gitea Server.
For further information visit [Git LFS](https://git-lfs.com/).

View File

@ -1,66 +0,0 @@
# Development Environment
![sandbox](https://uptime.monitoring.iuk.hdm-stuttgart.de/api/badge/1/status)
## Introduction
The **Sandbox Development Environment** enables students and lecturers to create and work with interactive case studies. It also provides a development environment for researchers and advanced programmers.
The Sandbox is available at: https://sandbox.iuk.hdm-stuttgart.de
Please carefully handle the resources and follow the fair use principle and as well as the German laws.
## Getting Started
Within the sandbox, the different disciplines of the IKID project can provide tasks to be worked on by the respective student groups. Both text-based tasks and programmatic tasks can be processed. For example, Markdown files can be created for editing textual tasks.
Currently, it is planned for the technical lectures that students get in touch with the programming language Python for the first time. Therefore, the Sandbox platform was created, in which experiments with Python can be carried out.
1. **Sign in**, use your **HdM Credentials** at [Sandbox](https://sandbox.iuk.hdm-stuttgart.de/)
2. Select the image you want to start (two options)
1. **Datascience environment**
2. **Datascience GPU environment** (choose only if you realy need the graphic card, otherwise you block resources from those who need them)
3. Create or upload a .ipynb file:
1. . **create a empty .ipynb file:**
![sandbox launcher](res/sandbox_launcher.png "Sandbox Launcher")
2. **upload a existing .ipynb file:**
![sandbox upload file](res/sandbox_upload_file_selector.png "Sandbox upload file")
4. **open** the file from the filebrowser & start working!
![sandbox .ipynb file](res/sandbox_ipynb_example.png "Sandbox Notebook").
5. **After you finished your work dont forget to shutdown your server!** Therefore, you should shutdown your server to release server resources. **Select File, Hub Control Panel**
![file menue](res/sandbox_file_menu.png "File menue")
7. Select **Stop Server**
![stop server](res/sandbox_stop_server.png "Stop Server")
1. Select the arrow in the upper right corner to **Logout**
![logout](res/sandbox_logout.png)
## Technical Overview
### Environments
The Sandbox provides multiple scientific environments. If any additional packages or libraries are need, please open an issue on our [GIT](https://git.sandbox.iuk.hdm-stuttgart.de/).
#### Datascience environment
* Available Data Science image is based on [Data Science Container](https://git.sandbox.iuk.hdm-stuttgart.de/grosse/-/packages/container/jupyterlab-datascience/latest)
* most common data analysis library's included for Julia, Python, R are available
Please open an [issue](https://git.sandbox.iuk.hdm-stuttgart.de/grosse/jupyterlab-datascience/issues/new) if any additional packages are needed or issues occurred.
#### Datascience GPU environment
* Available GPU image is based on [Datascience GPU Container](https://git.sandbox.iuk.hdm-stuttgart.de/grosse/-/packages/container/jupyterlab-datascience-gpu/latest)
* support added for the NVIDIA GPU A100 computations based on python most common GPU-able libraries like Tensorflow, PyTorch and Keras.
Please open an [issue](https://git.sandbox.iuk.hdm-stuttgart.de/grosse/jupyterlab-datascience-gpu/issues/new) if any additional packages are needed or issues occurred.
### Resources
Each instance has following resource limits:
- maximum 2 physical CPUs, guranteed 0.5 CPUs
- maximum 10GB DDR memory, guranteed 1GB
- maximum of 1GB HDD
For GPU enabled environments, 40GB shared (time sliced) GPU memory is availble. For additional information, please see the [official nvidia documentation](https://docs.nvidia.com/datacenter/tesla/mig-user-guide/index.html).
### Limitations
Please carefully follow the terms and conditions at [Sandbox](https://sandbox.iuk.hdm-stuttgart.de) website. The HDD free space can not be extended. The Sandbox should only be used for short running GPU tasks. For longer running trainings, please use the [Training Environment](sandbox/training.md). After GPU usage please stop your kernel/instance or free the blocked GPU resources manually. After 30 minutes of inactivity, the instanced will automatically removed. The HDD space is persistent, but will be deleted after 6 months.

View File

@ -1,15 +1,3 @@
# Overview Sandbox # Overview Sandbox
![Sandbox Architecture](res/sandbox-architecture.png) resources
## Development Environment
Playground...Getting started... Use cases
## Use Cases
## Data Pool
## Training Environment
(tbd)
## Data Generator
(tbd)

Binary file not shown.

Before

Width:  |  Height:  |  Size: 70 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 58 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 171 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 54 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 13 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 13 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 11 KiB

19
src/sandbox/services.md Normal file
View File

@ -0,0 +1,19 @@
# Services
## Status
![sandbox](https://uptime.monitoring.iuk.hdm-stuttgart.de/api/badge/1/status)
## Playground
## Training
## Storage
### Sandbox
1gb
### Tmp Binary
headless binary curl
### Data Pool
lfs

View File

@ -1,4 +0,0 @@
# Training Environment
Currently under heavy development, coming soon....stayed tuned!

View File

@ -1,12 +0,0 @@
# Use Cases
tbd
ipynb example syntax + markdown + tex + voila slider (interactive dashboards)
idee:
- bild use case example
- python code example
- markdown example (bild,headline, bullet points)
- tex
- voila