Merge branch 'develop' into 'main'

Develop

See merge request systemes-dinformation/alpha/pocs/scaleway-instances-template!1

README.md

-# Labo Rust
+# Scaleway Instances Template - Quickstart
 
-Infrastructure as Code project to deploy lab infrastructure and configure instances. Working with Atrium, Gitlab CI, Terraform, Cloud-init, bash scripts and Scaleway.
+## How does it work ?
 
-### How does it work ?
+The template will provision several resources (including your instances) to the Scaleway cloud provider using Terraform and Gitlab CI.
+After provisioning, configuration of the instances and deployment of the services (Atrium and TTYD) will be done using Cloud-init and bash scripts.
 
-The FORGE (Gitlab) manage the Infrastructure with Terraform at any state, plan, creation, modification and destruction with the CI. The FORGE store and will provide the configuration information
-such as credentials to Terraform only during the running state to improve security. Then, informations are destroyed with the Gitlab Agent container.
+> All your information must be stored in Gitlab and passed as variable to Terraform.
 
-On run state, Terraform will create, modify and destroy infrastructure resources in Scaleway to match the configuration described in the configuration files.
+### How the deployment is managed ?
 
-**Resources deployed :**
+The whole project is managed by Gitlab using the Gitlab CI. 
+Launching the pipeline will start the deployment of your infrastructure in 3 steps.  
 
-| resource-type                    | plan   | inbound port |
-|----------------------------------|--------|--------------|
-| scaleway_instance_ip             |        |              |
-| scaleway_domain_record           |        |              |
-| scaleway_domain_record           |        |              |
-| scaleway_instance_security_group |        | 22, 443      |
-| scaleway_instance_server         | DEV1-L |              |
+**Plan :**
+Terraform will perform a "fake deployment" and initialize everything he needs before deploying.  
+After performing, you will be able to see the plan of the future creation/modification/destruction in the log of the pipeline.
+If the modification could lead to break changes and errors, you must not launch the next step of the pipeline.
+**Creation :**
+Terraform will be applying the exact changes you reviewed in the Plan step.
+**Destruction :**
+Terraform will destroy the resources he managed to create (or manually imported one).
 
-After the resources provisionned with Terraform, Cloud-init will configure the instances by running bash scripts, creating files... And 3 services will be running for each instance.
+## How can I use it ?
 
-**Running services :**
+### Step 0 : Prerequisite
 
-- Atrium (Reverse proxy, TLS encryption and HTTPS to the others hosted services)
-- Code-server
-- Webtop
+**On The Forge (Gitlab)**, you must have or ask for the right to fork the repository.  
 
-After deployment, each resource can be accessed though HTTPS depending on their count number :
-- https://desktop-0.rust-0.daag.alpha.grandlyon.com/
-- https://code-0.rust-0.daag.alpha.grandlyon.com/
-- https://terminal-0.rust-0.daag.alpha.grandlyon.com/
-- https://rust-0.daag.alpha.grandlyon.com/
+**Scaleway** needs you to have credentials to use their services. You must go to [Scaleway API key page](https://console.scaleway.com/iam/api-keys) 
+and create one for your project. Please make sure to add a description starting with `Terraform - <your description>`.  
 
+**The Certificate Authority (CA)** Let's Encrypt needs an email. He will use it to send a notification before the certificate expires to proceed with
+the renewal.  
 
-- https://desktop-1.rust-1.daag.alpha.grandlyon.com/
-- https://code-1.rust-1.daag.alpha.grandlyon.com/
-- https://terminal-1.rust-1.daag.alpha.grandlyon.com/
-- https://rust-1.daag.alpha.grandlyon.com/
-
-### Credentials
-
-**Desktop username : abc**
-**Atrium username : admin**
-**Password : <Gitlab Stored>**
-
-## How to add new resources
-
-You can add Terraform resources directly to the main.tf file.
-Template are available to the link below.
-
-**Terraform Registry for Scaleway provider : https://registry.terraform.io/providers/scaleway/scaleway/latest/docs**
-
-## How to setup configuration file for services
-
-### cloud-init : file creation
-
-In this example, you will find how to create a file to a defined path with cloud-init.
-Content inside `%` are meant to be replaced with sed command to be able to use environment variables or user-data.
-
-**Example:**
-```yaml
-write_files:
-  - content: |
-      hostname: %atrium_hostname%.daag.alpha.grandlyon.com
-      debug_mode: false
-      letsencrypt_email: %atrium_letsencrypt_email%
-      tls_mode: Auto
-      apps: # optional : applications served by atrium
-        - id: 1
-          name: Code %atrium_count_index%
-          icon: web_asset
-          color: 4292030255
-          is_proxy: true
-          host: code-%atrium_count_index%
-          target: localhost:8080
-        - id: 2
-          name: Desktop %atrium_count_index%
-          icon: web_asset
-          color: 4292030255
-          is_proxy: true
-          host: desktop-%atrium_count_index%
-          target: localhost:8081
-    path: /root/atrium.yaml
-```
-
-**Bash commands example to replace `%` content :**
-```bash
-sed -i "s/%atrium_hostname%/$(scw-userdata atrium_hostname)/g" /root/atrium.yaml
-sed -i "s/%atrium_letsencrypt_email%/$(scw-userdata atrium_letsencrypt_email)/g" /root/atrium.yaml
-sed -i "s/%atrium_count_index%/$(scw-userdata atrium_count_index)/g" /root/atrium.yaml
-```
-
-## Downloading file
-
-The example show how to curl a configuration file from repository to a defined path with cloud-init.
+### Step 1 : Getting your own repository
+
+Fork the project from [this repository](https://forge.grandlyon.com/systemes-dinformation/alpha/pocs/scaleway-instances-template)
+
+### Step 2 : Setting up the variables with credentials and others information
+
+Go to the `Settings` menu, `CI/CD` and look for the `Variables` submenu.  
+Once it's done, you will need to create the variable bellow :  
+- **ENVIRONMENT** : Environment of the deployment, contained in domains, subdomains and resources name convention.  
+- **TF_VAR_INSTANCES_COUNT** : Number of instances to create.  
+- **TF_VAR_LETSENCRYPT_EMAIL** : Lets encrypt mail for renewal.  
+- **TF_VAR_PROJECT_DOMAIN** : External domain from Scaleway. (Example : <xxx>.alpha.grandlyon.com)  
+- **TF_VAR_SCW_ACCESS_KEY** : Scaleway Access Key.  
+- **TF_VAR_SCW_PROJECT_ID** : Project ID from your Scaleway project.  
+- **TF_VAR_SCW_SECRET_KEY** : Scaleway Secret Key.  
+- **TF_VAR_USER_PASSWORD** : User password for accessing services and instances.  
+
+### Step 3 : Every is set up, deployment is coming !
+
+Go the `CI/CD` menu and `Pipelines`.  
+From now, you will need to click on `Run pipeline` with your selected branch.  
+Since you added variables before, you don't need to put any in this step, click `Run Pipeline` again.  
+
+**Checking logs :** You can click on every step to check logs in case of errors or not.  
+**Check plan :** I recommend to always check the plan when you're trying things to do not perform destruction
+on production infrastructure.  
+
+Once plan is done and ok, click the `Apply button` to deploy the infrastructure. Please be sure to click the `Destroy
+button` to destroy the infrastructure when you don't need it anymore.
+
+### Step 4 : Accessing your services
+
+After deployment, each resource can be accessed though HTTPS, URL depends of the variable you defined before :
+- **TTYD :** https://terminal-<INSTANCE_COUNT>.<ENVIRONMENT>-<INSTANCE_COUNT>.<PROJECT_DOMAIN/
+- **Atrium :** https://<ENVIRONMENT>-<INSTANCE_COUNT>.<PROJECT_DOMAIN>/
 
 **Example :**
-```yaml
-runcmd:
-#
-# cloud-init execute cmd before HOME and USER are set.
-#
-  - "export HOME=/root"
-  - "export USER=root"
-#
-# Install Atrium
-#
-  - "curl https://forge.grandlyon.com/systemes-dinformation/alpha/projects/labo-rust/-/raw/develop/instance-scripts/config-files/atrium.yaml -o root/atrium.yaml"
-  - "curl https://forge.grandlyon.com/systemes-dinformation/alpha/projects/labo-rust/-/raw/develop/instance-scripts/config-files/atrium.sh | bash"
-```
-
-## Use Gitlab variables in instances
-
-From my instances, I should be able to access the variables as user-data using the command `scw-userdata <my-variable>` after declaring them in Terraform.
-**Some documentation : https://blog.scaleway.com/introducing-scaleway-cloud-init-support/**
-
-**Example : Declare in Terraform**
-```HCL
-user_data = {
-    atrium_count_index = count.index
-    atrium_hostname = scaleway_domain_record.subdomain_record[count.index].name
-    atrium_letsencrypt_email = var.LETSENCRYPT_EMAIL
-    user_password = var.USER_PASSWORD
-    cloud-init = file("../instance-scripts/cloud-init.yml") // this is not a variable, but the declaration of cloud-init file.
-}
-```
-
-** Example : Retrieve in instance
-```bash
-scw-userdata atrium_count_index
-```
-
-## Setup Terraform Locally
-
-First, you must setup 2 local files for your variables :
-
-### variables-local.tf 
-
-Create a file **variables-local.tf** containing the following code :
-```hcl
-variable "FORGE_PROJECT_ID" {
-  type        = string
-  description = "Forge Project ID"
-  default     = "your project id"
-  sensitive   = true
-}
-
-variable "FORGE_USERNAME" {
-  type        = string
-  description = "Forge Username"
-  default     = "your username"
-  sensitive   = true
-}
-
-variable "FORGE_ACCESS_TOKEN" {
-  type        = string
-  description = "Forge Access Token"
-  default     = "your access token"
-  sensitive   = true
-}
-```
-### variables-local.tfvars
-
-Now, you can create a file for your variables information called **variables-local.tfvars** containing the following code :
-```hcl
-### SCW variables
-
-SCW_PROJECT_ID         = ""
-SCW_ACCESS_KEY         = ""
-SCW_SECRET_KEY         = ""
-INSTANCES_COUNT        = ""
-ENVIRONMENT            = ""
-```
-
-### Terraform init - Gitlab remote tfstate
-
-You can grab a init command from your gitlab project on menu Infrastructure > Terraform.
-Select your environment and click the actions button, then you will only need to provide a gitlab project token.
-Command should look like :
-```bash
-export GITLAB_ACCESS_TOKEN=<YOUR-ACCESS-TOKEN>
-terraform init \
-    -backend-config="address=https://forge.grandlyon.com/api/v4/projects/<project-id>/terraform/state/<tfstate-name>" \
-    -backend-config="lock_address=https://forge.grandlyon.com/api/v4/projects/<project-id>/terraform/state/<tfstate-name>/lock" \
-    -backend-config="unlock_address=https://forge.grandlyon.com/api/v4/projects/<project-id>/terraform/state/<tfstate-name>/lock" \
-    -backend-config="username=xxxxxxx" \
-    -backend-config="password=$GITLAB_ACCESS_TOKEN" \
-    -backend-config="lock_method=POST" \
-    -backend-config="unlock_method=DELETE" \
-    -backend-config="retry_wait_min=5"
-
-terraform init -var-file=variables-local.tfvars
-```
-
-### Terraform plan - With variables file
-
-```bash
-terraform plan -var-file=variables-local.tfvars -out=tfplan
-```
-
-### Terraform apply - With plan
-
-```bash
-terraform apply tfplan
-```
-
-### Terraform destroy - With variables file
-
-```bash
-terraform destroy -var-file=variables-local.tfvars
-```
\ No newline at end of file
+- https://terminal-1.rust-1.daag.alpha.grandlyon.com/
+- https://rust-1.daag.alpha.grandlyon.com/
\ No newline at end of file

instance-scripts/cloud-init.yml

@@ -17,10 +17,6 @@ runcmd:
   - "export HOME=/root"
   - "export USER=root"
 #
-# Install Rust
-#
-  - "curl https://forge.grandlyon.com/systemes-dinformation/alpha/projects/labo-rust/-/raw/develop/instance-scripts/scripts/rust.sh | bash"
-#
 # Install Docker
 #
   - "curl https://forge.grandlyon.com/systemes-dinformation/alpha/projects/labo-rust/-/raw/develop/instance-scripts/scripts/docker.sh | bash"
@@ -29,18 +25,6 @@ runcmd:
 #
   - "curl https://forge.grandlyon.com/systemes-dinformation/alpha/projects/labo-rust/-/raw/develop/instance-scripts/scripts/atrium.sh | bash"
 #
-# Install Code Server
-#
-  - "curl https://forge.grandlyon.com/systemes-dinformation/alpha/projects/labo-rust/-/raw/develop/instance-scripts/scripts/code-server.sh | bash"
-#
-# Install Webtop
-#
-  - "curl https://forge.grandlyon.com/systemes-dinformation/alpha/projects/labo-rust/-/raw/develop/instance-scripts/scripts/webtop.sh | sh"
-#
 # Install ttyd
 #
-  - "curl https://forge.grandlyon.com/systemes-dinformation/alpha/projects/labo-rust/-/raw/develop/instance-scripts/scripts/ttyd.sh | bash"
-#
-# Install Repository
-#
-  - "git clone https://github.com/NathanRodet/rust-axum-demo /root/rust-axum-demo"
\ No newline at end of file
+  - "curl https://forge.grandlyon.com/systemes-dinformation/alpha/projects/labo-rust/-/raw/develop/instance-scripts/scripts/ttyd.sh | bash"
\ No newline at end of file

instance-scripts/config-files/atrium.yaml

@@ -5,26 +5,12 @@ letsencrypt_email: %atrium_letsencrypt_email%
 tls_mode: Auto
 apps: # optional : applications served by atrium
   - id: 1
-    name: Code
-    icon: web_asset
-    color: 4292030255
-    is_proxy: true
-    host: code-%atrium_count_index%
-    target: localhost:8080
-  - id: 2
-    name: Desktop
-    icon: web_asset
-    color: 4292030255
-    is_proxy: true
-    host: desktop-%atrium_count_index%
-    target: localhost:8081
-  - id: 3
     name: Terminal
     icon: terminal
     color: 4292030255
     is_proxy: true
     host: terminal-%atrium_count_index%
-    target: localhost:8082
+    target: localhost:8081
     secured: true
     login: admin
     password: %user_password%

instance-scripts/config-files/config.yaml

-# The content of this file is recreated by cloud init.
-bind-addr: 0.0.0.0:8080
-auth: password
-password: %user_password%
-cert: false
\ No newline at end of file

instance-scripts/config-files/ttyd.service

@@ -3,7 +3,7 @@ Description=ttyd web server to allow remote access to terminal
 After=network.target
 
 [Service]
-ExecStart=/root/ttyd/ttyd.x86_64 -p 8082 -i lo -c admin:%user_password% -T xterm bash
+ExecStart=/root/ttyd/ttyd.x86_64 -p 8081 -i lo -c admin:%user_password% -T xterm bash
 
 [Install]
 WantedBy=multi-user.target
\ No newline at end of file

instance-scripts/scripts/atrium.sh

@@ -4,11 +4,6 @@
 # Install Atrium
 #
 
-#
-# Example : Generate user_password Argon2 encoded hash for Atrium
-# echo $(scw-userdata user_password) | argon2 $(echo $RANDOM | md5sum | head -c 10) -id -l 32 -e
-#
-
 export USER_PASSWORD_HASH=$(echo -n "$(scw-userdata user_password)" | argon2 "$(echo $RANDOM | md5sum | head -c 12)" -id -l 32 -e)
 
 mkdir -p /root/atrium/data && mkdir -p /root/atrium/letsencrypt_cache

instance-scripts/scripts/code-server.sh

-#!/bin/bash
-
-#
-# Install Code Server
-#
-
-mkdir -p  /root/.config/code-server/
-curl https://forge.grandlyon.com/systemes-dinformation/alpha/projects/labo-rust/-/raw/develop/instance-scripts/config-files/config.yaml -o root/.config/code-server/config.yaml
-sed -i "s/%user_password%/$(scw-userdata user_password)/g" /root/.config/code-server/config.yaml
-curl -fsSL https://code-server.dev/install.sh | sh
-
-#
-# Run Code Server
-#
-
-sudo systemctl enable --now code-server@$USER
\ No newline at end of file

instance-scripts/scripts/rust.sh

-#!/bin/bash
-
-#
-# Install Rust
-#
-
-curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
-source "$HOME/.cargo/env"
\ No newline at end of file

instance-scripts/scripts/webtop.sh

-#!/bin/sh
-
-#
-# Run Webtop Container
-#
-
-docker run -d --name=webtop \
-       --security-opt seccomp=unconfined \
-       -e PUID=0 \
-       -e PGID=0 \
-       -e TZ=Europe/Paris \
-       -e SUBFOLDER=/ \
-       -e KEYBOARD=fr-fr-azerty \
-       -e AUTO_LOGIN=false \
-       -p 8081:3000 \
-       -v /root/webtop:/config \
-       -v /var/run/docker.sock:/var/run/docker.sock \
-       --shm-size=8gb \
-       --cap-add=NET_ADMIN \
-       --privileged \
-       --restart unless-stopped \
-       --sysctl net.ipv6.conf.all.disable_ipv6=0 lscr.io/linuxserver/webtop:ubuntu-xfce
-
-#
-# Modify default password
-#
-
-echo "$(scw-userdata user_password)\n$(scw-userdata user_password)" | docker exec -i webtop passwd abc
\ No newline at end of file

terraform/main.tf

@@ -13,7 +13,7 @@ resource "scaleway_instance_ip" "public_ip" {
 
 resource "scaleway_domain_record" "subdomain_record" {
   count    = var.INSTANCES_COUNT
-  dns_zone = "daag.alpha.grandlyon.com"
+  dns_zone = var.PROJECT_DOMAIN
   name     = "${var.ENVIRONMENT}-${count.index}"
   type     = "A"
   data     = scaleway_instance_ip.public_ip[count.index].address
@@ -22,7 +22,7 @@ resource "scaleway_domain_record" "subdomain_record" {
 
 resource "scaleway_domain_record" "host_subdomain_record" {
   count    = var.INSTANCES_COUNT
-  dns_zone = "daag.alpha.grandlyon.com"
+  dns_zone = var.PROJECT_DOMAIN
   name     = "*.${var.ENVIRONMENT}-${count.index}"
   type     = "A"
   data     = scaleway_instance_ip.public_ip[count.index].address

terraform/variables-gitlab.tf

@@ -27,7 +27,7 @@ variable "PROJECT_DOMAIN" {
 variable "LETSENCRYPT_EMAIL" {
   type        = string
   sensitive   = true
-  description = "User password for accessing services"
+  description = "Lets encrypt mail for renewal"
 }
 
 variable "USER_PASSWORD" {
@@ -43,5 +43,5 @@ variable "INSTANCES_COUNT" {
 
 variable "ENVIRONMENT" {
   type        = string
-  description = "The type of lab to create, ex: devrust, pentest..."
+  description = "Environment of the deployment, contained in domains, subdomains and resources name convention"
 }