diff --git a/README.md b/README.md index 32c3e7f1b3a18f5e52fc2f951273efefbd7d7b3c..4e550c23860c8ac9223de14347baaf98a8a6dc16 100644 --- a/README.md +++ b/README.md @@ -1,213 +1,74 @@ -# Labo Rust +# Scaleway Instances Template - Quickstart -Infrastructure as Code project to deploy lab infrastructure and configure instances. Working with Atrium, Gitlab CI, Terraform, Cloud-init, bash scripts and Scaleway. +## How does it work ? -### How does it work ? +The template will provision several resources (including your instances) to the Scaleway cloud provider using Terraform and Gitlab CI. +After provisioning, configuration of the instances and deployment of the services (Atrium and TTYD) will be done using Cloud-init and bash scripts. -The FORGE (Gitlab) manage the Infrastructure with Terraform at any state, plan, creation, modification and destruction with the CI. The FORGE store and will provide the configuration information -such as credentials to Terraform only during the running state to improve security. Then, informations are destroyed with the Gitlab Agent container. +> All your information must be stored in Gitlab and passed as variable to Terraform. -On run state, Terraform will create, modify and destroy infrastructure resources in Scaleway to match the configuration described in the configuration files. +### How the deployment is managed ? -**Resources deployed :** +The whole project is managed by Gitlab using the Gitlab CI. +Launching the pipeline will start the deployment of your infrastructure in 3 steps. -| resource-type | plan | inbound port | -|----------------------------------|--------|--------------| -| scaleway_instance_ip | | | -| scaleway_domain_record | | | -| scaleway_domain_record | | | -| scaleway_instance_security_group | | 22, 443 | -| scaleway_instance_server | DEV1-L | | +**Plan :** +Terraform will perform a "fake deployment" and initialize everything he needs before deploying. +After performing, you will be able to see the plan of the future creation/modification/destruction in the log of the pipeline. +If the modification could lead to break changes and errors, you must not launch the next step of the pipeline. +**Creation :** +Terraform will be applying the exact changes you reviewed in the Plan step. +**Destruction :** +Terraform will destroy the resources he managed to create (or manually imported one). -After the resources provisionned with Terraform, Cloud-init will configure the instances by running bash scripts, creating files... And 3 services will be running for each instance. +## How can I use it ? -**Running services :** +### Step 0 : Prerequisite -- Atrium (Reverse proxy, TLS encryption and HTTPS to the others hosted services) -- Code-server -- Webtop +**On The Forge (Gitlab)**, you must have or ask for the right to fork the repository. -After deployment, each resource can be accessed though HTTPS depending on their count number : -- https://desktop-0.rust-0.daag.alpha.grandlyon.com/ -- https://code-0.rust-0.daag.alpha.grandlyon.com/ -- https://terminal-0.rust-0.daag.alpha.grandlyon.com/ -- https://rust-0.daag.alpha.grandlyon.com/ +**Scaleway** needs you to have credentials to use their services. You must go to [Scaleway API key page](https://console.scaleway.com/iam/api-keys) +and create one for your project. Please make sure to add a description starting with `Terraform - <your description>`. +**The Certificate Authority (CA)** Let's Encrypt needs an email. He will use it to send a notification before the certificate expires to proceed with +the renewal. -- https://desktop-1.rust-1.daag.alpha.grandlyon.com/ -- https://code-1.rust-1.daag.alpha.grandlyon.com/ -- https://terminal-1.rust-1.daag.alpha.grandlyon.com/ -- https://rust-1.daag.alpha.grandlyon.com/ - -### Credentials - -**Desktop username : abc** -**Atrium username : admin** -**Password : <Gitlab Stored>** - -## How to add new resources - -You can add Terraform resources directly to the main.tf file. -Template are available to the link below. - -**Terraform Registry for Scaleway provider : https://registry.terraform.io/providers/scaleway/scaleway/latest/docs** - -## How to setup configuration file for services - -### cloud-init : file creation - -In this example, you will find how to create a file to a defined path with cloud-init. -Content inside `%` are meant to be replaced with sed command to be able to use environment variables or user-data. - -**Example:** -```yaml -write_files: - - content: | - hostname: %atrium_hostname%.daag.alpha.grandlyon.com - debug_mode: false - letsencrypt_email: %atrium_letsencrypt_email% - tls_mode: Auto - apps: # optional : applications served by atrium - - id: 1 - name: Code %atrium_count_index% - icon: web_asset - color: 4292030255 - is_proxy: true - host: code-%atrium_count_index% - target: localhost:8080 - - id: 2 - name: Desktop %atrium_count_index% - icon: web_asset - color: 4292030255 - is_proxy: true - host: desktop-%atrium_count_index% - target: localhost:8081 - path: /root/atrium.yaml -``` - -**Bash commands example to replace `%` content :** -```bash -sed -i "s/%atrium_hostname%/$(scw-userdata atrium_hostname)/g" /root/atrium.yaml -sed -i "s/%atrium_letsencrypt_email%/$(scw-userdata atrium_letsencrypt_email)/g" /root/atrium.yaml -sed -i "s/%atrium_count_index%/$(scw-userdata atrium_count_index)/g" /root/atrium.yaml -``` - -## Downloading file - -The example show how to curl a configuration file from repository to a defined path with cloud-init. +### Step 1 : Getting your own repository + +Fork the project from [this repository](https://forge.grandlyon.com/systemes-dinformation/alpha/pocs/scaleway-instances-template) + +### Step 2 : Setting up the variables with credentials and others information + +Go to the `Settings` menu, `CI/CD` and look for the `Variables` submenu. +Once it's done, you will need to create the variable bellow : +- **ENVIRONMENT** : Environment of the deployment, contained in domains, subdomains and resources name convention. +- **TF_VAR_INSTANCES_COUNT** : Number of instances to create. +- **TF_VAR_LETSENCRYPT_EMAIL** : Lets encrypt mail for renewal. +- **TF_VAR_PROJECT_DOMAIN** : External domain from Scaleway. (Example : <xxx>.alpha.grandlyon.com) +- **TF_VAR_SCW_ACCESS_KEY** : Scaleway Access Key. +- **TF_VAR_SCW_PROJECT_ID** : Project ID from your Scaleway project. +- **TF_VAR_SCW_SECRET_KEY** : Scaleway Secret Key. +- **TF_VAR_USER_PASSWORD** : User password for accessing services and instances. + +### Step 3 : Every is set up, deployment is coming ! + +Go the `CI/CD` menu and `Pipelines`. +From now, you will need to click on `Run pipeline` with your selected branch. +Since you added variables before, you don't need to put any in this step, click `Run Pipeline` again. + +**Checking logs :** You can click on every step to check logs in case of errors or not. +**Check plan :** I recommend to always check the plan when you're trying things to do not perform destruction +on production infrastructure. + +Once plan is done and ok, click the `Apply button` to deploy the infrastructure. Please be sure to click the `Destroy +button` to destroy the infrastructure when you don't need it anymore. + +### Step 4 : Accessing your services + +After deployment, each resource can be accessed though HTTPS, URL depends of the variable you defined before : +- **TTYD :** https://terminal-<INSTANCE_COUNT>.<ENVIRONMENT>-<INSTANCE_COUNT>.<PROJECT_DOMAIN/ +- **Atrium :** https://<ENVIRONMENT>-<INSTANCE_COUNT>.<PROJECT_DOMAIN>/ **Example :** -```yaml -runcmd: -# -# cloud-init execute cmd before HOME and USER are set. -# - - "export HOME=/root" - - "export USER=root" -# -# Install Atrium -# - - "curl https://forge.grandlyon.com/systemes-dinformation/alpha/projects/labo-rust/-/raw/develop/instance-scripts/config-files/atrium.yaml -o root/atrium.yaml" - - "curl https://forge.grandlyon.com/systemes-dinformation/alpha/projects/labo-rust/-/raw/develop/instance-scripts/config-files/atrium.sh | bash" -``` - -## Use Gitlab variables in instances - -From my instances, I should be able to access the variables as user-data using the command `scw-userdata <my-variable>` after declaring them in Terraform. -**Some documentation : https://blog.scaleway.com/introducing-scaleway-cloud-init-support/** - -**Example : Declare in Terraform** -```HCL -user_data = { - atrium_count_index = count.index - atrium_hostname = scaleway_domain_record.subdomain_record[count.index].name - atrium_letsencrypt_email = var.LETSENCRYPT_EMAIL - user_password = var.USER_PASSWORD - cloud-init = file("../instance-scripts/cloud-init.yml") // this is not a variable, but the declaration of cloud-init file. -} -``` - -** Example : Retrieve in instance -```bash -scw-userdata atrium_count_index -``` - -## Setup Terraform Locally - -First, you must setup 2 local files for your variables : - -### variables-local.tf - -Create a file **variables-local.tf** containing the following code : -```hcl -variable "FORGE_PROJECT_ID" { - type = string - description = "Forge Project ID" - default = "your project id" - sensitive = true -} - -variable "FORGE_USERNAME" { - type = string - description = "Forge Username" - default = "your username" - sensitive = true -} - -variable "FORGE_ACCESS_TOKEN" { - type = string - description = "Forge Access Token" - default = "your access token" - sensitive = true -} -``` -### variables-local.tfvars - -Now, you can create a file for your variables information called **variables-local.tfvars** containing the following code : -```hcl -### SCW variables - -SCW_PROJECT_ID = "" -SCW_ACCESS_KEY = "" -SCW_SECRET_KEY = "" -INSTANCES_COUNT = "" -ENVIRONMENT = "" -``` - -### Terraform init - Gitlab remote tfstate - -You can grab a init command from your gitlab project on menu Infrastructure > Terraform. -Select your environment and click the actions button, then you will only need to provide a gitlab project token. -Command should look like : -```bash -export GITLAB_ACCESS_TOKEN=<YOUR-ACCESS-TOKEN> -terraform init \ - -backend-config="address=https://forge.grandlyon.com/api/v4/projects/<project-id>/terraform/state/<tfstate-name>" \ - -backend-config="lock_address=https://forge.grandlyon.com/api/v4/projects/<project-id>/terraform/state/<tfstate-name>/lock" \ - -backend-config="unlock_address=https://forge.grandlyon.com/api/v4/projects/<project-id>/terraform/state/<tfstate-name>/lock" \ - -backend-config="username=xxxxxxx" \ - -backend-config="password=$GITLAB_ACCESS_TOKEN" \ - -backend-config="lock_method=POST" \ - -backend-config="unlock_method=DELETE" \ - -backend-config="retry_wait_min=5" - -terraform init -var-file=variables-local.tfvars -``` - -### Terraform plan - With variables file - -```bash -terraform plan -var-file=variables-local.tfvars -out=tfplan -``` - -### Terraform apply - With plan - -```bash -terraform apply tfplan -``` - -### Terraform destroy - With variables file - -```bash -terraform destroy -var-file=variables-local.tfvars -``` \ No newline at end of file +- https://terminal-1.rust-1.daag.alpha.grandlyon.com/ +- https://rust-1.daag.alpha.grandlyon.com/ \ No newline at end of file diff --git a/instance-scripts/cloud-init.yml b/instance-scripts/cloud-init.yml index c255c0aef7aea33eaaed99827cfdb1cbaf620f5e..98a97e773e44c56c1d3943f916b9fac562f6a826 100644 --- a/instance-scripts/cloud-init.yml +++ b/instance-scripts/cloud-init.yml @@ -17,10 +17,6 @@ runcmd: - "export HOME=/root" - "export USER=root" # -# Install Rust -# - - "curl https://forge.grandlyon.com/systemes-dinformation/alpha/projects/labo-rust/-/raw/develop/instance-scripts/scripts/rust.sh | bash" -# # Install Docker # - "curl https://forge.grandlyon.com/systemes-dinformation/alpha/projects/labo-rust/-/raw/develop/instance-scripts/scripts/docker.sh | bash" @@ -29,18 +25,6 @@ runcmd: # - "curl https://forge.grandlyon.com/systemes-dinformation/alpha/projects/labo-rust/-/raw/develop/instance-scripts/scripts/atrium.sh | bash" # -# Install Code Server -# - - "curl https://forge.grandlyon.com/systemes-dinformation/alpha/projects/labo-rust/-/raw/develop/instance-scripts/scripts/code-server.sh | bash" -# -# Install Webtop -# - - "curl https://forge.grandlyon.com/systemes-dinformation/alpha/projects/labo-rust/-/raw/develop/instance-scripts/scripts/webtop.sh | sh" -# # Install ttyd # - - "curl https://forge.grandlyon.com/systemes-dinformation/alpha/projects/labo-rust/-/raw/develop/instance-scripts/scripts/ttyd.sh | bash" -# -# Install Repository -# - - "git clone https://github.com/NathanRodet/rust-axum-demo /root/rust-axum-demo" \ No newline at end of file + - "curl https://forge.grandlyon.com/systemes-dinformation/alpha/projects/labo-rust/-/raw/develop/instance-scripts/scripts/ttyd.sh | bash" \ No newline at end of file diff --git a/instance-scripts/config-files/atrium.yaml b/instance-scripts/config-files/atrium.yaml index b9f6de3f6cc1e9de8850f93af647f93887b56e3e..50351098894d2404261b1620d86bbe12ddf937d2 100644 --- a/instance-scripts/config-files/atrium.yaml +++ b/instance-scripts/config-files/atrium.yaml @@ -5,26 +5,12 @@ letsencrypt_email: %atrium_letsencrypt_email% tls_mode: Auto apps: # optional : applications served by atrium - id: 1 - name: Code - icon: web_asset - color: 4292030255 - is_proxy: true - host: code-%atrium_count_index% - target: localhost:8080 - - id: 2 - name: Desktop - icon: web_asset - color: 4292030255 - is_proxy: true - host: desktop-%atrium_count_index% - target: localhost:8081 - - id: 3 name: Terminal icon: terminal color: 4292030255 is_proxy: true host: terminal-%atrium_count_index% - target: localhost:8082 + target: localhost:8081 secured: true login: admin password: %user_password% diff --git a/instance-scripts/config-files/config.yaml b/instance-scripts/config-files/config.yaml deleted file mode 100644 index f212ef59ce13e4a0428648bb1c1722762124bc20..0000000000000000000000000000000000000000 --- a/instance-scripts/config-files/config.yaml +++ /dev/null @@ -1,5 +0,0 @@ -# The content of this file is recreated by cloud init. -bind-addr: 0.0.0.0:8080 -auth: password -password: %user_password% -cert: false \ No newline at end of file diff --git a/instance-scripts/config-files/ttyd.service b/instance-scripts/config-files/ttyd.service index 545737074ff5cde21dbb3b1fd183c744e0188392..177aa70826e25a81b3d60e4f4436075e776bfb6d 100644 --- a/instance-scripts/config-files/ttyd.service +++ b/instance-scripts/config-files/ttyd.service @@ -3,7 +3,7 @@ Description=ttyd web server to allow remote access to terminal After=network.target [Service] -ExecStart=/root/ttyd/ttyd.x86_64 -p 8082 -i lo -c admin:%user_password% -T xterm bash +ExecStart=/root/ttyd/ttyd.x86_64 -p 8081 -i lo -c admin:%user_password% -T xterm bash [Install] WantedBy=multi-user.target \ No newline at end of file diff --git a/instance-scripts/scripts/atrium.sh b/instance-scripts/scripts/atrium.sh index b95a734fdba749ec98e56c693c308e16a6030ab6..42a9612d1c2f79d6f5479b6ce2d8c3d826c13439 100644 --- a/instance-scripts/scripts/atrium.sh +++ b/instance-scripts/scripts/atrium.sh @@ -4,11 +4,6 @@ # Install Atrium # -# -# Example : Generate user_password Argon2 encoded hash for Atrium -# echo $(scw-userdata user_password) | argon2 $(echo $RANDOM | md5sum | head -c 10) -id -l 32 -e -# - export USER_PASSWORD_HASH=$(echo -n "$(scw-userdata user_password)" | argon2 "$(echo $RANDOM | md5sum | head -c 12)" -id -l 32 -e) mkdir -p /root/atrium/data && mkdir -p /root/atrium/letsencrypt_cache diff --git a/instance-scripts/scripts/code-server.sh b/instance-scripts/scripts/code-server.sh deleted file mode 100644 index 8c267c08c2ed2453b7a21a102fe3eeb3c1deb2fd..0000000000000000000000000000000000000000 --- a/instance-scripts/scripts/code-server.sh +++ /dev/null @@ -1,16 +0,0 @@ -#!/bin/bash - -# -# Install Code Server -# - -mkdir -p /root/.config/code-server/ -curl https://forge.grandlyon.com/systemes-dinformation/alpha/projects/labo-rust/-/raw/develop/instance-scripts/config-files/config.yaml -o root/.config/code-server/config.yaml -sed -i "s/%user_password%/$(scw-userdata user_password)/g" /root/.config/code-server/config.yaml -curl -fsSL https://code-server.dev/install.sh | sh - -# -# Run Code Server -# - -sudo systemctl enable --now code-server@$USER \ No newline at end of file diff --git a/instance-scripts/scripts/rust.sh b/instance-scripts/scripts/rust.sh deleted file mode 100644 index 21df85f4b32fe2ead4be19dd72266633f1436f47..0000000000000000000000000000000000000000 --- a/instance-scripts/scripts/rust.sh +++ /dev/null @@ -1,8 +0,0 @@ -#!/bin/bash - -# -# Install Rust -# - -curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y -source "$HOME/.cargo/env" \ No newline at end of file diff --git a/instance-scripts/scripts/webtop.sh b/instance-scripts/scripts/webtop.sh deleted file mode 100644 index a4081a9253f2f643512eb1059fc0dc63bfa816ee..0000000000000000000000000000000000000000 --- a/instance-scripts/scripts/webtop.sh +++ /dev/null @@ -1,28 +0,0 @@ -#!/bin/sh - -# -# Run Webtop Container -# - -docker run -d --name=webtop \ - --security-opt seccomp=unconfined \ - -e PUID=0 \ - -e PGID=0 \ - -e TZ=Europe/Paris \ - -e SUBFOLDER=/ \ - -e KEYBOARD=fr-fr-azerty \ - -e AUTO_LOGIN=false \ - -p 8081:3000 \ - -v /root/webtop:/config \ - -v /var/run/docker.sock:/var/run/docker.sock \ - --shm-size=8gb \ - --cap-add=NET_ADMIN \ - --privileged \ - --restart unless-stopped \ - --sysctl net.ipv6.conf.all.disable_ipv6=0 lscr.io/linuxserver/webtop:ubuntu-xfce - -# -# Modify default password -# - -echo "$(scw-userdata user_password)\n$(scw-userdata user_password)" | docker exec -i webtop passwd abc \ No newline at end of file diff --git a/terraform/main.tf b/terraform/main.tf index ed0bee7f1aa8707ce939478ce3f6582cb13e5720..f41e2aa481e5f89d4e9f17d56002f72a95e875de 100644 --- a/terraform/main.tf +++ b/terraform/main.tf @@ -13,7 +13,7 @@ resource "scaleway_instance_ip" "public_ip" { resource "scaleway_domain_record" "subdomain_record" { count = var.INSTANCES_COUNT - dns_zone = "daag.alpha.grandlyon.com" + dns_zone = var.PROJECT_DOMAIN name = "${var.ENVIRONMENT}-${count.index}" type = "A" data = scaleway_instance_ip.public_ip[count.index].address @@ -22,7 +22,7 @@ resource "scaleway_domain_record" "subdomain_record" { resource "scaleway_domain_record" "host_subdomain_record" { count = var.INSTANCES_COUNT - dns_zone = "daag.alpha.grandlyon.com" + dns_zone = var.PROJECT_DOMAIN name = "*.${var.ENVIRONMENT}-${count.index}" type = "A" data = scaleway_instance_ip.public_ip[count.index].address diff --git a/terraform/variables-gitlab.tf b/terraform/variables-gitlab.tf index b8ac5e8094ce7917a34cdfd93069c9c5af6fd5bf..6c31d72e31aac81dc3e0d800806fc51e060d1f58 100644 --- a/terraform/variables-gitlab.tf +++ b/terraform/variables-gitlab.tf @@ -27,7 +27,7 @@ variable "PROJECT_DOMAIN" { variable "LETSENCRYPT_EMAIL" { type = string sensitive = true - description = "User password for accessing services" + description = "Lets encrypt mail for renewal" } variable "USER_PASSWORD" { @@ -43,5 +43,5 @@ variable "INSTANCES_COUNT" { variable "ENVIRONMENT" { type = string - description = "The type of lab to create, ex: devrust, pentest..." + description = "Environment of the deployment, contained in domains, subdomains and resources name convention" }