Merge branch 'authn-authz-documentation' into 'master'

First version of the authentication/authorization page

See merge request refonte-data/documentation!12

docs/architecture/services/legacy-auth-middleware.md

+
+# Legacy Authentication middleware
+
+## What it does
+
+
+
+## How it works
+
+<!-- ![legacy-auth-middleware-service](../../assets/legacy-auth-middleware-service.png) -->
+
+The entrypoint of the service is a REST API provided by a [NestJS](https://github.com/nestjs/nest) application. 
+
+<!-- It depends on two other services:
+
+* The Legacy Authentication Middleware to verify the identity of the user and retrieve its profile (called through our api gateway: Kong)
+* The Admin API of Kong to get the user specific secrets in order to sign the JSON Web Tokens -->
+
+## API documentation
+
+NestJS provides a [swagger module](https://docs.nestjs.com/recipes/swagger) that can be easily integrated. Using specific annotations alongside your endpoints declaration, this module will automatically generates a swagger documentation, reachable at `/api-doc`.
+
+## Service health
+
+NestJS provides a [health module](https://github.com/nestjs/terminus) based on Terminus, that gives you the opportunity to declare predefined or custom health indicators. It exposes the health status of the service at `/health`.  
+This service will return a `200` http status code when all indicators are healthy. Otherwise it will return a `503` http status code. 
+
+<!-- For this service we declared two health indicators that verify that: 
+* the Kong API gateway is up
+* the legacy authentication middleware is up -->
+
+## Stats
+
+We are using a Node module called [swagger-stats](http://swaggerstats.io/).  
+It traces API calls, monitors API performance and usage statistics. It exposes the metrics in different formats, such as Prometheus format, so you may use Prometheus and Graphana for API monitoring and alerting.  
+Those metrics are available at `/swagger-stats/metrics`.
+
+For more information about this module, visit the [official swagger-stat page](http://swaggerstats.io/docs.html).
+
+## Docker
+It is possible to run this service using Docker containers, using the `docker-compose.yml` and `Dockerfile` files.  
+For more information, refer to the project [legacy-auth-middleware][add a link]
\ No newline at end of file

docs/miscellaneous/authentication&authorization.md

+In the context of the Data portal, we needed an authentication/authorization system in order to control and restrict the access of some ressources.
+
+There is an existing system that let a user authenticate and then access or not some restricted access dataset. However, it does not implement those security features in a modern and very secure way. For exemple, to get the resources accessible by a user we need to pass its username and password. Obviously we cannot just store and pass the user's unencrypted credentials everytime we need it. To address this problem, we developped a middleware that implements a more modern authentication system using features of our API gateway and the legacy service to generate JSON Web Token.
+
+In the following sections we will explain in more details what as been implemented.
+
+## Authentication
+
+There are two kind of entities that can be authenticated: users and services.
+
+### Authenticating a user
+
+Lets explain step by step what is going on from the user account creation to the when he/she/it lists its accessible resources. 
+
+```plantuml
+@startuml
+
+!define BLACK   #333745
+!define RED     #d5232a
+!define GREEN   #37A77C
+
+' Base Setting
+skinparam BackgroundColor transparent
+
+skinparam Sequence {
+    ArrowThickness 1
+    ArrowColor RED
+    LifeLineBorderColor GREEN
+    ParticipantBorderThickness 1
+}
+skinparam Participant {
+    BackgroundColor #FFFFFF
+    BorderColor BLACK
+    FontColor BLACK
+}
+
+skinparam note {
+    BackgroundColor #FFFFFF
+    BorderColor BLACK
+    FontColor BLACK
+}
+
+participant "Front" as front
+participant "Middleware Legacy Auth" as middle
+
+    group Get Public Key
+        front -> middle : <b>GET</b> /publicKey
+        front <-- middle : { publicKey }
+    end
+   
+@enduml
+```
+Before anything, the front web application has to retrieve the public encryption key from the middleware. It will use it to encrypt the password entered by the user when he creates its account but also when he logs in. This key is part of a public/private key generated by the middleware when it starts up.
+
+```plantuml
+@startuml
+
+!define BLACK   #333745
+!define RED     #d5232a
+!define GREEN   #37A77C
+
+' Base Setting
+skinparam BackgroundColor transparent
+
+skinparam Sequence {
+    ArrowThickness 1
+    ArrowColor RED
+    LifeLineBorderColor GREEN
+    ParticipantBorderThickness 1
+}
+skinparam Participant {
+    BackgroundColor #FFFFFF
+    BorderColor BLACK
+    FontColor BLACK
+}
+
+skinparam note {
+    BackgroundColor #FFFFFF
+    BorderColor BLACK
+    FontColor BLACK
+}
+
+participant "Front" as front
+participant "Middleware Legacy Auth" as middle
+participant "Email Service" as email
+
+group User creation
+    front -> middle : <b>POST</b> /user
+    note over middle : Set token in redis with ttl 24h
+    middle -> email : email : <b>POST</b> /email/send (body contains account validation link)
+    middle <-- email : void
+    front <-- middle : void
+end
+@enduml
+```
+
+When a user has entered and submitted its information, the account is not directly created. We require the user to verify its email address. To do so the middleware stores temporarily the user information in redis as a key/value pair. The key is a generated unique Uuid4 token. This key/value pair is valid for 24h.
+
+Then, the middleware formats an email containing a link, that includes the Uuid token, to a particular page of the front web application.
+
+Finally it transfers the email to the email service that will handle the delivery job. 
+
+```plantuml
+@startuml
+
+!define BLACK   #333745
+!define RED     #d5232a
+!define GREEN   #37A77C
+
+' Base Setting
+skinparam BackgroundColor transparent
+
+skinparam Sequence {
+    ArrowThickness 1
+    ArrowColor RED
+    LifeLineBorderColor GREEN
+    ParticipantBorderThickness 1
+}
+skinparam Participant {
+    BackgroundColor #FFFFFF
+    BorderColor BLACK
+    FontColor BLACK
+}
+
+skinparam note {
+    BackgroundColor #FFFFFF
+    BorderColor BLACK
+    FontColor BLACK
+}
+
+participant "Front" as front
+participant "Middleware Legacy Auth" as middle
+participant "Email Service" as email
+participant "Legacy Auth (Neogeo)" as django
+
+group Validate User creation
+    front -> middle : <b>POST</b> /user/validateAccount
+    note over middle : Validate token existance in redis
+    middle -> django : <b>POST</b> /add_user/
+    middle <-- django : Ok
+    note over middle : Remove token from redis
+    front <-- middle : void
+end
+
+@enduml
+```
+
+When the user receives the "account validation" email, he/she/it is invited to click the link. This will open the web application that will get the Uuid4 token from the url and call a validate account endpoint of the middleware.
+
+If the token is still in Redis, then the middleware can process with the account creation calling the legacy Auth service.
+
+```plantuml
+@startuml
+
+!define BLACK   #333745
+!define RED     #d5232a
+!define GREEN   #37A77C
+
+' Base Setting
+skinparam BackgroundColor transparent
+
+skinparam Sequence {
+    ArrowThickness 1
+    ArrowColor RED
+    LifeLineBorderColor GREEN
+    ParticipantBorderThickness 1
+}
+skinparam Participant {
+    BackgroundColor #FFFFFF
+    BorderColor BLACK
+    FontColor BLACK
+}
+
+skinparam note {
+    BackgroundColor #FFFFFF
+    BorderColor BLACK
+    FontColor BLACK
+}
+
+participant "Front" as front
+participant "Authentication Service" as auth
+participant "Middleware Legacy Auth" as middle
+participant "Legacy Auth (Neogeo)" as django
+participant "Email Service" as email
+participant "Kong" as kong
+
+group Login
+    front -> auth : <b>POST</b> /login/legacy
+    auth -> middle : <b>POST</b> /user/login
+    middle -> django : <b>POST</b> /get_user/
+    middle <-- django : { userInfo }
+    middle --> auth : { userInfo with encrypted password as authzKey}
+    auth -> kong : <b>PUT</b> /consumers/:email
+    auth <-- kong : Ok
+    auth -> kong : <b>GET or POST</b> /consumers/:email/jwt (POST if no creadetials exist for this user)
+    auth <-- kong : { credentials }
+    front <-- auth : { token: jwt }
+end
+
+@enduml
+```
+
+Once the account creation  is done, the user can log in with its credentials. The front sends the username and the encrypted password to the authentication service.
+
+This authentication service will then start a process of a couple steps:
+*  Verify the user identity transfering the credentials to the middleware which will unencrypt the password and call the legacy authentication service to get user info. If the credentials are correct the authentication service will receive the information of the user profil.
+*  Once the profile verified, the authentication service will create the user identity in the API Gateway (Kong) (if not already existing). This is referenced as a [consumer](https://docs.konghq.com/0.14.x/admin-api/#consumer-object) in the API Gateway documentation.
+*  Based on the created consumer and thanks to the [JWT plugin](https://docs.konghq.com/hub/kong-inc/jwt/) of Kong, the authentication service will get the JWT Credentials of the user. Those are per user credentials containing a public and private key. If the user does not have credentials yet, the authentication service will generate them using the appropriate endpoint of the Kong JWT plugin.
+*  The service generates a unique random Uuid4 that we name "xsrf token" 
+*  At this point, the authentication service has everything it needs to generate authentication pieces for the user. Using the credentials coming from Kong it will sign a JSON Web Token containing some of the User info (firstname, lastname, username, email) as well as the user encrypted password and the xsrf token.
+* Finally the authentication service will return a response to the front application with the xsrf token and the user info in the body. It will also set a cookie `access_token` containing the JWT for the domaine name hosting the application.
+
+```plantuml
+@startuml
+
+!define BLACK   #333745
+!define RED     #d5232a
+!define GREEN   #37A77C
+
+' Base Setting
+skinparam BackgroundColor transparent
+
+skinparam Sequence {
+    ArrowThickness 1
+    ArrowColor RED
+    LifeLineBorderColor GREEN
+    ParticipantBorderThickness 1
+}
+skinparam Participant {
+    BackgroundColor #FFFFFF
+    BorderColor BLACK
+    FontColor BLACK
+}
+
+skinparam note {
+    BackgroundColor #FFFFFF
+    BorderColor BLACK
+    FontColor BLACK
+}
+
+participant "Front" as front
+participant "Authentication Service" as auth
+participant "Middleware Legacy Auth" as middle
+participant "Legacy Auth (Neogeo)" as django
+participant "Email Service" as email
+participant "Kong" as kong
+
+group Login
+    front -> auth : <b>POST</b> /login/legacy
+    auth -> middle : <b>POST</b> /user/login
+    middle -> django : <b>POST</b> /get_user/
+    middle <-- django : { userInfo }
+    middle --> auth : { userInfo with encrypted password as authzKey}
+    auth -> kong : <b>PUT</b> /consumers/:email
+    auth <-- kong : Ok
+    auth -> kong : <b>GET or POST</b> /consumers/:email/jwt (POST if no creadetials exist for this user)
+    auth <-- kong : { credentials }
+    front <-- auth : { token: jwt }
+end
+
+@enduml
+```
+
+At the end of this process, the front web application has the authentication pieces required to access restricted resources. Let's take the example of the user data accesses.
+
+```plantuml
+@startuml
+
+!define BLACK   #333745
+!define RED     #d5232a
+!define GREEN   #37A77C
+
+' Base Setting
+skinparam BackgroundColor transparent
+
+skinparam Sequence {
+    ArrowThickness 1
+    ArrowColor RED
+    LifeLineBorderColor GREEN
+    ParticipantBorderThickness 1
+}
+skinparam Participant {
+    BackgroundColor #FFFFFF
+    BorderColor BLACK
+    FontColor BLACK
+}
+
+skinparam note {
+    BackgroundColor #FFFFFF
+    BorderColor BLACK
+    FontColor BLACK
+}
+
+participant "Front" as front
+participant "Kong" as kong
+participant "Middleware Legacy Auth" as middle
+participant "Legacy Auth (Neogeo)" as django
+
+group User Info
+    front -> middle : <b>GET</b> /user/resources
+    middle -> django : <b>POST</b> /get_user_service/
+    middle <-- django : { userResources }
+    front <-- middle : { userResources }
+end
+
+@enduml
+```
+
+When a user navigates to the data accesses page, the web application makes a call to the middleware through the API gateway: Kong. The JWT plugin of Kong will verify that the request has a valid JWT (not expired, not modifed) in the dedicated cookie. In the failing case, Kong will return a `401` http error, otherwise it will proxify the request to the authentication service.
+
+Each endpoints that requires a user identity are protected by a "middleware". You can find the corresponding code in the `decode-jwt-payload.middleware.ts` file of our projects. The main role of this application middleware is to decode the JWT payload in order to get the user identity (username, encrypted password).
+
+However in order to get through the middleware and go on with the incoming request the following criteria must be met:
+* the `x-anonymous-consumer` header value of the request must not be `true` (meaning that kong as identified an existing user)
+* the request contains the two pieces of identity `JWT` and `xsrf-token`
+* the xsrf-token present in the header is the same as the one present in the JWT payload
+If one or more of those verifications fails the middleware will return a `401 Unauthenticated` error.
+
+If the get user resources request passes successfully those tests, the legacy middleware authentication will call the Legacy auth service and return the user resoures.
+
+### Authenticating a service
+
+In a micro-services architecture, it is very common that a particular service needs another one to accomplish a task. There could different ways to implement this depending on the needs. In our most common cases the first service needs are direct response of the second service. Thus, we decided to use HTTP requests. Just as the web application calls a service through the api gateway, a service calls another one through the proxy of the API Gateway.
+
+Some HTTP endpoints might be secured and the service will need to authenticate itself to the other service. Obviously a service can't log in with its own credentials as a user would do. The authentication method for services is consequently a bit different and based on API Keys which is another authentication method supported by Kong.
+
+Each service that needs to access protected endpoints will need to have its own key. You can use Kong admin API or [Konga](https://pantsel.github.io/konga/) that provides a GUI for the management of Kong in order to generate keys. In the following paragraph I will explain how to proceed using Konga interface.
+
+An API key is associated to a consumer, so the first step is to create a consumer that will represent the service. As a convention we decided to prefix the services related consumers by '__' which gives for example `__legacy-auth-middleware`. In the consumers section, we can create a new consumer and enter the username of our choice.
+
+![Kong consumer creation](../assets/kong-consumer.png "Kong consumer creation")
+
+Once the consumer is created, when clicking on its name, we can manage its credentials, its groups and a couple more things. In the credentials section and under the API KEYS tab we can create new keys for this specific user. After clicking the create button, juste leave the key field empty if you want to let kong generate it for you.
+
+![Kong consumer api key creation](../assets/kong-consumer-api-key.png "Kong consumer api key creation")
+
+There we go, the service has its own API key! Now every time it needs to make a call to a restricted access endpoint, it will be able to pass along its identity using this key. Though, at this point kong does not know were to look for api keys. To remedy to that we need to configure a `key-auth` plugin. It can be specific to a service or a route but can also be global. 
+
+Let's say we want a global plugin. In the plugin section, click the "add global plugins" button, and select `key-auth`. Then specify the name of the header where we will put the api key in key names. If you want this plugin blocking your request when you don't specify an api key you can add the id of your `ANONYMOUS` user. This means that when kong doesn't identify an existing user, it will transmit the request server with an `x-anonymous-consumer` header set to true.
+
+![Kong key auth plugin configuration](../assets/kong-key-auth-plugin.png "Kong key auth plugin configuration")
+
+So now the service just needs to pass a header `ApiKey:<the service api key>` when calling another service through kong to pass its identity.
+
+However specifying one by one which service is allowed to access a service endpoint is not very convenient, we were more interested in a group based approach. For example our authentication-middleware could belong to a `email-writter` group which would allow it to give send mail tasks to the email service.
+
+Thankfully kong consumers already support this functionnality. We will explain in the next section how we impleteded a group based authorization.
+
+## Authorization
+
+In the previous section we have seen how the identity of a user (or a service) is passed to a service through the API Gateway and how it can use it. Knowing the identity of the consumer was not sufficient in some cases. If we take the example of the organization service, we didn't really care about the specific identity of the user but we wanted to know whether the user was an admin or not: a group based authorization.
+
+First thing first let's see how to add a group to a consumer. Find the consumer in the list and click on it to see the details. Navigate to the group section. Click the add group button and enter the name of the group. The consumer now belongs to a group. This works both for user related consumers or service related consumers.
+
+![Kong consumer groups](../assets/kong-consumer-groups.png "Kong consumer groups")
+
+Then, we need to configure an Access Control List plugin in order for kong to pass the user groups to the upstream. The ACL plugin needs an authentication plugin (JWT plugin, key-auth plugin...) to have indentified an existing user to then get its groups. Go to the global plugin section, click the add plugin button and select `Acl` under security. We don't really want to whitelist or blacklist users, here we just want to propagate the user's groups to the upstream server. As the plugin require one of the two fields to be filled, we basically just enter a fake user id under blacklist.
+
+![Kong Acl plugins configuration](../assets/kong-acl-plugin.png "Kong Acl plugins configuration")
+
+At this point if a service receive an authenticated request it will find the user/service's groups in the following header: `x-consumer-groups`.
+
+Our services are developed using the framework Nest.js. This framework comes with a concept called guard. A guard is basically a middleware that intercepts the request and that will let it through only if it respects some conditions. In our case we need a group guard that would return a `403 Forbidden` error if the user does not belong to the required group.
+
+The corresponding code can be found in the `groups.guards.ts` file of the project that required this kind of authorization (ex: email service). In order for this guard to be applied to our service, we declared it as a provider in the `app.module.ts` file as following.
+
+```ts
+@Module({
+  imports: [...],
+  controllers: [...],
+  providers: [
+    {
+      provide: APP_GUARD,
+      useClass: GroupsGuard,
+    },
+  ],
+})
+export class AppModule { }
+```
+
+As we need this guard to work for any group it is used in combination with a custom decorator `@Groups` defined in the `groups.decorators.ts` file. Doing so, in the controller file, before the declaration of an endpoint we can specify `@Groups('emailWritter')` to restrict the access to users belonging to this group. Actually, 'emailWritter' is just the key that is used to get the real group name from the configuration of the service (please refer to the code).