dacc.md

# DACC (Data Aggregator Cross Cozy)

What is DACC ? A cozy blackbox where apps can send data in order to be anonymised and aggregate to provide usage stats.

## Infrastructure workflow

We have two environments, a dev one and a production. Both of them look like the following schema :

![dacc](/img/architecture/dacc.png)

On the development env, the 'cozy env' refers to our 'Alpha' VM.

## Service

There is a service named `aggregatorUsageEvent`, running on a daily basis, sending pre-processed data to cozy. This service is responsible for parsing traced events store in database and send them to cozy.

The service is configured to run every day randomly before **07:00AM**.

## Usage events

This section explains how the application is tracking usage events. Here are the main step:

- All events are stored during the use of the application in the doctype **com.grandlyon.ecolyo.usageevents**
- A service is responsible of the aggregation of these events to create ANONYMIZED indicators. These indicators are then sent to a remote docType.

| Title | Tech name | target | context | startDate | result | trigger | Cozy Dacc Name |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Complétion d'un nouveau profile | ProfileSetEvent | --- | --- | --- | --- | Lors de l'affichage de la page de félicitation profileTypeFinished | navigation-action-daily |
| Consultation du nouveau bilan mensuel | ReportFromEvent | --- | 'analysis' | --- | 1 | Lorsque l'utilisateur consulte sa page analyse quand la notification est activée | summary-subscription-monthly |

## Indicators

!!! warning "note"
    All indicators are only sent if a user validate first version of CGU, the one containing dacc validation

### Monthly Indicators

All monthly indicators call can be found after this comment on the code

```js
// Monthly events
```

| Indicator | Trigger | value |
| --- | --- | --- |
| consumption-variation-monthly | Tous les mois si l'utilisateur a été connecté au moins une fois dans le mois | La variation de consommation par type de fluide. Les paramètres différents sont le type de profil ainsi que le nombre de mois entre la première connexion de l'utilisateur et le moment où est remonté l'indicateur (seniority) |
| summary-subscription-monthly | Tous les mois si un utilisateur a souscrit a la newsletter ou c'est désabonné | 1 si l’utilisateur est inscrit sinon 0 |
| fluid-data-granularity-monthly | Tous les mois sans condition | 1 si l'utilisateur possède de la donnée a la demi heure sur le mois. Sinon 0 |
| profile-count-monthly | Tous les mois si le profil a été complété au moins une fois | Le nombre de fois ou le profile a été complété |

### Daily Indicators

The daily process get all events of previous day and process them in the following indicators

| Indicator | Trigger | value |
| --- | --- | --- |
| konnector-connected-daily | Tous les jours | La valeur est 1. Les valeurs sont catégorisées par type de fluide (electricity / electricity:water ...) |
| summary-subscription-monthly | Tous les jours | 1 si l'utilisateur est revenu sur Ecolyo via le lien présent dans la newsletter |

## Information sending

### Format

All indicator are post to a remote doctype, which will allow the application to read a defined external API. This doctype is created by Cozy itself.
The structure to send is the following :

```ts
{
  createdBy: string
  measureName: string
  startDate: string
  value: number | null
  group1?: object
  group2?: object
  group3?: object
}
```

with:

- *createdBy*: contains "ecolyo" value.
- *startDate*: used to defined the start date of the indicator
- *value*: contains a value relevant to the defined indicator (example: number of page viewed on a day).
- *groups*: contains all attributes defining a indicator. Please refer to [indicators definition](https://stats.cozycloud.cc/question#eyJkYXRhc2V0X3F1ZXJ5Ijp7ImRhdGFiYXNlIjo0LCJxdWVyeSI6eyJzb3VyY2UtdGFibGUiOjh9LCJ0eXBlIjoicXVlcnkifSwiZGlzcGxheSI6InRhYmxlIiwidmlzdWFsaXphdGlvbl9zZXR0aW5ncyI6e319)
- *mesureName*: contains one of the following value
  
| ENUM | value |
| --- | --- |
| CONSUMPTION_VARIATION_MONTHLY   | consumption-variation-monthly |
| FLUID_DATA_GRANULARITY    | fluid-data-granularity-monthly |
| KONNECTOR_CONNECTED_PER_DAY   | konnector-connected-daily |
| PROFILE_COUNT_MONTHLY    | profile-count-monthly |
| SUMMARY_SUBSCRIPTION_MONTHLY  | summary-subscription-monthly |

### Environments

There is two remote doctypes pointing to two different mode. `dacc-dev` is used on alpha.

```json
    "dacc": {
      "type": "cc.cozycloud.dacc",
      "verbs": ["ALL"]
    },
    "dacc-dev": {
      "type": "cc.cozycloud.dacc.dev",
      "verbs": ["ALL"]
    },
```

## How to local test ?

First you need to modify the service in order to be able to see data in local :

- Run a local server that will listen and log events.

```sh
node srcipts/server.js
```

- Edit dacc URL (line 67) to send requests to your local server at `http://localhost:8081`

```js
    await client
      .getStackClient()
      .fetchJSON(
        'POST',
        environments.isProduction()
          ? '/remote/cc.cozycloud.daccv_2'
          : 'http://localhost:8081',
        {
          data: JSON.stringify(indicator),
        }
      )
```

- For **monthly indicators**, remove date protection (ligne 1100)

```js
  if (
    profile
    // &&
    // DateTime.local()
    //   .setZone('utc', {
    //     keepLocalTime: true,
    //   })
    //   .startOf('day').day === profile.monthlyAnalysisDate.day
  ) {
```

- Generate usage events by navigating through the app

You will be able to see previous events and freshly generated events in the db `com-grandlyon-ecolyo-usageevent`

:warning: because the cron run every day before 7am, it will take events from the day before. So if you want to test specific usage, you will have to edit the `eventDate` of events you want to send.

- Then you are finally ready to test ! Just run the following :

*If aggregatorUsageEvents changed:*

```sh
yarn build-dev:browser && yarn run cozy-konnector-dev -m ./manifest.webapp ./build/services/aggregatorUsageEvents/ecolyo.js
```

*To save time, when running multiple times and if aggregatorUsageEvents didn't change, you don't have to build.*

```sh
yarn run cozy-konnector-dev -m ./manifest.webapp ./build/services/aggregatorUsageEvents/ecolyo.js
```

- You should see sent indicators in the logs of your local server

```json
{
  "data": {"createdBy":"ecolyo","measureName":"connection-count-monthly","startDate":"2021-12-09","value":3}
}
{
  "data": {"createdBy":"ecolyo","measureName":"connection-count-monthly","startDate":"2021-12-09","value":3}
}
```

## How to add new indicator ?

You have to reach cozy and discuss with them about new indicators. They will implement it for `dacc-dev` and `dacc` and you will be able to test it on alpha env. Remember you will need **at least 5 data** of each type to see it in metabase.

## Visualisation

All the data aggregated in the dacc can be visualized with a *Metabase* website hosted by [cozy](https://stats.cozycloud.cc/).

## Errors

More here in [DJU](/ecolyo/functionalities/profile_type/#dju).

If there is an authentication error, the api will returns an error in html format to cozy, so you'll get from cozy an error 502 as following, because cozy expects a json response :

```json
{
  "status": "502",
  "title": "Bad Gateway",
  "detail": "the content-type for the response is not authorized",
  "source": {}
}
```