1_collecte_donnee.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1 - Collecte des données"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nous allons avoir besoin des modules suivants :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...et des proxies HTTP(S) qui vont bien :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "proxies = {\n",
    "    'http': 'http://proxyhttp1pro:8080',\n",
    "    'https': 'http://proxyhttp1pro:8080'\n",
    "}\n",
    "\n",
    "s = requests.Session()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "s.proxies = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Préambule"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nous souhaitons collecter les données relatives au jeu de données présenté ici : https://data.beta.grandlyon.com/fr/datasets/arbres-alignement-metropole-lyon/info. Le portail data.grandlyon.com peut nous fournir un exemple de requête WFS permettant de récupérer 30 enregistrements dans le format GeoJSON. Voici cette URL:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=30\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "La commande suivante permet d'effectuer une requête HTTP vers cette URL, et de stocker le résultat dans la variable `res` : "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "res = s.get(url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On peut consulter le \"status code\" de la requête comme suit :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res.status_code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Si le status code est égal à 200, alors c'est bon : la requête a réussi !"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "La commande suivante permet de stocker la réponse de la requête dans la variable `data`, au format JSON :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = res.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Afin de visualiser son contenu, vous pouvez décommenter la ligne suivante et l'exécuter : "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['type', 'name', 'crs', 'features'])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Les données qui nous intéressent se trouvent dans la clé `features` du dictionnaire `data` : "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "records = data['features']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Comme \"promis\", nous avons collecté 30 enregistrements : "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "30"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(data['features'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Collecte \"paginée\" des données"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On introduit une fonction qui permet de construire les URLs relatives aux différentes pages que nous devons consulter : "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_url(maxfeatures_per_page, page_no):\n",
    "    \n",
    "    if page_no < 1:\n",
    "        raise Exception('page_no has to be greater than 1!')\n",
    "    \n",
    "    startindex = maxfeatures_per_page * (page_no-1)\n",
    "    \n",
    "    root_url = \"https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326\"\n",
    "\n",
    "    url = root_url + \"&maxfeatures=%i&startindex=%i\" % (maxfeatures_per_page, startindex)\n",
    "    \n",
    "    return url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "URL pour la première page de 100 éléments :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=100&startindex=0'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "build_url(100, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "URL pour la déuxième page de 1000 éléments :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=1000'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "build_url(1000, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "La commande `range(a, b)` permet de générer la liste de numéros entiers allant de `a` à `b`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 2, 3, 4, 5, 6, 7, 8, 9]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(range(1, 10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Le jeu de donnée en question incluant un peu plus que 94000 enregistrements, on aura besoin de pas plus que 95 pages de 1000 éléments :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=0\n",
      "2 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=1000\n",
      "3 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=2000\n",
      "4 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=3000\n",
      "5 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=4000\n",
      "6 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=5000\n",
      "7 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=6000\n",
      "8 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=7000\n",
      "9 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=8000\n",
      "10 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=9000\n",
      "11 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=10000\n",
      "12 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=11000\n",
      "13 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=12000\n",
      "14 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=13000\n",
      "15 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=14000\n",
      "16 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=15000\n",
      "17 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=16000\n",
      "18 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=17000\n",
      "19 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=18000\n",
      "20 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=19000\n",
      "21 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=20000\n",
      "22 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=21000\n",
      "23 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=22000\n",
      "24 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=23000\n",
      "25 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=24000\n",
      "26 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=25000\n",
      "27 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=26000\n",
      "28 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=27000\n",
      "29 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=28000\n",
      "30 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=29000\n",
      "31 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=30000\n",
      "32 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=31000\n",
      "33 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=32000\n",
      "34 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=33000\n",
      "35 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=34000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "36 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=35000\n",
      "37 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=36000\n",
      "38 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=37000\n",
      "39 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=38000\n",
      "40 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=39000\n",
      "41 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=40000\n",
      "42 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=41000\n",
      "43 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=42000\n",
      "44 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=43000\n",
      "45 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=44000\n",
      "46 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=45000\n",
      "47 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=46000\n",
      "48 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=47000\n",
      "49 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=48000\n",
      "50 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=49000\n",
      "51 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=50000\n",
      "52 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=51000\n",
      "53 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=52000\n",
      "54 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=53000\n",
      "55 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=54000\n",
      "56 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=55000\n",
      "57 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=56000\n",
      "58 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=57000\n",
      "59 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=58000\n",
      "60 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=59000\n",
      "61 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=60000\n",
      "62 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=61000\n",
      "63 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=62000\n",
      "64 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=63000\n",
      "65 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=64000\n",
      "66 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=65000\n",
      "67 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=66000\n",
      "68 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=67000\n",
      "69 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=68000\n",
      "70 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=69000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "71 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=70000\n",
      "72 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=71000\n",
      "73 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=72000\n",
      "74 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=73000\n",
      "75 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=74000\n",
      "76 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=75000\n",
      "77 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=76000\n",
      "78 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=77000\n",
      "79 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=78000\n",
      "80 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=79000\n",
      "81 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=80000\n",
      "82 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=81000\n",
      "83 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=82000\n",
      "84 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=83000\n",
      "85 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=84000\n",
      "86 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=85000\n",
      "87 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=86000\n",
      "88 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=87000\n",
      "89 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=88000\n",
      "90 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=89000\n",
      "91 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=90000\n",
      "92 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=91000\n",
      "93 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=92000\n",
      "94 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=93000\n",
      "95 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=94000\n",
      "96 https://download.data.grandlyon.com/wfs/grandlyon?SERVICE=WFS&VERSION=2.0.0&outputFormat=application/json; subtype=geojson&request=GetFeature&typename=abr_arbres_alignement.abrarbre&SRSNAME=EPSG:4326&maxfeatures=1000&startindex=95000\n"
     ]
    }
   ],
   "source": [
    "le_nombre_de_enregistrements_par_page = 1000\n",
    "\n",
    "# on utilisera la variable suivante pour comuler les réponses des différentes requêtes :\n",
    "records = []\n",
    "\n",
    "i = 1\n",
    "while True:    \n",
    "    url = build_url(le_nombre_de_enregistrements_par_page, i)\n",
    "    print(i, url)\n",
    "    \n",
    "    res = s.get(url)\n",
    "    data = res.json()\n",
    "    records += data['features']\n",
    "    \n",
    "    i +=1 \n",
    "\n",
    "    if len(data['features']) < le_nombre_de_enregistrements_par_page:\n",
    "        # cela veut dire que nous avons atteint la dernière page !\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "La commande suivante nous permet de stocker les données dans un fichier le résultat des requêtes précedentes :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = 'data/arbres_alignement_Lyon.json'\n",
    "\n",
    "with open(filename, 'w') as fp:\n",
    "    json.dump(records, fp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On utilisera ce fichier comme entrée des cahiers suivants..."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}