Exploring Weather and Climate Data, Scraping, and Connecting to the Dashboard

This vignette explains the process to create list of dates for user selection and to retrieve data from Iowa Mesonet Reanalysis (IEMRE)

##Load libraries

library(DSPGFarmer)
library(jsonlite)
library(tidyverse)
library(rmarkdown)

Data

Weather data is retrieved from Iowa Environmental Mesonet Reanalysis (IEMRE) alt text

IEM provides:
* A regular data set of Iowa Environmental Data.
* Access to hourly and daily interval data.
* Hourly data includes 2 Meter Air Temperature, 10 Meter Wind Speed, and Direction and Precipitation.
* Daily data includes High and Low 2 Meter Air Temperature and Precipitation.

Data can be accessed through some JSON web services; MultiDay Request, Single Day Request, and Single Day’s Hourly (CDT/CST) Request. Example MultiDay Request. Other request examples are available on the IEMRE page. Multi-year querying is not supported at this time.

Script to create list of dates for user selection

#Instantiate variables

Year= 2015
startMonth = 4
EndMonth = 10

##----------------------------------------------------------------------------

CurrentDate <- Sys.Date()-2 #Because there is a 2-3 day delay in data availability we subtract a few days from the current date
StartDates <- list()
EndDates <-list()

#Calculate the number of days remaining in the cureent month using the lubridate package
EndDay = lubridate::days_in_month(as.Date(paste(EndMonth,"/01/2000",sep="")))

#Format dates into m/d/y format
StartMonthCorrect <-format(as.Date(paste(startMonth,"/01/2000",sep=""), "%m/%d/%y"),"%m")
EndMonthCorrect <-format(as.Date(paste(EndMonth,"/01/2000",sep=""), "%m/%d/%y"),"%m")

YearRange = as.numeric(format(CurrentDate,"%Y"))-Year

for(yearVal in 0:YearRange){
  
    
  StartDate = paste(Year+yearVal,"-",StartMonthCorrect,'-',"01",sep="")
  EndDate = paste(Year+yearVal,"-",EndMonthCorrect,"-",EndDay,sep="")
  if(Year+yearVal==as.numeric(format(CurrentDate,"%Y"))){
    EndDate <- format(Sys.Date()-2,"%Y-%m-%d")
  }
  StartDates <-append(StartDates,StartDate)
  EndDates <-append(EndDates,EndDate)
}

Script to retrieve data from IEMRE

Table containing variable description.

Variable	Description
base_url	Url for IEMRE
target_request	Which data to access. Ex.”Multiday” – Multiday Request
End_of_url	Output is a JSON file

##----lat and lon are retrieved from user input but as an example, lat and lon will be provided here-----

lat <- 42
lon <- -98

base_url <- "https://mesonet.agron.iastate.edu/iemre/"
target_request <- "multiday/"
end_of_url <- "json"

#Create empty list to store url's
url_list <-list()

##----Loop through the start and end dates.-----
##----url format is: https://mesonet.agron.iastate.edu/iemre/multiday/StartDates/EndDates/lat/lon/json------

##----Create empty dataframe----
df <- data.frame()

for (i in 1:length(StartDates)) {
      
      url <- paste0(base_url, target_request, StartDates[i], "/", EndDates[i], "/", lat, "/", lon, "/", end_of_url)
        url_list <- c(url_list, url)
        print(url)  
        data <- as.data.frame(jsonlite::fromJSON(url)) #Extract JSON data and convert to a dataframe
        df <- rbind(df,data)
}

##----Save data to a csv file----
#write.csv(df,"path/to/save/filename.csv")

##----Display first few line of dataset
head(df)

Nabil Idris

Data

Script to create list of dates for user selection

Script to retrieve data from IEMRE