Exploring Weather and Climate Data, Scraping, and Connecting to the Dashboard

Nabil

Data

Weather data is retrieved from Iowa Environmental Mesonet Reanalysis (IEMRE) alt text

IEM provides:
* A regular data set of Iowa Environmental Data.
* Access to hourly and daily interval data.
* Hourly data includes 2 Meter Air Temperature, 10 Meter Wind Speed, and Direction and Precipitation.
* Daily data includes High and Low 2 Meter Air Temperature and Precipitation.

Data can be accessed through some JSON web services; MultiDay Request, Single Day Request, and Single Day’s Hourly (CDT/CST) Request. Example MultiDay Request. Other request examples are available on the IEMRE page. Multi-year querying is not supported at this time.

Methods

Script to create list of weather dates for user selection 

Year= 2015
startMonth = 4
EndMonth = 10

##----------------------------------------------------------------------------

CurrentDate <- Sys.Date()-2
StartDates <- list()
EndDates <-list()

EndDay = lubridate::days_in_month(as.Date(paste(EndMonth,"/01/2000",sep="")))

StartMonthCorrect <-format(as.Date(paste(startMonth,"/01/2000",sep=""), "%m/%d/%y"),"%m")
EndMonthCorrect <-format(as.Date(paste(EndMonth,"/01/2000",sep=""), "%m/%d/%y"),"%m")

YearRange = as.numeric(format(CurrentDate,"%Y"))-Year

for(yearVal in 0:YearRange){
  
    
  StartDate = paste(Year+yearVal,"-",StartMonthCorrect,'-',"01",sep="")
  EndDate = paste(Year+yearVal,"-",EndMonthCorrect,"-",EndDay,sep="")
  if(Year+yearVal==as.numeric(format(CurrentDate,"%Y"))){
    EndDate <- format(Sys.Date()-2,"%Y-%m-%d")
  }
  StartDates <-append(StartDates,StartDate)
  EndDates <-append(EndDates,EndDate)
}

Script to retrieve data from IEMRE

Variable Description
base_url Url for IEMRE
target_request Which data to access. Ex.”Multiday” – Multiday Request
End_of_url Output is a JSON file 

##----lat and lon are retrieved from user input but as an example, lat and lon will be provided here-----

lat <- 42
lon <- -98

base_url <- "https://mesonet.agron.iastate.edu/iemre/"
target_request <- "multiday/"
end_of_url <- "json"

#Create empty list to store url's
url_list <-list()

##----Loop through the start and end dates. First date is the start date, second date is the end date. Dates are paired.-----
##----url format is: https://mesonet.agron.iastate.edu/iemre/multiday/start_dates/end_dates/lat/lon/json------

##----Create empty dataframe----
df <- data.frame()

for (i in 1:length(StartDates)) {
      
      url <- paste0(base_url, target_request, StartDates[i], "/", EndDates[i], "/", lat, "/", lon, "/", end_of_url)
        url_list <- c(url_list, url)
        print(url)  
        data <- as.data.frame(jsonlite::fromJSON(url))
        df <- rbind(df,data)
}
#> [1] "https://mesonet.agron.iastate.edu/iemre/multiday/2015-04-01/2015-10-31/42/-98/json"
#> [1] "https://mesonet.agron.iastate.edu/iemre/multiday/2016-04-01/2016-10-31/42/-98/json"
#> [1] "https://mesonet.agron.iastate.edu/iemre/multiday/2017-04-01/2017-10-31/42/-98/json"
#> [1] "https://mesonet.agron.iastate.edu/iemre/multiday/2018-04-01/2018-10-31/42/-98/json"
#> [1] "https://mesonet.agron.iastate.edu/iemre/multiday/2019-04-01/2019-10-31/42/-98/json"
#> [1] "https://mesonet.agron.iastate.edu/iemre/multiday/2020-04-01/2020-10-31/42/-98/json"
#> [1] "https://mesonet.agron.iastate.edu/iemre/multiday/2021-04-01/2021-10-31/42/-98/json"
#> [1] "https://mesonet.agron.iastate.edu/iemre/multiday/2022-04-01/2022-07-31/42/-98/json"

##----Save data to a csv file----
#write.csv(df,"path/to/save/filename.csv")
##----Display first few line of dataset
head(df)
#>    data.date data.mrms_precip_in data.prism_precip_in data.daily_high_f
#> 1 2015-04-01         0.007874016                    0            84.992
#> 2 2015-04-02         0.275590551                    0            63.104
#> 3 2015-04-03         0.027559055                    0            51.512
#> 4 2015-04-04         0.000000000                    0            67.442
#> 5 2015-04-05         0.000000000                    0            71.924
#> 6 2015-04-06         0.000000000                    0            55.112
#>   data.12z_high_f data.climate_daily_high_f data.daily_low_f data.12z_low_f
#> 1          79.196                      54.1           51.944         34.034
#> 2          83.876                      54.5           37.004         38.336
#> 3          62.726                      54.8           30.254         28.958
#> 4          52.178                      55.1           28.688         26.960
#> 5          68.180                      55.5           37.940         29.246
#> 6          74.012                      55.8           48.164         38.498
#>   data.climate_daily_low_f data.daily_precip_in data.12z_precip_in
#> 1                     30.3           0.00000000                  0
#> 2                     30.6           0.00000000                  0
#> 3                     30.9           0.00000000                  0
#> 4                     31.3           0.00000000                  0
#> 5                     31.6           0.00000000                  0
#> 6                     31.9           0.03425197                  0
#>   data.climate_daily_precip_in
#> 1                         0.07
#> 2                         0.08
#> 3                         0.07
#> 4                         0.08
#> 5                         0.08
#> 6                         0.08