Frost has been identified as one of the leading causes of green seed in Western Canada. As the effects of climate change threaten to destabilize Albertan climate conditions, the weather-dependent exigencies of the canola industry can be remedied with long-term weather prediction models. The 2019 Calgary iGEM team has established such a weather prediction model through the use of a novel machine learning algorithm, the principal component neural network mean model (PNMM), which utilizes principal component analysis (PCA) and recurrent neural network (RNN) machine learning methodologies in tandem with traditional weather prediction factors. The PNMM was able to predict minimum daily temperature for the 2018 growing season with a mean absolute error of 2.101°C. Therefore, the PNMM can be utilized to predict inclement weather with a degree of accuracy for any location with sufficient historical weather data, allowing Albertan farmers to optimize their agricultural decision-making and minimize the occurrence of green seed.

Alberta’s climate is becoming increasingly volatile due to the effects of global warming. The high variability in contemporary weather patterns has turned the management and planning of canola crops into a risky and complicated affair (Deser et al. 2012). This is due to the inability of modern temperature prediction methods to accurately predict frost events. Generally, current methods of long-term weather prediction, such as with Environment and Climate Change Canada’s (ECCC’s) long-term probabilistic temperature and precipitation forecast maps (ECCC, 2019), give only probabilities of temperatures being below, at, or above normal temperatures. The lack of daily resolution in temperature detail lowers the usefulness of these models for use in agricultural contexts. Additionally, the application of predictions over a large area, such as with ECCC’s predictions and Old Farmer Almanac’s predictions, leads to reduced accuracy in comparison to localized predictions (Old Farmer’s Almanac, 2019).

To combat this problem, we utilized our experience in machine learning with MATLAB and TensorFlow to create a novel machine learning algorithm, dubbed the principal component neural network mean model (PNMM). The model is able to predict daily minimum temperatures for localized areas with a high level of accuracy. To generate an example use case, we randomly selected the southern Albertan town of Vulcan to train and test our model on. To train our model, we used multiple years worth of weather data from climate measurement stations around Vulcan.

By predicting daily minimum temperatures, our weather modelling methodology can be used to forecast inclement weather. Therefore, the knowledge gained from this model can aid farmers in planning their seed choice, crop planting, and harvest timing.

• The average data of the five weather stations examined that surround Vulcan, Alberta is representative of the weather of Vulcan, Alberta.
• Missing weather data that was interpolated is accurate to real values.

• TODO

Input weather data was collected using the Alberta Agriculture and Forestry’s Alberta Climate Information Service (ACIS). Special thanks to ACIS for allowing the use of their data in our research. The dataset involved weather data from 5 different weather stations surrounding the town of Vulcan, Alberta. These weather stations were Blackie AGCM, Champion AGDM, Mossleigh AGCM, Queenstown, and Travers AGCM, where most measurements besides those attributed to precipitation were collected two metres above the ground. The data parameters that were obtained were comprised of the following: average relative humidity (%), average air temperature (°C), maximum air temperature (°C), accumulated precipitation (mm), minimum relative humidity (%), maximum relative humidity (%), minimum air temperature (°C), precipitation (mm), and average wind speed (km/h) (ACIS 2019).

The target weather data were labeled as the daily minimum temperature values of the five examined weather stations 153 days into the future from input dates. The training data of the PCA model used weather data from the dates of June 13, 2012 to April 30, 2018. The labelled testing data was selected as the weather data from the dates of May 1, 2018, to September 30, 2018.

Figure 1: Location Map of the five examined weather stations (Blackie ACGM, Champion AGDM, Mossleigh AGCM, Queenstown, and Travers AGCM) around Vulcan, Alberta, Canada (ACIS, 2019).

• The value of the local minimum in error reached by training in the model is or is equal to the global minimum in error.
• Extrema of weather data outside of the training data do not excessively differ in magnitude from the extrema of the training weather data, ensuring the validity of using min-max normalization.

Recurrent neural networks (RNNs) are ANNs (click hereto read more) where the hidden layer nodes feed back into the neural network. This architecture allows the neural network to develop internal memory states, allowing it to exploit time series data to inform the predictions of the RNN.

However, typical RNNs suffer from the “vanishing gradient problem,” as described by Hochreiter, Bengio, Frasconi, and Schmidhuber (2001). The “vanishing gradient problem” describes the issue where the rate of change of a model’s parameters during training decreases to negligible values, hence “vanishing.”

Deeply layered models, such as unrolled RNNs with long-term memory, are highly prone to the “vanishing gradient problem.” Therefore, many RNNs are limited in the temporal memory they can possess, which can lower their capacity for learning. In order to nullify this difficulty, the team needed to explore more advanced solutions.

Therefore, our team implemented gated recurrent units (GRUs), as introduced by Cho et al. (2014). By replacing the standard nodes of an RNN with GRUs, which utilize update and reset gates, the neural network was able to maintain a long-term memory of temporal data to inform the decisions of the neural network.

The architecture of a GRU can be described as:

For an iteration \(t\), where \(x_{t}\) is the input vector, \(W\) and \(U\) are matrices of parameters, \(b\) is a parameter vector, \(g\) is the sigmoid activation function, and \(h\) is the hyperbolic tangent activation function, the general formulas for the update gate, reset gate, and output vectors are given as:

$$ z_{t}=\sigma _{g}(W_{z}x_{t}+U_{z}h_{t-1}+b_{z}), $$ $$ r_{t}=\sigma _{g}(W_{r}x_{t}+U_{r}h_{t-1}+b_{r}), and $$ $$ h_{t}=(1-z_{t})\circ h_{t-1}+z_{t}\circ\sigma _{h}(W_{h}x_{t}+U_{h}(r_{t}\circ h_{t-1})+b_{h})), respectively $$

A final neural network architecture of the form 21-256-256-256-5 was utilized by the team, with 3 consecutive GRU units in the hidden layers of the RNN, and a final output layer of temperature predictions for each examined weather station.

The input data for the RNN model were collected by using Environment and Climate Change Canada’s (ECCC) Historical Data tool. The data used were comprised of hourly weather data from the dates of December 24, 2012, to September 30, 2018, for the five examined weather stations. Aside from the wind speed data fields for Mossleigh AGCM and Travers AGCM, the data fields of each weather station in the input data were: air temperature (°C), dew point temperature (°C), relative humidity (%), and wind speed (km/h). However, some data was missing at certain times, requiring those data points to be interpolated. The interpolation method used was hot deck imputation.

The time interval of the collected training and validation weather data for the RNN was May 26, 2013, to April 30, 2018. Additionally, the test data was selected as the weather data from May 1, 2018, to September 30, 2018.

Data preprocessing is an indispensable initial step for neural network training. In order to enable compatibility between the input data and the RNN, input data was converted into numeric feature columns, which was especially relevant for providing date input information.

Additionally, min-max scaling was used to scale all input values to be in or almost in the range of [0,1]. Data normalization of this type was utilized to increase the efficiency of the training of the neural network, as numerically large inputs can disproportionately influence node values in early stages of training. The general formula for this type of scaling is given as:

$$x_{normalized}=\frac{x-min(v)}{max(v)-min(v)},$$

where \(x_{normalized}\) is the normalized value of \(x\), where \(x\) is a value of a data field \(v\). \(max(v)\) and \(min(v)\) are the maximum and minimum values of \(v\), respectively.

Training data was selected as the least recent 80% of the original preprocessed input data. The remaining most recent 20% of the original preprocessed input data was taken as the validation data. In neural network training, validation data is not used for training, but is instead used to evaluate the neural network’s ability to generalize to new weather data.

Additionally, the weather model training used a batch generator that produced each batch of data used in the training as a set of 64 continuous time series of 6 weeks of input data each, which were randomly sampled with replacement from the training data. These were all bundled with their corresponding labels. The neural network was trained to minimize the mean squared error (MSE) of its predicted temperature values versus the true temperature values for the training data, while ignoring a warm up period of the first 2 days (48 hours) of predictions. This period was ignored due to those early predictions lacking enough time series information to provide stable predictions.

In order to prevent overfitting, techniques like dropout and early stopping were utilized to preserve the ability for the model to generalize. Dropout is a technique whereby some nodes in the neural network are disabled during training, preventing them from memorizing the training data. The model used dropout rates of 0.1, 0.15, and 0.2 for the consecutive GRU layers in the RNN, respectively, corresponding to 25, 38, and 51 (randomly selected) disabled nodes during each training update.

Each epoch of the training involved only 10 learning steps. The training encompassed 53 epochs, using early stopping with a patience of 6 epochs, meaning that the final neural network model was selected once the previous 6 epochs resulted in no improvement on the best neural network of the previous 6 epochs. The early stopping technique was used in order to prevent the validation error from increasing due to model overfitting while training.

Additionally, the optimizer used in the RNN training was the Adam optimizer, as introduced by Kingma and Ba (2014). Adam was chosen due to its efficient performance when applied to problems with “non-stationary objectives and problems with very noisy and/or sparse gradients” (Kingma and Ba. 2014).

Once the training of the neural network concluded, the RNN weather model could then be used to forecast weather when given new preprocessed input data of the same dimensionality as the training data.

In order to help alleviate individual shortcomings of the RNN and PCA models, we devised a hybrid component model comprised of the two aforementioned models and the historical normal temperature of Vulcan, collected from ACIS (2018). The reason temperature normals were incorporated into this model is because of the integral role they play in most temperature models; ECCC and ACIS temperature predictions are relative to temperature models (ECCC, 2018) (ACIS, 2018).

The designation given to this combined model was: principal component neural network mean model (PNMM). The benefits of hybrid models lie in the use of multiple approaches to the same problem, increasing the confidence of classification when the component models corroborate each other. Conversely, when the underlying models’ predictions diverge, hybrid models normalize its component models’ variance spikes across time series.

Mathematically, we can represent the PNMM thusly:

PNMM Prediction = \(D(t)\), the PNMM Prediction Function

PCA Prediction = \(A(t|\Omega _{\alpha})\), the PCA Prediction Function supported by \(\Omega _{\alpha}\)

RNN Prediction = \(B(t|\Omega _{\beta})\), the RNN Prediction Function supported by \(\Omega _{\beta}\)

Normal Prediction = \(C(t|\Omega _{\alpha})\), the Temperature Normal Prediction Function supported by \(\Omega _{\alpha}\)

\(\Omega\) = Information Space(\(\alpha\) for daily data, \(\beta\) for hourly data)

\(m\) = (Number of Component Models) \(- 1\)

$$ D(t)=\frac{\sum_{k=0}^{m}[k(A(t|\Omega _{\alpha}))+(m-k)(B(t|\Omega _{\beta}))+(m-1)(C(t|\Omega _{\alpha}))]}{(m+1)^2} $$

Due to the nature of hybrid models, the performance of the PNMM is never the worst model when compared to its component models for specific time intervals. Additionally, when considered across a long enough time scale, the reliability and robustness of the PNMM typically results in it outperforming its individual component models.

In order to evaluate the performance of the models, minimum temperature predictions for each day of the hindcasted time period of May 1, 2018 to September 30, 2018 were compiled into a time series and compared to the actual daily minimum temperatures for those dates. The results of this comparison are displayed in Figure 2.