Time series forecasting plays an important role in various fields, such as energy, finance, transport, and weather. Temporal convolutional networks (TCNs) based on dilated causal convolution have been widely used in time series forecasting. However, two problems weaken the performance of TCNs. One is that in dilated casual convolution, causal convolution leads to the receptive fields of outputs being concentrated in the earlier part of the input sequence, whereas the recent input information will be severely lost. The other is that the distribution shift problem in time series has not been adequately solved. To address the first problem, we propose a subsequence-based dilated convolution method (SDC). By using multiple convolutional filters to convolve elements of neighboring subsequences, the method extracts temporal features from a growing receptive field via a growing subsequence rather than a single element. Ultimately, the receptive field of each output element can cover the whole input sequence. To address the second problem, we propose a difference and compensation method (DCM). The method reduces the discrepancies between and within the input sequences by difference operations and then compensates the outputs for the information lost due to difference operations. Based on SDC and DCM, we further construct a temporal subsequence-based convolutional network with difference (TSCND) for time series forecasting. The experimental results show that TSCND can reduce prediction mean squared error by 7.3% and save runtime, compared with state-of-the-art models and vanilla TCN.

Inferring the future state of data based on past information [

As a common type of deep learning model for time series modeling, temporal convolutional networks (TCNs) have been widely used in current research on time series forecasting. When dealing with long sequences, TCNs are not as prone to gradient disappearance problems as recurrent neural networks (RNNs). Compared with transformers, TCNs have advantages in memory consumption [

Dilated causal convolution plays an important role in TCNs [

However, for time series forecasting, it is unnecessary to employ causal convolution to prevent future information leakage into the past, as the input sequence is solely past information compared to the predicted sequence. Worse still, causal convolution leads to the following problems. On the one hand, in dilated causal convolution, the earlier an input element is located, the greater its effect on the output sequences. For example, in

In addition, the problem of distribution shift exists widely in time series, which significantly reduces the performance of forecasting models. The distribution shift problem is that the statistical properties (such as the mean) of the time series may change over time, which may lead to the distribution shift between training and test data. For example, in

The contributions of this paper are as follows:

To mitigate information loss in dilated convolution on long sequences, we propose a novel subsequence-based convolution method (SDC). The method extracts temporal features from a receptive field via a growing subsequence, and the subsequence has a richer representation than a single element.

To break the limitation of dilated causal convolution on the receptive field, we use multiple convolution filters to generate elements that share a receptive field in SDC without causal convolution. As the elements of the shared receptive field increase, eventually, all output elements will be able to look back at the entire input sequence.

To alleviate the distribution shift in time series, we propose a difference and compensation method (DCM) to reduce the discrepancies between and within input sequences by difference operations. As shown in

Based on SDC and DCM, we further construct a temporal subsequence-based convolutional network with difference (TSCND) for time series forecasting. Experimentally, compared with state-of-the-art methods and vanilla TCN, TSCND can reduce prediction mean squared error by 7.3% and save runtime. The results of the ablation experiments also demonstrate the effectiveness of SDC and DCM for time series forecasting.

Current time series forecasting methods can be divided into traditional statistics-based methods and deep learning-based methods.

Traditional statistics-based methods, such as the autoregressive integrated moving average (ARIMA) [

Deep learning can automatically learn and model the hidden features of complex series based on raw data and can achieve better forecasting accuracy on complex time series datasets. Therefore, more research is now based on deep learning methods.

Recurrent neural networks (RNNs) [

In recent years, transformer-based models [

TCNs are also popular for time series forecasting tasks [

Dilated causal convolution can capture the long-term dependencies of the time series, but the causal convolution structure limits the receptive field and results in severe information loss during dilated convolution. Therefore, we propose a new dilated convolution method to replace dilated causal convolution in TCNs.

Forecasting models often suffer badly from distribution shift in time series. Domain adaptation [

The overall architecture of our proposed model is shown in

DCM is used to address the problem of distribution shift. It contains a difference stage and a compensation stage.

In the difference stage, a difference sequence is obtained through differencing adjacent elements in an input sequence. For an input sequence

In the compensation stage, to compensate for the information loss caused by difference operation, the last element value of the original sequence is added back to the output. For the output sequence

To ensure that the sequence length can meet the requirements of the SDC, we pad the difference sequence

The process of padding and embedding the sequence

Due to the limitation of dilated causal convolution on the receptive field, we propose the SDC to be used instead of dilated causal convolution. Our SDC has the following two similarities with dilated causal convolution:

At the SDC layer, an input or output sequence is divided into several subsequences. For multi-layer SDC, the initial subsequence length is 1, and the length will increase with the number of SDC layers by SDC operation. Each SDC layer uses

Specifically, at SDC Layer

The detailed process of the SDC operation for convolving subsequences is as follows: firstly, the elements in the neighboring subsequences are convolved using several different filters, and elements in the same subsequence are not convolved with each other. Then the output elements generated by convolving the same elements but with different convolution filters are adjacent to each other, and the elements generated by the same subsequences are merged into a new subsequence. E.g., in

We first formalize SDC operations from the perspective of a single output element. The SDC operation on the

Next, based on the SDC operation for a single element, we extend to the SDC operation for the sequence. The SDC operation for the input sequence

For a SDC Layer

To enhance the comprehensibility of the SDC layer, we utilize pseudo-code to illustrate the process in Algorithm 1.

The Decoding module includes 2 fully-connected layers. The one maps

To evaluate our model’s performance, we conducted univariate time series forecasting on four popular real-world datasets ETTh1, ETTh2, ECL, and WTH.

To verify the superiority of our proposed method, we compared it with four SOTA models: PatchTST, FEDformer, Autoformer, and Informer. In addition, our proposed method was also compared with the classic models TCN and LSTM.

We used the following evaluation metrics:

(1) Mean squared error (MSE):

(2) Mean absolute error (MAE):

For all the compared models, the same parameter settings were used for the training process, with predicted sequence lengths of 24, 48, 168, 336, and 720. The models were optimized using the adaptive moment estimation (Adam) optimizer with learning rates starting at 1e-3. The total number of epochs is 8 with proper early stopping. We used Mean Squared Error (MSE) as our loss function. The inputs of each dataset were zero-mean normalized. Following the previous related work [

For better performance of TSCND, we set the convolutional filter size

Method | Ours | PatchTST | FEDformer | Autoformer | Informer | LSTM | TCN | |
---|---|---|---|---|---|---|---|---|

Metric | MSE | MSE | MSE | MSE | MSE | MSE | MSE | |

ETTh1 | 24 | 0.046 | 0.064 | 0.098 | 0.065 | 0.044 | ||

48 | 0.066 | 0.091 | 0.158 | 0.120 | 0.061 | |||

168 | 0.100 | 0.118 | 0.183 | 0.249 | 0.086 | |||

336 | 0.125 | 0.119 | 0.222 | 0.244 | 0.131 | |||

720 | 0.162 | 0.123 | 0.269 | 0.266 | 0.195 | |||

ETTh2 | 24 | 0.108 | 0.104 | 0.093 | 0.147 | 0.090 | ||

48 | 0.130 | 0.140 | 0.155 | 0.182 | 0.119 | |||

168 | 0.183 | 0.182 | 0.232 | 0.276 | 0.223 | |||

336 | 0.206 | 0.268 | 0.263 | 0.300 | 0.268 | |||

720 | 0.305 | 0.351 | 0.277 | 0.355 | 0.312 | |||

ECL | 24 | 0.371 | 0.366 | 0.246 | 0.774 | 0.182 | ||

48 | 0.350 | 0.450 | 0.285 | 0.909 | 0.213 | |||

168 | 0.295 | 0.644 | 0.373 | 0.908 | 0.287 | |||

336 | 0.421 | 0.703 | 0.416 | 0.955 | 0.314 | |||

720 | 0.502 | 0.453 | 0.677 | 0.408 | 0.995 | |||

WTH | 24 | 0.254 | 0.143 | 0.116 | 0.150 | 0.094 | ||

48 | 0.140 | 0.257 | 0.181 | 0.203 | 0.196 | |||

168 | 0.233 | 0.307 | 0.273 | 0.284 | 0.272 | |||

336 | 0.306 | 0.336 | 0.320 | 0.331 | 0.315 | |||

720 | 0.398 | 0.371 | 0.404 | 0.353 | 0.405 |

Note: A lower MSE indicates a better prediction. The best results are highlighted in bold and the next best result is highlighted with an underline.

Method | TSCND | PatchTST | FEDformer | Autoformer | Informer | LSTM | TCN | |
---|---|---|---|---|---|---|---|---|

Metric | MAE | MAE | MAE | MAE | MAE | MAE | MAE | |

ETTh1 | 24 | 0.162 | 0.204 | 0.247 | 0.205 | 0.163 | ||

48 | 0.191 | 0.237 | 0.319 | 0.288 | 0.190 | |||

168 | 0.246 | 0.267 | 0.346 | 0.431 | 0.225 | |||

336 | 0.284 | 0.271 | 0.387 | 0.424 | 0.286 | |||

720 | 0.319 | 0.278 | 0.435 | 0.449 | 0.364 | |||

ETTh2 | 24 | 0.251 | 0.255 | 0.240 | 0.307 | 0.235 | ||

48 | 0.279 | 0.291 | 0.314 | 0.342 | 0.270 | |||

168 | 0.335 | 0.332 | 0.389 | 0.424 | 0.384 | |||

336 | 0.361 | 0.403 | 0.417 | 0.442 | 0.423 | |||

720 | 0.443 | 0.472 | 0.431 | 0.484 | 0.450 | |||

ECL | 24 | 0.452 | 0.472 | 0.363 | 0.726 | 0.317 | ||

48 | 0.447 | 0.504 | 0.382 | 0.786 | 0.341 | |||

168 | 0.407 | 0.624 | 0.442 | 0.773 | 0.387 | |||

336 | 0.493 | 0.618 | 0.481 | 0.788 | 0.407 | |||

720 | 0.514 | 0.513 | 0.646 | 0.480 | 0.818 | |||

WTH | 24 | 0.386 | 0.278 | 0.255 | 0.291 | 0.212 | ||

48 | 0.386 | 0.315 | 0.338 | 0.329 | ||||

168 | 0.354 | 0.440 | 0.390 | 0.416 | 0.391 | |||

336 | 0.412 | 0.449 | 0.419 | 0.457 | 0.420 | |||

720 | 0.475 | 0.475 | 0.468 | 0.464 | 0.495 |

Note: A lower MAE indicates a better prediction. The best results are highlighted in bold and the next best result is highlighted with an underline.

Compared to TCN, our method brings improvement in all cases on ETTh1, ETTh2, and ECL datasets, and brings improvement in 7/10 cases on WHT datasets. It achieved better results than TCN, with an average MSE reduction of 19.4%. In the short-term forecasting of WTH, the results of TSCND and TCN are similar for the following reasons. The time series of the WTH dataset is relatively stationary, thus the DCM method is of little help to the model. And when the prediction length is short, this advantage of SDC using subsequences instead of elements to extract features is not obvious, so TSCND and TCN perform similarly. However, when predicting long-term time series, TSCND is superior to TCN.

To evaluate the efficiency of our proposed model in handling inputs of different lengths, we chose PatchTST and TCN for comparison. We compared the time required for training each model for 1 epoch on the ETTh2 dataset under different input sequence lengths. The results are shown in

Input length | Ours | PatchTST | TCN |
---|---|---|---|

24 | 5.23 s | 6.28 s | |

48 | 5.70 s | 6.30 s | |

168 | 6.26 s | 6.41 s | |

336 | 7.09 s | 7.13 s | |

720 | 7.18 s | 7.61 s |

To evaluate the proposed method for addressing the distribution shift problem, we conducted experiments on the ETTh1 dataset using our proposed method, the Revin method (Revin) [

Predicted length | Origin | Ours | Revin [ |
SubLast [ |
---|---|---|---|---|

24 | 0.044 | 0.029 | 0.028 | |

48 | 0.067 | 0.041 | ||

168 | 0.110 | 0.073 | 0.087 | |

336 | 0.131 | 0.095 | 0.089 | |

720 | 0.311 | 0.111 | 0.097 |

Predicted length | Ours | Revin | SubLast |
---|---|---|---|

168 | 3.31 ms | ||

336 | 15.62 ms | ||

720 | 21.45 ms |

To test whether a subsequence can capture more temporal dependencies than a single element, we tested three different structures of the Decoder Layer for making predictions using SDC output.

We conducted experiments using above structures on the ECL dataset, and the experimental results are shown in

This paper proposes a temporal subsequence-based convolutional network with difference for time series forecasting, with two effective modules: (i) SDC, which extracts information from a receptive field via a subsequence rather than a single element, and multiple convolutional filters are used at each layer to enable the elements in the same subsequence to share a receptive field. These designs allow the receptive fields of all final output elements to cover the entire input sequence, thus reducing the loss of information in the dilated convolution. (ii) DCM, which can effectively solve the problem of distribution shift. The method reduces the discrepancies between and within input sequences through difference operations. By restoring the original input information to the outputs, the method can compensate for the loss of information due to difference operations.

We conducted experiments on commonly used time series datasets. Firstly, TSCND outperforms the SOTA method on most of the MAE and MSE metrics, which indicates that TSCND is effective in both short-term and long-term time series forecasting. Secondly, the experiments on training time demonstrate the advantage of TSCND in efficiency. Finally, ablation experiments show that DCM is useful in mitigating the distribution shift problem and the use of subsequences instead of single elements in the SDC method can enhance the feature extraction capability.

In future work, we will further study from the following directions: Firstly, the convolution filter size and hidden layer dimension of the proposed model are important hyperparameters. They are currently set manually relying on experience, and it would be valuable to study the automatic selection of these hyperparameters. Then we will explore combining the proposed SDC with research on heterogeneous information systems for multivariate time series forecasting. Finally, we will consider the possibility of the SDC as an alternative to dilated causal convolution for time series classification and anomaly detection.

We thank the members of the MOE Research Center of Software/Hardware Co-Design Engineering for their contributions to this work.

This work was supported by the National Key Research and Development Program of China (No. 2018YFB2101300), the National Natural Science Foundation of China (Grant No. 61871186), and the Dean’s Fund of Engineering Research Center of Software/Hardware Co-Design Technology and Application, Ministry of Education (East China Normal University).

The authors confirm contribution to the paper as follows: study conception and design: Haoran Huang and Weiting Chen; data collection: Haoran Huang; analysis and interpretation of results: Haoran Huang and Weiting Chen; draft manuscript preparation: Haoran Huang, Weiting Chen, and Zheming Fan. All authors reviewed the results and approved the final version of the manuscript.

All datasets that support the findings of this study are openly available. The ETT dataset is available at

The authors declare that they have no conflicts of interest to report regarding the present study.