<p>If I understood correctly the gap is in the middle of the data. In such cases you should not use forecasts that extrapolate the data, but some interpolation method.</p>
<p>If there is relatively large amount of variation in the data you would have best results using something like <a href="https://en.wikipedia.org/wiki/Cubic_Hermite_spline#Catmull%E2%80%93Rom_spline" rel="nofollow noreferrer">Catmull–Rom spline</a>. Catmull–Rom spline has some nice properties (see <a href="https://splines.readthedocs.io/en/latest/euclidean/catmull-rom-properties.html" rel="nofollow noreferrer">here</a>). The main advantage of the Catmull-Rom spline is that all real observations you have will become part of it and it allows you to estimate missing points in non-linear way.</p>
<p>If you work in R you can easily implement it with <a href="https://rdrr.io/github/jedalong/PathInterpolatR/man/splineCR.html" rel="nofollow noreferrer">splineCR</a>, if you work with Python you can have look at <a href="https://github.com/vmichals/python-algos/blob/master/catmull_rom_spline.py" rel="nofollow noreferrer">this github</a> code. Nowadays you will find it also part of programs such as EViews or Stata.</p>

CC BY-SA 4.0
2021-09-19T15:51:13.017Z
Yes, the gap is in the middle. Your proposalseems clearly superior to the forecast solution, but... Is there any way to use the information coming from the correlated series to improve the interpolation?
Khairon
CC BY-SA 4.0
2021-09-19T15:57:47.213Z
@Khairon the catmull-rom spline should do a decent job but if you want you can separately try to fit missing variables based on an auxiliary regression between variable of interest and that other highly correlated variable and see what gives you better results, although personally I think the solution above would be considered more elegant by people reviewing your work, replacing it by proxy * $\beta$ from the auxiliary regression would be crude because that is based on an average coeff. of the relationship over the whole sample rather than what the relationship was at the point of discont.
1muflon1
CC BY-SA 4.0
2021-09-20T09:03:22.243Z
(just a second, I am editing the comment)
Khairon
CC BY-SA 4.0
2021-09-20T09:16:27.583Z
True, it is the average coefficient, but if the series move in the same direction should not it be a good approximation?
If I don’t know the values of the gaps (and this is the problem in the first place) I cannot know which result is better.
I found another series in which there is no gap, the data is not identical (I found the first release data, while the gaps are in the revised data), but is in general quite near. Is I assume that their difference is in average zero, then the method with the auxiliary regression was actually more performing, i.e. nearer to the first release data
Khairon
CC BY-SA 4.0
2021-09-20T09:20:18.930Z
@Khairon well you can guess which one is better based on how fitted points behave, if you get flat line (like you would get with linear interpolation) but you see that your series has a lot of jumps then it is very likely the method was not appropriate. In any case you can use auxiliary reg if you think it works for you
1muflon1
CC BY-SA 4.0
2021-09-20T09:20:56.550Z
indicator: 
2016/12 -0.60 (revised data)
2017/01 1.76 (auxReg) -0.44 (interp) 1.9 (first rel)
2017/02 -2.35 (auxReg) 0.85 (interp) -1.1 (first rel)
2017/03 0.70 (revised data)

The auxiliary regression at least guesses the right direction, while interpolation does not. The problem is that normally I would not have the data for making this test, and therefore I cannot know which method is in general more performing.
Khairon
CC BY-SA 4.0
2021-09-20T09:30:15.197Z
in this case it is fine. But in general you would say that using interpolation is more performing, I would like to understand why.
My intuition would say that the more information you use, the more the estimate is accurate, therefore the combination of methods (interpolation + proxy) should be better to use one of the two. The only exception is that if the proxy is weak, then the added variance could not be worth to use the proxy. Why not?
In any case I am grateful for the suggestions you gave me so far, if not today it will be useful in in the future.
Khairon
CC BY-SA 4.0
2021-09-20T13:30:47.117Z
@Khairon yes because in general, if you have data issues it is hard to have good solid rigorous model to fit the missing observation, basing it only off a high level of correlation with other variable is not very rigorous because there could be some stochastic trends present, you don’t really know how stable that relationship is, also if you have excellent proxy then why not just use proxy to begin with that is always an option. Interpolation avoids these issues, but then in the end as long as you can justify what you are doing it’s up to you
1muflon1
CC BY-SA 4.0
2021-09-18T19:43:09.567Z
2021-09-18T19:43:09.567Z
1muflon1
<p>I have two time series, of different length.
A time series is GDP growth.</p>
<p>The gdp growth is the series I need, and it is also the longer series, but it has two gaps in two periods one after the other. The aggregate economic indicator is shorter, but it covers the period where I find the two gaps, and for the period in which both data are available, they are strictly correlated (r=0.94).</p>
<p>How could I fill these two gaps?</p>
<p>One possibility would be to use the autoregressive forecast. AR(1) describes quite well the series R²=0.91, and no significant autocorrelation in the residuals.
But I do not think it is the optimal solution, because:</p>
<ol>
<li>I have two gaps, the second gap will be filled with the two step ahead forecast, which is less precise,</li>
<li>I have data AFTER the point, and neglecting part of the information does not seem to be the best solution,</li>
<li>I have also some external information (the correlated series), which I could also exploit.</li>
</ol>
<p>Which method would be the most appropriate?</p>
<p>Thanks for the tips!</p>

<econometrics><applied-econometrics><statistics><time-series>
Filling gap in data with correlated series
Khairon
# Filling gap in data with correlated series

(Asked by: username_0 on 2021-09-18)

I have two time series, of different length.
A time series is GDP growth.

The gdp growth is the series I need, and it is also the longer series, but it has two gaps in two periods one after the other. The aggregate economic indicator is shorter, but it covers the period where I find the two gaps, and for the period in which both data are available, they are strictly correlated (r=0.94).

How could I fill these two gaps?

One possibility would be to use the autoregressive forecast. AR(1) describes quite well the series R²=0.91, and no significant autocorrelation in the residuals.
But I do not think it is the optimal solution, because:

1. I have two gaps, the second gap will be filled with the two step ahead forecast, which is less precise,
2. I have data AFTER the point, and neglecting part of the information does not seem to be the best solution,
3. I have also some external information (the correlated series), which I could also exploit.

Which method would be the most appropriate?

Thanks for the tips!

**Tags:** econometrics, applied-econometrics, statistics, time-series


---

## 1 answers

### Answer by username_1 on 2021-09-18. Score: 1 | Accepted answer

If I understood correctly the gap is in the middle of the data. In such cases you should not use forecasts that extrapolate the data, but some interpolation method.

If there is relatively large amount of variation in the data you would have best results using something like [Catmull–Rom spline](https://en.wikipedia.org/wiki/Cubic_Hermite_spline#Catmull%E2%80%93Rom_spline). Catmull–Rom spline has some nice properties (see [here](https://splines.readthedocs.io/en/latest/euclidean/catmull-rom-properties.html)). The main advantage of the Catmull-Rom spline is that all real observations you have will become part of it and it allows you to estimate missing points in non-linear way.

If you work in R you can easily implement it with [splineCR](https://rdrr.io/github/jedalong/PathInterpolatR/man/splineCR.html), if you work with Python you can have look at [this github](https://github.com/vmichals/python-algos/blob/master/catmull_rom_spline.py) code. Nowadays you will find it also part of programs such as EViews or Stata.

> **Comments:**
> **username_0**: Yes, the gap is in the middle. Your proposalseems clearly superior to the forecast solution, but... Is there any way to use the information coming from the correlated series to improve the interpolation?  
> **username_1**: @username_0 the catmull-rom spline should do a decent job but if you want you can separately try to fit missing variables based on an auxiliary regression between variable of interest and that other highly correlated variable and see what gives you better results, although personally I think the solution above would be considered more elegant by people reviewing your work, replacing it by proxy * $\beta$ from the auxiliary regression would be crude because that is based on an average coeff. of the relationship over the whole sample rather than what the relationship was at the point of discont.  
> **username_0**: (just a second, I am editing the comment)  
> **username_0**: True, it is the average coefficient, but if the series move in the same direction should not it be a good approximation? If I don’t know the values of the gaps (and this is the problem in the first place) I cannot know which result is better. I found another series in which there is no gap, the data is not identical (I found the first release data, while the gaps are in the revised data), but is in general quite near. Is I assume that their difference is in average zero, then the method with the auxiliary regression was actually more performing, i.e. nearer to the first release data  
> **username_1**: @username_0 well you can guess which one is better based on how fitted points behave, if you get flat line (like you would get with linear interpolation) but you see that your series has a lot of jumps then it is very likely the method was not appropriate. In any case you can use auxiliary reg if you think it works for you  
> **username_0**: indicator:  2016/12 -0.60 (revised data) 2017/01 1.76 (auxReg) -0.44 (interp) 1.9 (first rel) 2017/02 -2.35 (auxReg) 0.85 (interp) -1.1 (first rel) 2017/03 0.70 (revised data)  The auxiliary regression at least guesses the right direction, while interpolation does not. The problem is that normally I would not have the data for making this test, and therefore I cannot know which method is in general more performing.  
> **username_0**: in this case it is fine. But in general you would say that using interpolation is more performing, I would like to understand why. My intuition would say that the more information you use, the more the estimate is accurate, therefore the combination of methods (interpolation + proxy) should be better to use one of the two. The only exception is that if the proxy is weak, then the added variance could not be worth to use the proxy. Why not? In any case I am grateful for the suggestions you gave me so far, if not today it will be useful in in the future.  
> **username_1**: @username_0 yes because in general, if you have data issues it is hard to have good solid rigorous model to fit the missing observation, basing it only off a high level of correlation with other variable is not very rigorous because there could be some stochastic trends present, you don’t really know how stable that relationship is, also if you have excellent proxy then why not just use proxy to begin with that is always an option. Interpolation avoids these issues, but then in the end as long as you can justify what you are doing it’s up to you  

---

