Testing out new indicators & its impact on your forecasted numbers

We’ve established previously that selecting the right leading indicators for your business is important (regardless of which stage you’re at in the supply chain.)
‍
[Learn more about how to identify your relevant leading, not lagging indicators.]

It can admittedly be quite a maze to pinpoint the relevant indicators. How do you know if you’re focusing on the relevant ones? Are there new indicators that you should include?

You might also be depending on the same indicators for a while now but would like to assess and test out the plausibility of new indicators. And crucially, its impact on your forecasted numbers.

Let’s break it down.

"You might also be depending on the same indicators
for a while now but would like to assess and test out the plausibility of new indicators. And crucially, its impact
on your forecasted numbers."

Common variable search methodologies & how they perform

We’ve found that many analysts do actively build and test out linear regressions models to identify the indicators relevant to their main variable. However, testing out the various methods typically takes quite a fair bit of time when running them using different search methodologies. 

To begin with, what are these search methodologies and what do they do best?

‍Coefficient search
‍Let’s start with using coefficient search. It includes three available methods; main equation only, all equations, and all equations - lagwise. 

All methods start by building a VAR model with LASSO penalty, using the struct selected in the variable selection settings.

This model will penalize variables that are not useful for prediction, meaning that their coefficients will shrink toward zero. Essentially, coefficient search examines all the selected variables and their individual contributions to the variable of interest. While it allows for a comprehensive examination of all factors, it may not automatically identify the most important ones.
 
‍Stepwise search
‍Stepwise search is an iterative process that starts with an initial set of factors, and then proceeds to add or remove variables based on statistical criteria (e.g., p-values or information criteria). It is aimed at providing measures that reflect how good an indicator will be out-of-sample. Essentially, it will reflect how well an indicator will perform in situations it has not been tested on before.

‍Backwards search: In a forecasting context, the model will never have access to the data points that we want to predict, posing a different problem than just describing the past movements of a variable.

While a complicated model will often fit the data well, it may suffer from overfitting, meaning that it will be very good at describing what has happened, but not what will happen. It is somewhat similar to the stepwise search, but starts with a model that includes all factors and then progressively removes the least significant ones based on statistical criteria. 

Backwards search is intended to produce values of how well an indicator is at predicting the main variable. It builds on the idea that while an indicator may be useful at describing the main variable in-sample, we are usually more interested in how well the indicator can strengthen the predictive power of a model. Simply comparing the efficacy of all three search methods without contextually fitting it would be an oversimplification and sorely inaccurate.

‍It would be worth trying out all three methods and determining the most suitable and effective approach in your particular context.

Here’s a quick guide on what to think about when testing out the three different methodologies:

Are the results displaying a low signal-to-noise ratio?
‍With a low signal-to-noise ratio in the data, it becomes more challenging to differentiate between the truly influential variables and the random fluctuations. It would mean that the meaningful information is overshadowed or obscured by random fluctuations or noise. 

Essentially, decreased forecast accuracy. 

If you identify that there is low signal-to-noise in the data using one method, that’s an indication that it might be worth trying out the next method.

Do the results vary quite a bit across the methods?
‍Separately run one or all three methods automatically and compare the forecasted results. If the results forecasted vary quite a bit across the three methods, it’s worth taking a look at which indicator(s) is still consistently prominent toward the main variable in your forecast.

This gives you a better sense of which variables (according to the methods tested) are worth considering moving forward.

Easily test all three methods in Indicio

Indicio includes all three methods you can easily switch between when performing an indicator analysis.

Choose the variables you think will be most relevant for forecasting the primary data point you have selected. Select your desired method (stepwise, coefficient or backwards search) and run the indicator analysis. Indicio will then calculate and rank each explanatory variable according to its causality to your main variable of interest, using partial correlation and LASSO regularisation to reduce noise and remove irrelevant coefficients. 

This gives you the ability to test more than 20 macroeconomic indicators in Indicio, all with the goal to test whether they have predictive power in forecasting your main variable, be it total sales or market demand.

Tests are then run based on a VAR model with Lasso effects that will then display which variables are insignificant and would not add predictive power.

No more guesswork. [Try it out for yourself.]

Benefits of testing out these methods

Gain robust insights on the variables that matter
‍Each method invariably has its own assumptions and criteria for selecting the relevant variables that you should pay attention to.

By using multiple methods, you can assess the consistency of the results. If a variable consistently appears as significant across different methods, it provides stronger evidence of its importance. Conversely, if a variable appears significant in one method but not in others, it may require further investigation.

Room to apply the method most appropriate for the context  
‍The value of each method may vary depending on the specific context and the nature of the variables being considered.

- By trying out all three methods, you can evaluate the most suitable and effective approach in your particular context. This allows for a more customized and context-specific analysis, increasing the likelihood of uncovering the most relevant variables for forecasting future sales.

- Trying out all three methods ensures that key variables are not overlooked and increases confidence in the findings. However, it is important to interpret and compare the results critically, considering the strengths and limitations of each method.

[Try it out in Indicio]