Forecasting is an everyday activity in today’s financial organizations. Estimating quarter-end revenue figures, balancing the books for incoming and outgoing cash flow, and forecasting operational expenditures are necessary to enable the best possible estimate of a given proposal’s outcome. These activities can be time and resource-intensive when using traditional methodologies due to the specific domain knowledge required and the volume of necessary forecasts. By using forecast automation and machine learning techniques in the forecasting process, organizations can reduce the time and effort needed to produce a batch of forecasts at month-end or quarter-close activities.

Current forecast challenges 

In today’s forecasting landscape, many companies still rely on the knowledge and expertise of subject matter experts in their domain to also generate the forecasts for their specific areas of business. Generating these forecasts typically require the forecaster to consolidate several data sources (sometimes from different systems), review historical values and forecast performance, and evaluate current business trends and market indicators to generate an informed forecast for the next period. Much of this work is time-consuming and repetitive, which takes away from other more valuable work the subject matter expert could be doing with their time.

Forecast automation 

One technique for assisting subject matter experts in their forecast cycle is through the application of forecast automation. One example of this could be collecting and consolidating different data sources used as input for the subject matter expert. Instead of manually refreshing or downloading data sources from various spreadsheets, we can use automation pipelines to pull the relevant information periodically, aggregate it to meet the forecaster’s needs, and publish the data to a central location readily available to those who need it.

Automating this process has additional benefits beyond simply reducing the time it takes to compile the data sources:  

• Validation and auditing steps can reduce potential errors from forgetting to refresh a particular data source or pulling the data before it’s ready in the source system.  
• Subject matter experts can more frequently evaluate their forecasts and the source data throughout the forecast period as they no longer need to compile their data manually. 
• Employees can more easily transfer domain knowledge by making the datasets available to all who work in a particular area and retain more knowledge when employees leave or move to different departments 
• Additional data sources can be incorporated and made available to everyone, not just the forecaster who thought to include that data source 

Other examples of forecast automation include collection and aggregation of forecasts generated by subject matter experts into a centralized repository, reporting and evaluating current and historical forecast accuracy, auditing source data systems for discrepancies or anomalies, and many more.

Introducing machine learning 

While forecasting automation can assist with gathering and consolidating the required materials for a forecaster to more efficiently produce a forecast, organizations can incorporate machine learning (ML) techniques into the process to help with (or even fully automate) the generation of the forecast numbers themselves.

Machine learning models can leverage multiple data sources, other forecasts (such as market indicators or consumer indices), and categorical features into the training process to produce a robust forecast product. Subject matter experts can work with data scientists to help identify critical data points to include to leverage existing domain expertise in the models. Depending on performance and business needs, the output from these models can either be taken as-is directly into the reporting process or used as a base point for the forecasters to evaluate and adjust as necessary.

Machine learning forecasting differs from traditional statistical forecasting algorithms like ARIMA (Autoregressive Integrated Moving Average) and ETS (error, trend, seasonal) in a few ways:

• Historical actual values vs. Features: Traditional methods can generate a forecast using only the historical actual values. They rely on the computation of statistical information (such as trends, moving averages, autoregressive coefficients) about the historical time series to generate future forecast values. ML models need features to relate back to the historical actuals. These features can be similar to the statistical information from the traditional methods but provide the flexibility to include a large variety of different types of information. 
• Variety of variables: Machine learning methods can incorporate a wider variety of additional variables into the model, such as categorical or text-based features (ML models can even use the output from ARIMA or ETS as a feature!) 
• Time series forecasting: Traditional methods are designed for time series forecasting, while organizations need to adapt machine learning approaches to achieve time series forecasting. In particular, machine learning models need to ensure cross-validation happens in a time-aware manner to avoid “information leakage” in the training process. 

Traditional time series methods and machine learning techniques can be leveraged in an automated forecasting process to quickly produce a forecast for direct consumption or review and adjustment by a subject matter expert.

Due to the efficiency of generating automated forecasts, organizations could make forecasts at a much higher frequency (weekly instead of quarterly) and less initial effort. Forecasting more frequently allows the subject matter expert to focus more on specific problem areas, update their forecasts more regularly, and better track their forecast accuracy.

Conclusion 

Producing regular forecasts in any domain can be manual and time-consuming. By introducing process automation into the forecasting workflow, we can reduce the time required to refresh and assemble the data points needed for the forecaster to produce a projection. By incorporating machine learning techniques into the forecasting process, we can further reduce the time and effort required to produce a preliminary forecast. Automating the data pipelines and forecast generation allows subject matter experts to focus more time on areas that need additional scrutiny or are particularly tricky to forecast. 

Interested in bringing forecast automation and machine learning into your business? Contact us!