[autoscaler] Smart Autoscaling using time-series forecasting

We received an interesting suggestion to forecast build volume and automatically scale the number of runners accordingly, via the autoscaler. One example use case would be detecting patterns such as cron jobs that create bursts of builds.

There are a few open questions we would need to figure out

  1. how do we store and the time-series data? we would likely snapshot the server demand and timestamp every time the planner is executed. This information could be used to forecast
  2. can we efficiently store this data without introducing a time-series database? Can we efficiently store this data directly in the database? how much history should we store?
  3. will we need to smooth and denoise the data?
  4. what time-series forecasting algorithm should we use?

I need to put a good amount of thought into this, but wanted to get this documented and include in our 2019 roadmap.