In this post I showed you how to calculate mean incrementally (i.e. incorporate new sample values to an existing mean variable).

There is a slightly different way of “approaching” (pun intended) the mean. Every time you get a new sample, steer your mean a little closer to the value of this new sample! Simple as that!

The algorithm in pseudocode (python like):

mean = 0 # or some initial guess
for each new sample encountered:
    mean += rate * (new sample - mean)

new sample - mean gives you the direction and magnitude the mean needs to move in order to be one with the sample, but you don’t wanna move it right on the sample, you wanna move a little towards the sample, so you multiply this by a rate which is 0-1. A rate of 0.5 would mean you move halfway along the “mean-sample delta” (the vector that takes you from your mean to your sample).

As your number of samples approaches infinity, and as long as you decay your rate properly, you will approach the true mean! If you do not decay your rate, you might get stuck jumping around the true mean value!

This is kinda cool! It’s like saying that as long as you are taking little steps in the right direction (in response to sample values), you will approach the expected value eventually! It’s inevitable! This is a gritty (as opposed to greedy :grin:) algorithm!