How to Compute Standard Deviation in One Pass

Unbiased Estimator of Sample Standard Deviation

The formula for the sample standard deviation is

where

N is the number of data points in your sample. The first formula is known as the unbiased estimator of sample standard deviation. This is a technical detail but the (N-1) in the denominator makes the estimator unbiased. When N is large N vs. (N-1) does not matter but when N is small you should use (N-1).

Computing std in 1 pass

If someone asks you to write a simple program to compute the standard deviation and if you decide to use the formula shown above then you have to compute the average first then compute the standard deviation which means you have to make 2 passes over the data points.

You should know that there is an algebraic identity you can take advantage of to compute the standard deviation in 1 pass.

using this algebraic identity we can write the std as

Example Program in C#

The C# code below implements the last formula for the sample standard deviation

```using System;
using System.Collections.Generic;
using System.Text;
using System.Collections;

namespace example_std_stderr
{
class Program
{
static void Main(string[] args)
{
// we need a data set
// let’s populate an array with data points
// it does not matter what the distribution looks like
// replace this section with your own data retrieval code
int N = 100;
ArrayList al = new ArrayList();
for(int i=0; i<N; i++)
{
double val = Math.Exp(-(double)i/N);
}
double[] arr = (double[])al.ToArray(typeof(double));

// compute the sampling standard deviation in 1 pass
double sum1 = 0;
double sum2 = 0;
for(int i=0; i < arr.Length; i++)
{
sum1 += arr[i] * arr[i];
sum2 += arr[i];
}

double mean_val = sum2 / arr.Length;
double sample_variance =
(sum1 – arr.Length * mean_val * mean_val) / (arr.Length -1);

double sample_std_deviation = Math.Sqrt(sample_variance);
}
}
}```
{"found_math": false, "script_math_tex": 0, "script_math_asciimath": 0, "math_annotations": 0, "math_alttext": 0, "mathml": 0, "mathjax_tag": 0, "mathjax_inline_tex": 0, "mathjax_display_tex": 0, "mathjax_asciimath": 0, "img_math": 0, "codecogs_latex": 0, "wp_latex": 0, "mimetex.cgi": 0, "/images/math/codecogs": 0, "mathtex.cgi": 0, "katex": 0, "math-container": 0, "wp-katex-eq": 0, "align": 0, "equation": 0, "x-ck12": 0, "texerror": 0, "url": "https://sureshemre.wordpress.com/2012/04/21/how-to-compute-standard-deviation-in-one-pass/", "fetch_time": 1500913841000000000, "content_mime_type": "text/html", "warc_filename": "crawl-data/CC-MAIN-2017-30/segments/1500549424889.43/warc/CC-MAIN-20170724162257-20170724182257-00569.warc.gz", "warc_record_offset": 720517103, "warc_record_length": 38566, "token_count": 447, "char_count": 1914, "score": 3.921875, "int_score": 4, "crawl": "CC-MAIN-2017-30", "snapshot_type": "longest", "language": "en", "language_score": 0.7805337309837341, "file_path": "/sailhome/yjruan/project/latent-lm/data/processed/finemath-4plus/finemath-4plus/train-00014-of-00064.parquet"}