## Apparent puzzles with goodness-of-fit tests

by

Given two data sets it is a not unreasonable question to ask if they have similar distributions.

For example, if we produce one data set of 500 numbers between 0 and 1, chosen uniformly and randomly (at least as randomly as we can using a pseudo-random number generator), and another data set of 500 numbers distributed normally, with mean 0 and variance 1, then eyeballing their histograms tells us , even if we did not know, that they are not similarly distributed:

A more considered way of approaching the question of whether two data sets are similarly distributed is to utilize one or more goodness of fit tests. There are several of these in common use, including:

Mathematica® incorporates these, and other goodness-of-fit tests in the function DistributionFitTest[]

These goodness-of-fit tests basically perform a hypothesis test with the null hypothesis $H_0$ being that the  data sets are identically  distributed,  and an alternative hypothesis $H_a$ that they are not.

The goodness-of-fit tests return a p-value, and a small p-value indicates it is unlikely the data sets are similarly distributed.

So, if we carry out the Pearson Chi-Square test on the uniform and normal data sets, as above, we get the exceptionally small p-value $5.32085\times10^{-85}$ indicating, very strongly, that the two data sets are not similarly distributed.

The p-value from a Pearson Chi-Square test is a random variable: if we carry out the test on two other data sets, one from a uniform distribution, the other from a normal distribution, we will get a somewhat different p-value. How different? The plot below shows the variation in p-values when we simulated choosing 500 uniformly distributed numbers and 500 normally distributed numbers 1,000 times:

We see that despite some variation the values are all very small, as we would expect.

Now let’s see what happens if we choose  500 points from a uniform distribution, and 500 more points from the same uniform distribution. We expect the Pearson Chi-Square test to return a reasonably high p-value, indicating that we cannot reject the idea that the data come from the same distribution.  We did this once and got a satisfying 0.741581 as the p-value.

But what if we repeat this experiment 1,000 times. How will the p-values vary?

The plot below shows the result of 1,000 simulations of choosing two data sets of 500 points, each from the same uniformly distribution:

These p-values seem reasonably uniformly spread between 0 and 1. Are they? The Cramér-von Mises goodness-of-fit test indicates that we cannot reject the hypothesis that these p-values are uniformly distributed in the interval [0,1].

We set the confidence level for the Pearson Chi-Square test at 0.01, so we could expect that 1 time in 100 the Pearson Chi-Square test will indicate that the two data sets are not from the same distribution, even though they are. In 1,000 trials we could expect about 10 such instances, and that is more or less what we find.

The uniform distribution of the p-values is, at first glance, quite surprising, but since the p-values themselves are random values we expect that they will indicate something other than what we know to be the case every so often, dependent on the confidence level we set beforehand. For example, with the confidence level set at 0.05, we see that about 5% of the time the Pearson Chi-Square test indicates that the two data sets are not from the same distribution even though they are. :

Notes:

1. We randomly reset the seed for the pseudo-random number generator in Mathematica® at each of the 1,000 simulations.
2. The result of uniformly distributed p-values for data sets from the same distribution  is not peculiar to the Pearson Chi-Square test.
3. The uniform distribution of p-values under the null hypothesis is proved here.

## Excel statistical functions: AVERAGE

by

Excel has many statistical functions. Here is a description of one of the most commonly used: AVERAGE

## Questions about some infinite series and necklaces of partitions of labeled beads

by

We learn early in a study of infinite series that the geometric series $S=\sum_{k=1}^{\infty}1/2^k$ sums to 1.

Sometimes you will see this sort of reasoning:

$S=1/2+1/4+1/8+ \ldots$ so $2S=1+(1+1/2+1/4+\ldots =1+S$ so $S=1$

which is somewhat suspect in light of Euler’s “argument”:

$1+2+4+8+16+\ldots=1+2(1+2+4+8+\ldots)$ so $1+2+4+8+16+\ldots = -1$.

We need first to know that $S=\sum_{k=1}^{\infty}1/2^k$ converges absolutely, which we can do, for example by a use of the ratio test:

the $(k+1)^{\textrm{th}}$ term of $S=\sum_{k=1}^{\infty}1/2^k$ divided by the $k^{\textrm{th}}$ term is 1/2 <1.

Another application of the ratio test shows that the series $\sum_{k=1}^{\infty}k/2^k$ is absolutely convergent:

the $(k+1)^{\textrm{th}}$ of this series is $(k+1)/2^{k+1}$, the $k^{\textrm{th}}$ term is $k/2^k$ and their ratio is $\frac{(k+1)}{2^{k+1}}\frac{2^k}{k}=\frac{1}{2}(1+1/k)$ which approaches $1/2$ as $k \to \infty$.

If we denote $T=\sum_{k=1}^{\infty}k/2^k$ then a simple, and legitimate, calculation shows that $2T-T=1+1/2+1/4+1/8+\ldots$which we  know to be $1+1=2$.

In fact, for any natural number $p$ the ratio test shows that the series $S(p):=\sum_{k=1}^{\infty}k^p/2^k$ is absolutely convergent.

A Mathematica® calculation:

TableForm[Table[{p, Limit[Sum[k^p/2^k, {k, 1, n}], n -> Infinity]}, {p, 0, 10}], TableHeadings -> {None, {“p”, “S(p)”}}]

yields the following results for $S(p)$:

The first curious thing is that the results are all whole numbers. Why is that?

The second curious thing is that if we enter this sequence of whole numbers into the The On-Line Encyclopedia of Integer Sequences we get a match:

these whole numbers match the number of necklaces of partitions of p+1 labeled beads. They also match the sequence of cumulants of the probability distribution of the number of tails before the first head in a sequence of fair coin tosses. Is that right? If so, why? $p=2$ is the first really interesting case.

A detailed account of these and similar series is dealt with by Mircea Cîrnu “Determinantal formulas for sum of generalized arithmetic-geometric series” Boletn de la Asociacion Matematica Venezolana, Vol. XVIII, No. 1 (2011), 13-25.

Enjoy! It’s connections like these that give mathematicians a buzz.

## The CATs of mathematics

November 2, 2013

TranslationMathematics has numerous cats. For instance, there’s: Catalan numbers, which appear all over the place in counting situations. The Catalan constant Category theory The Catenary curve The Catenoid surface A Catalan surface Catastrophe theory But here’s a Cat that will never appear in the Encyclopedia of Mathematics: The totally pissed Cat:  Yeah, I’m looking at […]

## Why Jared and Brittany will never be good at math.2

September 19, 2013

TranslationJared was pleased how easy algebra was. He thought it might be hard, but he was finding it much easier than he had imagined. “Hey, Brittany, look at this,” he said. “How simple is this?” Brittany looked at Jared’s work on the problem of calculating how far the point is from the origin of the […]

## Why Jared and Brittany will never be good at math.1

September 15, 2013

TranslationJared “simplified” in algebra class. His math teacher asked how he got that. “Easy!” said Jared. “Just cancel the x’s.” Jared was pretty sure he was becoming skilled at canceling. “But what if you put into ?” the teacher asked Jared. “The result would be 2, not 1.” Brittany jumped into the conversation: “You can’t do […]

## When is an integral not an integral?

September 14, 2013

Translation No surprise to anyone really that students get confused by the difference between definite and indefinite integrals. The so-called indefinite integral is not really an integral at all, not in the sense of area: it’s the solution set to a differential equation. It’s not even usually a single function at all, but a whole […]

## The Rime of the Data Scientist

September 13, 2013

TranslationThe Rime of the Data Scientist (with apologies to Samuel Taylor Coleridge) Part I It is a Data Scientist, And he stoppeth one of three. `By thy Python code and glittering eye, Now wherefore stopp’st thou me?   The classroom doors are opened wide, And I am next one in; The others are met, the […]

## The leading (base 10) digit of an integer

June 25, 2013

TranslationSurely the leading (= left-most) digit of a positive integer is an obvious thing? Just stare at the integer (e.g. 7823) and observe the left-most digit (7, in this example)? Suppose, however, that you wanted to find the leading digit of a very large list of positive integers, a list so large it was hard […]