Statistical Inference/Probability Theory

Set Theory

Sample Space

The set, S, of all possible outcomes of a particular experiment is called the sample space for the experiment. [Definition 1.1.1]

Event

An event is any collection of possible outcomes of an experiment, that is, any subset of S (including S itself). [Definition 1.1.2]

Let A be an event, a subset of S. We say the event A occurs if the outcome of the experiment is in the set A. When speaking of probabilities, we generally speak of the probability of an event, rather than a set. But we may use the terms interchangeably.

Base relationships

We first need to define formally the following two relationships, which allow us to order and equate sets:

Containment: $A\subset B\Leftrightarrow x\in A\Rightarrow x\in B$

Equality: $A=B\Leftrightarrow A\subset B\land B\subset A$

Elementary operations

Given any two events (or sets) A and B , we have the following elementary set operations:

Union: The union of A and B, written $A\cup B$, is the set of elements that belong to either A or B or both:

$A\cup B=\{x:x\in A\lor x\in B\}$

Intersection: The intersection of A and B, written $A\cap B$, is the set of elements that belong to both A and B:

$A\cap B=\{x:x\in A\land x\in B\}$

Complementation: The complement of A, written ${\overline {A}}$, is the set of all elements that are not in A:

${\overline {A}}=\{x:x\notin A\}$

Event operations

The elementary set operations can be combined: for any three events, A, B, and C, defined on a sample space S, the following relationships hold [Theorem 1.1.4].

Commutativity

• $A\cup B=B\cup A$
• $A\cap B=B\cap A$

Associativity

• $A\cup (B\cup C)=(A\cup B)\cup C$
• $A\cap (B\cap C)=(A\cap B)\cap C$

Distributive Laws

• $A\cap (B\cup C)=(A\cap B)\cup (A\cap C)$
• $A\cup (B\cap C)=(A\cup B)\cap (A\cup C)$

DeMorgan Laws

• ${\overline {A\cup B}}={\overline {A}}\cap {\overline {B}}$
• ${\overline {A\cap B}}={\overline {A}}\cup {\overline {B}}$

Disjoint events

Two events A and B are disjoint (or mutually exclusive) [Definition 1.1.5] if $A\cap B=\emptyset$.

The events A1, A2, ... are pairwise disjoint (or mutually exclusive) if $A_{i}\cap A_{j}=\emptyset ,\forall _{i\neq j}$.

Disjoint sets are sets with no points in common. If we draw a Venn diagram for two disjoint sets, the sets do not overlap. The collection $A_{i}=[i,i+1),i=0,1,2,\dots$ consists of pairwise disjoint sets. Note further that $\cup _{i=0}^{\infty }A_{i}=[0,\infty )$.

Event space partitions

If A1, A2,... are pairwise disjoint and $\cup _{i=0}^{\infty }A_{i}=S$, then the collection A1, A2, . . . forms a partition of S. [Definition 1.1.6]

The sets $A_{i}=[i,i+1)$ form a partition of $[0,\infty )$. In general, partitions are very useful, allowing us to divide the sample space into small, non-overlapping pieces.

Basics of Probability Theory

Axiomatic foundations

For each event A in the sample space S we want to associate with A a number between zero and one that will be called the probability of A, denoted by P(A).

Sigma Algebra

A collection of subsets of S is called a sigma algebra (or Borel field) [Definition 1.2.1], denoted by ${\cal {B}}$, if it satisfies the following three properties:

• (a) $\emptyset \in {\cal {B}}$ (the empty set is an element of ${\cal {B}}$).
• (b) if $A\in {\cal {B}}$, then ${\overline {A}}\in {\cal {B}}$ (${\cal {B}}$ is closed under complementation).
• (c) if $A_{1},A_{2},\dots \in {\cal {B}}$, then $\cup _{i=1}^{\infty }A_{i}\in {\cal {B}}$ (${\cal {B}}$ is closed under countable unions).

The empty set $\emptyset$ is a subset of any set. Thus, $\emptyset \subset S$. Property (a) states that this subset is always in a sigma algebra. Since $S={\overline {\emptyset }}$, properties (a) and (b) imply that S is always in ${\cal {B}}$ also. In addition, from DeMorganªs Laws it follows that ${\cal {B}}$ is closed under countable intersections. If $A_{1},A_{2},\dots \in {\cal {B}}$, then ${\overline {A_{1}}},{\overline {A_{2}}},\dots \in {\cal {B}}$ by property (b), and therefore $\cup _{i=1}^{\infty }{\overline {A_{i}}}\in {\cal {B}}$ . However, using DeMorgan's Law, we have ${\overline {\cup _{i=1}^{\infty }{\overline {A_{i}}}}}=\cap _{i=1}^{\infty }A_{i}$. Thus, again by property (b), $\cap _{i=1}^{\infty }A_{i}\in {\cal {B}}$.

Associated with sample space S we can have many different sigma algebras. For example, the collection of the two sets $\{\emptyset ,S\}$ is a sigma algebra, usually called the trivial sigma algebra. The only sigma algebra we will be concerned with is the smallest one that contains all of the open sets in a given sample space S .

Probability Function

Given a sample space S and an associated sigma algebra ${\cal {B}}$, a probability function [Definition 1.2.4] is a function P with domain ${\cal {B}}$ that satisfies

• 1. $P(A)\geq 0,\forall _{A\in {\cal {B}}}$.
• 2. $P(S)=1$.
• 3. If $A_{1},A_{2},\dots \in {\cal {B}}$ are pairwise disjoint, then $P(\cup _{i=1}^{\infty }A_{i})=\Sigma _{i=1}^{\infty }P(A_{i})$. [Axiom of Countable Additivity]

These three properties are usually referred to as the Axioms of Probability (or the Kolmogorov Axioms, after A. Kolmogorov, one of the fathers of probability theory). Any function P that satisfies the Axioms of Probability is called a probability function. The axiomatic definition makes no attempt to tell what particular function P to choose; it merely requires P to satisfy the axioms. For any sample space many different probability functions can be defined. Which one(s) reflects what is likely to be observed in a particular experiment is still to be discussed.

We need general methods of defining probability functions that we know will always satisfy Kolmogorov's Axioms. We do not want to have to check the Axioms for each new probability function. The following gives a common method of defining a legitimate probability function.

Defining Probability Functions

Let $S=\{s_{1},\dots ,s_{n}\}$ be a finite set. Let ${\cal {B}}$ be any sigma algebra of subsets of S. Let $p_{1},\dots ,p_{n}$ be nonnegative numbers that sum to 1. For any $A\in {\cal {B}}$, define $P(A)=\Sigma _{\{i:s_{i}\in A\}}p_{i}$ (The sum over an empty set is defined to be 0.) Then P is a probability function on ${\cal {B}}$. This remains true if $S=\{s_{1},s_{2},\dots \}$ is a countable set [Theorem 1.2.6].

The physical reality of the experiment might dictate the probability assígnment.

The Calculus of Probabilities

Properties of the probability function applied to a single event

If P is a. probability function and A is any set in ${\cal {B}}$, then [Theorem 1.2.8]

• (a) $P(\emptyset )=0$, where $\emptyset$ is the empty set;
• (b) $P(A)\leq 1$;
• (c) $P({\overline {A}})=1-P(A)$.

Properties of the probability function applied to any set pairs

If P is a probability function and A and B are any sets in ${\cal {B}}$, then [Theorem 1.2.9]

• (a) $P(B\cap {\overline {A}})=P(B)-P(A\cap B)$;
• (b) $P(A\cup B)=P(A)+P(B)-P(A\cap B)$;
• (c) If $A\subset B$, then $P(A)\leq P(B)$.

Bonferroni's Inequality

Formula (b) of Theorem 1.2.9 gives a useful inequality for the probability of an intersection. Since $P(A\cup B)\leq 1$, we have, after some rearranging, $P(A\cap B)\geq P(A)+P(B)-1$.

This inequality is a special case of what is known as Bonferroni's Inequality. Bonferroni's Inequality allows us to bound the probability of a simultaneous event (the intersectíon) in terms of the probabilities of the individual events.

Properties of the probability function applied to collections of sets

If P is a probability function, then [Theorem 1.2.11]

• (a) $P(A)=\Sigma _{i=1}^{\infty }P(A\cap C_{i})$, for any partition $C_{1},C_{2},\dots$;
• (b) $P(\cup _{i=1}^{\infty }A_{i})\leq \Sigma _{i=1}^{\infty }P(A_{i})$, for any sets $A_{1},A_{2},\dots$ (Boole's Inequality)

There is a similarity between Boole's Inequality and Bonferroni's Inequality. In fact, they are essentially the same thing. We could have used Boole's Inequality to derive that $P(A\cap B)\geq P(A)+P(B)-1$ (the special case of Bonferroni's Inequality, presented above). If we apply Boole*s Inequality to ${\overline {A}}$, we have $P(\cup _{i=1}^{n}{\overline {A_{i}}})\leq \Sigma _{i=1}^{n}P({\overline {A_{i}}})$ and using the facts that $\cup {\overline {A_{i}}}={\overline {\cap A_{i}}}$ and $P({\overline {A_{i}}})=1-P(A_{i})$, we obtain $1-P(\cap _{i=1}^{n}A_{i})\leq n-\Sigma _{i=1}^{n}P(A_{i})$

This becomes, on rearranging terms, $P(\cap _{i=1}^{n}A_{i})\geq \Sigma _{i=1}^{n}P(A_{i})-(n-1)$, which is a more general version of the Bonferroni Inequality presented before.

Counting

Most often, methods of counting are used in order to construct probability assignments on finite sample spaces, although they can be used to answer other questions also.

Counting problems, in general, sound complicated, and often we must do our counting subject to many restrictions. The way to solve such problems is to break them down into a series of simple tasks that are easy to count, and employ known rules of combining tasks. The following theorem is a first step in such a process and is sometimes known as the Fundamental Theorem of Counting.

Fundamental Theorem of Counting

If a job consists of $k$ separate tasks, the $i$-th of which can be done in $n_{i}$ ways, $i=1,\dots ,k$, then the entire job can be done in $n_{1}\times n_{2}\times \dots \times n_{k}$ ways.

Possible Methods of Counting

Counting can be made with without replacement. Also, the order in which the task is performed may matter. Four types counting scenarios are thus possible: ordered or unordered, both of which may be done with or without replacement.

Factorial

For a positive integer n, n! (read n factorial) is the product of all of the positive integers less than or equal to n [Definition 1.2.16]. That is, $n!=n\times (n-1)\times (n-2)\times \dots \times 3\times 2\times 1$. Furthermore, we define 0! = 1.