Statistical Inference/Probability Theory

Set Theory

Sample Space

The set, S, of all possible outcomes of a particular experiment is called the sample space for the experiment. [Definition 1.1.1]

Event

An event is any collection of possible outcomes of an experiment, that is, any subset of S (including S itself). [Definition 1.1.2]

Let A be an event, a subset of S. We say the event A occurs if the outcome of the experiment is in the set A. When speaking of probabilities, we generally speak of the probability of an event, rather than a set. But we may use the terms interchangeably.

Base relationships

We first need to define formally the following two relationships, which allow us to order and equate sets:

Containment:

<amsmath>A \subset B \Leftrightarrow x \in A \Rightarrow x \in B</amsmath> 

Equality:

<amsmath>A = B \Leftrightarrow A \subset B \land B \subset A</amsmath>

Elementary operations

Given any two events (or sets) A and B , we have the following elementary set operations:

Union: The union of A and B, written <amsmath>A \cup B</amsmath>, is the set of elements that belong to either A or B or both:

<amsmath>A \cup B = \{ x : x \in A \lor x \in B \}</amsmath>

Intersection: The intersection of A and B, written <amsmath>A \cap B</amsmath>, is the set of elements that belong to both A and B:

<amsmath>A \cap B = \{ x : x \in A \land x \in B \}</amsmath>

Complementation: The complement of A, written <amsmath>\overline{A}</amsmath>, is the set of all elements that are not in A:

<amsmath>\overline{A} = \{ x : x \notin A \}</amsmath>

Event operations

The elementary set operations can be combined: for any three events, A, B, and C, defined on a sample space S, the following relationships hold [Theorem 1.1.4].

Commutativity

• <amsmath>A \cup B = B \cup A</amsmath>
• <amsmath>A \cap B = B \cap A</amsmath>

Associativity

• <amsmath>A \cup (B \cup C) = (A \cup B) \cup C</amsmath>
• <amsmath>A \cap (B \cap C) = (A \cap B) \cap C</amsmath>

Distributive Laws

• <amsmath>A \cap (B \cup C) = (A \cap B) \cup (A \cap C)</amsmath>
• <amsmath>A \cup (B \cap C) = (A \cup B) \cap (A \cup C)</amsmath>

DeMorgan Laws

• <amsmath>\overline{A \cup B} = \overline{A} \cap \overline{B}</amsmath>
• <amsmath>\overline{A \cap B} = \overline{A} \cup \overline{B}</amsmath>

Disjoint events

Two events A and B are disjoint (or mutually exclusive) [Definition 1.1.5] if <amsmath>A \cap B = \emptyset</amsmath>.

The events A1, A2, ... are pairwise disjoint (or mutually exclusive) if <amsmath>A_i \cap A_j = \emptyset, \forall_{i\ne j}</amsmath>.

Disjoint sets are sets with no points in common. If we draw a Venn diagram for two disjoint sets, the sets do not overlap. The collection <amsmath>A_i = [i, i + 1), i = 0,1,2, \dots</amsmath> consists of pairwise disjoint sets. Note further that <amsmath>\cup^\infty_{i=0} A_i = [0, \infty)</amsmath>.

Event space partitions

If A1, A2,... are pairwise disjoint and <amsmath>\cup^\infty_{i=0} A_i = S</amsmath>, then the collection A1, A2, . . . forms a partition of S. [Definition 1.1.6]

The sets <amsmath>A_i = [i, i + 1)</amsmath> form a partition of <amsmath>[0, \infty)</amsmath>. In general, partitions are very useful, allowing us to divide the sample space into small, nonoverlapping pieces.

Basics of Probability Theory

For each event A in the sample space S we want to associate with A a number between zero and one that will be called the probability of A, denoted by P(A).

Sigma Algebra

A collection of subsets of S is called a sigma algebra (or Borel field) [Definition 1.2.1], denoted by <amsmath>\cal{B}</amsmath>, if it satisfies the following three properties:

• (a) <amsmath>\emptyset \in \cal{B}</amsmath> (the empty set is an element of <amsmath>\cal{B}</amsmath>).
• (b) if <amsmath>A \in \cal{B}</amsmath>, then <amsmath>\overline{A} \in \cal{B}</amsmath> (<amsmath>\cal{B}</amsmath> is closed under complementation).
• (c) if <amsmath>A_1, A_2, \dots \in \cal{B}</amsmath>, then <amsmath>\cup^\infty_{i=1} A_i \in \cal{B}</amsmath> (<amsmath>\cal{B}</amsmath> is closed under countable unions).

The empty set <amsmath>\emptyset</amsmath> is a subset of any set. Thus, <amsmath>\emptyset \subset S</amsmath>. Property (a) states that this subset is always in a sigma algebra. Since <amsmath>S = \overline{\emptyset}</amsmath>, properties (a) and (b) imply that S is always in <amsmath>\cal{B}</amsmath> also. In addition, from DeMorganªs Laws it follows that <amsmath>\cal{B}</amsmath> is closed under countable intersections. If <amsmath>A_1, A_2, \dots \in \cal{B}</amsmath>, then <amsmath>\overline{A_1}, \overline{A_2}, \dots \in \cal{B}</amsmath> by property (b), and therefore <amsmath>\cup^\infty_{i=1} \overline{A_i} \in \cal{B}</amsmath> . However, using DeMorgan's Law, we have <amsmath>\overline{\cup^\infty_{i=1} \overline{A_i}} = \cap^\infty_{i=1} A_i</amsmath>. Thus, again by property (b), <amsmath>\cap^\infty_{i=1} A_i \in \cal{B}</amsmath>.

Associated with sample space S we can have many different sigma algebras. For example, the collection of the two sets <amsmath>\{\emptyset, S\}</amsmath> is a sigma algebra, usually called the trivial sigma algebra. The only sigma algebra we will be concerned with is the smallest one that contains all of the open sets in & given sample space S .