## Overview

**Runtime Minimization.** One of the most important properties of a program is
the time it takes to execute. One goal as a programmer is to minimize the time
(in seconds) that a program takes to complete.

**Runtime Measurement.** Some natural techniques:

- Measure the number of seconds that a program takes to complete using a stopwatch (either physical or in software). This tells you the actual runtime, but is dependent on the machine and inputs.
- Count the number of operations needed for inputs of a given size. This is a machine independent analysis, but still depends on the input, and also doesn’t actually tell you how long the code takes to run.
- Derive an algebraic expression relating the number of operations to the size of an input. This tells you how the algorithm scales, but does not tell you how long the code takes to run.

**Algorithm Scaling.** While we ultimately care about the runtime of an
algorithm in seconds, we’ll often say that one algorithm is better than another
simply because of how it scales. By scaling, we mean how the runtime of a piece
of code grows as a function of its input size. For example, inserting at the
beginning of ArrayList on an old computer might take $R(N) = 0.0001N$ seconds,
where $N$ is the size of the list.

For example, if the runtime of two algorithms is $R_1(N) = N^2$, and $R_2(N) = 5000 + N$, we’d say algorithm 2 is better, even though R1 is much faster for small N.

A rough justification for this argument is that performance critical situations are exactly those for which N is “large”, though this is not an obvious fact. In almost all cases we’d prefer the linear algorithm. In some limited real-world situations like matrix multiplication, one might select one algorithm for small N, and another algorithm for large N. We won’t do this in 61B.

**Simplfying Algebraic Runtime.** We utilize several simplifications to make
runtime analysis simpler.

- Pick an arbitrary option to be our
*cost model*, e.g. # of array accesses. - Ignore small inputs, e.g. treat $2N+1$ just like $2N$.
- Ignore constant scaling factor, e.g. treat $2N$ just like $N$.

As an example, if we have an algorithm that performs $2N + 1$ increment operations and $4N^2 + 2N + 6$ compares, our intuitive simplifications will lead us to say that this algorithm has a runtime proportional to $N^2$.

The cost model is simply an operation that we’re picking to represent the entire piece of code. Make sure to pick an appropriate cost model! If we had chosen the number of increment operations as our cost model, we’d mistakenly determine that the runtime was proportional to $N$. This is incorrect since for large N, the comparisons will vastly outnumber the increments.

**Big Theta.** To formalize our intuitive simplifications, we introduce
Big-Theta notation. We say that a function $R(N) \in \Theta(f(N))$ if there
exists positive constants $k_1$ and $k_2$ such that $k_1 f_1(N) \leq
R(N) \leq k_2f_2(N)$.

Many authors write $R(N) = \Theta(f(N))$ instead of $R(N) \in \Theta(f(N))$. You may use either notation as you please. I will them interchangeably.

An alternate non-standard definition is that $R(N) \in \Theta(f(N))$ iff the $\lim_{N\to\infty} \frac{R(N)}{f(N)} = k$, where $k$ is some positive constant. We will not use this calculus based definition in class. I haven’t thought carefully about this alternate definition, so it might be slightly incorrect due to some calculus subtleties.

When using $\Theta$ to capture a function’s asymptotic scaling, we avoid unnecessary terms in our $\Theta$ expression. For example, while $4N^2 + 3N + 6 \in \Theta(4N^2 + 3N)$, we will usually make the simpler claim that is $4N^2

- 3N + 6 \in \Theta(N^2)$.

**Order of Growth.** If a function $R(N) \in \Theta(f(N))$, we saw the the
order of growth is $f(N)$. For example $4N^2 + 3N + 6 \in \Theta(N^2)$, so
we say its order of growth is $N^2$.

The terms “constant”, “linear”, and “quadratic” are often used for algorithms with order of growth $1$, $N$, and $N^2$, respectively. For example, we might say that an algorithm with runtime $4N^2 + 3N + 6$ is quadratic.

**Big O.** Big O notation is similar to BigTheta. However, instead of bounding
from below and above, big O only bounds from above.

In other words $R(N) \in O(f(N))$ iff there exists a positive constant $k_2$ such that $R(N) \leq k_2 f_2(N)$.

In terms of limits, $R(N) \in O(f(N))$ if $\lim_{N\to\infty} \frac{R(N)}{f(N)} = k$, where $0 \leq k \leq 1$.

You can think of $\Theta$ sort of like $=$ for orders of growth, and big O sort of like $\leq$.

For example the following facts are true:

- $N + N^2 \in O(N^{500})$, runtime is $\leq$ 500th power
- $N + N^2 \in O(N^2)$, runtime is $\leq$ quadratic
- $N + N^2 \in \Theta(N^2)$, runtime is $=$ quadratic

**Best Case vs. Always vs. Worst Case.** One particularly vexing topic is what
happens when a piece of code has a runtime that depends on the nature of its
input. For example, consider a `containsZero`

function that checks an array to
see if it has zeros. The runtime of this algorithm for arrays seems like it
should just be $\Theta(N)$, meaning that the runtime should grow linearly with
the array size.

However, this isn’t quite correct. This code has order of growth equal to $N$ only in the worst case. Suppose that we test this algorithm by feeding it increasingly large arrays that are all zeros: The runtime function we measure will be completely flat, since it will always find a zero at the front of the array. In the special case of an all-zeros array, the runtime is $\Theta(1)$.

Here, the problem isn’t with $\Theta$ notation, but rather the fact that asking
for the runtime of `containsZero`

is an imprecise question.

It’s somewhat like asking “If you have 1000 dollars in savings, how much will you have next year?”. There is no answer. If it’s invested in the stock market, it will grow (or decline) exponentially. If it is invested in the space under your mattress, it will be constant.

**Big Theta in the Worst Case vs. Big O.** In the real world, it is very common
for people to use $O$ notation when they really mean to say “$\Theta$ in the
Worst Case”. For example, the `containsZero`

function above is $\Theta(N)$ in
the worst case, but most posts on the internet would say it is simply $O(N)$
(with no explicit mention of the phrase ‘worst case’).

Both statements are correct! However, “$Theta(N)$ in the worst case” is a slightly stronger statement.

The difference is identical to the one between the English sentences “Every member of my family is less than or equal to 100 years old” vs. “The oldest member of my family is 100 years old.” The latter statement is a stronger statement. That is, any family for which the latter statement applies is also a family for which the former statement applies (but not necessarily the other way around).

This is not particular important, but since we’re students at arguably the best CS program in the country, it seems like a thing worth mentioning given the ubiquity of $O$ notation.

Please forgive your TAs if they are unfamiliar with this distinction. It is a subtle difference that often goes overlooked.

## Recommended Problems

For more code analysis problems, See lec19 guide instead.

### B level

- Suppose we have a function
`bleepBlorp`

, and its runtime $R(N)$ has order of growth $\Theta(N^2)$. Which of the following can we say?- $R(N) \in \Theta(N^2)$ true, this is what order of growth means!
- $R(N) \in \Theta(N^2)$ for any inputs true, this statement is exactly the same as the one above
- $R(N) \in \Theta(N^2)$ for worst case inputs true, since also true for ANY input
- For large N, if we run bleepBlorp on an input of size $N$, and an input of size $10N$, we will have to wait roughly 100 times as long for the larger input. true, this is the nature of quadratics
- If we run bleepBlorp on an input of size 1000, and an input of size 10000, we will have to wait roughly 100 times as long for the larger input. false, 1000 may not be a large enough N to exhibit quadratic behavior

- Suppose we have a function called
`binarySearch`

, and that its runtime is $O(N^2)$. Which of the following can we say?- For large N, if we run binarySearch on an input of size $N$, and an input of size $10N$, we will have to wait roughly 100 times as long for the larger input.false, big O just says we'll wait no longer than roughly 100 times as long
- For large N, if we run binarySearch on a worst case input of size $N$, and a worst case input of size $10N$, we will have to wait roughly 100 times as long for the larger input.false, big O just says we'll wait no longer than roughly 100 times as long
- For large N, if we run binarySearch on a worst case input of size $N$, and a worst case input of size $10N$, we won’t have to wait longer than roughly 100 times as long for the larger input.

- The worst case runtime of
`binarySearch`

is $\Theta(N^2)$.

- The worst case runtime of
`binarySearch`

is not $\Theta(N^2)$. false, same reason as above

### A level

- Suppose we find that an algorithm has a runtime R(N) that is quadratic in the worst case, and linear in the best case. Which of the
following statements are true? Highlight the text after each bullet point for the answer.
- $R(N) \in O(N)$ false, runtime may be worse
- $R(N) \in O(N^2)$ true, never worse than quadratic
- $R(N) \in O(N^3)$ true, never worse than cubic
- $R(N) \in \Theta(N)$ false, order of growth of runtime varies
- $R(N) \in \Theta(N^2)$ false, order of growth of runtime varies
- $R(N) \in \Theta(N^3)$ false, order of growth of runtime varies
- In the worst case, $R(N) \in O(N)$ false, quadratic is not less than or equal to linear
- In the worst case, $R(N) \in O(N^2)$ true, quadratic is less than or equal to quadratic
- In the worst case, $R(N) \in O(N^3)$ true, quadratic is less than or equal to cubic
- In the worst case, $R(N) \in \Theta(N)$ false, quadratic is not equal to linear
- In the worst case, $R(N) \in \Theta(N^2)$ true, quadratic is equal to quadratic
- In the worst case, $R(N) \in \Theta(N^3)$ false, quadratic is not equal to cubic
- In the best case, $R(N) \in O(N)$ true, linear is less than or equal to linear
- In the best case, $R(N) \in O(N^2)$ true, linear is less than or equal to quadratic
- In the best case, $R(N) \in O(N^3)$ true, linear is less than or equal to cubic
- In the best case, $R(N) \in \Theta(N)$ true, linear is equal to linear
- In the best case, $R(N) \in \Theta(N^2)$ false, linear is not equal to quadratic
- In the best case, $R(N) \in \Theta(N^3)$ false, linear is not equal to cubic