Mathematical foundations in machine learning Day 1

Introduction to Big O Notation

Big O notation is used to describe the asymptotic behavior of an algorithm's time or space complexity. It provides a high-level understanding of how the performance of an algorithm changes with the size of the input \( n \). Big O focuses on the worst-case scenario and disregards constant factors, allowing comparisons between different algorithms in terms of growth rate.

1. Definition

Formally, an algorithm's time complexity is \( O(f(n)) \) if there exist constants \( c > 0 \) and \( n_0 \geq 1 \) such that:

\[
T(n) \leq c \cdot f(n) \quad \text{for all} \quad n \geq n_0
\]

Here:
- \( T(n) \) is the actual time (or steps) taken by the algorithm for input size \( n \),
- \( f(n) \) is a function representing the upper bound of \( T(n) \),
- \( c \) is a constant multiplier, and
- \( n_0 \) is the threshold after which the growth is dominated by \( f(n) \).

2. Common Big O Notations

- Constant Time: \( O(1) \)
  
Accessing an element in an array by index takes the same amount of time regardless of array size.

  \[
  T(n) = c \quad \Rightarrow \quad O(1)
  \]

- Logarithmic Time: \( O(\log n) \)
  
Binary search algorithm halves the search space at every step.

  \[
  T(n) = c \cdot \log n \quad \Rightarrow \quad O(\log n)
  \]

- Linear Time: \( O(n) \)
  
Iterating through all elements of an array.

  \[
  T(n) = c \cdot n \quad \Rightarrow \quad O(n)
  \]

- Linearithmic Time: \( O(n \log n) \)
  
Merge sort algorithm splits the array and merges it in sorted order.

  \[
  T(n) = c \cdot n \log n \quad \Rightarrow \quad O(n \log n)
  \]

- Quadratic Time: \( O(n^2) \)
  
Nested loops that each iterate through the array.

  \[
  T(n) = c \cdot n^2 \quad \Rightarrow \quad O(n^2)
  \]

- Exponential Time: \( O(2^n) \)
  
Solving the traveling salesman problem using brute force.

  \[
  T(n) = c \cdot 2^n \quad \Rightarrow \quad O(2^n)
  \]

3. Big O Simplification

Big O disregards lower-order terms and constant factors because they become insignificant as \( n \) grows. For example:

If the time complexity is \( T(n) = 3n^2 + 5n + 7 \), we simplify it to \( O(n^2) \) since \( n^2 \) dominates as \( n \to \infty \).

\[
\lim_{n \to \infty} \frac{T(n)}{n^2} = \lim_{n \to \infty} \frac{3n^2 + 5n + 7}{n^2} = 3
\]

Thus, the complexity is \( O(n^2) \).

4. Comparison of Growth Rates

The growth rates of common time complexities can be compared as follows:

\[
O(1) < O(\log n) < O(n) < O(n \log n) < O(n^2) < O(2^n) < O(n!)
\]

These are arranged in increasing order of their asymptotic growth. For large \( n \), algorithms with lower time complexities will be significantly more efficient.

5. Graphical Representation

For better intuition, consider a graph plotting these complexities for increasing \( n \):

\[
\text{y-axis: Time complexity}, \quad \text{x-axis: Input size } n
\]

On this graph, \( O(1) \) will be flat, \( O(\log n) \) will grow slowly, while \( O(n!) \) will explode exponentially. This visual aid highlights the differences in performance for larger inputs.

6. Worst-Case, Best-Case, and Average-Case

Big O typically refers to worst-case complexity, but we can also define:

- Best-case: The smallest possible number of operations (denoted by \( \Omega(f(n)) \)).
- Average-case: The expected number of operations over all possible inputs (denoted by \( \Theta(f(n)) \)).

For example, quicksort has a worst-case complexity of \( O(n^2) \) but an average-case complexity of \( O(n \log n) \).

7. Mathematical Derivation of Big O

For a function \( T(n) \), determining Big O often involves limiting behavior. For example:

If \( T(n) = 2n^3 + 4n^2 \), we consider the dominant term:

\[
\lim_{n \to \infty} \frac{T(n)}{n^3} = 2
\]

Thus, \( T(n) = O(n^3) \).

Big O notation provides a mathematical way to describe the efficiency of an algorithm by focusing on the dominant factors that influence runtime or space usage as input size grows. It is a critical tool for comparing algorithms and making performance-driven decisions in algorithm design.

Introduction to Limit Derivatives and Derivation Methods

The concept of a derivative is foundational in calculus, representing the rate of change of a function as its input changes. Derivatives are defined using limits, which formalize the idea of instantaneous change. This discussion will cover both the limit definition of a derivative and common derivation techniques.

1. Limit Definition of the Derivative

The derivative of a function \( f(x) \) at a point \( x = a \) is defined as the limit of the difference quotient:

\[
f'(a) = \lim_{h \to 0} \frac{f(a+h) - f(a)}{h}
\]

Here, \( h \) represents a small increment in \( x \), and the quotient measures the average rate of change of \( f(x) \) over the interval \( [a, a+h] \). As \( h \to 0 \), this becomes the instantaneous rate of change at \( x = a \).

Geometric Interpretation

- The derivative at \( x = a \) represents the slope of the tangent line to the curve \( y = f(x) \) at \( (a, f(a)) \).
  

2. One-Sided Derivatives

Sometimes, we are interested in the behavior of a function from only one side. The right-hand derivative and left-hand derivative are defined as:

- Right-hand derivative:
  
  \[
  f'_+(a) = \lim_{h \to 0^+} \frac{f(a+h) - f(a)}{h}
  \]

- Left-hand derivative:
  
  \[
  f'_-(a) = \lim_{h \to 0^-} \frac{f(a+h) - f(a)}{h}
  \]

If \( f'_+(a) = f'_-(a) \), the derivative exists at \( x = a \). If these limits are different, the function has a corner or a cusp at \( x = a \), and the derivative does not exist.

3. Derivative Notation

Derivatives are denoted in various ways, depending on the context:

- \( f'(x) \): Lagrange's notation.
- \( \frac{dy}{dx} \): Leibniz's notation, useful for expressing derivatives in terms of differentials.
- \( Df(x) \): Operator notation, emphasizing the derivative as a function itself.

4. Common Derivation Methods

(a) Power Rule

For any polynomial function \( f(x) = x^n \), the derivative is given by:

\[
\frac{d}{dx} \left( x^n \right) = n x^{n-1}
\]

Example:

\[
\frac{d}{dx} \left( x^3 \right) = 3x^2
\]

(b) Sum and Difference Rules

For functions \( f(x) \) and \( g(x) \), the derivative of their sum or difference is:

\[
\frac{d}{dx} \left( f(x) + g(x) \right) = f'(x) + g'(x)
\]
\[
\frac{d}{dx} \left( f(x) - g(x) \right) = f'(x) - g'(x)
\]

Example:

\[
\frac{d}{dx} \left( x^2 + 3x \right) = 2x + 3
\]

(c) Product Rule

For two functions \( f(x) \) and \( g(x) \), the derivative of their product is:

\[
\frac{d}{dx} \left( f(x) \cdot g(x) \right) = f'(x) \cdot g(x) + f(x) \cdot g'(x)
\]

Example:

\[
\frac{d}{dx} \left( x^2 \cdot \sin(x) \right) = 2x \cdot \sin(x) + x^2 \cdot \cos(x)
\]

(d) Quotient Rule

For two functions \( f(x) \) and \( g(x) \), the derivative of their quotient is:

\[
\frac{d}{dx} \left( \frac{f(x)}{g(x)} \right) = \frac{f'(x) \cdot g(x) - f(x) \cdot g'(x)}{g(x)^2}
\]

Example:

\[
\frac{d}{dx} \left( \frac{x^2}{\cos(x)} \right) = \frac{2x \cdot \cos(x) - x^2 \cdot (-\sin(x))}{\cos^2(x)}
\]

(e) Chain Rule

If a function \( f(x) \) is composed of two functions \( u(x) \) and \( g(x) \), such that \( f(x) = g(u(x)) \), then the derivative of \( f(x) \) is:

\[
\frac{d}{dx} f(x) = \frac{d}{du} g(u) \cdot \frac{d}{dx} u(x)
\]

Example:

\[
f(x) = \sin(x^2), \quad u(x) = x^2, \quad g(u) = \sin(u)
\]
\[
f'(x) = \cos(x^2) \cdot 2x
\]

5. Higher-Order Derivatives

The second derivative \( f''(x) \) represents the derivative of \( f'(x) \). It can be used to analyze the concavity of a function. For example:

\[
f''(x) = \frac{d^2}{dx^2} f(x)
\]

Higher-order derivatives are denoted as \( f^{(n)}(x) \), where \( n \) is the order.

Example:

\[
f(x) = x^3 \quad \Rightarrow \quad f'(x) = 3x^2 \quad \Rightarrow \quad f''(x) = 6x
\]

6. Special Derivatives

- Exponential functions:
  
  \[
  \frac{d}{dx} e^x = e^x
  \]
  
- Logarithmic functions:
  
  \[
  \frac{d}{dx} \ln(x) = \frac{1}{x}
  \]

- Trigonometric functions:
  
  \[
  \frac{d}{dx} \sin(x) = \cos(x), \quad \frac{d}{dx} \cos(x) = -\sin(x)
  \]

7. Using Derivatives for Approximation

The linear approximation or tangent line approximation uses derivatives to estimate the value of a function near a point \( a \):

\[
f(x) \approx f(a) + f'(a) \cdot (x - a)
\]

This is particularly useful in analyzing functions locally and making predictions about their behavior.

The derivative is a fundamental concept in calculus that measures the rate of change of a function. Using limits, we rigorously define the derivative and develop various methods for finding derivatives of different types of functions. The power rule, product rule, quotient rule, and chain rule are essential techniques that allow us to differentiate a wide variety of functions.

Introduction to Fermat's Theorem and Function Approximation

Fermat's theorem in calculus is closely related to the concept of critical points of a function, where the derivative either vanishes or does not exist. This theorem plays a fundamental role in optimization problems. Alongside this, function approximation uses derivatives to estimate or approximate the behavior of a function near certain points, especially using techniques like Taylor and Maclaurin series.

1. Fermat's Theorem (for Stationary Points)

Fermat's theorem states that if a function \( f(x) \) has a local maximum or local minimum at some point \( x = a \), and if \( f(x) \) is differentiable at \( x = a \), then the derivative at that point must be zero:

\[
f'(a) = 0
\]

This means that the slope of the tangent line to the curve at \( x = a \) is horizontal.

Proof Sketch:

Let \( f(x) \) have a local maximum at \( x = a \). Then for values of \( x \) close to \( a \):

\[
f(x) \leq f(a) \quad \text{for} \quad x \text{ near } a
\]

If \( h > 0 \), the right-hand derivative at \( x = a \) is:

\[
\lim_{h \to 0^+} \frac{f(a+h) - f(a)}{h} \leq 0
\]

If \( h < 0 \), the left-hand derivative at \( x = a \) is:

\[
\lim_{h \to 0^-} \frac{f(a+h) - f(a)}{h} \geq 0
\]

Thus, if \( f(x) \) is differentiable at \( x = a \), both of these limits must equal zero:

\[
f'(a) = 0
\]

A similar argument applies for a local minimum. Fermat's theorem does not guarantee that \( f'(a) = 0 \) implies a maximum or minimum. These points are called critical points, and further analysis is needed to classify them as maxima, minima, or saddle points.

2. Critical Points and Fermat's Theorem

- A point \( a \) is called a critical point if \( f'(a) = 0 \) or if the derivative does not exist at \( a \).
- Fermat's theorem identifies necessary conditions for maxima and minima, but not sufficient conditions.

Example:

Consider the function \( f(x) = x^2 \). We compute its derivative:

\[
f'(x) = 2x
\]

Setting \( f'(x) = 0 \) gives \( x = 0 \) as the only critical point. We can confirm that \( x = 0 \) is a local minimum by analyzing the second derivative:

\[
f''(x) = 2
\]

Since \( f''(0) > 0 \), \( x = 0 \) is indeed a local minimum by the second derivative test.

3. Second Derivative Test

The second derivative test helps determine whether a critical point is a local maximum, minimum, or a saddle point:

- If \( f''(a) > 0 \), \( f(a) \) is a local minimum.
- If \( f''(a) < 0 \), \( f(a) \) is a local maximum.
- If \( f''(a) = 0 \), the test is inconclusive, and further analysis is required.

4. Function Approximation Using Taylor Series

Taylor series is a powerful tool in function approximation, allowing us to approximate a differentiable function around a point using its derivatives. The Taylor series of a function \( f(x) \) centered at \( x = a \) is given by:

\[
f(x) = f(a) + f'(a)(x - a) + \frac{f''(a)}{2!}(x - a)^2 + \frac{f^{(3)}(a)}{3!}(x - a)^3 + \cdots
\]

This formula uses derivatives of all orders to approximate \( f(x) \) near \( x = a \).

Taylor Polynomial Approximation

For practical purposes, we often truncate the Taylor series after a few terms, creating a Taylor polynomial of degree \( n \):

\[
P_n(x) = f(a) + f'(a)(x - a) + \frac{f''(a)}{2!}(x - a)^2 + \cdots + \frac{f^{(n)}(a)}{n!}(x - a)^n
\]

This gives an approximation of \( f(x) \) near \( x = a \), where the accuracy depends on the number of terms used.

Maclaurin Series

The Maclaurin series is a special case of the Taylor series centered at \( a = 0 \):

\[
f(x) = f(0) + f'(0)x + \frac{f''(0)}{2!}x^2 + \frac{f^{(3)}(0)}{3!}x^3 + \cdots
\]

5. Example of Function Approximation

Consider the function \( f(x) = e^x \). The Taylor series expansion of \( e^x \) around \( x = 0 \) (Maclaurin series) is:

\[
e^x = 1 + x + \frac{x^2}{2!} + \frac{x^3}{3!} + \cdots
\]

For small \( x \), we can approximate \( e^x \) using just the first few terms:

\[
e^x \approx 1 + x + \frac{x^2}{2}
\]

This approximation becomes more accurate as more terms are added.

6. Lagrange Remainder Theorem

When using a Taylor polynomial to approximate a function, the Lagrange remainder provides an estimate of the error involved. The remainder term after \( n \) terms is given by:

\[
R_n(x) = \frac{f^{(n+1)}(c)}{(n+1)!} (x - a)^{n+1}
\]

where \( c \) lies between \( a \) and \( x \). This expression helps quantify the difference between the exact function value and the Taylor polynomial approximation.

7. Application of Fermat's Theorem in Optimization

In optimization problems, Fermat's theorem is used to locate critical points, which may be potential solutions to optimization problems. These critical points are further analyzed using the second derivative test or other methods to classify them as maxima, minima, or saddle points.

Fermat's theorem plays a crucial role in identifying critical points in optimization problems. It states that at local maxima or minima, the derivative of a function must vanish. On the other hand, function approximation techniques like the Taylor series allow us to approximate functions using polynomials derived from their derivatives. Together, these tools form the foundation for both theoretical analysis and practical problem-solving in calculus and optimization.

Introduction to Taylor Expansion and Convex Functions

Taylor expansion and convex functions are essential concepts in mathematical analysis, calculus, and optimization. Taylor expansion provides a powerful method to approximate functions near a given point using polynomials, while convex functions play a critical role in optimization due to their unique properties that simplify finding global minima or maxima.

1. Taylor Expansion

The Taylor series of a function \( f(x) \) is a representation of the function as an infinite sum of terms calculated from the derivatives of the function at a specific point. The Taylor expansion around \( x = a \) is given by:

\[
f(x) = f(a) + f'(a)(x - a) + \frac{f''(a)}{2!}(x - a)^2 + \cdots + \frac{f^{(n)}(a)}{n!}(x - a)^n + R_n(x)
\]

Here:
- \( f^{(n)}(a) \) is the \( n \)-th derivative of \( f(x) \) evaluated at \( x = a \),
- \( n! \) is the factorial of \( n \),
- \( R_n(x) \) is the remainder term, which measures the error between the actual function and the \( n \)-th order polynomial approximation.

Maclaurin Series

The Maclaurin series is a special case of the Taylor series where the expansion is centered at \( a = 0 \):

\[
f(x) = f(0) + f'(0)x + \frac{f''(0)}{2!}x^2 + \cdots + \frac{f^{(n)}(0)}{n!}x^n + R_n(x)
\]

Example: Taylor Expansion of \( e^x \)

Consider \( f(x) = e^x \). The Taylor expansion of \( e^x \) around \( x = 0 \) (Maclaurin series) is:

\[
e^x = 1 + x + \frac{x^2}{2!} + \frac{x^3}{3!} + \cdots
\]

This infinite series provides an accurate approximation of \( e^x \) for values of \( x \) near zero.

Lagrange Remainder Term

The Lagrange remainder provides a bound on the error when truncating a Taylor series at the \( n \)-th term. The remainder after \( n \) terms is given by:

\[
R_n(x) = \frac{f^{(n+1)}(c)}{(n+1)!} (x - a)^{n+1}
\]

where \( c \) is a value between \( a \) and \( x \). This expression ensures the accuracy of the approximation and gives insight into how quickly the Taylor series converges.

2. Convex Functions

A function \( f(x) \) is called convex if its graph lies below the line segment connecting any two points on the graph. More formally, \( f(x) \) is convex on an interval \( I \) if for all \( x_1, x_2 \in I \) and \( \lambda \in [0, 1] \):

\[
f(\lambda x_1 + (1-\lambda)x_2) \leq \lambda f(x_1) + (1-\lambda)f(x_2)
\]

This definition ensures that a convex function exhibits a "bowl-shaped" behavior, which makes it easier to identify global minima.

First and Second Derivative Conditions for Convexity

- A function \( f(x) \) is convex if and only if its second derivative is non-negative for all \( x \) in the interval:

\[
f''(x) \geq 0 \quad \text{for all} \quad x
\]

- If \( f(x) \) is differentiable, it is convex if:

\[
f(y) \geq f(x) + f'(x)(y - x) \quad \text{for all} \quad x, y
\]

This inequality states that the tangent line at any point \( x \) on the curve lies below the curve for all other points \( y \), which is a key characteristic of convex functions.

Strict Convexity

A function is strictly convex if the inequality in the definition of convexity is strict for all \( x_1 \neq x_2 \):

\[
f(\lambda x_1 + (1-\lambda)x_2) < \lambda f(x_1) + (1-\lambda)f(x_2)
\]

Strict convexity implies that the function has a unique global minimum.

3. Examples of Convex Functions

(a) Quadratic Function

The function \( f(x) = x^2 \) is convex since:

\[
f''(x) = 2 \geq 0
\]

This means the graph of \( f(x) = x^2 \) is convex for all \( x \), with a global minimum at \( x = 0 \).

(b) Exponential Function

The function \( f(x) = e^x \) is convex because:

\[
f''(x) = e^x \geq 0 \quad \text{for all } x
\]

Since the second derivative is always positive, \( e^x \) is strictly convex, and its graph curves upward for all \( x \).

4. Convexity and Optimization

Convex functions are crucial in optimization because any local minimum of a convex function is also a global minimum. This simplifies many optimization problems, especially in machine learning and operations research, where minimizing a convex function ensures finding the best possible solution.

Gradient Descent on Convex Functions

When applying gradient descent to minimize a convex function \( f(x) \), we iterate by updating the parameter \( x \) using the derivative:

\[
x_{\text{new}} = x_{\text{old}} - \eta f'(x_{\text{old}})
\]

where \( \eta \) is the step size or learning rate. For convex functions, gradient descent converges to the global minimum because the gradient always points in the direction of steepest descent.

5. Relationship Between Taylor Expansion and Convexity

The Taylor expansion of a function can help us analyze its convexity by approximating the function using its second derivative. Consider the Taylor expansion of a function \( f(x) \) around a point \( x = a \):

\[
f(x) \approx f(a) + f'(a)(x - a) + \frac{f''(a)}{2!}(x - a)^2
\]

The second-order term \( \frac{f''(a)}{2!}(x - a)^2 \) gives information about the curvature of the function at \( x = a \). If \( f''(a) > 0 \), the function is locally convex at \( x = a \).
 

Taylor expansion allows us to approximate functions using polynomials by leveraging their derivatives. Convex functions, defined by the property that their second derivative is non-negative, play a fundamental role in optimization because they guarantee global solutions. Both concepts are deeply intertwined, as Taylor expansion provides insight into a function's curvature, helping to determine whether it is convex and thereby simplifying optimization strategies.