NonLinear Regression Problem & Ridge Regression

Part One : Basis for Nonlinear Regression

In this part we’re going to investigate how to use linear regression to approximate and estimate complicated functions. For example, suppose we want to fit the following function on the interval [0,1] :

Import Modules

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

Create a problem

def y(x):
    ret = .15*np.sin(40*x) 
    ret = ret + .25*np.sin(10*x)
    step_fn1 = np.zeros(len(x))
    step_fn1[x >= .25] = 1
    step_fn2 = np.zeros(len(x))
    step_fn2[x >= .75] = 1
    ret = ret - 0.3*step_fn1 + 0.8 *step_fn2 
    return ret

x = np.arange(0.0, 1.0, 0.001)
plt.plot(x, y(x))
plt.show()

Plot the graph and see the functions

Here the input space, shown on the x-axis, is very simple – it’s just the interval $[0,1]$. The output space is reals ($\mathcal{R}$), and a graph of the function { ( x , y ( x ) ) ∣ x ∈ [ 0 , 1 ] } \{(x,y(x)) \mid x \in [0,1]\} { (x,y(x))∣x∈[0,1]} is shown above. Clearly a linear function of the input will not give a good approximation to the function. There are many ways to construct nonlinear functions of the input. Some popular approaches in machine learning are regression trees, neural networks, and local regression (e.g. LOESS). However, our approach here will be to map the input space $[0,1]$ into a “feature space” ${\mathcal{R}}^d$, and then run standard linear regression in the feature space.

Feature extraction, or “featurization”, maps an input from some input space $\mathcal{X}$ to a vector in ${\mathcal{R}}^d$. Here our input space is $\mathcal{X}=[0,1]$, so we could write our feature mapping as a function $\Phi :[0,1]\to {\mathcal{R}}^d$. The vector $\Phi (x)$ is called a feature vector, and each entry of the vector is called a feature. Our feature mapping is typically defined in terms of a set of functions, each computing a single entry of the feature vector. For example, let’s define a feature function ${\varphi }_1(x)=1(x\ge 0.25)$. The $1(\cdot )$ denotes an indicator function, which is 1 if the expression in the parenthesis is true, and 0 otherwise. So ${\varphi }_1(x)$ is $1$ if $x\ge 0.25$ and $0$ otherwise. This function produces a “feature” of x. Let’s define two more features: ${\varphi }_2(x)=1(x\ge 0.5)$ and ${\varphi }_3(x)=1(x\ge 0.75)$. Now we can define a feature mapping into ${\mathcal{R}}^3$ as:
$$\Phi (x)=({\varphi }_1(x),{\varphi }_2(x),\varphi$$