YouTip

Linear Regression is one of the most fundamental and widely used algorithms in machine learning.\\n\\nLinear Regression is a basic machine learning algorithm used for predicting continuous values. It assumes a linear relationship between the target variable y and the feature variable x, and attempts to find a best-fit line to describe this relationship.\\n\\ny = w * x + b\\nWhere:\\n\\n* `y` is the predicted value\\n\\n* `x` is the feature variable\\n\\n* `w` is the weight (slope)\\n\\n* `b` is the bias (intercept)\\n\\nThe goal of linear regression is to find the optimal `w` and `b` that minimize the error between the predicted value `y` and the true value. The commonly used error function is Mean Squared Error (MSE):\\n\\nMSE = 1/n * Σ(y_i - y_pred_i)^2\\nWhere:\\n\\n* y_i is the actual value.\\n* y_pred_i is the predicted value.\\n* n is the number of data points.\\n\\nOur goal is to adjust w and b to minimize the MSE.\\n\\n## How to Solve Linear Regression?\\n\\n### 1. Least Squares Method\\n\\nThe least squares method is a commonly used approach to solve linear regression. It finds the optimal ( w ) and ( b ) by solving the following equations.\\n\\nThe goal of least squares is to minimize the Residual Sum of Squares (RSS), with the formula:\\n\\n$$\\ntext{RSS} = sum_{i = 1}^{n} left(right. y_{i} - left(hat{y}right)_{i} left.right)^{2}\\n$$\\n\\nWhere:\\n\\n* `yi y_i` is the actual value.\\n* `y^i hat{y}_i` is the predicted value, calculated by the linear regression model `y^i=wxi+b hat{y}_i = w x_i + b`.\\n\\nBy minimizing RSS, we obtain the following normal equations:\\n\\n$$\\nleft{right. w sum_{i = 1}^{n} x_{i}^{2} + b sum_{i = 1}^{n} x_{i} = sum_{i = 1}^{n} x_{i} y_{i} \\\\ w sum_{i = 1}^{n} x_{i} + b n = sum_{i = 1}^{n} y_{i}\\n$$\\n\\n### Matrix Form\\n\\nWriting the normal equations in matrix form:\\n\\n$$\\nleft[right. sum_{i = 1}^{n} x_{i}^{2} & sum_{i = 1}^{n} x_{i} \\\\ sum_{i = 1}^{n} x_{i} & n left]right. left[right. w \\\\ b left]right. = left[right. sum_{i = 1}^{n} x_{i} y_{i} \\\\ sum_{i = 1}^{n} y_{i} left]right.\\n$$\\n\\n### Solution Method\\n\\nBy solving the above matrix equation, we can obtain the optimal `w w` and `b b`:\\n\\n$$\\nleft[right. w \\\\ b left]right. = left(left[right. sum_{i = 1}^{n} x_{i}^{2} & sum_{i = 1}^{n} x_{i} \\\\ sum_{i = 1}^{n} x_{i} & n left]right.right)^{- 1} left[right. sum_{i = 1}^{n} x_{i} y_{i} \\\\ sum_{i = 1}^{n} y_{i} left]right.\\n$$\\n\\n### 2. Gradient Descent Method\\n\\nThe goal of gradient descent is to minimize the loss function `J(w,b) J(w, b)`. For linear regression problems, Mean Squared Error (MSE) is typically used as the loss function:\\n\\n$$\\nJ left(right. w , b left.right) = frac{1}{2 m} sum_{i = 1}^{m} left(right. y_{i} - left(hat{y}right)_{i} left.right)^{2}\\n$$\\n\\nWhere:\\n\\n* `m m` is the number of samples.\\n* `yi y_i` is the actual value.\\n* `y^i hat{y}_i` is the predicted value, calculated by the linear regression model `y^i=wxi+b hat{y}_i = w x_i + b`.\\n\\nThe gradient is the partial derivative of the loss function with respect to the parameters, representing the direction of change of the loss function in the parameter space. For linear regression, the gradient is calculated as follows:\\n\\n$$\\nfrac{partial J}{partial w} = - frac{1}{m} sum_{i = 1}^{m} x_{i} left(right. y_{i} - left(hat{y}right)_{i} left.right)\\n$$\\n\\n$$\\nfrac{partial J}{partial b} = - frac{1}{m} sum_{i = 1}^{m} left(right. y_{i} - left(hat{y}right)_{i} left.right)\\n$$\\n\\n### Parameter Update Rules\\n\\nGradient descent updates the parameters `w w` and `b b` according to the following rules:\\n\\n$$\\nw : = w - alpha frac{partial J}{partial w}\\n$$\\n\\n$$\\nb : = b - alpha frac{partial J}{partial b}\\n$$\\n\\nWhere:\\n\\n* `α alpha` is the learning rate, controlling the step size of each update.\\n\\n### Steps of Gradient Descent\\n\\n1. **Initialize parameters**: Initialize `w w` and `b b` (usually set to 0 or random values).\\n2. **Calculate loss function**: Calculate the loss function value `J(w,b) J(w, b)` with current parameters.\\n3. **Calculate gradient**: Calculate the partial derivatives of the loss function with respect to `w w` and `b b`.\\n4. **Update parameters**: Update `w w` and `b b` based on the gradient.\\n5. **Repeat iteration**: Repeat steps 2 to 4 until the loss function converges or the maximum number of iterations is reached.\\n\\n## Implementing Linear Regression with Python\\n\\nBelow we demonstrate how to implement linear regression using Python through a simple example.\\n\\n### 1. Import Necessary Libraries\\n\\n## Example\\n\\nimport numpy as np\\n\\nimport matplotlib.pyplot as plt\\n\\nfrom sklearn.linear_model import LinearRegression\\n\\n### 2. Generate Simulated Data\\n\\n## Example\\n\\nimport numpy as np\\n\\nimport matplotlib.pyplot as plt\\n\\nfrom sklearn.linear_model import LinearRegression\\n\\n# Generate some random data\\n\\n np.random.seed(0)\\n\\n x =2 * np.random.rand(100,1)\\n\\n y =4 + 3 * x + np.random.randn(100,1)\\n\\n# Visualize data\\n\\n plt.scatter(x, y)\\n\\n plt.xlabel('x')\\n\\n plt.ylabel('y')\\n\\n plt.title('Generated Data From ')\\n\\n plt.show()\\n\\nDisplay as follows:\\n\\n!(#)\\n\\n### 3. Using Scikit-learn for Linear Regression\\n\\n## Example\\n\\nimport numpy as np\\n\\nimport matplotlib.pyplot as plt\\n\\nfrom sklearn.linear_model import LinearRegression\\n\\n# Generate some random data\\n\\n np.random.seed(0)\\n\\n x =2 * np.random.rand(100,1)\\n\\n y =4 + 3 * x + np.random.randn(100,1)\\n\\n# Create linear regression model\\n\\n model = LinearRegression()\\n\\n# Fit model\\n\\n model.fit(x, y)\\n\\n# Output model parameters\\n\\nprint(f"slope (w): {model.coef_}")\\n\\nprint(f"intercept (b): {model.intercept_}")\\n\\n# Predict\\n\\n y_pred = model.predict(x)\\n\\n# Visualize fitting results\\n\\n plt.scatter(x, y)\\n\\n plt.plot(x, y_pred, color='red')\\n\\n plt.xlabel('x')\\n\\n plt.ylabel('y')\\n\\n plt.title('Linear Regression Fit')\\n\\n plt.show()\\n\\nOutput:\\n\\nslope (w): 2.968467510701019intercept (b): 4.222151077447231\\nDisplay as follows:\\n\\n!(#)\\n\\nWe can use the `score()` method to evaluate model performance, which returns the R^2 value.\\n\\n## Example\\n\\nimport numpy as np\\n\\nfrom sklearn.linear_model import LinearRegression\\n\\n# Generate some random data\\n\\n np.random.seed(0)\\n\\n x =2 * np.random.rand(100,1)\\n\\n y =4 + 3 * x + np.random.randn(100,1)\\n\\n# Create linear regression model\\n\\n model = LinearRegression()\\n\\n# Fit model\\n\\n model.fit(x, y)\\n\\n# Calculate model score\\n\\n score = model.score(x, y)\\n\\nprint("model score:", score)\\n\\nOutput:\\n\\nmodel score: 0.7469629925504755\\n### 4. Manual Implementation of Gradient Descent\\n\\n## Example\\n\\nimport numpy as np\\n\\nimport matplotlib.pyplot as plt\\n\\nfrom sklearn.linear_model import LinearRegression\\n\\n# Generate some random data\\n\\n np.random.seed(0)\\n\\n x =2 * np.random.rand(100,1)\\n\\n y =4 + 3 * x + np.random.randn(100,1)\\n\\n# Initialize parameters\\n\\n w =0\\n\\n b =0\\n\\n learning_rate =0.1\\n\\n n_iterations =1000\\n\\n# Gradient descent\\n\\nfor i in range(n_iterations):\\n\\n y_pred = w * x + b\\n\\n dw = -(2/len(x)) * np.sum(x * (y - y_pred))\\n\\n db = -(2/len(x)) * np.sum(y - y_pred)\\n\\n w = w - learning_rate * dw\\n\\n b = b - learning_rate * db\\n\\n# Output final parameters\\n\\nprint(f"Manually implemented slope (w): {w}")\\n\\nprint(f"Manually implemented intercept (b): {b}")\\n\\n# Visualize manual fitting results\\n\\n y_pred_manual = w * x + b\\n\\n plt.scatter(x, y)\\n\\n plt.plot(x, y_pred_manual, color='green')\\n\\n plt.xlabel('x')\\n\\n plt.ylabel('y')\\n\\n plt.title('Manual Gradient Descent Fit')\\n\\n plt.show()\\n\\nOutput:\\n\\nManually implemented slope (w): 2.968467510701028Manually implemented intercept (b): 4.222151077447219\\nDisplay as follows:\\n\\n!(#)