### Python - Softmax Regression

Wang Tiven July 17, 2018
425 favorite favorites
bookmark bookmark
share share

$f(z) = \frac{e^{z}}{1 + e^{z}} = \frac{1}{1+e^{-z}}$ $z = wx + b$

$z = w_1x_1 + ... + w_mx_m + b= \sum_{l=1}^{m} w_l x_l + b= \mathbf{w}^T\mathbf{x} + b$

\begin{align} \begin{bmatrix} z_1 \\ z_2 \\ \vdots \\ z_k \end{bmatrix} = \begin{bmatrix} w_{1,1} & w_{1,2} & \cdots & w_{1,m} \\ w_{2,1} & w_{2,2} & \cdots & w_{2,m} \\ \vdots & \vdots & \ddots & \vdots \\ w_{k,1} & w_{k,2} & \cdots & w_{k,m} \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_m \end{bmatrix} + b = \begin{bmatrix} w_{1,1} x_1 + w_{1,2} x_2 + \cdots + w_{1,m} x_m + b \\ w_{2,1} x_1 + w_{2,2} x_2 + \cdots + w_{2,m} x_m + b \\ \vdots \\ w_{k,1} x_1 + w_{k,2} x_2 + \cdots + w_{k,m} x_m + b \end{bmatrix} \end{align}

\begin{align} h_w(x) = \begin{bmatrix} P(y = 1 | x; w) \\ P(y = 2 | x; w) \\ \vdots \\ P(y = k | x; w) \end{bmatrix} = \frac{1}{ \sum_{j=1}^{k}{\exp(z_j) }} \begin{bmatrix} \exp(z_1) \\ \exp(z_2) \\ \vdots \\ \exp(z_k) \\ \end{bmatrix} \end{align}

## Cost Function

\begin{align} J(w) = - \left[ \sum_{k=1}^{K} 1\left\{y = k\right\} \log \frac{\exp(z_k)}{\sum_{j=1}^K \exp(z_j)}\right] \end{align}

\begin{align} J(w) = - \left[ \sum_{i=1}^{n}\sum_{k=1}^{K} 1\left\{y^i = k\right\} \log \frac{e^{(z_{ik})}}{\sum_{j=1}^K e^{(z_{ij})}}\right] \end{align}

## Softmax regression

Softmax regression (or multinomial logistic regression) is a generalization of logistic regression to the case where we want to handle multiple classes. In logistic regression we assumed that the labels were binary: y(i)∈{0,1}. We used such a classifier to distinguish between two kinds of hand-written digits. Softmax regression allows us to handle y(i)∈{1,…,K} where K is the number of classes.